# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Goal

Build an OPC UA server (.NET 10) that exposes industrial data sources —
including AVEVA System Platform (Wonderware) Galaxy — under a unified
Equipment-based address space. Galaxy access flows through the in-process
`GalaxyDriver` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/`) talking gRPC
to a separately installed **mxaccessgw** gateway process. The gateway owns the
MXAccess COM bitness constraint (its worker is x86 net48); everything in this
repo is .NET 10. PR 7.2 retired the legacy in-process `Galaxy.Host` /
`Galaxy.Proxy` / `Galaxy.Shared` projects + the `OtOpcUaGalaxyHost` Windows
service.

See `docs/v2/Galaxy.Performance.md` for the runtime perf surface
(tracing, metrics, soak harness).

## Related Projects (scadaproj umbrella)

This repo is one of a family of related SCADA / OT / Wonderware / OPC UA
**sister projects** that live as sibling directories under `~/Desktop/`. They are
**separate repos and separate processes**, coupled at runtime over wire protocols
(gRPC + OPC UA) — not by project/compile references — and share the
`ZB.MOM.WW.*` product namespace. The common subject is AVEVA System Platform
(Wonderware) "Galaxy" data, and `mxaccessgw` is the linchpin the others connect
through.

- **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) —
  the umbrella/index workspace that aggregates the whole family (a high-level scan
  of each sister project: purpose, location, stack, primary commands). It also hosts
  the shared `ZB.MOM.WW.*` libraries — `ZB.MOM.WW.Auth`, `ZB.MOM.WW.Theme`,
  `ZB.MOM.WW.Health`, `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Configuration`, and
  `ZB.MOM.WW.GalaxyRepository`. **This repo is indexed there** — see the OtOpcUa
  entry in `../scadaproj/CLAUDE.md`.
- **`~/Desktop/MxAccessGateway`** (`mxaccessgw`,
  [`../MxAccessGateway/CLAUDE.md`](../MxAccessGateway/CLAUDE.md)) — the gRPC gateway
  that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session
  x86 net48 worker). **OtOpcUa's `GalaxyDriver` depends on this** for live Galaxy
  read/write/subscribe and Galaxy Repository browse.
- **`~/Desktop/HistorianGateway`**
  ([`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process
  gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse
  (via the shared `ZB.MOM.WW.GalaxyRepository` lib). Patterned on `mxaccessgw` but with
  no COM / no x86 worker. **OtOpcUa's historian backend** (`Driver.Historian.Gateway`)
  consumes it as the sole historian path.
- **`~/Desktop/ScadaBridge`**
  ([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) — the distributed SCADA
  platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's
  OPC UA address space.

**Propagate cross-repo changes to the umbrella index.** When a fact the index records
about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow,
shared-lib consumption, or per-project commands — update the **OtOpcUa entry in
`../scadaproj/CLAUDE.md`** in the same change so the index never drifts from this repo.

**The index edit belongs on `scadaproj`'s `main`, and must be pushed.** `scadaproj` is a
separate repo with its own checkout, so committing there inherits whatever branch it
happens to be on — which is usually *not* `main` and is often an unrelated feature branch
with no upstream. That silently drifts the index for as long as that branch stays
unmerged, which is the exact failure the rule above exists to prevent. Do this explicitly:

```bash
cd ~/Desktop/scadaproj
git rev-parse --abbrev-ref HEAD            # note it, to restore afterwards
git stash list                             # never commit onto a dirty unrelated branch
git checkout main && git pull
#   …edit the OtOpcUa entry in CLAUDE.md…
git commit -am "docs(index): <what changed about OtOpcUa>"
git push origin main
git checkout -                             # restore the original branch
```

Observed 2026-07-27: the `Sql` poll driver's index entry had sat unpushed on an unrelated
feature branch since 2026-07-24 because it was committed onto the current checkout, so the
index disagreed with reality for three days despite this rule. If you find index commits
stranded on a feature branch, cherry-pick them onto `main` rather than merging that branch.

## Architecture Overview

### Data Flow

1. **Galaxy Repository DB (ZB)** — SQL Server database holding the
   deployed object hierarchy and attribute definitions. The
   mxaccessgw's `GalaxyRepositoryClient` queries it via gRPC; the
   driver consumes the materialised hierarchy through
   `IGalaxyHierarchySource`.
2. **MXAccess (via mxaccessgw)** — Live read/write/subscribe over a
   gRPC session. The gateway owns the COM apartment + STA pump
   server-side; the driver speaks `MxCommand` / `MxEvent` protos
   exclusively.
3. **OPC UA Server** — Exposes the deployed tree under **two OPC UA namespaces**
   (see below). Galaxy tags are bound by `TagConfig.FullName`
   (`tag_name.AttributeName`); reads/writes/subscriptions are translated to that
   reference for MXAccess.

### v3 OPC UA Address Space (Batch 4): dual namespace

The server exposes **two OPC UA namespaces** (replacing the single
`https://zb.com/otopcua/ns`), built by `AddressSpaceComposer` /
`AddressSpaceApplier` and served by `OtOpcUaNodeManager` (which registers both
URIs; `V3NodeIds` + `AddressSpaceRealm` are the identity authority):

| Namespace URI | Realm | Subtree | NodeId `s=` scheme |
|---|---|---|---|
| `https://zb.com/otopcua/raw` | `AddressSpaceRealm.Raw` | the `/raw` device tree (Folder → Driver → Device → TagGroup → Tag) | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) |
| `https://zb.com/otopcua/uns` | `AddressSpaceRealm.Uns` | the UNS tree (Area → Line → Equipment → signal) | slash-joined **`Area/Line/Equipment/EffectiveName`** |

- **Single source, fanned to both.** Every device value has exactly one source —
  the raw tag's node. A `UnsTagReference` projects that raw tag into an equipment
  as a UNS-namespace variable that carries an **`Organizes` reference to its raw
  node** and mirrors it. One driver publish for a RawPath fans (in `DriverHostActor`)
  to the raw NodeId AND every referencing UNS NodeId with identical
  value/quality/timestamp — the mux key stays single (RawPath).
- **Writes via either NodeId** route to the same driver ref under the same
  `WriteOperate` gating (the node-manager write gate is realm-qualified and fails
  closed); a failed device write reverts both NodeIds via the shared fan-out
  (write-outcome self-correction).
- **Native alarms** materialize once at the raw tag (`ConditionId = RawPath`) and
  fan via SDK notifiers to the raw device folder AND every referencing equipment
  folder — one `ReportEvent`, one Server-object copy; ack/confirm/shelve route on
  `ConditionId = RawPath`.
- **Historian dual-registration.** Both NodeIds register the **same** historian
  tagname; the mux `HistorizedTagRef` stays single (RawPath). HistoryRead works
  through either NodeId.

The retired `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme and
the single-namespace `EquipmentTags` materialization path are gone. See
`docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` +
`docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`.

### Key Concept: Tag Name and FullName

Galaxy objects have a **tag_name** — a globally unique system name used for
MXAccess read/write — and attributes are referenced as `tag_name.AttributeName`
(e.g. `DelmiaReceiver_001.DownloadPath`). This dot-separated reference is what
`TagConfig.FullName` stores for a Galaxy equipment tag. The Galaxy address
picker browses the live hierarchy using contained names for navigation, then
resolves the selection to the `tag_name.AttributeName` form.

**Galaxy is a standard Equipment-kind driver.** Galaxy points are ordinary
equipment `Tag`s bound to `GalaxyMxGateway` via `TagConfig.FullName`
(`tag_name.AttributeName`), authored through the standard Tag modal + Galaxy
address picker on the equipment page's Tags tab — the same flow as Modbus or
S7. There is no alias machinery, no `SystemPlatform` namespace kind, and no
relay→alias converter. See
`docs/plans/2026-06-12-galaxy-standard-driver-design.md` for the full design.

### Data Type Mapping

Galaxy `mx_data_type` values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`.

### Change Detection

`DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys. The server's `DriverHost` consumes the signal and rebuilds the address space.

## mxaccessgw

The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3.

## Build Commands

```bash
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx
dotnet test ZB.MOM.WW.OtOpcUa.slnx                                # all tests
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests               # a single test project
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod"    # a single test
```

CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`)
is **resolved** — the temporary `NuGetAuditSuppress` was removed once the patched bundle
shipped. `Core.AlarmHistorian` now carries a surgical direct pin of
`SQLitePCLRaw.bundle_e_sqlite3` to `2.1.12` (the first fixed native bundle, embedding the
SQLite 3.50.2+ patch), overriding the `2.1.11` that `Microsoft.Data.Sqlite` pulls
transitively. `dotnet restore`/`build` is clean with the audit active.

Test projects live under `tests/<module>/` (Core, Server, Drivers,
Drivers/Cli, Client, Tooling) — there is no single unit-test project.
Unit suites are named `*.Tests`; integration suites are `*.IntegrationTests`
and need their Docker fixture up (see Docker Workflow). DB-backed tests in
`*.Configuration.Tests`, `*.Admin.Tests`, and `*.Server.Tests` require the
central SQL Server.

## Docker Workflow (driver fixtures + central SQL Server)

> **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing +
> deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate**
> recipes (S7 blackhole, GLAuth outage, HistorianGateway LiveIntegration), the integration-test
> harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail.

> **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28).

> ⚠️ **The `project: lmxopcua` label is aspirational, not universal.** This note used to claim every fixture compose file carries it. As of 2026-07-24 only the **MQTT** fixture actually does — `grep -rn "labels:" tests --include=*.yml` matches `Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml` and nothing else. So `docker ps --filter label=project=lmxopcua` returns the MQTT fixture alone, not the fleet. Add the label when you touch an older fixture; until then enumerate by container name.

Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-<driver>/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`.

**`docker -H ssh://...` does NOT work from the Windows VM.** Windows OpenSSH ↔ docker.exe stdio bridging hangs (`docker system dial-stdio` runs server-side but no API data flows). Use the helper below — it SSHes into the docker host and runs `docker compose` server-side.

> **On macOS there is no `lmxopcua-fix` — drive the host over SSH directly.** The helper is a Windows-VM
> script; `~/bin` is empty on the Mac and the commands below will not resolve. Use passwordless SSH
> (and `rsync` in place of `sync`), which is what `infra/README.md` §1 documents as the Mac path:
>
> ```bash
> ssh dohertj2@10.100.0.35 'docker ps --format "{{.Names}}\t{{.Status}}"'
> ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mqtt && docker compose up -d'
> rsync -av tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/ \
>     dohertj2@10.100.0.35:/opt/otopcua-mqtt/          # the `sync` step, by hand
> ```
>
> A local container runtime *does* exist on the Mac (OrbStack), so the `docker-dev/` rig itself runs
> locally there — it is only the shared **fixtures** that live on `10.100.0.35`.

**Use `lmxopcua-fix.ps1` (in `~/bin`) to control fixtures from the Windows VM:**

```powershell
lmxopcua-fix ls                            # list all lmxopcua-tagged containers on the host
lmxopcua-fix up   modbus      standard     # bring a profile up
lmxopcua-fix up   abcip       controllogix
lmxopcua-fix up   s7          s7_1500
lmxopcua-fix up   opcuaclient                # single-service stack, no profile arg
lmxopcua-fix down modbus                   # tear stack down
lmxopcua-fix logs modbus
lmxopcua-fix sync modbus                   # rsync this repo's tests/.../Docker/ → /opt/otopcua-modbus/
```

**`sync` is the deployment step.** When you edit a fixture's compose file or Dockerfile under `tests/.../Docker/`, run `lmxopcua-fix sync <driver>` to push the changes to the docker host before bringing the stack up. The repo files are the source of truth; `/opt/otopcua-<driver>/` is a mirrored deployment.

**Endpoints (defaults already point at the docker host):**
- SQL Server (always-on): `10.100.0.35,14330` — used by `appsettings.json` for `ConfigDb`.
- Modbus: `10.100.0.35:5020` (`MODBUS_SIM_ENDPOINT`)
- AB CIP: `10.100.0.35:44818` (`AB_SERVER_ENDPOINT`)
- S7: `10.100.0.35:1102` (`S7_SIM_ENDPOINT`)
- OPC UA reference (opc-plc): `opc.tcp://10.100.0.35:50000` (`OPCUA_SIM_ENDPOINT`)
- MQTT (Mosquitto, **auth on both listeners — no anonymous fallback**): `10.100.0.35:8883` TLS+auth
  (`MQTT_FIXTURE_ENDPOINT`) · `10.100.0.35:1883` plaintext-but-authenticated (`MQTT_FIXTURE_PLAIN_ENDPOINT`).
  Also needs `MQTT_FIXTURE_USERNAME` / `MQTT_FIXTURE_PASSWORD`, and `MQTT_FIXTURE_CA_CERT` to pin the
  fixture CA (`scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt`). A co-running publisher loops
  JSON under `otopcua/fixture/…`.
- MQTT **Sparkplug B**: the same broker, plus the `sparkplug` compose profile's `otopcua-sparkplug-sim`
  (project-owned C# edge-node simulator) — group **`OtOpcUaSim`**, edge nodes **`EdgeA`**/**`EdgeB`**,
  `EdgeA` device **`Filler1`**, ~2 s cadence. It subscribes to `spBv1.0/OtOpcUaSim/NCMD/{node}` and
  re-births on request. **Births are never retained**, so `docker restart otopcua-sparkplug-sim` is how
  you force a fresh one while a browse window is open.

Override any endpoint via the env var to point at a real PLC. The local OtOpcUa server runs on this VM at `opc.tcp://localhost:4840` — **that's not on the docker host**.

**Local docker-dev rig — login is DISABLED, so do live `/run` verification yourself (don't wait for the user to sign in).** The local `docker-dev/docker-compose.yml` stack (AdminUI at `http://localhost:9200` via Traefik; OPC UA `opc.tcp://localhost:4840` central-1 / `:4841` central-2) runs the AdminUI with `Security__Auth__DisableLogin: "true"` — **no sign-in form; it's auto-authenticated as a full-access admin.** So AdminUI / Razor `/run` verification (deploy a config, drive a page, confirm behavior — e.g. via the Chrome browser-automation tools against `http://localhost:9200`) does **not** require the user to log in. Run it yourself; do not defer it as "user-driven sign-in required." (Caveat: OPC UA *data-plane* auth is still real LDAP against the shared GLAuth on `10.100.0.35:3893` — that only gates Client.CLI read/write **role** operations, e.g. binding a `multi-role` / `opc-writeop` user, and is independent of the AdminUI login. Things genuinely outside the local rig — real PLCs, or the AVEVA Historian reached via the `ZB.MOM.WW.HistorianGateway` sidecar — still need the user.)

See `docs/v2/dev-environment.md` for the full inventory and rationale.

## Build & Runtime Constraints

- Language: C#, .NET 10, AnyCPU. The MXAccess COM bitness constraint
  is owned by the mxaccessgw worker (x86 net48), not by anything in
  this repo.
- The gateway's MXAccess worker requires a deployed ArchestrA Platform
  on the machine running the gateway. The OtOpcUa server itself does
  not.

## Transport Security

The server supports configurable OPC UA transport security via the `OpcUa:EnabledSecurityProfiles` list in `appsettings.json`. Phase 1 profiles (the `OpcUaSecurityProfile` enum members): `None` (default), `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`. Security policies are built from the enabled profiles by `BuildSecurityPolicies` at startup. The server certificate is always created even for `None`-only deployments because `UserName` token encryption depends on it. See `docs/security.md` for the full guide.

## Redundancy

The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.

**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**

- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.

**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".

**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard".

Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.

**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.

**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:

- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).

See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.

## Mesh command transport (`MeshTransport`)

Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
change, not a redeploy. Things worth knowing before touching it:

- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
  ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
  `DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
  actor — it would deploy to a single node while every other node silently kept its old configuration,
  and the deployment could still seal green.
- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When
  this was written the fleet was still one mesh, so a receptionist served the whole cluster and
  `SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
  the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
  `Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
  the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
  behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
  within its own mesh.
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
  central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
  Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
  comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
  on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise
  it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node
  applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
  replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay.

Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.

**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:

- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.

## Config source (`ConfigSource` / `ConfigServe`)

Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
fail-closed). Things worth knowing before touching it:

- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
  admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
  dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
  hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
  `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
  stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
  mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
  and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
  `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
  empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
  SQL today; `StoreAsync` is idempotent and the cache replicates).
- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
  both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
  ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
  `Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
  ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
  is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.

On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
`Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`.
Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up`. See
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.

## Live telemetry transport (`Telemetry` / `TelemetryDial`)

Per-cluster mesh **Phase 5** (code-complete on `feat/mesh-phase5`, live gate pending) added a second
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
touching it:

- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
  `driver-health` (`DriverHealthChanged`), `driver-resilience-status`
  (`DriverResilienceStatusChanged`) — feeding the exact same central sinks
  (`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
  either mode.
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
  delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
  `Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
  observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
  the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
  never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient`
  transport as a command-plane reply, not telemetry.
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
  into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
  server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
  `TelemetryDial:Mode` on central actually switches anything: which of the two always-available
  sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
  makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
  [MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
  `TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
  (`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
  every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
  for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
  `ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
  first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
  is ignored, and it never permanently gives up (observability, not data plane). The node-local hub
  replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
  so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
  tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
  node stream is up) — the same semantics the DPS pill has always had.

See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
§`Telemetry`/`TelemetryDial` for the config keys, and
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
gate procedure.

## LocalDb pair-local store (Phases 1 + 2)

Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).

**Phase 1** caches the
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
never register it.** The cache can optionally replicate to the node's redundant pair peer over the
library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
exception).

**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
same event accepted on both nodes — which happens in every boot window, since
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.

See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), `docs/AlarmHistorian.md`, and the runbook
`docs/operations/2026-07-20-localdb-pair-replication.md`.

## Phase 4 — cut the driver-side ConfigDb connection

Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles`
has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs`
gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is
unaffected (it keeps ConfigDb via its admin role). Consequences:

- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator`
  fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused
  node may stay `Direct`.
- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not
  spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db
  factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and
  `Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes
  **240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture
  — where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself
  is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less
  (driver-only) nodes".
- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore`
  (replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces
  `EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed
  `ScriptedAlarmState` table is now unused (dropping it is a deferred follow-up, tracked as Task 9).
- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on
  a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is
  the system of record.
- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty
  `NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so
  `OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only —
  the same rows a driver node never received via config either, so this is not a regression.

Code-complete on `feat/mesh-phase4`; the table-drop (Task 9) and the live gate (Task 10) are still
open. See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and
`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4.

## LDAP Authentication

The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.

Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)

## Library Preferences

- **Logging**: Serilog with rolling daily file sink
- **Unit tests**: xUnit + Shouldly for assertions
- **Service hosting (Server, Admin)**: .NET generic host with `AddWindowsService` (decision #30 — replaced TopShelf in v2; see `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs`)
- **OPC UA**: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: `OPCFoundation.NetStandard.Opc.Ua.Server`

## OPC UA .NET Standard Documentation

Use the DeepWiki MCP (`mcp__deepwiki`) to query documentation for the OPC UA .NET Standard stack: `https://deepwiki.com/OPCFoundation/UA-.NETStandard`. Tools: `read_wiki_structure`, `read_wiki_contents`, and `ask_question` with repo `OPCFoundation/UA-.NETStandard`.

## Testing

Use the Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/` for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See `docs/Client.CLI.md` for full documentation.

```bash
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode"
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500
```

Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`.

The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`.

**v3 (Batch 2): device I/O is authored in the `/raw` project tree** (`GlobalRaw.razor` → `RawTree.razor`, `IRawTreeService`) — a cluster-rooted `Folder → Driver → Device → TagGroup → Tag` hierarchy with per-node context menus (the reusable `ContextMenu`), lazy loading, driver/device config modals (endpoint moved to `Device.DeviceConfig`; Test-connect probes the `DriverDeviceConfigMerger`-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed `/clusters/{id}/drivers` flow (`DriverTypePicker`, `DriverEditRouter`, the 8 `*DriverPage` shells, the Drivers list) is **retired** — its form bodies live on inside the `/raw` modals. The `DriverType` dispatch maps are keyed off the `DriverTypeNames` constants (fixing the `TwinCat`/`Focas` drift). A `Calculation` pseudo-driver (`DriverTypeNames.Calculation`, `IDependencyConsumer`, deploy-time `scriptId`-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See `docs/Raw.md`.

**v3 (Batch 3): UNS Equipment is reference-only.** The equipment **Tags** tab no longer authors or binds tags — it holds `UnsTagReference` rows pointing at raw tags authored in `/raw`. "+ Add reference" opens the `RawTree` in picker mode, **cluster-scoped** (structurally + server-enforced `tag.cluster == equipment.cluster`); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. **Effective-name uniqueness** (references + VirtualTags + ScriptedAlarms share the `{EquipmentId}/{EffectiveName}` NodeId space) is enforced at authoring (`IEffectiveNameGuard`) and at the deploy gate (`DraftValidator.UnsEffectiveNameCollision` — catches rename-induced collisions). Scripts use **`ctx.GetTag("{{equip}}/<RefName>")`** — resolved per-equipment through `UnsTagReference` effective names to the backing RawPath at both compose seams (`AddressSpaceComposer` + `DeploymentArtifact`, via the shared `EquipmentReferenceMap`, byte-parity); an unresolved `<RefName>` is a deploy error (`EquipReferenceUnresolved`) AND a live Monaco diagnostic (`OTSCRIPT_EQUIPREF`) — editor accepts ⇔ publish accepts. `EquipmentScriptPaths.DeriveEquipmentBase` is deleted (the dot-joint `{{equip}}.X` became the slash-joint `{{equip}}/<RefName>`). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (`RawTreeService` substring scan). `ImportEquipmentModal` dropped the `DriverInstanceId` column. See `docs/Uns.md` + `docs/ScriptEditor.md`.

**v3 (Batch 4 = v3.0): dual-namespace address space.** The OPC UA server exposes **two namespaces** — `https://zb.com/otopcua/raw` (Raw device tree, `s=<RawPath>`) + `https://zb.com/otopcua/uns` (UNS tree, `s=<Area>/<Line>/<Equipment>/<EffectiveName>`) — replacing the single `.../ns` and the retired `EquipmentNodeIds` scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in `DriverHostActor` to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable `Organizes`-references its raw node. Writes route via **either** NodeId to the same driver ref under the same realm-qualified `WriteOperate` gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (`ConditionId = RawPath`) and fan via SDK `AddNotifier` to the raw device folder + every referencing equipment folder (one `ReportEvent`, one Server-object copy); both NodeIds register the SAME historian tagname (mux `HistorizedTagRef` stays single, keyed by RawPath). Identity authority: `V3NodeIds` + `AddressSpaceRealm` (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, `docs/Uns.md`, and `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`.

The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (only Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `<Driver>TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`.

## Scripting / Script Editor

C# virtual-tag scripts are authored in a **Roslyn-backed Monaco editor** on the
ScriptEdit page (`/scripts/{id}`) and inline inside the virtual-tag modal on the
`/uns` page. The editor provides completions, live diagnostics, hover, signature
help, document formatting, and tag-path completions inside `ctx.GetTag("…")` /
`ctx.SetVirtualTag("…")` literals — all backed by the same compiler context the
runtime publish gate uses, so what the editor accepts/rejects matches publish
exactly. The backend lives in
`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/` (six minimal-API
endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
(Administrator or Designer) via `RequireAuthorization` in
`ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`).
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.

## Scripted Alarm Ack/Shelve

Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` LDAP role via `RoleCarryingUserIdentity`; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`.

## Historian / HistoryRead

**Backend: HistorianGateway (sole historian backend).** As of the gateway-integration cutover, the
historian read, alarm-write, and continuous-historization paths are all served by the
**`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed
**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin
`IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke
Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were
retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`,
now a retired stub).

A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the
raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default
historian tagname, which is the tag's driver `FullName`.

### Read path (`ServerHistorian` section)

The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
(historized nodes return `GoodNoData`, never an error). Supported variants: Raw, Processed
(Average/Minimum/Maximum/Total/Count aggregates), and AtTime over historized variable nodes; Events over
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.

`ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`):

| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` |
| `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) |
| `ApiKey` | `""` | Peppered-HMAC key `histgw_<id>_<secret>` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** |
| `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme |
| `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) |
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |

### Alarm-history path (`AlarmHistorian` section)

Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).

### Continuous historization (`ContinuousHistorization` section)

When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a
durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed
`IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The
recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the
crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**)
with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries
only the recorder + outbox knobs:

| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder |
| `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage |
| `CommitMode` | `PerEntry` | `PerEntry` = fsync before each append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs` |
| `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` |
| `DrainBatchSize` | `64` | Entries peeked + written per drain pass |
| `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) |
| `Capacity` | `0` | Max un-acked outbox entries before drop-oldest; `0` = unbounded |
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |

### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)

`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync`
hook for added historized value tags — the gateway-backed `GatewayTagProvisioner` calls the gateway's
`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues`
land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a
deploy. Non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not provisioned); the
recorder likewise drops + meters non-numeric values. Continuous historization is **numeric-analog only** in
v1 (`UInt16→UInt4` is a documented fallback).

The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`),
built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the
read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier.
When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
the "Boolean historization maps to Int2" section in `docs/Historian.md`.

### Gateway-side prerequisites

The target HistorianGateway OtOpcUa points at **must** run with:

- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.

### Migration note (deployments upgrading from the Wonderware backend)

The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env:

| Old (Wonderware) key | New (gateway) key |
|---|---|
| `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) |
| `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) |
| `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) |

The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`)
were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
`ServerHistorian`. See `docs/Historian.md` for the full guide.

### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")

**No longer a limitation.** The cutover was live-validated against the real gateway on
`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
populate `Source_Object` — not a fixable bug). Evidence:
`docs/plans/2026-06-27-otopcua-historian-followups.md`.

Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
`wonder-sql-vd03`, gateway running the prerequisites above):

```bash
export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret>             # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag>                         # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag>          # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag>            # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name>                      # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"
```

The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
gate the operator runs on the VPN before trusting the cutover.

### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)

The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built
later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the
`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it
through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the
recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the
deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe
`Tell`; a faulting feed can never break a deploy). **Reads and alarm-writes work today, and the value-capture
path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway
(deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test
proving the recorder re-registers the deployed set after a process restart. Until that live verification runs,
treat continuous value-capture as unproven-in-production rather than absent. Tracked as **issue #491**.