lmxopcua/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Goal

Build an OPC UA server (.NET 10) that exposes industrial data sources —
including AVEVA System Platform (Wonderware) Galaxy — under a unified
Equipment-based address space. Galaxy access flows through the in-process
`GalaxyDriver` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/`) talking gRPC
to a separately installed **mxaccessgw** gateway process. The gateway owns the
MXAccess COM bitness constraint (its worker is x86 net48); everything in this
repo is .NET 10. PR 7.2 retired the legacy in-process `Galaxy.Host` /
`Galaxy.Proxy` / `Galaxy.Shared` projects + the `OtOpcUaGalaxyHost` Windows
service.

See `docs/v2/Galaxy.Performance.md` for the runtime perf surface
(tracing, metrics, soak harness).

## Related Projects (scadaproj umbrella)

This repo is one of a family of related SCADA / OT / Wonderware / OPC UA
**sister projects** that live as sibling directories under `~/Desktop/`. They are
**separate repos and separate processes**, coupled at runtime over wire protocols
(gRPC + OPC UA) — not by project/compile references — and share the
`ZB.MOM.WW.*` product namespace. The common subject is AVEVA System Platform
(Wonderware) "Galaxy" data, and `mxaccessgw` is the linchpin the others connect
through.

- **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) —
  the umbrella/index workspace that aggregates the whole family (a high-level scan
  of each sister project: purpose, location, stack, primary commands). It also hosts
  the shared `ZB.MOM.WW.*` libraries — `ZB.MOM.WW.Auth`, `ZB.MOM.WW.Theme`,
  `ZB.MOM.WW.Health`, `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Configuration`, and
  `ZB.MOM.WW.GalaxyRepository`. **This repo is indexed there** — see the OtOpcUa
  entry in `../scadaproj/CLAUDE.md`.
- **`~/Desktop/MxAccessGateway`** (`mxaccessgw`,
  [`../MxAccessGateway/CLAUDE.md`](../MxAccessGateway/CLAUDE.md)) — the gRPC gateway
  that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session
  x86 net48 worker). **OtOpcUa's `GalaxyDriver` depends on this** for live Galaxy
  read/write/subscribe and Galaxy Repository browse.
- **`~/Desktop/HistorianGateway`**
  ([`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process
  gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse
  (via the shared `ZB.MOM.WW.GalaxyRepository` lib). Patterned on `mxaccessgw` but with
  no COM / no x86 worker. **OtOpcUa's historian backend** (`Driver.Historian.Gateway`)
  consumes it as the sole historian path.
- **`~/Desktop/ScadaBridge`**
  ([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) — the distributed SCADA
  platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's
  OPC UA address space.

**Propagate cross-repo changes to the umbrella index.** When a fact the index records
about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow,
shared-lib consumption, or per-project commands — update the **OtOpcUa entry in
`../scadaproj/CLAUDE.md`** in the same change so the index never drifts from this repo.

## Architecture Overview

### Data Flow

1. **Galaxy Repository DB (ZB)** — SQL Server database holding the
   deployed object hierarchy and attribute definitions. The
   mxaccessgw's `GalaxyRepositoryClient` queries it via gRPC; the
   driver consumes the materialised hierarchy through
   `IGalaxyHierarchySource`.
2. **MXAccess (via mxaccessgw)** — Live read/write/subscribe over a
   gRPC session. The gateway owns the COM apartment + STA pump
   server-side; the driver speaks `MxCommand` / `MxEvent` protos
   exclusively.
3. **OPC UA Server** — Exposes the deployed tree under **two OPC UA namespaces**
   (see below). Galaxy tags are bound by `TagConfig.FullName`
   (`tag_name.AttributeName`); reads/writes/subscriptions are translated to that
   reference for MXAccess.

### v3 OPC UA Address Space (Batch 4): dual namespace

The server exposes **two OPC UA namespaces** (replacing the single
`https://zb.com/otopcua/ns`), built by `AddressSpaceComposer` /
`AddressSpaceApplier` and served by `OtOpcUaNodeManager` (which registers both
URIs; `V3NodeIds` + `AddressSpaceRealm` are the identity authority):

| Namespace URI | Realm | Subtree | NodeId `s=` scheme |
|---|---|---|---|
| `https://zb.com/otopcua/raw` | `AddressSpaceRealm.Raw` | the `/raw` device tree (Folder → Driver → Device → TagGroup → Tag) | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) |
| `https://zb.com/otopcua/uns` | `AddressSpaceRealm.Uns` | the UNS tree (Area → Line → Equipment → signal) | slash-joined **`Area/Line/Equipment/EffectiveName`** |

- **Single source, fanned to both.** Every device value has exactly one source —
  the raw tag's node. A `UnsTagReference` projects that raw tag into an equipment
  as a UNS-namespace variable that carries an **`Organizes` reference to its raw
  node** and mirrors it. One driver publish for a RawPath fans (in `DriverHostActor`)
  to the raw NodeId AND every referencing UNS NodeId with identical
  value/quality/timestamp — the mux key stays single (RawPath).
- **Writes via either NodeId** route to the same driver ref under the same
  `WriteOperate` gating (the node-manager write gate is realm-qualified and fails
  closed); a failed device write reverts both NodeIds via the shared fan-out
  (write-outcome self-correction).
- **Native alarms** materialize once at the raw tag (`ConditionId = RawPath`) and
  fan via SDK notifiers to the raw device folder AND every referencing equipment
  folder — one `ReportEvent`, one Server-object copy; ack/confirm/shelve route on
  `ConditionId = RawPath`.
- **Historian dual-registration.** Both NodeIds register the **same** historian
  tagname; the mux `HistorizedTagRef` stays single (RawPath). HistoryRead works
  through either NodeId.

The retired `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme and
the single-namespace `EquipmentTags` materialization path are gone. See
`docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` +
`docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`.

### Key Concept: Tag Name and FullName

Galaxy objects have a **tag_name** — a globally unique system name used for
MXAccess read/write — and attributes are referenced as `tag_name.AttributeName`
(e.g. `DelmiaReceiver_001.DownloadPath`). This dot-separated reference is what
`TagConfig.FullName` stores for a Galaxy equipment tag. The Galaxy address
picker browses the live hierarchy using contained names for navigation, then
resolves the selection to the `tag_name.AttributeName` form.

**Galaxy is a standard Equipment-kind driver.** Galaxy points are ordinary
equipment `Tag`s bound to `GalaxyMxGateway` via `TagConfig.FullName`
(`tag_name.AttributeName`), authored through the standard Tag modal + Galaxy
address picker on the equipment page's Tags tab — the same flow as Modbus or
S7. There is no alias machinery, no `SystemPlatform` namespace kind, and no
relay→alias converter. See
`docs/plans/2026-06-12-galaxy-standard-driver-design.md` for the full design.

### Data Type Mapping

Galaxy `mx_data_type` values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`.

### Change Detection

`DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys. The server's `DriverHost` consumes the signal and rebuilds the address space.

## mxaccessgw

The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3.

## Build Commands

```bash
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx
dotnet test ZB.MOM.WW.OtOpcUa.slnx                                # all tests
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests               # a single test project
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod"    # a single test
```

CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`)
is **resolved** — the temporary `NuGetAuditSuppress` was removed once the patched bundle
shipped. `Core.AlarmHistorian` now carries a surgical direct pin of
`SQLitePCLRaw.bundle_e_sqlite3` to `2.1.12` (the first fixed native bundle, embedding the
SQLite 3.50.2+ patch), overriding the `2.1.11` that `Microsoft.Data.Sqlite` pulls
transitively. `dotnet restore`/`build` is clean with the audit active.

Test projects live under `tests/<module>/` (Core, Server, Drivers,
Drivers/Cli, Client, Tooling) — there is no single unit-test project.
Unit suites are named `*.Tests`; integration suites are `*.IntegrationTests`
and need their Docker fixture up (see Docker Workflow). DB-backed tests in
`*.Configuration.Tests`, `*.Admin.Tests`, and `*.Server.Tests` require the
central SQL Server.

## Docker Workflow (driver fixtures + central SQL Server)

> **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing +
> deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate**
> recipes (S7 blackhole, GLAuth outage, HistorianGateway LiveIntegration), the integration-test
> harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail.

> **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. The driver fixture compose files under `tests/.../Docker/docker-compose.yml` now carry a `project: lmxopcua` label on every service. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28).

Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-<driver>/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`.

**`docker -H ssh://...` does NOT work from this VM.** Windows OpenSSH ↔ docker.exe stdio bridging hangs (`docker system dial-stdio` runs server-side but no API data flows). Use the helper below — it SSHes into the docker host and runs `docker compose` server-side.

**Use `lmxopcua-fix.ps1` (in `~/bin`) to control fixtures from this VM:**

```powershell
lmxopcua-fix ls                            # list all lmxopcua-tagged containers on the host
lmxopcua-fix up   modbus      standard     # bring a profile up
lmxopcua-fix up   abcip       controllogix
lmxopcua-fix up   s7          s7_1500
lmxopcua-fix up   opcuaclient                # single-service stack, no profile arg
lmxopcua-fix down modbus                   # tear stack down
lmxopcua-fix logs modbus
lmxopcua-fix sync modbus                   # rsync this repo's tests/.../Docker/ → /opt/otopcua-modbus/
```

**`sync` is the deployment step.** When you edit a fixture's compose file or Dockerfile under `tests/.../Docker/`, run `lmxopcua-fix sync <driver>` to push the changes to the docker host before bringing the stack up. The repo files are the source of truth; `/opt/otopcua-<driver>/` is a mirrored deployment.

**Endpoints (defaults already point at the docker host):**
- SQL Server (always-on): `10.100.0.35,14330` — used by `appsettings.json` for `ConfigDb`.
- Modbus: `10.100.0.35:5020` (`MODBUS_SIM_ENDPOINT`)
- AB CIP: `10.100.0.35:44818` (`AB_SERVER_ENDPOINT`)
- S7: `10.100.0.35:1102` (`S7_SIM_ENDPOINT`)
- OPC UA reference (opc-plc): `opc.tcp://10.100.0.35:50000` (`OPCUA_SIM_ENDPOINT`)

Override any endpoint via the env var to point at a real PLC. The local OtOpcUa server runs on this VM at `opc.tcp://localhost:4840` — **that's not on the docker host**.

**Local docker-dev rig — login is DISABLED, so do live `/run` verification yourself (don't wait for the user to sign in).** The local `docker-dev/docker-compose.yml` stack (AdminUI at `http://localhost:9200` via Traefik; OPC UA `opc.tcp://localhost:4840` central-1 / `:4841` central-2) runs the AdminUI with `Security__Auth__DisableLogin: "true"` — **no sign-in form; it's auto-authenticated as a full-access admin.** So AdminUI / Razor `/run` verification (deploy a config, drive a page, confirm behavior — e.g. via the Chrome browser-automation tools against `http://localhost:9200`) does **not** require the user to log in. Run it yourself; do not defer it as "user-driven sign-in required." (Caveat: OPC UA *data-plane* auth is still real LDAP against the shared GLAuth on `10.100.0.35:3893` — that only gates Client.CLI read/write **role** operations, e.g. binding a `multi-role` / `opc-writeop` user, and is independent of the AdminUI login. Things genuinely outside the local rig — real PLCs, or the AVEVA Historian reached via the `ZB.MOM.WW.HistorianGateway` sidecar — still need the user.)

See `docs/v2/dev-environment.md` for the full inventory and rationale.

## Build & Runtime Constraints

- Language: C#, .NET 10, AnyCPU. The MXAccess COM bitness constraint
  is owned by the mxaccessgw worker (x86 net48), not by anything in
  this repo.
- The gateway's MXAccess worker requires a deployed ArchestrA Platform
  on the machine running the gateway. The OtOpcUa server itself does
  not.

## Transport Security

The server supports configurable OPC UA transport security via the `OpcUa:EnabledSecurityProfiles` list in `appsettings.json`. Phase 1 profiles (the `OpcUaSecurityProfile` enum members): `None` (default), `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`. Security policies are built from the enabled profiles by `BuildSecurityPolicies` at startup. The server certificate is always created even for `None`-only deployments because `UserName` token encryption depends on it. See `docs/security.md` for the full guide.

## Redundancy

The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.

**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**

- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.

**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".

Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.

**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started).

**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:

- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.

## LocalDb pair-local store (Phases 1 + 2)

Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).

**Phase 1** caches the
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
never register it.** The cache can optionally replicate to the node's redundant pair peer over the
library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
exception).

**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
same event accepted on both nodes — which happens in every boot window, since
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.

See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), `docs/AlarmHistorian.md`, and the runbook
`docs/operations/2026-07-20-localdb-pair-replication.md`.

## LDAP Authentication

The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.

Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)

## Library Preferences

- **Logging**: Serilog with rolling daily file sink
- **Unit tests**: xUnit + Shouldly for assertions
- **Service hosting (Server, Admin)**: .NET generic host with `AddWindowsService` (decision #30 — replaced TopShelf in v2; see `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs`)
- **OPC UA**: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: `OPCFoundation.NetStandard.Opc.Ua.Server`

## OPC UA .NET Standard Documentation

Use the DeepWiki MCP (`mcp__deepwiki`) to query documentation for the OPC UA .NET Standard stack: `https://deepwiki.com/OPCFoundation/UA-.NETStandard`. Tools: `read_wiki_structure`, `read_wiki_contents`, and `ask_question` with repo `OPCFoundation/UA-.NETStandard`.

## Testing

Use the Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/` for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See `docs/Client.CLI.md` for full documentation.

```bash
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode"
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500
```

Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`.

The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`.

**v3 (Batch 2): device I/O is authored in the `/raw` project tree** (`GlobalRaw.razor` → `RawTree.razor`, `IRawTreeService`) — a cluster-rooted `Folder → Driver → Device → TagGroup → Tag` hierarchy with per-node context menus (the reusable `ContextMenu`), lazy loading, driver/device config modals (endpoint moved to `Device.DeviceConfig`; Test-connect probes the `DriverDeviceConfigMerger`-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed `/clusters/{id}/drivers` flow (`DriverTypePicker`, `DriverEditRouter`, the 8 `*DriverPage` shells, the Drivers list) is **retired** — its form bodies live on inside the `/raw` modals. The `DriverType` dispatch maps are keyed off the `DriverTypeNames` constants (fixing the `TwinCat`/`Focas` drift). A `Calculation` pseudo-driver (`DriverTypeNames.Calculation`, `IDependencyConsumer`, deploy-time `scriptId`-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See `docs/Raw.md`.

**v3 (Batch 3): UNS Equipment is reference-only.** The equipment **Tags** tab no longer authors or binds tags — it holds `UnsTagReference` rows pointing at raw tags authored in `/raw`. "+ Add reference" opens the `RawTree` in picker mode, **cluster-scoped** (structurally + server-enforced `tag.cluster == equipment.cluster`); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. **Effective-name uniqueness** (references + VirtualTags + ScriptedAlarms share the `{EquipmentId}/{EffectiveName}` NodeId space) is enforced at authoring (`IEffectiveNameGuard`) and at the deploy gate (`DraftValidator.UnsEffectiveNameCollision` — catches rename-induced collisions). Scripts use **`ctx.GetTag("{{equip}}/<RefName>")`** — resolved per-equipment through `UnsTagReference` effective names to the backing RawPath at both compose seams (`AddressSpaceComposer` + `DeploymentArtifact`, via the shared `EquipmentReferenceMap`, byte-parity); an unresolved `<RefName>` is a deploy error (`EquipReferenceUnresolved`) AND a live Monaco diagnostic (`OTSCRIPT_EQUIPREF`) — editor accepts ⇔ publish accepts. `EquipmentScriptPaths.DeriveEquipmentBase` is deleted (the dot-joint `{{equip}}.X` became the slash-joint `{{equip}}/<RefName>`). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (`RawTreeService` substring scan). `ImportEquipmentModal` dropped the `DriverInstanceId` column. See `docs/Uns.md` + `docs/ScriptEditor.md`.

**v3 (Batch 4 = v3.0): dual-namespace address space.** The OPC UA server exposes **two namespaces** — `https://zb.com/otopcua/raw` (Raw device tree, `s=<RawPath>`) + `https://zb.com/otopcua/uns` (UNS tree, `s=<Area>/<Line>/<Equipment>/<EffectiveName>`) — replacing the single `.../ns` and the retired `EquipmentNodeIds` scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in `DriverHostActor` to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable `Organizes`-references its raw node. Writes route via **either** NodeId to the same driver ref under the same realm-qualified `WriteOperate` gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (`ConditionId = RawPath`) and fan via SDK `AddNotifier` to the raw device folder + every referencing equipment folder (one `ReportEvent`, one Server-object copy); both NodeIds register the SAME historian tagname (mux `HistorizedTagRef` stays single, keyed by RawPath). Identity authority: `V3NodeIds` + `AddressSpaceRealm` (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, `docs/Uns.md`, and `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`.

The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (only Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `<Driver>TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`.

## Scripting / Script Editor

C# virtual-tag scripts are authored in a **Roslyn-backed Monaco editor** on the
ScriptEdit page (`/scripts/{id}`) and inline inside the virtual-tag modal on the
`/uns` page. The editor provides completions, live diagnostics, hover, signature
help, document formatting, and tag-path completions inside `ctx.GetTag("…")` /
`ctx.SetVirtualTag("…")` literals — all backed by the same compiler context the
runtime publish gate uses, so what the editor accepts/rejects matches publish
exactly. The backend lives in
`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/` (six minimal-API
endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
(Administrator or Designer) via `RequireAuthorization` in
`ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`).
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.

## Scripted Alarm Ack/Shelve

Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` LDAP role via `RoleCarryingUserIdentity`; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`.

## Historian / HistoryRead

**Backend: HistorianGateway (sole historian backend).** As of the gateway-integration cutover, the
historian read, alarm-write, and continuous-historization paths are all served by the
**`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed
**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin
`IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke
Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were
retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`,
now a retired stub).

A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the
raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default
historian tagname, which is the tag's driver `FullName`.

### Read path (`ServerHistorian` section)

The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
(historized nodes return `GoodNoData`, never an error). Supported variants: Raw, Processed
(Average/Minimum/Maximum/Total/Count aggregates), and AtTime over historized variable nodes; Events over
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.

`ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`):

| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` |
| `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) |
| `ApiKey` | `""` | Peppered-HMAC key `histgw_<id>_<secret>` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** |
| `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme |
| `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) |
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |

### Alarm-history path (`AlarmHistorian` section)

Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).

### Continuous historization (`ContinuousHistorization` section)

When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a
durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed
`IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The
recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the
crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**)
with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries
only the recorder + outbox knobs:

| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder |
| `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage |
| `CommitMode` | `PerEntry` | `PerEntry` = fsync before each append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs` |
| `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` |
| `DrainBatchSize` | `64` | Entries peeked + written per drain pass |
| `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) |
| `Capacity` | `0` | Max un-acked outbox entries before drop-oldest; `0` = unbounded |
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |

### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)

`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync`
hook for added historized value tags — the gateway-backed `GatewayTagProvisioner` calls the gateway's
`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues`
land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a
deploy. Non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not provisioned); the
recorder likewise drops + meters non-numeric values. Continuous historization is **numeric-analog only** in
v1 (`UInt16→UInt4` is a documented fallback).

The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`),
built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the
read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier.
When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
the "Boolean historization maps to Int2" section in `docs/Historian.md`.

### Gateway-side prerequisites

The target HistorianGateway OtOpcUa points at **must** run with:

- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.

### Migration note (deployments upgrading from the Wonderware backend)

The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env:

| Old (Wonderware) key | New (gateway) key |
|---|---|
| `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) |
| `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) |
| `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) |

The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`)
were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
`ServerHistorian`. See `docs/Historian.md` for the full guide.

### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")

**No longer a limitation.** The cutover was live-validated against the real gateway on
`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
populate `Source_Object` — not a fixable bug). Evidence:
`docs/plans/2026-06-27-otopcua-historian-followups.md`.

Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
`wonder-sql-vd03`, gateway running the prerequisites above):

```bash
export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret>             # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag>                         # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag>          # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag>            # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name>                      # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"
```

The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
gate the operator runs on the VPN before trusting the cutover.

### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)

The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built
later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the
`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it
through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the
recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the
deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe
`Tell`; a faulting feed can never break a deploy). **Reads and alarm-writes work today, and the value-capture
path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway
(deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test
proving the recorder re-registers the deployed set after a process restart. Until that live verification runs,
treat continuous value-capture as unproven-in-production rather than absent. Tracked as **issue #491**.