lmxopcua/docs/security.md

# Security

> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.

> **v2 status (2026-05-26).** The four security concerns below are unchanged in v2.
> Paths + project names moved: `OtOpcUa.Server/Security/` → `OtOpcUa.Security/`
> (`Ldap/`, `Jwt/`, `Endpoints/AuthEndpoints.cs`), `OtOpcUa.Admin` is gone (its
> auth + role-grant pages live in `OtOpcUa.AdminUI`), and Admin auth policies
> register from `OtOpcUa.Host/Program.cs` via `AddOtOpcUaAuth`
> (`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs`) rather
> than in a separate Admin process. The Admin UI session is a **Cookie**
> scheme; the cookie stores the `ClaimsPrincipal` directly. JWT is **opt-in by
> `Security:Jwt:SigningKey`** (validated at host start, ≥32 bytes): with a key,
> `/auth/token` mints an HS256 token for a **cookie session only** (a bearer
> cannot renew itself), and a first-party `AddJwtBearer` scheme (`Bearer`,
> `ValidAlgorithms = HS256`, parameters from `JwtTokenService.BuildValidationParameters`)
> accepts it on `/api/*` and every `RequireAuthorization` endpoint through the
> `OtOpcUa.CookieOrBearer` default policy scheme. Without a key neither exists. DataProtection
> keys persist to the shared Config DB (`PersistKeysToDbContext<OtOpcUaConfigDbContext>`)
> so cookies survive failover between admin-role nodes.
>
> See `docs/plans/2026-05-26-akka-hosting-alignment-design.md` §5 for the v2
> auth + DataProtection rationale.

OtOpcUa has four independent security concerns. This document covers all four:

1. **Transport security** — OPC UA secure channel (signing, encryption, X.509 trust).
2. **OPC UA authentication** — Anonymous / UserName / X.509 session identities. A UserName token is an **API key** (username literally `apikey`, password `otua_<keyId>_<secret>`) verified by peppered HMAC — **LDAP backs Admin UI login only** since #520 (2026-08-01).
3. **Data-plane authorization** — who can do what on an OtOpcUa endpoint. ⚠️ Governed by **one flag** (#520 stages 5–6, 2026-08-01). With `Security:Acl:Enabled` **false — the default** — the only enforcement is two coarse, server-wide role gates (`WriteOperate` on Value writes, `AlarmAck` on the alarm methods), and Read, Browse, Subscribe and HistoryRead are **ungated** for any admitted session. With it **true**, scoped `NodeAcl` grants (the `PermissionTrie` subsystem) decide **every** data-plane operation — read, browse, subscribe, write, alarm commands and HistoryRead — *replacing* the server-wide roles. See [Data-Plane Authorization](#data-plane-authorization).
4. **Control-plane authorization** — who can view or edit fleet configuration in the Admin UI. Gated by the `AdminRole` (`Viewer` / `Designer` / `Administrator`) claim resolved from `LdapGroupRoleMapping`.

Transport security and OPC UA authentication are per-node concerns configured in the Server's bootstrap `appsettings.json`. Data-plane ACLs and Admin role grants live in the Config DB.

---

## Transport Security

### SDK 2.0.1 compatibility (#782)

The application targets exact OPC Foundation SDK **2.0.1**, replacing **1.5.378.182**.
The upgrade retains the existing host, security profiles, anonymous-admission setting,
API-key authentication and ACL flag semantics. It adds no companion-specification or
PubSub capability. Release evidence and operator gates are linked from the
[rollout runbook](plans/2026-11-10-opcua-sdk-2-rollout.md); source changes alone do not establish release approval.

SDK role augmentation preserves the application's principal, API-key id and application
roles. A rejected API key must remain rejected rather than falling through to another
identity provider. Per-IP authentication miss accounting uses the public transport-observed
`SecureChannelContext.PeerAddress` during activation; an unknown peer retains the existing
process-wide budget fallback. This is accounting evidence, not a client-supplied address or
a new IP admission policy.

SDK event receive checks compose with application ACL filtering. For the application's
rejected-write `AuditWriteUpdateEvent` only, a temporary validation context supplies
`ReceiveEvents` to the recipient's SDK roles after the current source variable, effective
identity, source `Read` grant and audit/encryption guard are checked. It expires before
queueing; shared role-permission metadata is not widened. Prior permission callback
errors veto delivery. SDK-generated audits retain SDK defaults. This does not enable
auditing or grant a general audit feed.

Certificate configuration uses the SDK certificate manager, and certificate ownership
lasts through the owning host/session's lifetime. Preserve the existing application
certificate, private key, issuer/peer stores and filesystem permissions during rollout;
verify key access and CA usage under the service identity. Do not rotate certificates,
enable auto-accept, add `None`, or relax revocation checks to get the candidate running.
Continuation points remain session-owned and bound to node/topology and current authority;
restart requires fresh sessions and fresh points.

### Overview

The OtOpcUa Server supports configurable OPC UA transport security profiles that control how data is protected on the wire between OPC UA clients and the server.

There are two distinct layers of security in OPC UA:

- **Transport security** -- secures the communication channel itself using TLS-style certificate exchange, message signing, and encryption. This is what the `OpcUa:EnabledSecurityProfiles` setting controls.
- **UserName token encryption** -- protects user credentials (username/password) sent during session activation. The OPC UA stack encrypts UserName tokens using the server's application certificate regardless of the transport security mode. UserName authentication therefore works on `None` endpoints too — the credentials themselves are always encrypted. A secure transport profile adds protection against message-level tampering and eavesdropping of data payloads.

### Supported security profiles

The profiles are the members of the `OpcUaSecurityProfile` enum (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`). The server ships **three** baseline profiles; the config value is the bare enum-member name (no hyphens, no underscores):

| Enum member                     | Security Policy  | Message Security Mode | Description                                      |
|---------------------------------|------------------|-----------------------|--------------------------------------------------|
| `None`                          | None             | None                  | No signing or encryption. Suitable for development and isolated networks only. |
| `Basic256Sha256Sign`            | Basic256Sha256   | Sign                  | Messages are signed but not encrypted. Protects against tampering but data is visible on the wire. |
| `Basic256Sha256SignAndEncrypt`  | Basic256Sha256   | SignAndEncrypt        | Messages are both signed and encrypted. Full protection against tampering and eavesdropping. |

`BuildSecurityPolicies` (`OpcUaApplicationHost.cs`) maps each configured profile to an SDK `ServerSecurityPolicy`. The server exposes a separate endpoint per configured profile and clients select the one they prefer at session open. The enum's XML doc notes that Aes128/Aes256 variants can be added later by extending the enum + `BuildSecurityPolicies` — the wiring is profile-agnostic — but they are **not implemented today**. There is no `SecurityProfileResolver` class.

> **Config value form.** The enum binds by member name, so a profile string with hyphens (e.g. `Basic256Sha256-Sign`) does **not** bind — use the exact enum-member spelling above. If `EnabledSecurityProfiles` is empty, the server falls back to a single `None` endpoint (logged, very visible) so it still has a listening endpoint.

### Configuration

Transport security is configured in the `OpcUa` section of the Host process's bootstrap `appsettings.json` (bound to `OpcUaApplicationHostOptions`):

```json
{
  "OpcUa": {
    "ApplicationName": "OtOpcUa",
    "ApplicationUri": "urn:node-a:OtOpcUa",
    "PublicHostname": "0.0.0.0",
    "OpcUaPort": 4840,
    "PkiStoreRoot": "C:/ProgramData/OtOpcUa/pki",
    "AutoAcceptUntrustedClientCertificates": false,
    "EnabledSecurityProfiles": [ "Basic256Sha256Sign", "Basic256Sha256SignAndEncrypt" ]
  }
}
```

`EnabledSecurityProfiles` is a **list** — the server publishes one endpoint per entry. The default (when the key is omitted) is all three baseline profiles (`None`, `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`); production deployments typically drop `None`. The list must contain at least one entry (`OpcUaApplicationHostOptionsValidator` enforces `MinCount(…, 1)`).

The server certificate is auto-generated on first start if none exists in `PkiStoreRoot/own/`. Always generated even for `None`-only deployments because UserName token encryption depends on it.

### PKI directory layout

```
{PkiStoreRoot}/
  own/        Server's own application certificate and private key
  issuer/     CA certificates that issued trusted client certificates
  trusted/    Explicitly trusted client (peer) certificates
  rejected/   Certificates that were presented but not trusted
```

### Certificate trust flow

When a client connects using a secure profile (`Sign` or `SignAndEncrypt`), the following trust evaluation occurs:

1. The client presents its application certificate during the secure channel handshake.
2. The server checks whether the certificate exists in the `trusted/` store.
3. If found, the connection proceeds.
4. If not found and `AutoAcceptUntrustedClientCertificates` is `true`, the certificate is automatically copied to `trusted/` and the connection proceeds.
5. If not found and `AutoAcceptUntrustedClientCertificates` is `false`, the certificate is copied to `rejected/` and the connection is refused.

The Admin UI `Certificates.razor` page (`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Certificates.razor`) lists the contents of each PKI sub-store (own / trusted / issuer / rejected) by reading the `OpcUa:PkiStoreRoot` path from configuration. **FleetAdmin** users get per-row actions on the **trusted** and **rejected** stores: **Trust** (move a rejected cert into `trusted/`), **Untrust** (move a trusted cert back into `rejected/`), and **Delete** (remove a cert from either store). Each action requires an inline confirmation, is re-checked against the `FleetAdmin` policy server-side, and is honored live — the SDK trust list re-enumerates the directory store on the next handshake, no restart required. (own / issuer remain read-only.) The actions are filesystem moves/deletes performed by `CertificateStoreManager`, which finds the target by matching the certificate thumbprint — never by building a path from caller input.

### Production hardening

- Set `AutoAcceptUntrustedClientCertificates = false`.
- Drop `None` from `EnabledSecurityProfiles`.
- Promote trusted client certs with the Admin UI Certificates page's **Trust** action (FleetAdmin) — or by moving the `.der` from `rejected/` to `trusted/certs/` — rather than relying on the auto-accept fallback.
- Periodically audit the `rejected/` directory; an unexpected entry is often a misconfigured client or a probe attempt.

### Internal gRPC planes (h2c) — accepted cleartext posture

Everything above concerns the **OPC UA** endpoint — the surface external clients bind to. OtOpcUa
also runs **five internal gRPC planes** between its own processes (central ↔ driver node, and node ↔
pair partner). Those are a separate transport story, recorded here so it is written down once rather
than rediscovered per plane.

| # | Plane | gRPC service | Listener (config key) | Credential (config key) | Interceptor |
|---|---|---|---|---|---|
| 1 | Live telemetry stream (central dials the node) | `telemetry.v1.TelemetryStreamService/Subscribe` | driver node, `Telemetry:GrpcListenPort` | `Telemetry:ApiKey` (dialer presents `TelemetryDial:ApiKey`) | `TelemetryStreamAuthInterceptor` |
| 2 | Config-serve / artifact fetch (node fetches from central) | `deployment_artifact.v1.DeploymentArtifactService/Fetch` | central (admin role), `ConfigServe:GrpcListenPort` | `ConfigServe:ApiKey` (fetcher presents `ConfigSource:ApiKey`) | `ConfigServeAuthInterceptor` |
| 3 | LocalDb pair sync (one node dials its partner) | `localdb_sync.v1.LocalDbSync/Sync` | driver node, `LocalDb:SyncListenPort` | `LocalDb:Replication:ApiKey` (byte-identical on both nodes of the pair) | `LocalDbSyncAuthInterceptor` |
| 4 | Mesh commands (central drives the node) | `mesh_transport.v1.NodeCommandService/Execute` | **shares** the node's `Telemetry:GrpcListenPort` listener | `MeshTransport:GrpcPsk` — **per application `Cluster`**, not fleet-wide | `MeshCommandAuthInterceptor` |
| 5 | Mesh deploy acks (node reports to central) | `mesh_transport.v1.CentralCommandService/ReportApply` | **shares** central's `ConfigServe:GrpcListenPort` listener | per-cluster key resolved by `IMeshPskProvider` — `MeshTransport:ClusterPsks:<clusterId>` override, else the `OT-MESH-PSK-<clusterId>` secret | `CentralCommandAuthInterceptor` |

Planes 4 and 5 claim no port of their own; they ride the listener their sibling already binds. On the
docker-dev rig those two config keys resolve to `:4056` (driver nodes) and `:4055` (central) — rig
values of the keys above, **not** hardcoded ports.

**The posture.** All five listeners are bound `HttpProtocols.Http2` on a **cleartext** port, and every
client dials a prior-knowledge `http://host:port` address — h2c, no TLS, no ALPN. (Cleartext is
structural, not incidental: a cleartext `Http1AndHttp2` endpoint cannot negotiate HTTP/2 without ALPN,
which is why these are dedicated HTTP/2-only ports rather than a multiplex onto the main one.)
Authentication on all five is a **shared bearer token in an `Authorization` header**, compared in
constant time and fail-closed when unconfigured. Because the transport is cleartext, **the bearer
itself crosses the wire in the clear** — anyone who can observe or interpose on the link can read a
key and then replay it.

What that buys an attacker is not uniform:

- Planes 1–2 are **read** surfaces: the live telemetry feed, and the full deployed configuration.
- Plane 3 **writes** rows into a node's deployment-artifact cache — which is what that node boots
  from when central SQL is unreachable.
- Planes 4–5 are the **command** surfaces, and are the sharpest: plane 4 carries `RestartDriver`,
  `AlarmCommand` and deployment dispatch; plane 5 carries the apply acks the deploy coordinator
  counts, so forged acks can seal a deployment nodes never applied.

**Decision: risk accepted, 2026-08-06** (Gitea [`lmxopcua#545`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/545),
option 1). Accepted once, for all five planes together, on the **trusted-link assumption every one of
them already makes**: the central↔site and pair-local links are private (VPN or dedicated WAN), not
routed over anything an untrusted party can observe. Two things bound the consequences if that
assumption is wrong:

- **Per-cluster PSKs bound blast radius on the two command planes.** A captured `SITE-A` key drives
  `SITE-A` only; it is refused by `SITE-B`'s node and by central when it claims to be `SITE-B`.
  A single fleet-wide key would have made one observed link equal to fleet-wide command authority.
- **Production keys are delivered as `${secret:...}` references**, never committed literals, and
  **MUST NOT be reused across clusters** — reusing one key collapses the isolation above back into a
  fleet-wide key with extra steps. The three literal `dev-mesh-psk-*` keys exist only on the
  docker-dev rig.

**Future work.** The recorded alternative is a `UseTls`/scheme option on these planes (dial `https://`,
bind the listener with a certificate). If it is ever built it **must cover all five planes together**
— a mixed posture, where some internal links are encrypted and others are not, is harder to reason
about than either uniform answer, and invites exactly the "which one was it again?" mistake this note
exists to prevent. See issue #545.

---

## OPC UA Authentication

The Server accepts three OPC UA identity-token types:

| Token | Handler | Notes |
|---|---|---|
| Anonymous | No `IOpcUaUserAuthenticator` call — the SDK admits anonymous sessions at the channel. | ⚠️ Since #520 an anonymous session is a **normal grantable principal** (`anonymous`), not a bypass — see [Anonymous is a normal principal](#anonymous-is-a-normal-principal). With `Security:Acl:Enabled` **off (the default)** it typically holds no roles, so every write and alarm ack is refused — but it can **Browse, Read, Subscribe and HistoryRead the entire address space** (those operations are ungated). With the flag **on**, those operations too are decided by the `anonymous` principal's `NodeAcl` grants (see [Data-Plane Authorization](#data-plane-authorization)). Do not expose an Anonymous-admitting endpoint to an untrusted network. |
| UserName/Password | `ApiKeyOpcUaUserAuthenticator.AuthenticateUserNameAsync` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/ApiKeyOpcUaUserAuthenticator.cs`, implements `IOpcUaUserAuthenticator`), backed by an `IApiKeyStore`. | The username is the literal `apikey` and the password is an `otua_<keyId>_<secret>` credential (#520). One indexed lookup plus one peppered HMAC-SHA256, no network I/O. The key resolves to a **principal**, whose `NodeAcl` grants become roles on a `RoleCarryingUserIdentity`. **LDAP is not on this path** — it backs AdminUI login only. |
| X.509 Certificate | Stack-level acceptance during the secure-channel handshake. | The certificate must be trusted (see PKI trust flow). It carries **no** roles, so a cert-only session is read-everything / write-nothing. |

When no authenticator is supplied, `OpcUaApplicationHost` falls back to `NullOpcUaUserAuthenticator`;
the Host wires the real `ApiKeyOpcUaUserAuthenticator` as a singleton in `Program.cs`. Which
**`IApiKeyStore`** backs it depends on the node's role:

| Node | Store | An `Enabled = 0` revocation reaches a **new activation** | …and an **already-open session** |
|---|---|---|---|
| `admin` (including fused `admin,driver`) | `ConfigDbApiKeyStore` — reads the config database per activation, wrapped in `BoundedApiKeyStore` | On the **next activation**, bounded by `Security:ApiKeys:StoreCacheSeconds` (default 5 s) | On the **next deployment** |
| `driver`-only | `ArtifactApiKeyStore` — reads the keys carried by the deployment artifact, held in memory | On the **next deployment** | On the **next deployment** |
| neither role | `NullOpcUaUserAuthenticator` — denies every UserName token | n/a | n/a |

⚠️ **The last column is the correction, and it used to be missing.** This table once said a ConfigDb node
revokes "on the next request", which is true only of a **new session activation**. An operation on a
session that is *already open* does not re-read `IApiKeyStore` at all — `DataPlaneAuthorizer.Evaluate`
re-checks the key against the in-memory **artifact snapshot** (`DataPlaneAuthority.FindUsableKey`), which
`OpcUaPublishActor` installs on every apply. So clearing `Enabled` in the ConfigDb stops the *next*
connect on an admin node and nothing else until a deployment lands, on **both** node kinds.

**Expiry is the exception, and it is immediate on both.** `FindUsableKey` compares `ExpiresUtc` against
the authorizer's clock on every call, so an `ExpiresUtc` that has already passed denies at once on an
open session — and since arch-review 03/S19 it reaches a live **subscription** too, on a timer (see
[Data-Plane Authorization](#data-plane-authorization)).

The deny-all is registered **explicitly**, not left absent: `OtOpcUaServerHostedService` takes
`IOpcUaUserAuthenticator` as a required constructor parameter, so an unregistered seam fails DI and the
node does not boot at all.

### Bounding the key store (`StoreTimeoutMs` / `StoreCacheSeconds`)

The SDK raises impersonation **synchronously, holding a `SessionManager`-wide event lock**, so the key
lookup does not stall one activation — it serializes **every** session activation on the server,
Anonymous and X.509 included. On a driver-only node that is harmless (the artifact store is two
dictionary probes). On an admin or fused node it is two EF round trips to central SQL, so a ConfigDb that
is *slow or hung* rather than *refusing* could hold that lock indefinitely. `BoundedApiKeyStore`
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/BoundedApiKeyStore.cs`) wraps the ConfigDb store — and only it —
with two `Security:ApiKeys` knobs:

| Key | Default | Meaning |
|---|---|---|
| `StoreTimeoutMs` | `2000` | Hard per-call deadline on the store. Exceeding it **denies** the activation. `≤ 0` disables the deadline (the pre-R3-05 unbounded behaviour — do not do this on an admin node). |
| `StoreCacheSeconds` | `5` | TTL of a read-through cache in front of the store, so a burst of activations costs one round trip. `0` disables it (a read per activation). |

- ⚠️ **The deadline is a HARD race, not a cooperative token — and the difference was a live-gate
  failure.** It was originally a `CancellationTokenSource(TimeSpan, TimeProvider)` linked into the inner
  call, which bounds the call only if the callee chooses to observe the token.
  `Microsoft.Data.SqlClient` does not, while a connect or login is in flight on a black-holed socket: the
  2026-08-20 R3-05 gate paused the SQL container and watched a keyed activation hang **42.9 s** past its
  2000 ms bound (the Warning below fired only at unpause), stalling an unrelated already-open
  subscription **57 s** behind the `SessionManager` lock. `BoundedApiKeyStore` now races the inner call
  against a `Task.Delay` on the injected clock and returns control the moment the delay wins.
- **An abandoned lookup runs on, and its answer is discarded.** Nothing can cancel a SqlClient call that
  is not listening, so the query may still complete (or fault) in the background long after the
  activation was denied. Its result is **never** cached — it was taken before the deadline and would be
  served after it, so a key revoked or expired in the meantime could be admitted from a lookup nobody
  was still waiting for. The token is still cancelled, so a *cooperative* inner store does stop, and the
  abandoned task's outcome is observed and logged at Debug (*"an abandoned key store lookup finished /
  faulted after its deadline; discarded"*) so it can never surface as an `UnobservedTaskException`.
- **A timeout denies; it never admits.** The deadline surfaces as an `OperationCanceledException` into
  `ApiKeyOpcUaUserAuthenticator`'s existing catch-all, which returns the same opaque denial as a bad
  credential. There is no second denial path to get wrong, and the decorator can only ever return what
  the inner store returned or throw.
- **What a timeout looks like:** a distinct Warning — *"ApiKeyStore: the key store did not answer within
  N ms — denying this activation. Every OPC UA session activation on this server is serialized behind
  this call; check ConfigDb health"* — plus
  `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. It is deliberately **not** the ordinary
  *"no enabled, unexpired key with id X"* line, which would send an operator hunting for a key problem
  that does not exist.
- **What to watch:** `otopcua.opcua.apikey_store.bounded{outcome=cache_hit}`. A busy fused central with no
  `cache_hit` stream means `StoreCacheSeconds` is 0.
- **Faults and timeouts are never cached** — caching a deny would extend an outage past the outage. Only
  a completed lookup (a hit or an authoritative miss) is stored, and a cached hit whose `ExpiresUtc` has
  since passed is re-read rather than served, so the TTL can never resurrect an expired key.

#### Pre-authentication guards (`KnownIdSet` / `MissBurst` / `MissRefillPerSecond`)

The negative cache above is keyed by **key id**, so a flood of *distinct*, well-formed `otua_<random>_x`
credentials missed it every time and cost one indexed SQL query per attempt — under the SDK's
`SessionManager`-wide lock, with no credential required (review 03 §3.3). `BoundedApiKeyStore` now puts two
guards in front of the store, bound from the same `Security:ApiKeys` section
(`ApiKeyStoreGuardOptions`, Host-owned):

| Key | Default | Meaning |
|---|---|---|
| `KnownIdSet` | `true` | Keep an in-memory set of every `ApiKeyId` the ConfigDb holds, refreshed every `StoreCacheSeconds` (single-flight; a failed refresh keeps the previous set). An id outside it is refused in memory — `otopcua.opcua.apikey_store.bounded{outcome=unknown_id}` — and the store is never asked. A key minted less than `StoreCacheSeconds` ago is refused until the next refresh (the same window `Enabled` revocation already has). `StoreCacheSeconds = 0` disables the set with the cache. |
| `MissBurst` | `100` | Token bucket on lookups that **reach** the store and **miss**, for when the set is unavailable (not loaded yet, or the store cannot list ids). Exhausted ⇒ unknown ids are refused without a store read (`{outcome=miss_budget}`, a Warning-free deny through the authenticator's catch-all). A hit refunds its token, so legitimate clients are never charged. `0` disables the budget. |
| `MissRefillPerSecond` | `20` | Refill rate of the miss bucket. |
| `MissBurstPerEndpoint` | `10` | Per-client miss bucket, keyed by the client's **IP address** (#751). A miss is charged to the client's own bucket first and then to the process-wide one, so one noisy client exhausts its own budget (`{outcome=endpoint_miss_budget}`) long before it can drain `MissBurst` for everyone. A refusal by either bucket takes from neither; a hit refunds both. `0` disables per-client accounting. |
| `MissRefillPerSecondPerEndpoint` | `2` | Refill rate of each per-client bucket. |

**Per-client accounting and its limits (#751).** `IOpcUaUserAuthenticator` has an endpoint-carrying
`AuthenticateUserNameAsync(username, password, remoteEndpoint, ct)` overload (default: ignore the endpoint
and delegate), which the server calls and `ApiKeyOpcUaUserAuthenticator` forwards to
`IApiKeyStore.FindAsync(..., remoteAddress, ...)`. SDK 2.0.1 supplies the public transport-observed
`SecureChannelContext.PeerAddress` on session activation. `OtOpcUaSdkServer` scopes it to the current
asynchronous activation and restores the prior context in `finally`; `OpcUaApplicationHost` adapts it to
the existing IP-budget seam with port zero (no peer port is exposed). The former private-channel-map
reflection helper is removed. An unknown address is charged to the process-wide bucket only, which
therefore stays as the backstop. `ApiKeyMissBudgetPerEndpointTests` (OpcUaServer.IntegrationTests)
pins per-client accounting over the wire; concurrent IPv4/IPv6 activation probes check peer isolation. At most `BoundedApiKeyStore.MaxTrackedEndpoints`
(4096) addresses are tracked; when full, idle (refilled) buckets are pruned and a new address beyond that is
charged process-wide only — never handed a fresh budget. Clients behind one NAT address share a bucket. Neither
guard applies to a driver-only node: its `ArtifactApiKeyStore` is two in-memory dictionary probes and is never
wrapped.

### Failed-credential lockout (`OpcUa:CredentialLockout`)

**A failed sign-in never locks a client out of anonymous access (#833).** OPC Foundation SDK 2.0.1's
`SessionManager` has its own lockout: after `ServerConfiguration.MaxFailedAuthenticationAttempts` (SDK
default **5**) failed identity validations — each within 60 s of the previous — it refuses **every**
`ActivateSession` from that client for **5 minutes** with `BadUserAccessDenied` ("Too many failed
authentication attempts"). Its check runs before the identity token is looked at, so it refused
**Anonymous** too (even a namespace-array read needs an activated session), a successful anonymous
activation did not clear it, and the authenticator was never called (no "anonymous admitted" line). Its key
is the client certificate thumbprint when one is presented, else the peer IP — so certificate-less clients
behind one NAT share a key, and on a `None` endpoint a presented certificate is not proven. That is what
the docker-dev live gate hit: four failed `multi-role` UserName activations against central-1, then every
anonymous session refused for five minutes while central-2 (which never saw the failures) kept admitting.

The host now sets `MaxFailedAuthenticationAttempts = 0` on every configuration it builds or loads
(`OpcUaApplicationHost.DisableSdkAuthenticationLockout`) and enforces a narrower replacement,
`FailedCredentialLockout`, in its identity-registry adapter:

- **UserName activations only.** **Anonymous activations never consult it**, from any client.
- **Two counts per client (#844).** Each *definite* failure is charged to the (client, **credential identity**)
  pair and to the client as a whole. The credential identity is `IOpcUaUserAuthenticator.DescribeCredential` —
  for API keys `key:<keyId>` (the public lookup index, never the secret), for anything that is not a well-formed key
  `user:<username>`.
  - **Per credential:** at `MaxFailedAttempts` (5) failures of one identity, each within `FailureWindowSeconds` (60) of
    the previous, that client is refused **that identity** `BadUserAccessDenied` for `LockoutSeconds` (300), without
    the key store being consulted. A success with **that same identity** clears its count.
  - **Per client:** at `MaxFailedAttemptsPerClient` failures across **every** identity (default `0` ⇒
    `2 × MaxFailedAttempts` = 10), the client is refused **every** UserName activation for `LockoutSeconds`.
    ⚠️ **A success never resets this count** — it only ages out through `FailureWindowSeconds` or an expired lockout.
  - Why both: before #844 one count per client was cleared by *any* success, so a client holding a valid key could make
    four guesses at another key id, sign in with its own key, and repeat forever. The per-credential count alone would
    fix that but would let a client spray four guesses at each of unboundedly many key ids; the client-wide cap that a
    success cannot reset bounds the spray (≤ 10 unverified guesses per window chain, then 5 minutes locked), while a
    legitimate client with one stale/revoked key and one good key loses only the stale one.
  - Timing per count: a refusal during a lockout is not counted (it never extends itself); an **expired lockout always
    starts a fresh count**, whatever `FailureWindowSeconds` is (an expired client-wide lockout clears every identity's
    count too). A Warning is logged when a count locks (naming the scope and identity); the first refusal of each
    lockout episode is logged at Information, the rest at Debug. Identities per client are bounded at 64; further ones
    share one overflow count (conservative — never laxer).
- **Only a definite bad credential counts (#845).** A wrong secret, an unknown/disabled/expired key id, a malformed
  credential or the wrong username counts. A denial that says nothing about the credential does **not**: a key-store
  timeout (`Security:ApiKeys:StoreTimeoutMs`), a miss-budget refusal, any other store exception, an unset pepper, an
  authenticator that throws, or a cancelled activation. The authenticator reports those with
  `OpcUaUserAuthResult.Unavailable(...)` (`Transient = true`) and the host releases the slot uncounted — so a ConfigDb
  stall can no longer lock every keyed client out for 5 minutes. The client sees the same opaque
  `BadIdentityTokenRejected` either way (no oracle).
- **Parallel activations wait — they are never refused for parallelism.** An activation gets a verification
  **slot** only while `identityFailures + identityInFlight < MaxFailedAttempts` **and**
  `clientFailures + clientInFlight < MaxFailedAttemptsPerClient`; beyond that it **queues** until one frees (a queued
  activation whose identity has room is not held behind one whose identity has none). So no more unverified guesses
  than either threshold are ever in progress-or-failed per window, while a legitimate consumer opening many keyed
  sessions at once (e.g. a ScadaBridge site node behind one NAT IP on a `None` endpoint) just waits milliseconds and is
  admitted. A waiter is refused only when what it needs actually **locks out** (`BadUserAccessDenied`, without reaching
  the authenticator), or after `SlotWaitSeconds` (default 10) with `BadTimeout` — **not** counted as a failure. A
  cancelled activation leaves the queue and frees any slot it held.
- **Keyed on certificate thumbprint *and* peer address together.** That separates clients only when they present
  a certificate — on a `Sign`/`SignAndEncrypt` endpoint the certificate is proven, and two clients behind one address
  with different certificates are locked out separately (pinned over the wire). ⚠️ On a `None` endpoint clients
  commonly present none, so the key is **effectively the IP alone**: NAT neighbours there **do** share the client-wide
  count (for UserName activations only — never for anonymous ones; the per-credential count is shared only by
  neighbours presenting the same key id). When the transport reports no peer address at all, every certificate-less
  client shares the single key `-@-`. (Observed on docker-dev: every host client arrives as `-@192.168.97.1`.)
- **The client `ApplicationUri` is deliberately NOT part of the key (#845, decided).** It would separate NAT
  neighbours on a `None` endpoint, but there it is a free, unauthenticated string in `CreateSession` (and the
  presented certificate is unproven), so an attacker would reset both counts simply by sending a new `ApplicationUri`
  with every guess — turning the lockout off for exactly the client it exists to stop. Keying on it is safe only
  where the certificate is proven, and there the thumbprint already separates clients. Operators who need NAT
  neighbours isolated should enable a signed endpoint (each client then presents its own certificate), not widen the key.
- **Validated at host start:** with `MaxFailedAttempts > 0`, `LockoutSeconds`, `FailureWindowSeconds` and
  `SlotWaitSeconds` must all be `> 0`, and `MaxFailedAttemptsPerClient` must be `0` or `>= MaxFailedAttempts`
  (`OpcUaApplicationHostOptionsValidator`); `MaxFailedAttempts <= 0` disables the lockout.
- **Bounded at 4096 tracked clients.** Entries with nothing left to remember are pruned when the table fills; a client
  arriving at a still-full table is not tracked. When a prune frees nothing, the next one is deferred until the earliest
  entry could have aged out (#845) — a flood of new client keys at a full table used to rescan all 4096 entries under the
  global lock on every failure. A release that frees an entry re-arms pruning immediately. API-key secrets are not
  guessable, so this is defence in depth, not the brute-force control.
- **The SDK lockout is off on both configuration paths** — the configuration the host builds and one loaded from
  `OpcUa:ApplicationConfigPath` (whose own `MaxFailedAuthenticationAttempts` is overwritten with `0`; pinned over the
  wire with a file carrying `2`).

Pinned over the wire by `CredentialLockoutWireTests` (OpcUaServer.IntegrationTests — anonymous admitted after
twelve failures; a valid key refused while client-wide locked out; own-key successes do not launder guesses at another
key id; transient denials never lock; distinct client certificates on a `Sign` endpoint are locked out separately; the
`ApplicationConfigPath` branch keeps the SDK lockout off; `MaxFailedAttempts = 0` admits after seven failures,
proving the SDK lockout is off) and unit-tested by `FailedCredentialLockoutTests`, `OpcUaApplicationHostImpersonationTests`
and `ApiKeyAuthenticationTests` (Host.Tests).

### LDAP bind flow (`OtOpcUaLdapAuthService`)

LDAP is configured under the `Security:Ldap` section (bound to `LdapOptions`, `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs`, `SectionName = "Security:Ldap"`). The app authenticator is `OtOpcUaLdapAuthService` — a thin wrapper around the shared `ZB.MOM.WW.Auth.Ldap` directory client that adds two app-only concerns the shared library deliberately does not model: the `Enabled` master switch and `DevStubMode`. Since #520 it serves the **Admin UI cookie login (`/auth/login`) only** — `LdapOpcUaUserAuthenticator` is deleted and the OPC UA UserName path authenticates API keys instead (see [API-key identity](#api-key-identity-opc-ua-data-plane)).

`OtOpcUaLdapAuthService.AuthenticateAsync`:

1. If `Enabled = false`, denies outright — no bind, no DevStub bypass (the master switch wins).
2. If `DevStubMode = true`, accepts any non-empty credentials and grants the `Administrator` role **without any network bind** (dev only — must be `false` in production).
3. Refuses to bind over a plaintext transport (`Transport = None`) unless `AllowInsecure = true` (dev/test only). This is enforced at login, not at startup.
4. Delegates the real path to the shared `ZB.MOM.WW.Auth.Ldap` client: it binds (search-then-bind via `ServiceAccountDn`, or direct-bind `cn={user},{SearchBase}` when no service account is set), verifies the password, and reads the user's group memberships.
5. Returns an `LdapAuthResult` carrying the validated username + the **groups** (never roles). Failure codes are folded into opaque user-facing error strings so a probe cannot distinguish "unknown user" from "wrong password".

**Group → role mapping happens downstream**, not in the auth service: the `/auth/login` handler (`AuthEndpoints`) resolves `IGroupRoleMapper<string>` (`OtOpcUaGroupRoleMapper`) per login — it applies the appsettings `GroupToRole` baseline, merges system-wide DB grants from the user's LDAP groups, and unions any pre-resolved roles (the DevStub grant) — and the resulting roles land in the Admin UI cookie principal. A mapper fault (e.g. a Config DB outage) falls back to the pre-resolved baseline rather than denying an otherwise-authenticated login. **Since #520 nothing on the OPC UA data plane consumes LDAP groups** — data-plane session roles come from the authenticated principal's `NodeAcl` grants (see [Role grant source](#role-grant-source-data-plane)).

`Transport` replaces the former `UseTls` bool: `Ldaps` (implicit TLS), `StartTls` (upgrade), or `None` (plaintext, requires `AllowInsecure`). Configuration example (Active Directory production):

```json
{
  "Security": {
    "Ldap": {
      "Enabled": true,
      "DevStubMode": false,
      "Server": "dc01.corp.example.com",
      "Port": 636,
      "Transport": "Ldaps",
      "AllowInsecure": false,
      "SearchBase": "DC=corp,DC=example,DC=com",
      "ServiceAccountDn": "CN=OtOpcUaSvc,OU=Service Accounts,DC=corp,DC=example,DC=com",
      "ServiceAccountPassword": "<from your secret store>",
      "GroupAttribute": "memberOf",
      "DisplayNameAttribute": "cn",
      "UserNameAttribute": "sAMAccountName",
      "GroupToRole": {
        "OPCUA-Designers": "Designer",
        "OPCUA-Admins": "Administrator",
        "OPCUA-Operators": "Operator"
      }
    }
  }
}
```

`GroupToRole` maps LDAP group names → roles (case-insensitive); a user gets every role whose source group is in their membership. The values are the canonical control-plane role strings (`Viewer` / `Designer` / `Administrator`, plus the appsettings-only `Operator` for the `DriverOperator` policy); the same map also supplies data-plane role strings (`ReadOnly`, `WriteOperate`, `WriteTune`, `WriteConfigure`, `AlarmAck`) — see [Role grant source (data-plane)](#role-grant-source-data-plane) below. `UserNameAttribute: "sAMAccountName"` is the critical AD override — the GLAuth dev default is `cn`, which is not how AD users are looked up; use `userPrincipalName` instead if operators log in with `user@corp.example.com` form. `LdapOptionsValidator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LdapOptionsValidator.cs`) fails startup when `Transport = None` and `AllowInsecure = false` on a real-LDAP (non-DevStub) config.

### API-key identity (OPC UA data plane)

The OPC UA data plane is **machine-to-machine**, so a consuming system authenticates with an API key
rather than an LDAP user (Gitea #520). The client presents an OPC UA **UserName** token whose username
is the fixed literal `apikey` and whose password is the key:

```
otua_<keyId>_<secret>
```

`ApiKeyOpcUaUserAuthenticator` verifies it — one indexed lookup on `keyId` plus one peppered
HMAC-SHA256 over the secret, compared with `CryptographicOperations.FixedTimeEquals`. No network I/O.
The key resolves to a **principal**, and `NodeAcl` rows naming that principal are what grant it
permissions.

| Key | Notes |
|---|---|
| `Security:ApiKeys:Username` | The static username clients present. Default `apikey`. Checked rather than ignored, so a typo is an explicit denial. |
| `Security:ApiKeys:Pepper` | Server-side pepper for the stored hashes. **Supply via `Security__ApiKeys__Pepper` or `${secret:}` — never commit it.** Rotating it invalidates every existing key. Has no default: an empty pepper still hashes "successfully", so `ApiKeySecret.ComputeHash` throws on one rather than producing a working-looking deployment with no protection. |

Secrets are never stored — only the hash. The plaintext is shown **once** at creation and is
unrecoverable afterwards.

**Every rejection returns the same opaque message** ("Invalid API key"), with the real reason logged
server-side only, so the server cannot be used as an oracle for whether a key id exists or which half
of a credential was wrong.

⚠️ **Revocation requires a deploy.** Per-cluster mesh Phase 4 leaves driver-only nodes with no
ConfigDb, so keys reach them inside the deployment artifact exactly as `NodeAcl` rows do. Clearing
`Enabled` takes effect on a node only once a new artifact reaches it.

**Galaxy alarm-ack identity per key (lmxopcua#749).** mxaccessgw acknowledges Galaxy alarms as a real
Galaxy account, so a key can carry an optional Galaxy ack identity (AdminUI `/apikeys` → "Ack
identity"). It is held only in the encrypted Secrets store
(`otopcua/apikeys/<keyId>/galaxy-ack-{user,password}`), never in the ConfigDb or the artifact; keys
without one and anonymous sessions use the Galaxy driver's `AlarmAck` fallback account, and with
neither the ack is refused before it reaches the gateway. Deleting a key on `/apikeys` removes its
identity. See [docs/drivers/Galaxy.md § Alarm acknowledge credential](drivers/Galaxy.md#alarm-acknowledge-credential).

### Anonymous is a normal principal

Anonymous sessions are **admitted by default** (`OpcUa:AllowAnonymous`, default `true` — flipping the
default would silently deny every client that has no key the moment this code deploys, including
ScadaBridge). Set it to `false` to make keys mandatory: the Anonymous token policy is then not
advertised, so a client is refused at *discovery* rather than only at activation, and anonymous
activation is rejected independently in case a client attempts a token type the endpoint never offered.

When admitted, an anonymous session is **not a special case** — it authenticates as the reserved
principal `anonymous`, and `NodeAcl` rows naming it grant permissions exactly as they would a key's
principal. It can therefore be granted **write**, not merely tolerated as read-only. It arrives at the
downstream gates as the same `RoleCarryingUserIdentity` a key produces, so no gate carries an
anonymous-shaped branch and there is no bypass path to get wrong.

Two consequences worth knowing:

- `anonymous` is **reserved**: an API key may not use it as its principal, or a credentialled client
  would silently inherit whatever anonymous was granted. Rejected at authentication as well as at
  authoring.
- If the anonymous principal's grants **cannot be read**, the session is **denied**, not admitted with
  no roles. Admitting is the pre-#520 behaviour, and defaulting to it precisely when the server cannot
  determine what anonymous may do stops being harmless once reads are gated.

⚠️ **Node ACLs are enforced only when `Security:Acl:Enabled` is set.** With the flag **off — the
default** — a key's grants are collapsed into the two server-wide role gates and **only
`Cluster`-scoped grants count**, so a narrower grant confers nothing rather than silently widening to
server-wide authority, and read, browse, subscribe and HistoryRead are ungated for any admitted
session. With it **on**, **every** data-plane operation is decided by scoped `NodeAcl` grants — read,
browse, subscribe, write, HistoryRead and the alarm commands (stage 6). See
[Node ACL enforcement](#node-acl-enforcement-securityaclenabled) and
`docs/plans/2026-07-31-apikey-acl-enforcement-design.md`.

### LDAP resilience — mostly retired with the data-plane authenticator (#520)

> **Changed 2026-07-31.** The OPC UA **data plane** no longer uses LDAP at all — it authenticates with
> API keys (see [API-key identity](#api-key-identity-opc-ua-data-plane)). `LdapOpcUaUserAuthenticator`
> is deleted, and with it four of the five `Security:Ldap` resilience keys: **`AuthTimeoutMs`**,
> **`MaxConcurrentAuthentications`**, **`OutageFailureThreshold`** and **`OutageCooldownSeconds`**.
> Setting them now does nothing (options binding ignores unknown keys); remove them from any config
> that carries them.

Those four existed for one specific reason. The OPC UA SDK raises its impersonation callback
**synchronously**, inside a `SessionManager`-wide event lock (verified against
`OPCFoundation.NetStandard.Opc.Ua.Server` 1.5.378), so a slow or hung LDAP bind in that callback did
not stall one SDK thread — it **serialized every session activation server-wide**, Anonymous and X509
included. That constraint is gone: key verification is one indexed lookup plus one HMAC, with no
network I/O on the path.

LDAP now runs only on **AdminUI login**, an ordinary async request path where a slow directory affects
the one request making it. One key remains:

| Key | Default | Bounds |
|---|---|---|
| `ConnectionTimeoutMs` | `10000` | Projected into the shared library's `ConnectionTimeoutMs` — bounds the socket connect **and** each per-operation search. Validated `> 0` at startup by `LdapOptionsValidator`. |



---

## Data-Plane Authorization

> ⚠️ **Read this first (updated 2026-08-01, #520 stages 5–6).** Whether the `NodeAcl` / `PermissionTrie`
> subsystem runs depends on one flag. With **`Security:Acl:Enabled` false — the default** — it does not:
> the only authorization in the server is the coarse role checks in
> [What is enforced with the flag off](#what-is-enforced-with-the-flag-off), and read, browse, subscribe
> and HistoryRead are ungated for any admitted session including Anonymous. With it **true**, those role
> checks are *replaced* by scoped ACL decisions and **every** data-plane operation is gated — see
> [Node ACL enforcement](#node-acl-enforcement-securityaclenabled).

### What is enforced with the flag off

There are exactly **three** authorization checks in the whole OPC UA server layer. All three are
coarse, **server-wide role string checks** with no per-node component.

| Operation | Gate | Requires |
|---|---|---|
| Write (Value attribute, either namespace) | `OtOpcUaNodeManager.EvaluateEquipmentWriteGate` | role `WriteOperate` |
| Alarm Acknowledge / Confirm / AddComment / Shelve / Unshelve (scripted alarms) | `OtOpcUaNodeManager.HandleAlarmCommand` | role `AlarmAck` |
| Alarm acknowledge (driver-fed native alarms) | `OtOpcUaNodeManager.HandleNativeAlarmAck` | role `AlarmAck` |

Both gates fail closed — a session with no identity or no matching role gets
`BadUserAccessDenied` — and read `identity.Roles` off the session's `RoleCarryingUserIdentity`
(`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/RoleCarryingUserIdentity.cs`). Role strings are
the constants in `OpcUaDataPlaneRoles`, derived from the principal's cluster-scoped `NodeAcl` grants
(see [Role grant source](#role-grant-source-data-plane)).

**Both gates also re-check the session's API key on every call, flag off or on** (Gitea #563). A
session's roles are resolved once, at activation, so before this an open session kept `WriteOperate` /
`AlarmAck` after its key's `ExpiresUtc` passed — or after a deployment disabled or removed the key —
until the client chose to disconnect, which on a machine-to-machine data plane can be never. Now each
write and each Part 9 method call first asks `DataPlaneAuthorizer.AuthorizeKeyUsable`: the same
`FindUsableKey` probe, against the same deployed snapshot and the same clock, that the ACL path makes
(one dictionary lookup, no I/O). An unusable key gets `BadUserAccessDenied` (logged as
`AclGate: denied … key … is no longer enabled or has expired`), checked *before* the "no gateway ⇒
`BadNotWritable`" availability answer. Things to know:

- **Only these two gates — plus HistoryUpdate.** (HistoryUpdate, added with the pluggable historian, is a third gated
  operation in both modes; see [HistoryUpdate](#historyupdate-historian-writes). It re-checks the key the same way.)
  Read, Browse, Subscribe and HistoryRead stay ungated with the flag off —
  that is the flag-off posture, and an expired key's session can still read and keep its subscriptions.
  Turn `Security:Acl:Enabled` on to have expiry reach those too.
- **Anonymous is unaffected.** It carries no key id, so there is nothing to expire; its roles stay what
  activation resolved from the `anonymous` principal's grants.
- **No flag.** An expired or disabled key keeping write authority is a defect, not a rollout choice.
- ⚠️ **On a fused admin+driver node a key minted after the last deployment authenticates (the ConfigDb
  store is read at activation) but its writes and acks are refused until a deployment carries it** — the
  check reads the deployed snapshot, which is also what the flag-on path has always done. Deploy after
  minting a key.

**What has no authorization check at all:**

- **Read, Browse, TranslateBrowsePaths** — no override, no handler. Any admitted session, including
  an Anonymous one, sees and reads the entire address space.
- **HistoryRead** — all four overrides (`HistoryReadRawModified`, `HistoryReadProcessed`,
  `HistoryReadAtTime`, `HistoryReadEvents`) run without an identity check.
- **CreateMonitoredItems / subscriptions / TransferSubscriptions**.
- **Non-Value attribute writes** — the gate hangs off `OnWriteValue`.

**And what the write gate does *not* distinguish:**

- **`WriteTune` and `WriteConfigure` are never checked.** They exist in the role vocabulary and in
  `NodePermissions`, but every write — whatever the tag's security classification — is gated on the
  single `WriteOperate` string.
- **`AlarmAcknowledge` / `AlarmConfirm` / `AlarmShelve` are not distinguished.** One `AlarmAck` role
  covers all five alarm methods; the `operation` argument is passed to the router but never consulted
  by the gate.
- **There is no per-node granularity.** A session that may write one tag may write **every** writable
  tag on the node.

The gate *is* realm-aware in one narrow sense: it is attached per-node during materialization for both
the Raw and UNS realms, and the write dispatch it guards is realm-qualified. The role decision itself
takes no realm and no node.

### Node ACL enforcement (`Security:Acl:Enabled`)

> **ns=0 standard nodes stay ungated, deliberately (#617).** A session with no grants can still connect
> and read the standard OPC UA server nodes (`ServerStatus`, `NamespaceArray`, `ServiceLevel`, ...) and
> browse ns=0. Those are served by the SDK's core node manager outside our authorizer, and gating them
> would break session setup (clients read them for connection and redundancy health). They carry no plant
> data; every raw/UNS node, event subscription and HistoryRead is still gated. The boot warning says so
> ("can still connect and read the standard server nodes ... but sees no plant data") — earlier wording
> ("can connect and see nothing") overstated it.

| Key | Default | Effect |
|---|---|---|
| `Security:Acl:Enabled` | `false` | `true` makes scoped `NodeAcl` grants decide **every** data-plane operation — read, browse, subscribe, write, alarm commands and HistoryRead |

**Live-gated on docker-dev 2026-08-01** (site-a-1 with the flag on, central-1 as the same-artifact
control): scoped read/browse/write behave as described below, and revoking a grant stopped a running
subscription mid-stream — the client received `BadUserAccessDenied`, then 25 seconds of silence, then
the feed resumed by itself when the grant was restored. Evidence + what the gate did *not* cover (UNS
grants, alarm scoping) in `docs/plans/2026-07-31-apikey-acl-enforcement-design.md`.

**It is a config change, not a redeploy.** Both paths are wired in both modes — the node always builds
its permission trie on every deploy and always constructs the gate; the flag only chooses which decision
the seams consult. Same dark-switch discipline as `MeshTransport`, `ConfigSource` and `Telemetry`.

#### What changes when it is on

| Operation | Off | On |
|---|---|---|
| Write (Value) | role `WriteOperate` | `NodePermissions.WriteOperate` on that node's scope |
| Alarm Acknowledge / AddComment / Enable / Disable | role `AlarmAck` | `AlarmAcknowledge` on the condition's scope |
| Alarm Confirm | role `AlarmAck` | `AlarmConfirm` |
| Alarm OneShotShelve / TimedShelve / Unshelve | role `AlarmAck` | `AlarmShelve` |

**Whose scope a condition's command is decided at (Gitea #797).** A *native* condition's id is its raw tag's
RawPath, so its commands are decided on that path (a `Tag` / `RawDevice` / … grant covering the raw tag), as
before. A *scripted* condition's id is its opaque, operator-authored `ScriptedAlarmId` (e.g. `scripted-eqa-hi`)
and is **never** split into a scope: `OtOpcUaNodeManager` binds each LIVE scripted condition instance to the
`EquipmentId` it was materialised under (only when it actually hangs under that equipment's folder), and
`DataPlaneAuthorizer.AuthorizeScriptedCondition` resolves that equipment to its `[uns, Area, Line, Equipment]`
path through the CURRENT authority generation's `EquipmentScopePaths` (decoded from the artifact by the same
`AclScopePathResolver` call that places an `Equipment` grant). So a grant on the owning equipment or any
ancestor (line, area, UNS namespace, cluster) authorizes its scripted alarms; a sibling equipment's grant or a
grant on one tag beneath the equipment does not. The same generation also carries each scripted alarm's
deployed owner (`ScriptedAlarmOwners`, from the artifact's `ScriptedAlarms` rows), and the live binding must
**agree** with it: the publish actor installs a generation before applying its address space, and a failed apply
keeps the old one, so a moved alarm is denied under its old and new owner's grants alike until the address space
re-binds it. Fail-closed: an unbound condition (owner folder missing), a
removed/replaced instance, or an owner the generation cannot place is denied for **every** grant, `Cluster`
included — there is no realm-root or id-split fallback. The key, grant and generation are re-read on every call
(uncached). NodeIds and `AlarmCommand.AlarmId` are unchanged. With the flag **off** none of this is consulted.

**Where an opaque-id UNS node is placed for Read / Browse / Subscribe / HistoryRead (Gitea #808).** UNS Area /
Line / Equipment folders are minted under their opaque entity ids (`s=UA-…`, `s=UL-…`, `s=EQ-…`), an equipment's
virtual tags under `s=EQ-…/Name`, and scripted conditions plus their instance-scoped children under
`s=<ScriptedAlarmId>[/…]` — none of which is a grant path, so splitting them reached only `Cluster` / `uns` grants.
`DataPlaneAuthorizer` now asks the node manager (`ResolveUnsAclSubject`, attached when the authorizer is set) how
the node sits in the LIVE address space and places it against the CURRENT generation: a hierarchy folder at its
entity's path from the generation's `UnsFolderScopes` (built by the same `AclScopePathResolver` call that plants
`UnsArea`/`UnsLine`/`Equipment` grants), **only while the live folder-id ancestry equals the one the generation
deploys** (a moved equipment is denied under its old and new line's grants until the rebuild re-parents it); a
virtual tag / its sub-folder at its equipment's path plus a reserved segment; a scripted condition and its children
at the owning equipment's path plus that reserved segment, through exactly the #797 owner rule. The reserved segment
means only grants on the equipment or an ancestor confer them — a `Tag` grant beneath the equipment never does, and
its Browse transit does not expose the equipment's alarms or virtual tags. A scripted condition additionally requires
its owner folder's live ancestry to match the generation's (the folder rule). UNS reference variables
(`s=Area/Line/Equipment/Name`) keep their id as their path — but only while it equals their live equipment folder's
placement plus one name, so a moved or renamed equipment's variables (still on the old-name id) are denied for Read
and Write under old and new names until the address space re-points them. An id that no live node holds (a removed
folder / virtual tag / condition, a probe, a NodeDeleted model change) is decided only by a `Cluster` or `uns`
`Namespace` grant — so narrow grantees do not receive NodeDeleted for removed opaque-id nodes, and a removed id can
never match an unrelated narrow grant that shares its name; a four-segment (reference-shaped) id keeps its split.
Identifiers containing NUL are refused. A UNS variable the reference pass wired to its raw node (the `Organizes`
edge) is placed as a reference variable whatever its id spells (Gitea #826 — the id rule alone made an equipment's
references folder-owned when an Area was named exactly like that equipment's id, losing their `Tag`-grant match).
Server-object (`i=2253`) delivery of a
**scripted** condition is now decided on the condition alone (its folders add nothing in steady state and would
leak a moved alarm to its old owner). Direct event items on a folder get the same per-event decision (Gitea #826,
below). So a key holding only an `Equipment` grant can navigate to its equipment,
read its alarm children, receive its scripted alarm events through the Server object and Acknowledge / Confirm /
AddComment with the EventId it received. Fail-closed throughout: an unplaceable node is denied under every grant.
| HistoryRead (all four variants) | **nothing** | `HistoryRead` on each node's scope, decided **per node in the batch** |
| HistoryUpdate (insert / replace / update / delete) | role `HistoryUpdate` (from a `Cluster`-scoped grant) + key re-check | `HistoryUpdate` on that node's scope, decided **per node in the batch** |
| Read (every attribute, not just Value) | nothing | `Read` on that node's scope, decided per node in the batch |
| Browse / TranslateBrowsePath | nothing | `Browse` on the source node, **and** on each returned reference |
| CreateMonitoredItems / SubscribeToEvents | nothing | `Subscribe` on that node's scope, decided per item |

⚠️ **The ACL decision REPLACES the role check; the two are not chained.** Chaining them would require a
grant to confer the server-wide role *as well as* pass the scope check, and only `Cluster`-scoped grants
confer a role (`ApiKeyInterimRoleMapper`) — so every narrow grant would remain inert and the flag would
look like it did nothing. Narrow grants working is the whole point of the stage.

Behaviours worth knowing before enabling it:

- **A deploy takes effect on sessions that are already open.** The gate stamps each decision with the
  current authority generation rather than the one the session activated under, so an ACL edit lands
  without bouncing clients — in both directions, widening and narrowing.
- **Revocation also reaches an open session — but read the two axes separately.** Every decision
  re-checks the authenticating key against the current snapshot, so an open session is re-authorized on
  every operation. What that check can *see* differs by axis:
  - **The deployment axis** (a key's `Enabled` cleared, a grant edited or removed). The snapshot only
    changes when an artifact is applied, so this lands **on the next deployment**, on both node kinds —
    including on a fused admin node, whose ConfigDb store is consulted at *activation* only. On a
    driver-only node the artifact is the only revocation channel there is.
  - **The clock axis** (a key's `ExpiresUtc` passing). `FindUsableKey` compares it against the
    authorizer's clock on every call, so this lands **immediately**, with no deployment.
- **Revocation reaches a live subscription too** (#520 stage 6, extended by arch-review 03/S19). Every
  other operation re-asks on each request; a subscription is authorized once and then streams, so it
  needed a push. A denied item is queued a `BadUserAccessDenied` and then detached from its node's value
  fan-out; restoring its authority re-attaches it. There are **two triggers**:
  - a **deployment**, which re-authorizes every live monitored item (the deployment axis);
  - the host's **low-frequency timer**, `Security:Acl:RevalidationIntervalSeconds` (default 60 s, `0`
    disables it, inert while `Security:Acl:Enabled` is false). Without it a key that expired by the
    clock kept streaming until a deployment happened to land — **indefinitely** on a fleet that has been
    deployed once and left running, which is exactly the shape a production fleet settles into.

  ⚠️ **An expired-key session is silenced, not disconnected.** Its items go `BadUserAccessDenied` and the
  session stays open, deliberately: a clock-driven denial must look to a client exactly like a
  deploy-driven one, OPC UA has no honest "expire this session" verb (deleting it server-side gives the
  client a transport failure rather than an authorization status), and every other operation on that
  session already denies. **A client that re-activates with a renewed key recovers on the next tick with
  no reconnect** — closing the session instead would force a full reconnect-and-resubscribe on every
  machine consumer in the fleet at once when a shared key expired.

  **Event/alarm monitored items are revalidated by both triggers too** (Gitea #562). A direct event
  subscription — one on an equipment or device folder's event notifier — is tracked in a second map
  alongside the data-change items and re-asked the creation gate's question (`Subscribe` on the
  notifier's scope, for the identity of the session that created it). A denied item is **detached** from
  the notifier's event fan-out through the SDK's own unsubscribe — exactly what the client's delete would
  do — never set `MonitoringMode.Disabled`, so restoring its authority re-attaches it and alarms resume
  with no client action. Events raised while it was stood down are not replayed. Two differences from a
  data-change item:
  - **The client is not told.** An event item has no value slot to carry `BadUserAccessDenied`, and a
    fabricated "access denied" event would land in every alarm consumer's stream, so the feed simply goes
    quiet. The operator-facing signal is the `AclGate: RevalidateEventSubscriptions withheld N of M` log line.
  - **A client may delete a stood-down item** normally; the delete is absorbed (the item is already off
    the fan-out) and a later restoring deployment does not resurrect it.

  **Server-object (`i=2253`, all-events) event subscriptions are filtered per event at delivery** (Gitea
  #586). They used to be outside the ACL entirely: the SDK attached such an item to the fan-out of every
  one of our root notifiers through `SubscribeToAllEvents`, and an equipment folder with no direct
  subscriber forwarded its events to the core node manager's Server object, which our node manager never
  sees — so a session holding no grant received every alarm in the server. Now:
  - The node manager attaches a Server-object item to **nothing**. It records it, and each condition hands
    its events to the recorded items itself (the condition's own `OnReportEvent`, which fires exactly once
    per event, so the Server object still sees one copy). The root-notifier forward to the core Server
    object is dropped.
  - Each event is delivered to an item only if the item's session holds `Subscribe` on **any** of: the
    condition itself (`ConditionId` = RawPath, raw realm — so a grant on the alarm's raw tag admits it), a
    root-notifier folder the condition bubbles through (its raw device folder; a scripted alarm's
    equipment folder), or an equipment folder wired as its extra notifier. That is the per-folder path's
    question — a session that may subscribe to any notifier the alarm reports through already receives it
    there — asked with the same authorizer and the same operation as the per-folder creation gate. The
    folders must be asked as well as the condition because the trie inherits downward only within a realm:
    an `Equipment` grant covers the UNS folder, not the condition at its RawPath.
  - **Revalidation is implicit, not trigger-driven.** Every event re-asks the authorizer, whose decision
    cache is keyed per generation and which re-checks key revocation and expiry on every call — so a
    deployment takes effect on the next event, and an expired key's feed stops on the next event, without
    waiting for the `AclRevalidationTimer` tick. Nothing is detached, so restoring a grant needs no
    re-attach: the next event simply passes. Events raised while withheld are not replayed, and the client
    is not told (the same reasoning as a stood-down direct item).
  - Creation is **not** refused — an all-events item is legitimate for a partially-granted session; what
    is withheld is each event it may not see.
  - **`ConditionRefresh` on a Server-object item** replays only the conditions the session passes the same
    check on (once per condition, not once per notifier it is wired to). Without that, a refresh would
    replay every condition's current state to any session.
  - Inert with the flag off: every Server-object item receives every condition, as before.
  - The replacement delivery keeps the two behaviours of the SDK path it replaced. **Audit events** a
    condition reports through itself (`AuditConditionAcknowledgeEventType`, Comment, Enable, Shelving)
    reach a Server-object item only while the server is auditing, over SignAndEncrypt or https-binary,
    which is exactly the SDK's own guard. And while any Server-object item exists, every condition counts as
    **monitored** (`ConditionState.EventsMonitored()`), which gates the SDK's method **audit** events on a
    server whose only subscribers are Server-object items. Both were caught in review: the
    first cut queued `i=8944` to every Server-object item with auditing off, and left conditions
    unmonitored. The second also needed the SDK's `AddRootNotifier` neutralised: its "subscribe existing
    all-events items with `unsubscribe: true`" bug would otherwise take the unit back from every
    condition rebuilt under a live item. (Method-driven **state-change** events no longer depend on this
    bookkeeping at all — the node manager reports them itself, Gitea #587; see `docs/AlarmTracking.md`.
    The SDK's counter stays imprecise in one known way — a folder subscriber's unit never reaches a
    condition materialised under that folder later, yet its unsubscribe still lowers it — which can now
    only cost an audit event, and only while auditing is on.)
  - Materialising a condition raises only **that** condition's monitored unit; the full reconcile (every
    condition) runs on removal, rebuild and Server-object subscribe/delete. A deploy of N alarms under a
    live Server-object item is therefore N bookkeeping steps, not N²/2 under the node-manager lock.

  **Direct (per-folder) event items are filtered per event too** (Gitea #826). An item on an equipment or raw
  device folder is fed by the SDK's own per-notifier fan-out (`MonitoredNode2`), which the #586 dispatcher never
  touches — so after a deployment moved a scripted alarm (or dropped the reference projecting a native alarm into
  an equipment), the OLD folder's subscribers kept receiving it, EventId included, until the rebuild landed, and
  indefinitely after a failed apply (commands were already denied). With the flag **on**, each event to such an
  item now also needs, against the CURRENT state:
  - the event's condition to be the **live** instance (a torn-down one is withheld);
  - the folder to still be a legitimate notifier of it — a node it hangs beneath, or a folder it is wired into
    **that the current generation still projects its raw tag into** (`DataPlaneAuthoritySnapshot.TagReferencingEquipment`,
    `RawPath → EquipmentIds`, decoded from the artifact's `UnsTagReferences` by the same resolver as the grants);
  - and the condition to pass exactly the Server-object rule above for the item's session.

  The generation check also narrows the **Server-object** rule: a wired folder whose reference the generation
  dropped no longer admits the native alarm there either. **`ConditionRefresh`** on a folder item is filtered the
  same way. The hook is `INodeManager3.ValidateEventRolePermissions`, which the node manager re-implements (the
  base method is not virtual); the SDK memoizes that verdict per `(item, event type, SourceNode)`, so the node
  manager clears each notifier's cache whole (sessions and orphaned, session-less subscriptions alike — an orphaned
  subscription must not queue a moved alarm's events for a later `TransferSubscriptions`) on every new generation or
  authorizer, on each `AclRevalidationTimer` tick, and at the end of every apply, and not at all while enforcement is
  off. A verdict is decided on the condition currently live under the event's NodeId, so a replaced instance's late
  event cannot leave a deny behind for its successor. Only alarm-bearing raw tags are carried in the reference map. Withheld events are not replayed and the client is not told; restoring the reference or
  grant resumes the feed on the next event. ⚠️ A native alarm reaches an equipment folder's items under
  enforcement only while the installed generation carries that reference — a hand-built authority (tests) must
  supply `tagReferencingEquipment`. Inert with the flag off.

  **Which permissions an event on a folder follows — checked against Part 3 (Gitea #838).** OPC UA Part 3's
  `RolePermissions` model gives events their own permission, `ReceiveEvents` (PermissionType bit 11), and the pinned
  SDK (2.0.1, `CustomNodeManager2.ValidateEventReceivePermissions`, documented there against Part 3 §8.55) applies it to
  **both the event's EventType and its SourceNode** — not to the notifier the item happens to sit on. The notifier's own
  permission only decides whether the item may be created/attached. So the spec's intent is *the folder AND the
  source*, never "an event reported through a folder inherits the folder's permissions". This server keeps that shape:
  the re-implemented `ValidateEventRolePermissions` runs the SDK's type + SourceNode check **first** and our decision
  only on top (both must pass). For the event kinds that reach folder items:
  - **Condition events** — SourceNode is the condition. Our decision is the condition rule above (the condition, or a
    folder it legitimately notifies through for the session), which in this ACL model is the deliberate
    generalisation of "the source": a `Tag` grant covers a native alarm's raw node and every UNS projection of it, and
    an `Equipment` grant covers the alarms it owns or references.
  - **A condition's Part 9 method audit events** (Acknowledge, Comment, Shelve…) — SourceNode is the condition, so they
    are resolved to it and follow exactly the condition rule (plus the SDK's audit/SignAndEncrypt guard on the
    Server-object path).
  - **Any other event on a folder item** (none is produced today; `GeneralModelChangeEvent` and the rejected-write
    `AuditWriteUpdateEvent` go to the Server object, #588) names no live condition and is **withheld** under
    enforcement — fail closed, which is stricter than the folder's permission and never laxer than Part 3.
  With the flag off only the SDK's own type + SourceNode `ReceiveEvents` check applies, as before.

  **Event bursts never block under the node-manager lock (Gitea #838).** Each notifier with event items has an SDK
  `MonitoredNode2` whose delivery task drains a **bounded** channel (4096, `FullMode = Wait`), and the synchronous
  `NodeState.ReportEvent` **blocks** while that channel is full. Condition events are reported under the node
  manager's `Lock`, and the delivery task needs that same `Lock` on every per-event verdict-cache miss — in the SDK's
  own `GetNodeMetadata` (so **with the flag off too**, i.e. this predates #826) and in the #826 check. A burst of more
  than 4096 events under one hold of `Lock`, starting at a miss (a new item, or right after a deploy invalidated the
  verdicts), therefore deadlocked the node manager: `FolderEventFloodWireTests` stalled at exactly 4097 events, flag on
  and off. Condition events now go through `ReportEventWithoutBlocking`: `ReportEventAsync` completes inline while the
  channels have room (the normal case, unchanged); when one is full that report continues asynchronously, holding no
  thread and no lock, and later reports queue behind it so events still leave **in report order**. The deferred work
  holds only frozen snapshots — bounded by the burst — and drains once `Lock` is released; what a client receives is
  still bounded by its item's queue (oldest discarded). Pinned by `FolderEventFloodWireTests` (6000 events under one
  `Lock` hold with publishing disabled: completes, deferred reports drain to zero, the item delivers at most its
  revised `QueueSize` in increasing order ending with the last event, and the next event is delivered).

  **Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
  own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the
  `AuditWriteUpdateEvent` of a rejected device write — used to go straight to `Server.ReportEvent`, where
  the core Server object queued them to every Server-object item ungated: a session with no grant learned
  the RawPath / UNS path of every node added, deleted or retyped, and (with auditing on) the written and
  prior values of a rejected write. With the flag **on**, the node manager now delivers them itself, item by
  item, to the Server-object items it already holds for #586 (the same audience — only Server-object items
  ever received them):
  - A **model-change** event keeps only the changes whose `Affected` node the session may **Browse**
    (ancestors of a granted node included, as for Browse itself). Every event the server raises today carries
    one change, so in practice it is delivered whole or not at all; a multi-change event would be delivered
    as a copy holding only the visible changes, with a neutral `Message` (the original names the nodes).
  - An **audit write** event needs `Read` on its source node — it carries the node's prior value and the
    attempted one — on top of the SDK's own guard (server auditing on, SignAndEncrypt or https-binary).
    Auditing stays **off by default**: this repo never sets `ServerConfiguration.AuditingEnabled`.
  - Any other server-wide event type is withheld (fail closed); the node manager raises none today.
  - With the flag **off**, these events go to `Server.ReportEvent` exactly as before.
- **An ancestor of a granted node is browsable**, or no narrow grant would be reachable: a client
  arrives at a deep node by browsing its ancestors, and the trie otherwise inherits downward only. Note
  what this necessarily discloses — the *names* of the folders on the path to a granted node are visible
  to whoever holds that grant. It is conferred only by grants that actually carry `Browse`, and it
  confers nothing but `Browse`.
- **…including the shared root folder every browse starts at.** Both realm trees hang under a single
  anchor folder (`s=OtOpcUa`, minted in the **raw** namespace by `CreateAddressSpace`; top-level raw
  folders and top-level UNS areas alike parent under it). Its identifier is a display name rather than
  a path, so splitting it produced a segment no grant path can contain, and the Browse gate refuses the
  *whole* call when its source node is denied — every narrow `Browse` grant was therefore unreachable
  by navigation, denied at the front door. The gate maps that one identifier onto the trie's
  zero-segment **realm-root** scope, and because the anchor fronts both trees its browse considers
  transit in **either** realm: a principal whose only grant lives in the UNS tree still enters through
  this raw-namespace node. The mapping is realm-qualified — in the UNS namespace `s=OtOpcUa` is an
  ordinary Area an operator happened to name that way and is evaluated as the path it is.
- **A denied node reports `BadUserAccessDenied` rather than `BadNodeIdUnknown`**, because the gate runs
  before node resolution. That is the information-hiding direction — error codes can no longer be used
  to enumerate the address space — but it does mean "access denied" is not evidence the node exists.
- **The shared root folder remains *listed*** for any admitted session. It is referenced from the ns=0
  `Objects` folder, whose browse the SDK's own node manager serves, so a session with no grants sees the
  anchor in that listing — and is refused the moment it tries to browse *into* it (the transit above is
  what a granted session enters on). There is one anchor, not one per namespace.

#### Cost

Read, Browse and subscription creation ask the same question many times per request — one Browse of a
device folder asks once per child. Those seams go through a per-`(session, node, operation, realm)`
decision cache (`SessionAclDecisionCache`), cleared wholesale whenever the authority generation
advances.

⚠️ **The cache holds the grant walk, not the decision.** Grants, principals and the trie are all fixed
for the life of a generation, so memoizing the walk is sound. A key's `ExpiresUtc` passes with the
*clock*, not with a deployment — so revocation and expiry are re-checked on every call, outside the
cache. Caching them would have let an expired key keep working until the next deploy.

**The cache is sharded per calling KEY, because a flat one let any client evict every other client's
decisions.** It used to be a single dictionary with the caller identity folded *into* the composite key
and one process-wide bound of `MaxEntries = 100_000`, whose only eviction was a wholesale `Clear()` at
the cap. So one machine consumer crawling a large address space past 100k entries wiped the memoized
walks of every other caller on the server, and each store paid a `ConcurrentDictionary.Count` — an
all-locks snapshot — to test the cap. It is now a dictionary of shards keyed by
**`ApiKeyId ?? Principal`**, each holding its own `(operation, realm, nodeIdentifier)` entries, with two
bounds:

| Bound | Value | What it clears |
|---|---|---|
| `MaxEntriesPerSession` | 20 000 | that one **key's** shard |
| `MaxEntries` | 100 000 | every shard (the shard *objects* are dropped, not emptied) |

⚠️ **The shard key is the API key, not the session — the constant's name predates it.** Every session
authenticating with the same `otua_<keyId>_…` credential shares one shard and one 20 000-entry budget,
so a crawl by one of them *can* clear the memoized decisions of its sibling sessions. What the sharding
guarantees is isolation **across distinct keys**: one key's crawl can never evict another key's entries.
(An Anonymous session shards under the reserved `anonymous` principal, so all of them likewise share
one shard.)

Both bounds are compile-time constants, not appsettings keys. Five keys simultaneously at their
per-key ceiling reach the global cap; **the number of shards is not itself bounded** — the bound is
entries.

Things to know before changing it:

- **There is no session-close eviction hook**, deliberately. The shard of a key with no live sessions
  left is shed the next time a wholesale clear happens — a generation bump, or the global cap tripping. Dropping the shard
  objects rather than emptying them is what makes that sweep work.
- **The counters are approximate under concurrency, by design.** The cap test is a check-then-act over
  `Volatile.Read` of `Interlocked` counters rather than a lock, so concurrent stores can overshoot a cap
  by roughly one shard's worth. Every clear resets the counters, so drift cannot ratchet. Inserts use
  `TryAdd` (not an indexer assignment) so that only a genuine insert moves a counter — otherwise two
  threads racing the same key would inflate them and permanently pull the ceilings in.
- **Only the CAS winner clears on a generation bump.** `InvalidateIfStale` compare-exchanges the
  generation, so a post-deploy burst of requests repopulates the cache once instead of each thread
  clearing it out from under the others.

**The hot path looks the entry up before it builds anything.** The gate used to construct the node scope
— splitting the `s=` identifier and allocating a `NodeScope` — and then allocate a five-capture closure
for the walk, on *every* call including cache hits. Now the identifier is extracted once, the cache is
consulted immediately, and a hit returns having built no scope, no closure and no
`UserAuthorizationState`; the scope and the walk happen only on a miss. `GetOrAdd` was split into an
allocation-free `TryGet` plus a `Set` to make that possible, and `TryGet` deliberately does not
create a shard, so a hit-only session never mints one and a just-cleared shard is not resurrected until
something is actually stored.

⚠️ **Revocation is checked ahead of the cache, not behind it** — the key-usability lookup runs *above*
the `TryGet`, so a cache hit cannot serve an allow for a key that has since been disabled or expired.
Reversing those two lines is the subtle way to reintroduce the bug the paragraph above warns about: the
generation never moves when a key merely expires, so nothing would ever drop the stale allow. Two tests
pin the ordering (`Key_revocation_is_rechecked_on_a_cache_hit` and
`An_expired_key_is_denied_even_though_its_allow_is_cached`).

**Deny reasons are composed lazily.** A denial on the cached path logs at `Debug`, and the interpolated
reason strings (which generation, which key, which operation) used to be built at every call site before
the log level was consulted — so a Browse denied across a thousand children formatted a thousand strings
with `Debug` off. The interpolating denials now guard on `_log.IsEnabled(level)` first, and the
`BadUserAccessDenied` `ServiceResult` is a shared static instance rather than one allocation per denied
node.

#### Everything denies when something is missing

No identity, no principal on the identity, **no cluster identity on the node**, a NodeId with no string
identifier, or a trie that was never installed — each returns `BadUserAccessDenied`. The client is told
nothing beyond that; the reason is logged server-side (`AclGate: denied …`) so the server cannot be used
as an oracle for which nodes exist or which grants a principal holds.

The consequence to plan for: a wiring or configuration mistake presents as **everything denied**, not
everything allowed. Confirm the trie exists at all before debugging a specific rule — the
`OpcUaPublish: data-plane authority generation N installed` line reports the grant count, and a separate
warning names every rule the build **dropped** (a dangling `ScopeId`, a blank one, a retired scope kind,
or a scope path that does not start at a realm). A dropped rule can never match anything.

### The `NodeAcl` + `PermissionTrie` subsystem

What is real about it:

- Operators **can** author `NodeAcl` rows in the Admin UI (`/clusters/{id}/acls`), and those rows are
  validated, versioned and persisted.
- `ConfigComposer` **does** snapshot every `NodeAcl` row into every deployment artifact, so an ACL
  edit shifts the artifact's `RevisionHash` and ships to every node.
- The node side **does** deserialize them (#520 stage 4) —
  `DeploymentArtifact.ParseDataPlaneAuthority` reads the artifact's `NodeAcls` (scoped to the node's own
  cluster) and `ApiKeys`, and `OpcUaPublishActor` installs both on the node's
  `IDataPlaneAuthoritySource` on every apply. The node manager holds that source, so it has a cluster
  identity.
- The same `Apply` call **builds and installs the permission trie** (#520 stage 5), under the same
  generation as the snapshot. Deliberately one call rather than two wirings: an uninstalled trie denies
  everything, which is indistinguishable from working enforcement until someone checks that an ALLOW is
  possible at all.
- `DataPlaneAuthorizer` **evaluates** them at every data-plane seam (#520 stages 5–6), when the flag is
  on. With the flag off an operator can still author a grant, deploy it green, and have it change nothing
  beyond the two coarse role gates — that is the documented default, not an unfinished state.

**The four things blocking a wire-up** (all verified, none of them merely "call the evaluator"):

1. ~~**Raw-realm nodes have no representable scope.**~~ **Closed by #520 stage 3.** `NodeScope` is now
   `(ClusterId, Kind, Segments)` with a variable-length path, and `NodeHierarchyKind` carries both
   `Uns` and `Raw`. The previous fixed-arity shape could not address a real raw tree at all — folder
   chains and tag-group chains both nest without bound — so it silently capped at five levels and
   mislabelled every one of them. Four `Raw*` scope kinds were appended for authoring.
2. ~~**The identity carries roles, not groups.**~~ **Dissolved by #520 stage 2.** The column is now
   `NodeAcl.Principal` and the principal comes from the presented API key (or the reserved `anonymous`),
   not from a directory — so there is no group list to lose.
3. ~~**`PermissionTrieBuilder`'s `scopePaths` argument has no producer.**~~ **Closed by #520 stage 3.**
   `AclScopePathResolver` is that producer, built on both sides of the split — EF entities at the deploy
   gate, artifact JSON on the node — and `scopePaths` is now a required argument with no fallback: an
   unresolvable scope is **dropped with a diagnostic** rather than planted at a trie position the walk
   can never reach. A scope resolves to *several* paths when one entity is reachable at several NodeIds
   (a raw tag projected into N equipments has one raw path plus N UNS paths), because a grant that
   covered only some of them would be bypassable through the one it missed.
4. ~~**No session lifecycle exists.**~~ **Closed by #520 stage 5, though not as the blocker imagined.**
   The generation exists and advances (stage 4), and `DataPlaneAuthorizer` now builds a
   `UserAuthorizationState` stamped with it — but **per authorization call, not per session**. The
   per-session design was LDAP-shaped: it assumed a directory could change a session's group membership
   out of band, so a long-lived session had to re-resolve on a timer and serve cached memberships in
   between. Under API keys nothing can change out of band — the principal comes from the key, and keys
   and grants both ride the deployment artifact — so the generation stamp is the entire invalidation
   signal, and rebuilding a small record per call picks it up for free while a cached one would have to
   be explicitly invalidated. The consequence is that `UserAuthorizationState.IsStale` and `NeedsRefresh`
   can never fire on this path; they are vestigial here rather than load-bearing. Caching becomes
   necessary at stage 6's read/browse/subscribe seams, which are on the publish path — and it will want
   a per-`(session, node)` decision cache, not this type.

`NodeAclScopeKind.FolderSegment` is the retired v2 kind with no v3 entity behind it. The stage-7 picker
does not offer it at all, so it can only arrive on a legacy row or a hand-edited one; it can never
resolve, and every end now says so specifically rather than reporting it as a missing id: the AdminUI
refuses to save it, the deploy gate rejects it with
`NodeAclScopeKindRetired`, and the trie builder drops it with a `RetiredScopeKind` diagnostic (Gitea
#533 — it used to arrive as `UnresolvedScope`, which sent an operator holding a legacy row hunting for a
typo in an id that was never the problem). One inconsistency remains: `NodePermissions.AlarmRead` has no
`OpcUaOperation` mapping, so it is grantable but unreachable.

**String identity in this subsystem** (Gitea #533): **identifiers** — `ScopeId`, `ClusterId`,
`Principal` — compare `OrdinalIgnoreCase`, because they are opaque handles whose spelling genuinely
varies by source (a `ClusterId` arrives lower-cased through `OTOPCUA_ROLES` and verbatim through
`Cluster:Roles`; a `ScopeId` comes from the stage-7 picker but a legacy row's may be anything).
**Path segments** —
every level of a scope path, and therefore `PermissionTrieNode.Children` — compare `Ordinal`, because
they are browse names and `Speed` / `speed` are two different nodes that can coexist under one device.
The rule is stated in full on `AclScopeRef`. It matters most where the deploy gate and the node are fed
from *different* snapshots, which is exactly the split `AclScopePathResolver` exists to bridge: case
drift there would silently drop every grant on the affected scope.

### Authoring grants (#520 stage 7)

Grants are authored at **`/clusters/{clusterId}/acls`**. Two things about the surface are worth knowing
before relying on it, because both close a way a grant could previously be saved and deployed while
authorizing nobody.

**The scope kind and the scope target are chosen together, from a tree.** "Choose scope…" opens a
picker showing the cluster's **raw device tree** (lazily expanded, exactly the `/raw` hierarchy) beside
its **UNS tree**, plus a whole-namespace row on each side. Selecting a node sets `ScopeKind` *and*
`ScopeId` in one action. Until this landed the two were an unrelated dropdown and a free-text box, so
`RawDevice` + a TagId was a well-formed row: it saved, it deployed, and it resolved to nothing.

Consequences of the trees being rooted at the grant's own cluster and of a node supplying its own kind:

- A **cross-cluster** scope is structurally unreachable — the picker's counterpart to the deploy gate's
  `NodeAclScopeCrossCluster` rule, rather than a second copy of it.
- A **Cluster** row deliberately produces **no** `ScopeId`. The deploy gate rejects a Cluster grant that
  carries one (`NodeAclScopeIdUnexpected`), so a picker that helpfully filled in the cluster id would
  author rows that cannot be deployed.
- **UNS tags are not offered.** A UNS signal is a `UnsTagReference` to a raw tag, and a `Tag` grant
  already covers the raw node *and* every UNS projection of it — so a UNS tag row would produce a grant
  that reads narrower than the one it creates. Pick the tag in the raw tree.
- `FolderSegment` is not offered at all (see above).

**Rows are resolved against the live configuration, at authoring time.** `IAclScopeService.CheckScopeAsync`
mirrors `DraftValidator.ValidateNodeAclScopes` rule for rule, so the page rejects what the deployment
would reject:

- The editor **refuses to save** a grant whose scope does not resolve — re-checked on submit, not
  trusted from when the form loaded, because an existing row's entity may have been deleted since and
  the operator may never touch the picker.
- The **list page** resolves every row and marks a broken one `unresolved` with the reason, so a grant
  that has rotted since it was authored is visible without waiting for a deploy to fail.
- The **principal** field suggests every known principal (API keys, existing grants, `anonymous`) and
  reports how many *live* keys currently authenticate as the one typed. Reaching nobody is a **warning,
  not a block** — authoring a grant before minting its key is legitimate; a typo is not, and the two are
  indistinguishable to the server.

The checks are in `AclScopeService` rather than in the Razor pages because this repo has no bUnit:
logic only a page can reach is untestable, and "the grant authorizes nobody" is not a failure mode to
leave unpinned.

### Hierarchy (design)

ACLs are evaluated against the node's scope path. `NodeScope` (`src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/NodeScope.cs`) carries a `Kind` that selects between two hierarchy shapes:

```
Equipment (UNS) kind:        Cluster → Namespace → UnsArea → UnsLine → Equipment → Tag
SystemPlatform (Galaxy) kind: Cluster → Namespace → FolderSegment(s) → Tag
```

On the Galaxy/SystemPlatform path each folder segment takes one trie level, so a deeply-nested Galaxy folder reaches the same depth as a full UNS path. Unset mid-path levels leave the corresponding id `null` and the evaluator walks only as far as the scope goes.

Each level can carry `NodeAcl` rows (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/NodeAcl.cs`) that grant a permission bundle to a set of `LdapGroups`.

### Permission flags

`NodePermissions` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Enums/NodePermissions.cs`), stored as an `int` bitmask in `NodeAcl.PermissionFlags`:

```csharp
[Flags]
public enum NodePermissions : int
{
    None = 0,

    Browse            = 1 << 0,
    Read              = 1 << 1,
    Subscribe         = 1 << 2,
    HistoryRead       = 1 << 3,
    WriteOperate      = 1 << 4,
    WriteTune         = 1 << 5,
    WriteConfigure    = 1 << 6,
    AlarmRead         = 1 << 7,
    AlarmAcknowledge  = 1 << 8,
    AlarmConfirm      = 1 << 9,
    AlarmShelve       = 1 << 10,
    MethodCall        = 1 << 11,
    HistoryUpdate     = 1 << 12,   // OPC UA annotation / insert / delete (separate from HistoryRead)

    ReadOnly  = Browse | Read | Subscribe | HistoryRead | AlarmRead,
    Operator  = ReadOnly | WriteOperate | AlarmAcknowledge | AlarmConfirm,
    Engineer  = Operator | WriteTune | AlarmShelve,
    Admin     = Engineer | WriteConfigure | MethodCall,
}
```

The three Write tiers map to Galaxy's v1 `SecurityClassification` — `FreeAccess`/`Operate` → `WriteOperate`, `Tune` → `WriteTune`, `Configure` → `WriteConfigure`. `SecuredWrite` / `VerifiedWrite` / `ViewOnly` classifications remain read-only from OPC UA regardless of grant.

`HistoryUpdate` (bit 12) is a **separate** permission from `HistoryRead` so a read-only grant cannot also authorize OPC UA HistoryUpdate operations. `TriePermissionEvaluator` maps `OpcUaOperation.HistoryUpdate` to this bit. It is **not** included in any composite bundle (`ReadOnly` / `Operator` / `Engineer` / `Admin`) — it must be granted explicitly, and it is offered as its own checkbox in the AdminUI ACL editor (`AclEdit.razor`). The HistoryUpdate service itself is now implemented (SQL Server historian backend only) — see the next section.

### HistoryUpdate (historian writes)

HistoryUpdate rewrites recorded history, so unlike HistoryRead it is gated **whatever `Security:Acl:Enabled` says**, and `WriteOperate` (live writes) never implies it.

- **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`.
- **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help.
- **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate).
- **Primary-only, after authorization (Gitea #674).** An authorized HistoryUpdate on a node that is not the redundancy Primary is refused `BadInvalidState` (the native-alarm-ack verdict, #597); an unauthorized one is told `BadUserAccessDenied` first, so the refusal never reveals the node's role to a caller who may not update.
- **Every applied change is attributed (Part 11 modified-value history, Gitea #668).** The session's principal (the API key's
  `NodeAcl` principal, or the reserved `anonymous`) and key id are passed to the backend with every HistoryUpdate and recorded in the
  modification trail next to the prior value; a modified-value HistoryRead (`IsReadModified = true`) returns it as
  `ModificationInfo.UserName`. Reading the trail is gated by the **same `HistoryRead` decision** as a raw read — there is no separate
  permission — so with `Security:Acl:Enabled` off it is as ungated as any HistoryRead, and anyone who may read a node's history can see
  who rewrote it and what it held before. See [`docs/Historian.md` §Modified-value history](Historian.md#modified-value-history-part-11).

### Evaluator — `PermissionTrie`

`src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/`:

| Class | Role |
|---|---|
| `PermissionTrie` | Cluster-scoped trie; each node carries `(GroupId → NodePermissions)` grants. |
| `PermissionTrieBuilder` | Builds a trie from the current `NodeAcl` rows in one pass and installs it into the cache. |
| `PermissionTrieCache` | Process-singleton cache keyed on `(ClusterId, GenerationId)`. Generation-sealed: `Install(trie)` adds a new generation + advances the "current" pointer; older generations are retained (in-flight requests still resolve) and GC'd by `Prune`. `Invalidate(clusterId)` drops every cached trie for a cluster. There is **no** `AclChangeNotifier` — a publish installs a new generation rather than signalling an invalidation. |
| `TriePermissionEvaluator` | Implements `IPermissionEvaluator.Authorize(session, operation, scope)`. Walks the cluster trie for the supplied `NodeScope`, unions grants along the path, and returns an `AuthorizationDecision`. Evaluates against the **session's bound generation** (`session.AuthGenerationId`), not just "current", so a grant added/removed in a newer generation cannot take effect mid-session. |

`NodeScope` is described above (Equipment-kind vs SystemPlatform-kind). The evaluator unions the matched grants along the path — a tag-level ACL and an area-level ACL both contribute.

### Dispatch gate — `DataPlaneAuthorizer` over `IPermissionEvaluator`

`IPermissionEvaluator.Authorize(UserAuthorizationState session, OpcUaOperation operation, NodeScope scope)` (default impl `TriePermissionEvaluator` at `src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/TriePermissionEvaluator.cs`) returns an `AuthorizationDecision`.

**The dispatch gate exists since #520 stages 5–6** (2026-08-01): `DataPlaneAuthorizer`
(`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/DataPlaneAuthorizer.cs`, registered in
`Host/Program.cs`, consulted by `OtOpcUaNodeManager`) calls `Authorize` at every data-plane seam —
read, browse, subscribe, write, alarm commands and HistoryRead — **when `Security:Acl:Enabled` is
true**. With the flag **off (the default)** the authorizer is constructed but not consulted, and the
effective posture for Read/Browse/Subscribe/HistoryRead is default-ALLOW (see
[What is enforced with the flag off](#what-is-enforced-with-the-flag-off)).

Properties of the wired path:

- **Driver-agnostic.** No driver-level code participates in authorization decisions. Drivers report `SecurityClassification` as metadata on tag discovery; everything else flows through the evaluator.
- **Strictly fail-closed (default-deny) when consulted.** Every guard path returns `NotGranted` — a stale session (past the staleness ceiling, decision #152), a cluster mismatch between session and scope, a missing trie, a pruned bound generation, or simply no matching grant. A misconfigured node therefore denies everything rather than allowing everything, which presents as an outage of the gated operations rather than a subtle permission bug.
- **Evaluator stays pure.** `TriePermissionEvaluator` has no OPC UA stack dependency and is tested directly from xUnit.

### Full model

See [`docs/v2/acl-design.md`](v2/acl-design.md) for the complete design: trie invalidation, flag semantics, per-path override rules, and the reasoning behind additive-only (no Deny). Most of it shipped with #520 stages 5–7 (behind `Security:Acl:Enabled`); where it and this document disagree, this document plus `docs/plans/2026-07-31-apikey-acl-enforcement-design.md` describe the running behaviour.

Note also that the `SystemPlatform` hierarchy kind named above was retired with the v3 dual-namespace address space; `NodeHierarchyKind` now has only `Equipment`.

### Role grant source (data-plane)

Since #520 (2026-08-01), data-plane roles come from the authenticated principal's **`NodeAcl` grants**,
not from LDAP. `ApiKeyInterimRoleMapper` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/ApiKeyInterimRoleMapper.cs`,
#520 stage 2) derives the session's server-wide role set from the grants naming the principal — a key's
principal, or the reserved `anonymous` — and attaches it to the `RoleCarryingUserIdentity` the gates
read. Neither `Security:Ldap:GroupToRole` nor the Config-DB `LdapGroupRoleMapping` table touches this
path any more; both feed **Admin UI login only**.

Two properties of that derivation matter to an operator:

- ⚠️ **Only `Cluster`-scoped grants confer a role.** This is deliberate, and it under-grants: honouring
  a narrower grant's flags while unable to honour its scope would turn "WriteOperate on Equipment X"
  into server-wide write authority. A scope-narrower-than-cluster grant is simply **inert with the flag
  off** (the AdminUI banner says so) — it takes effect only when `Security:Acl:Enabled` turns scoped
  evaluation on.
- **Flag vocabulary ≠ role vocabulary.** The `WriteOperate` permission flag maps to the `WriteOperate`
  role; the `AlarmAcknowledge` flag (the spelling an operator authors) maps to the `AlarmAck` role,
  which covers acknowledge/confirm/shelve as one gate. `ReadOnly`, `WriteTune` and `WriteConfigure`
  flags confer **no role** — with the flag off no gate reads them, and with it on the scoped ACL
  decision replaces roles entirely.

If a principal has no cluster-scoped grant carrying `WriteOperate` / `AlarmAcknowledge`, **writes and
alarm acknowledgement default-deny**: they return `BadUserAccessDenied` even for sessions that
authenticate successfully. (The same requirement gates both the scripted-alarm and the native
Galaxy-alarm Part-9 ack/confirm/shelve paths.) **Reads, browses, subscribes and history reads are
unaffected with the flag off** — they succeed with or without any grant.

---

## Control-Plane Authorization

Control-plane authorization governs **the Admin UI** — who can view fleet config, edit drafts, publish generations, manage cluster nodes + credentials.

Per decision #150 control-plane roles are **deliberately independent of data-plane ACLs**. An operator who can read every OPC UA tag in production may not be allowed to edit cluster config; conversely a `Designer` may not have any data-plane grants at all.

### Roles

The `AdminRole` enum (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Enums/AdminRole.cs`) defines three roles. Task 1.7 standardized the member names on the canonical `ZB.MOM.WW.Auth` `CanonicalRole` vocabulary (`ConfigViewer → Viewer`, `ConfigEditor → Designer`, `FleetAdmin → Administrator`); a data migration (`CanonicalizeAdminRoles`) rewrote existing rows. This was a rename, not a permission change.

| Role | Capabilities |
|---|---|
| `Viewer` | Read-only access to drafts, generations, audit log, fleet status. (Was `ConfigViewer`.) |
| `Designer` | Viewer plus the whole `ConfigEditor` surface: authoring (UNS, equipment, tags, ACLs, driver instances, reservations, CSV imports, **scripts** — virtual tags, Calculation tags, scripted-alarm predicates) **and deploy** (`/deployments`). A Designer can therefore put operator-authored C# into production; scripts run in-process on driver nodes (see [VirtualTags.md](VirtualTags.md) §"Known resource limits"). (Was `ConfigEditor`.) |
| `Administrator` | Designer plus the `FleetAdmin`-only surface: role-grant management, API keys, certificate trust/untrust/delete and manual redundancy failover. Satisfies both the `FleetAdmin` and `DriverOperator` authorization policies. (Was `FleetAdmin`.) |

`DriverOperator` is an **authorization policy name** (kept stable), not an `AdminRole` member. It gates **Reconnect** / **Restart** commands against live driver instances from the Admin UI `DriverStatusPanel` and requires the canonical role `Operator` or `Administrator` (`policy.RequireRole("Operator", "Administrator")` in `AddAuthorization`, `src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs`). `Operator` is an appsettings-only string role (not an `AdminRole` member); map an LDAP group to it via `GroupToRole`, e.g. `"ot-driver-operator": "Operator"`. The `FleetAdmin` policy requires the `Administrator` role.

#### AdminUI page authorization policies (R2-05)

Every routable AdminUI page carries an explicit **named-policy** `@attribute [Authorize(Policy = …)]` — there are no bare `[Authorize]` (authenticated-only) config pages and no per-page `Roles="…"` strings. The policy names are constants on `AdminUiPolicies` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Auth/AdminUiPolicies.cs`), the single source of truth referenced by pages, `AuthorizeView` blocks, imperative `IAuthorizationService.AuthorizeAsync` checks, and endpoint groups (never string literals). All four are registered by `AddOtOpcUaAuth`:

| Policy | Satisfied by roles | Gates |
|---|---|---|
| `AdminUiPolicies.ConfigEditor` | `Administrator`, `Designer` | The config-authoring surface: `/uns`, equipment, cluster/node/namespace/ACL editors, all 8 driver pages (which author live retry/breaker `ResilienceConfig`), Deployments, Scripts, ScriptEdit, and the `/api/script-analysis/*` endpoint group. |
| `AdminUiPolicies.FleetAdmin` | `Administrator` | `RoleGrants` page; the `Certificates` Trust/Untrust/Delete actions (`AuthorizeView` + server-side re-check). |
| `AdminUiPolicies.DriverOperator` | `Operator`, `Administrator` | Imperative live-ops actions only (Alerts ack/shelve, DriverStatusPanel reconnect/restart, live address pickers). |
| `AdminUiPolicies.AuthenticatedRead` | any authenticated user | The read-only dashboards, lists, and live tails (Home, Fleet, Hosts, Alerts, Certificates, cluster read views, etc.). Semantically identical to the `FallbackPolicy`; named so read pages carry a non-bare attribute. |

`Login` is the only `[AllowAnonymous]` page (Admin-001). A reflection guard — `PageAuthorizationGuardTests` (`tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Authorization/PageAuthorizationGuardTests.cs`) — enumerates every routable type in the AdminUI assembly and fails the build if any page carries a bare `[Authorize]`, a `Roles="…"` idiom, an unknown policy name, or is missing from the classification map. This is the anti-drift anchor (the repo has no bUnit): a new page fails the test until its author consciously classifies it. `AdminUiPoliciesTests` (`tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/AdminUiPoliciesTests.cs`) pins each policy's role semantics.

In v2 the authentication + authorization stack is wired centrally by `AddOtOpcUaAuth` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs`), which also installs a `FallbackPolicy` that requires an authenticated user. Razor pages gate inline with the named-policy attribute above, e.g. `@attribute [Authorize(Policy = AdminUiPolicies.ConfigEditor)]`. Nav-menu sections hide via `<AuthorizeView>`.

#### Address-picker browse is an outbound-connect capability (trust boundary)

Worth stating plainly, because "browse a configured database" reads weaker than what it is: the AdminUI's
live address pickers **dial a target named in the browse form, from the AdminUI host, before anything is
persisted.** The `Sql` picker is the sharpest case — `SqlDriverBrowser.OpenAsync`
(`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser/SqlDriverBrowser.cs`) accepts an **ad-hoc
`connectionString` literal** an operator pastes for a database that has no `connectionStringRef` yet, and
hands it to the ADO.NET provider verbatim. That is an arbitrary host and port plus whatever the provider's
keyword set allows (`Microsoft.Data.SqlClient` alone offers `AttachDbFilename`, `Failover Partner`,
`Enlist`). The OPC UA client and MQTT pickers have the same shape against their own endpoint fields.

**The control is the page policy, and it is admin-grade.** The only surfaces that reach a browse session
are `/raw` and `/uns`, both `@attribute [Authorize(Policy = AdminUiPolicies.ConfigEditor)]` — satisfied by
`Administrator` or `Designer` only. There is **no HTTP endpoint** for browse: `IBrowserSessionService` is a
scoped service consumed from those pages' Blazor circuits, so it inherits the page's policy and the
`FallbackPolicy` beneath it, and `PageAuthorizationGuardTests` fails the build if either page's attribute
drifts. A `Viewer` cannot open a picker; `AdminUiPolicies.DriverOperator` gates only the *live-ops* actions
inside one (Sparkplug rebirth), never the connect.

Supporting hygiene, none of which is the control:

- The pasted literal is **never persisted** — `SqlDriverConfigDto` has no such property, so the driver
  factory would drop it; the picker composes a session-only document and the browser keeps no field.
- A literal supplied alongside a `connectionStringRef` **logs a warning** naming the shadowed ref (never
  any connection text), so the override is visible rather than silent.
- Every failure on the open path is **redacted** (`SqlDriverBrowser.Sanitize`), and the provider's own
  connection-string-parser message is withheld outright because it quotes connection-string text verbatim.
- `NodeEdit.razor` warns against pasting a literal into a *saved* `DriverConfig`, which would persist it.

If a deployment needs the browse surface narrower than "anyone who may author config", the lever is the
page policy (and the LDAP group→role mapping behind it), not the browser.

#### Credential references are namespaced per driver (#815)

The outbound-connect capability above has a sharper edge whenever the config carries a **credential
reference**: a Designer authors both the endpoint *and* the reference, and the node resolves the reference
and sends the value to that endpoint — on Test Connect, on a live browse, and at runtime. Before #815 an
unscoped reference let anyone who may author config exfiltrate any value the admin node could read, with no
deploy: point a Galaxy `Gateway.Endpoint` at their own host, set `ApiKeySecretRef` to
`env:ConnectionStrings__ConfigDb` (or `env:ServerHistorian__ApiKey`, or `file:<any path>`), and click Test
Connect / browse; or set an OpcUaClient `Password` to `secret:<any store entry>`.

Every operator-authored reference is now confined to its driver's namespace — one table,
`DriverCredentialPolicy` (`Core.Abstractions`), one parser/resolver, `CredentialReferences`:

| Driver | Fields | Allowed references | Literal |
|---|---|---|---|
| `GalaxyMxGateway` | `Gateway.ApiKeySecretRef` | `secret:galaxy/<name>`, `secret:otopcua/galaxy/<name>`, `env:OTOPCUA_GALAXY_<NAME>`, legacy `env:GALAXY_MXGW_<NAME>` | allowed (warns), `dev:<key>` |
| `GalaxyMxGateway` | `AlarmAck.FallbackPasswordSecretRef` | same | `dev:<value>` only |
| `OpcUaClient` | `Password`, `UserCertificatePassword` | `secret:opcuaclient/<name>`, `env:OTOPCUA_OPCUACLIENT_<NAME>` | allowed |
| `Heidenhain` | `InspectPassword`, `DncPassword`, `DataPassword` | `secret:heidenhain/<name>`, `env:OTOPCUA_HEIDENHAIN_<NAME>` (#803) | refused |

Rules (all drivers): the prefixes are lower-case and exact (`SECRET:` / `Env:` are refused, never treated
as a literal); `${secret:<name>}` is accepted as a spelling of `secret:<name>`; secret names are
`[A-Za-z0-9._-]` segments with no empty, `.` or `..` segment; the variable suffix is `[A-Z0-9_]+`; **`file:` is
refused everywhere** (Galaxy used to read any file the node could open). A field is matched at any depth of
the DriverConfig / DeviceConfig, case-insensitively, as the drivers bind it.

Enforced, fail-closed, at every seam — the earlier ones exist for a clear message, the last one is the guarantee:

1. **AdminUI save** — `DriverConfigSaveGate` (`RawTreeService` driver/device create + update): the config is
   never stored.
2. **Deploy gate** — `CredentialReferenceDeployChecks`, code **`CredentialReferenceOutOfScope`** (Heidenhain keeps
   `HeidenhainCredentialPersisted`).
3. **Test Connect** — `AdminOperationsActor` refuses before any probe runs (`"Test Connect refused: …"`).
4. **Live browse** — `BrowserSessionService.OpenAsync` refuses before any bespoke or universal browser opens.
5. **Resolve time** — `GalaxySecretRef`, `GalaxyAckCredentialResolver`, `OpcUaUserIdentityFactory` and
   `HeidenhainCredentials` all resolve through `CredentialReferences`, which re-checks the scope **before** any
   secret-store or environment read. This is hard-fail on every admin-node path (the Galaxy and OpcUaClient
   AdminUI browsers, every Test Connect probe) and everywhere for Heidenhain. The one exception is the deprecated
   warn-only compatibility path for already-deployed drivers, below.

Messages name the field and the allowed forms, never the value. Out of scope by design: `${secret:}` tokens in
`appsettings.json` (host config, not operator-authored), and the `Sql` driver's `connectionStringRef`, which is
already a fixed prefix (`Sql__ConnectionStrings__<ref>`) whose value carries its own server — a Designer can
choose *which* provisioned database a ref names, but cannot redirect its credentials to another host.

**Migration (existing configs).** A stored config with an out-of-namespace reference is listed by the upgrade
preflight (below), refused by its next deploy (`CredentialReferenceOutOfScope` — the message names the
migration), and refused by Test Connect / live browse. Values most likely to be in real config databases, with
the exact replacement (create the new secret / variable first — on every node that runs the driver **and** on
the admin nodes, which run Test Connect and browse — then edit the field):

| Old value | Why it is refused | Replace with |
|---|---|---|
| Galaxy `env:MX_API_KEY` — the **old AdminUI Galaxy form default** (pre-#815 `GalaxyDriverForm.razor`), so the likeliest value in a real DB | variable outside the namespace | rename the variable to `OTOPCUA_GALAXY_API_KEY` and set `env:OTOPCUA_GALAXY_API_KEY` (or store it and use `secret:galaxy/<inst>/apikey`) |
| Galaxy `env:<ANY_OTHER_NAME>` (e.g. `env:MY_API_KEY`) | same | variable `OTOPCUA_GALAXY_<NAME>` → `env:OTOPCUA_GALAXY_<NAME>`. `env:GALAXY_MXGW_API_KEY` (docker-dev, parity docs) needs **no** change |
| Galaxy `file:<path>` — `docs/Configuration.md` used to recommend it for production | `file:` is refused everywhere | put the file's contents in the store as `galaxy/<inst>/apikey` → `secret:galaxy/<inst>/apikey` (or a variable `OTOPCUA_GALAXY_<NAME>`) |
| Galaxy `secret:<name>` not under `galaxy/` (e.g. the old `docs/v2/driver-specs.md` example `secret:galaxy-gw-api-key`) | secret outside the namespace | re-create it as `galaxy/<name>` (e.g. `galaxy/gw/apikey`) → `secret:galaxy/gw/apikey`. Names already under `galaxy/` or `otopcua/galaxy/` need no change |
| Galaxy `Env:X`, `SECRET:x`, `File:x`, … (any non-lower-case prefix) — **pre-#815 resolved these case-insensitively** | prefixes are now exact | the lower-case namespaced form from the rows above |
| Galaxy `AlarmAck.FallbackPasswordSecretRef` = `env:` / `file:` / `secret:` outside the namespace | same | as above; the documented `secret:otopcua/galaxy/<name>` needs no change |
| OpcUaClient `Password` / `UserCertificatePassword` = `secret:<name>` not under `opcuaclient/` (e.g. `secret:opcua/plant/pw`) | secret outside the namespace | re-create the secret as `opcuaclient/<name>` → `secret:opcuaclient/<name>` |
| OpcUaClient `SECRET:x` / `Secret:x` — pre-#815 resolved case-insensitively | prefixes are now exact | `secret:opcuaclient/<name>` |
| OpcUaClient **literal** password that happens to start with `env:`, `file:`, `secret:`/`${secret:` in any casing (e.g. a real password `Env:Prod1`) — pre-#815 sent it verbatim | it now parses as a reference | store it: `secret:opcuaclient/<name>` (a literal that does not start with one of those prefixes is unaffected) |

**Warn-only compatibility for already-deployed drivers (⚠️ deprecated — a later release fails closed).** Refusing
at resolve time would fault running plant drivers at the upgrade restart — and, for Galaxy, at **every gateway
reconnect**: `GalaxyDriver.ReopenAsync` (the `ReconnectSupervisor` callback) rebuilds the client options through
`BuildClientOptionsAsync`, which re-resolves `Gateway.ApiKeySecretRef` each time, so a running driver would fault
at its next reconnect, not at a deploy. So, for one compatibility window, the running-driver call sites — Galaxy
`InitializeAsync` / `ReopenAsync` / the ack-fallback resolver, and OpcUaClient session open — resolve an
out-of-namespace reference with the **old** semantics (Galaxy: case-insensitive `env:`/`file:`/`dev:`/`secret:`,
literal otherwise; OpcUaClient: case-insensitive `secret:`, literal otherwise) and log a Warning naming the
driver, field and allowed form (never the value), at most once per field per hour. The use also rides the
driver's diagnostics bag (`credential-compat:<field>`) to `/hosts` and to the `/raw` credential banner. This is
chosen **by call site**, not by a flag (`LegacyCredentialReferenceTracker`): the AdminUI browsers and every probe
never get the tracker and stay hard-fail, and Heidenhain (new driver, no legacy configs) is hard-fail everywhere.
It cannot be used to exfiltrate: save, Test Connect, browse and the deploy gate all refuse such a reference, so it
can only be the reference an administrator deployed **before** the upgrade, and it lives only until that driver's
next successful deploy (which requires migrating it). The hard-fail release deletes the tracker, the per-driver
`Legacy…` resolve methods and their `legacy` parameters — nothing else changes.

**Upgrade preflight.** On an admin node, `CredentialReferencePreflightService` runs once, 15 s after
`ApplicationStarted` (never inside an Akka starter, and it resolves nothing — ConfigDb reads only). It scans
every credential-resolving driver's stored `DriverConfig` / `DeviceConfig` rows (**Draft**) and the latest
deployment artifact (**Deployed** — what the running drivers were given) and logs one Warning per finding:
driver, device, field, allowed form, never the value. The same list, plus running drivers that reported a
warn-only resolution (**Running**, from driver-health diagnostics, fleet-wide), is shown as a banner on `/raw`,
with a Re-check button.

Residual (accepted): within one namespace, a Designer can still point one driver instance at another
instance's secret (e.g. `secret:galaxy/<other-inst>/apikey`) — the value then reaches the endpoint the Designer
chose. Per-instance scoping is a possible follow-up.

### Role grant source

Admin reads `LdapGroupRoleMapping` rows from the Config DB (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/LdapGroupRoleMapping.cs`) — the same pattern as the data-plane `NodeAcl` but scoped to Admin roles + (optionally) one cluster for multi-site fleets (a system-wide row, `IsSystemWide = true`, stacks additively with cluster-scoped rows). The `RoleGrants.razor` page lets `Administrator`s edit these mappings without leaving the UI.

### Headless deploy API (`POST /api/deployments`)

For CI / scripts that need to trigger a deployment without driving the Blazor "Deploy current configuration" button, admin-role nodes expose `POST /api/deployments` (`DeployApiEndpoints`, `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Api/DeployApiEndpoints.cs`). It forwards to the same `IAdminOperationsClient.StartDeploymentAsync` the button calls.

Auth is a **single configured secret** checked from the `X-Api-Key` header in fixed time — deliberately orthogonal to the cookie-only web auth (`OPC UA Authentication` above) so automation needs no LDAP login round-trip. The endpoint is `AllowAnonymous` so the `FallbackPolicy` doesn't 401 it, and enforces the key itself. **It self-disables (503) until `Security:DeployApiKey` is set**, so it is never open by default.

```bash
curl -X POST https://<admin-host>/api/deployments \
     -H 'X-Api-Key: <Security:DeployApiKey>' \
     -H 'Content-Type: application/json' \
     -d '{"createdBy":"ci-bot"}'
```

Responses: `202 Accepted` (`{ outcome, deploymentId, revisionHash }`) when a deployment was sealed, `200` for `NoChanges`, `409` when another deployment is in flight, `422` when rejected, `401` for a missing/wrong key, `503` when unconfigured. Set the secret via `Security:DeployApiKey` (env `Security__DeployApiKey`) on admin nodes only; treat it like any deploy credential (rotate, keep out of source).

---

## Mesh gRPC ingress is `AllowAnonymous` by design

Archreview 03/S14 (Critical, closed R3-01). On an admin-role node `AddOtOpcUaAuth` installs a global
`FallbackPolicy = RequireAuthenticatedUser` bound to the cookie scheme (the same fallback that protects
every AdminUI page above). ASP.NET Core's authorization middleware applies that fallback to **every
endpoint that carries no authorization metadata of its own** — gRPC endpoints included. The five mesh
gRPC endpoints (config-serve artifact fetch, central command ack, telemetry stream, node command, LocalDb
sync) authenticate with a per-cluster/per-node **PSK bearer inside a gRPC interceptor**, and that
interceptor runs *inside* the endpoint invocation — i.e. strictly *after* the authorization middleware.
Without an explicit `.AllowAnonymous()` opt-out, a correctly-keyed node call is challenged by the cookie
handler before the interceptor's verdict ever matters: on a login-enabled admin node this is a fleet-wide
deploy-plane outage (artifact fetches fail, deploy acks are dropped, deployments seal `TimedOut`), and it
went unnoticed for seven phases of prior live gates because the docker-dev rig always ran
`Security__Auth__DisableLogin=true` (an always-succeeding handler registered *under the cookie scheme
name*, so the fallback trivially passes) and every prior mesh-gRPC test built its own auth-free pipeline
by hand.

**Why `AllowAnonymous` is the correct fix, not a hole.** All five interceptors are fail-closed: an unset
key rejects every call, and the presented token is compared with `CryptographicOperations.FixedTimeEquals`.
`Program.cs:574` states the design contract directly — "the interceptor is the ONLY inbound auth on any
endpoint (the libraries verify nothing)" — which was already true of the gRPC pipeline; the missing opt-out
made it false of the HTTP pipeline in front of it. The pattern itself predates this fix: static assets
(`MapStaticAssets().AllowAnonymous()`), the metrics endpoint (`MapZbMetrics().AllowAnonymous()`), and the
headless deploy API above are all `AllowAnonymous` for the identical reason — each is gated by something
other than the cookie, and the fallback would otherwise 401/redirect it regardless.

| Endpoint | Role gate | Affected without `AllowAnonymous`, on a login-enabled node |
|---|---|---|
| `DeploymentArtifactService.Fetch` | `hasAdmin && ConfigServe:GrpcListenPort > 0` | Always — every `FetchAndCache` node's artifact fetch. |
| `CentralCommandService.ReportApply` | same | Always — every node's deploy ack, from every cluster. |
| `TelemetryStreamService.Subscribe` | `hasDriver && Telemetry:GrpcListenPort > 0` | Only on a **fused** `admin,driver` node (e.g. docker-dev's central pair) — a driver-only node never registers the cookie stack at all, so it is structurally unaffected either way. |
| `NodeCommandService.Execute` | same | Only on a fused node, same reasoning. |
| `localdb_sync.v1.LocalDbSync/Sync` | `hasDriver && LocalDb:SyncListenPort > 0` | Only on a fused node that also sets a sync port — no node in the fleet currently has that shape, but the map is anonymised anyway. |

**A fused central pair is affected on its OWN dispatch, not only inbound from sites.** `CentralCommunicationActor` dials every enabled `ClusterNode` row, its own pair included, and central's telemetry dialer dials its own pair too — so a login-enabled fused central breaks its own command/telemetry plane, not just the sites'.

**Mapping mechanics.** All five maps live in one place, `MeshGrpcEndpoints.MapOtOpcUaMeshGrpcServices`
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshGrpcEndpoints.cs`) — `Program.cs` calls it once and
carries no map sites of its own. Four maps chain `.AllowAnonymous()` directly onto `MapGrpcService<T>()`.
The fifth, `MapZbLocalDbSync()`, returns `IEndpointRouteBuilder` rather than an
`IEndpointConventionBuilder`, so it cannot take the same chain; it is wrapped in an empty-prefix
`app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync()`, which carries the convention without
changing the wire route pattern.

**Three standing guards**, all in `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/`, so a future map
cannot silently reopen this:

- `MeshGrpcAnonymousMetadataTests` — an endpoint-metadata census that calls the SAME
  `MapOtOpcUaMeshGrpcServices` production helper across all three role shapes (admin-only, driver-only,
  fused) and asserts every mesh endpoint carries `IAllowAnonymous`, with an exact expected count per shape
  and a scope check that the opt-out never leaks onto an unrelated endpoint.
- `MeshGrpcMapConventionTests` — a `Program.cs` source scan asserting zero direct `MapGrpcService</
  MapZbLocalDbSync(` occurrences and exactly one call to the mapper, so a map added straight into
  `Program.cs` (bypassing the helper, and therefore invisible to the census) fails `MeshGrpcMapConventionTests`.
- `MeshGrpcAuthStackTests` / `MeshGrpcDriverOnlyPipelineTests` drive the mesh endpoints through the REAL
  `AddOtOpcUaAuth` + `UseAuthentication`/`UseAuthorization` pipeline (`DisableLogin=false`) and the
  driver-only (no-auth-at-all) shape respectively — the CI-shaped reproduction of the outage, with a
  correctly-keyed call **succeeding** as the load-bearing assertion (a blocked call and a refused call both
  "fail"; only a genuine success through the real cookie stack proves the opt-out works) and an
  un-anonymised negative control proving the fixture's auth stack is actually live.

**Rig:** `docker-dev/docker-compose.yml`'s central-1/central-2 `Security__Auth__DisableLogin` is
`${OTOPCUA_DISABLE_LOGIN:-true}` — default unchanged, but `OTOPCUA_DISABLE_LOGIN=false docker compose
... up -d central-1 central-2` drives the rig through the real cookie login (`alice`/`bob`, password
`password`, against the shared GLAuth) for exactly this kind of verification, instead of every prior mesh
live gate running with the one middleware that breaks production neutralized.

See `archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md` (R3-01) for the full verification and design
record, and `docs/Configuration.md` §`MeshTransport` / §`ConfigSource`-`ConfigServe` and `docs/Telemetry.md`
for the affected listener ports.

### The known-cluster gate is fail-OPEN until central's first ClusterNode load (review 03 §4.2)

`CentralCommandAuthInterceptor` verifies an inbound deploy ack against the PSK of the cluster named in
`x-otopcua-cluster`. `MeshPskProvider` consults a **known-cluster gate** first — the set of cluster ids
central read from `ClusterNode` on its last contact refresh — and refuses anything outside it in O(1),
touching neither the config map nor the secret store. That gate is **inactive until the first successful
load** (`CentralCommunicationActor` publishes it from `ContactsLoaded`; an empty publication also
deactivates it, deliberately, so "the DB told me nothing" never becomes a fleet-wide authentication
outage). During a central boot with the ConfigDb down, therefore:

- an unauthenticated caller can name *any* `x-otopcua-cluster` value and make central perform one bounded
  (3 s) secret-store read plus one Warning line per request — the cost/oracle the gate exists to remove;
- **keys are still required**: nothing is admitted without the matching PSK, so this is a cost exposure,
  not an authentication bypass;
- the window is bounded by the bootstrap retry (15 × 2 s) and the `ContactRefreshSeconds` (60 s) periodic
  refresh, and closes on the first load that succeeds. The actor logs the condition on every failed load
  (*"The known-cluster gate was NOT refreshed either … it is INACTIVE"*).

Since review 03 §3.4 the per-cluster keys are **refreshed asynchronously** from the same handler
(`IMeshPskProvider.RefreshAsync`: concurrent bounded reads, atomic snapshot swap, a failed read keeps the
previous key) instead of being invalidated on every refresh, so the first ack per cluster per minute no
longer pays a synchronous store read on a gRPC thread. Accepted as documented, bounded behaviour; the
alternative — refusing every cluster until SQL answers — would turn a ConfigDb outage into a deploy-ack
outage on top of it.

---

## OTOPCUA0001 Analyzer — Compile-Time Guard

Per-capability resilience (retry, timeout, circuit-breaker) is applied by `CapabilityInvoker` in `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/`. A driver-capability call made **outside** the invoker bypasses resilience entirely — which in production looks like inconsistent timeouts, un-wrapped retries, and unbounded blocking.

`OTOPCUA0001` (Roslyn analyzer at `src/Tooling/ZB.MOM.WW.OtOpcUa.Analyzers/UnwrappedCapabilityCallAnalyzer.cs`) fires with category `OtOpcUa.Resilience` and default severity **Warning** (per `AnalyzerReleases.Shipped.md`) when a method on one of the six guarded capability interfaces (`IReadable`, `IWritable`, `ITagDiscovery`, `ISubscribable`, `IHostConnectivityProbe`, `IAlarmSource` — all in `ZB.MOM.WW.OtOpcUa.Core.Abstractions`) is invoked **outside** a lambda passed to `CapabilityInvoker.ExecuteAsync` / `ExecuteWriteAsync`. `AlarmSurfaceInvoker` is **not** a wrapper home — its own implementation is covered transitively because it routes through the inner `CapabilityInvoker.ExecuteAsync`. The analyzer walks up the syntax tree from the call site, finds any enclosing invoker invocation, and verifies the call lives transitively inside that invocation's anonymous-function argument — a sibling pattern (do the call, then invoke `ExecuteAsync` on something unrelated nearby) does not satisfy the rule.

The xunit.v3 + Shouldly suite at `tests/Tooling/ZB.MOM.WW.OtOpcUa.Analyzers.Tests/UnwrappedCapabilityCallAnalyzerTests.cs` covers the common fail/pass shapes + the sibling-pattern regression guard.

The rule is intentionally scoped to async surfaces — pure in-memory accessors like `IHostConnectivityProbe.GetHostStatuses()` return synchronously and do not require the invoker wrap.

---

## Write-Outcome Self-Correction

When an OPC UA client writes a value and the driver reports a failure, the server automatically
reverts the variable node to its prior value and surfaces the Bad-quality status code to the client.
This prevents the node from showing a phantom Good value after a device-level write error. The
revert is local to the node that received the write — no cluster coordination is required.

Additionally, an `AuditWriteUpdateEvent` is emitted on every write attempt (success or failure),
consistent with OPC UA Part 4 audit requirements. The event carries the source node id, the
requested value, and the final status code so audit log consumers can trace write outcomes without
polling the node.

(Implemented in master `1d797c1c`.)

---

## Audit Logging

- **Server**: authentication, certificate-validation, write, and write-denial events are logged through the regular Serilog rolling file sink; write outcomes also emit an `AuditWriteUpdateEvent` (see [Write-Outcome Self-Correction](#write-outcome-self-correction)).
- **Admin**: `AuditWriterActor` (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Audit/AuditWriterActor.cs`) writes `ConfigAuditLog` rows (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ConfigAuditLog.cs`) to the Config DB for publish, rollback, cluster-node CRUD, and credential rotation. Visible on the cluster Audit page (`ClusterAudit.razor`) for operators with `Viewer` or above.

---

## Config secrets (`${secret:}` delivery)

OtOpcUa never commits secret values to `appsettings*.json`. Every real secret is
supplied at deploy time — either as a plain environment variable, or as a
`${secret:<name>}` token backed by the shared **`ZB.MOM.WW.Secrets`** encrypted store.

A pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
`OtOpcUa.Host/Program.cs`) walks the assembled configuration and rewrites every
`${secret:<name>}` token into its resolved plaintext **before** the host is built —
so every downstream binder/validator (`AddZbSerilog`, `AddOtOpcUaConfigDb`, the first
`ValidateOnStart`) sees resolved values. The store's key-encryption key (KEK) comes from
the `ZB_SECRETS_MASTER_KEY` environment variable (`Secrets:MasterKey:Source=Environment`);
the encrypted SQLite store lives at `Secrets:SqlitePath`.

The expander is **fail-closed and section-agnostic**: a `${secret:<name>}` token whose
secret is absent throws `SecretNotFoundException` at startup, regardless of which feature
owns the key or whether that feature is enabled. Keys prefixed with `_` (comment keys) are
skipped, so a `${secret:...}` example inside a `_…Comment` value is never resolved.

The five config secrets and their canonical secret names:

| Config key | Owner | Secret name |
|---|---|---|
| `Security:Jwt:SigningKey` | `JwtOptions` | `otopcua/jwt/signing-key` |
| `Security:Ldap:ServiceAccountPassword` | `LdapOptions` | `otopcua/ldap/service-account-password` |
| `Security:DeployApiKey` | `DeployApiEndpoints` | `otopcua/deploy/api-key` |
| `ConnectionStrings:ConfigDb` | `AddOtOpcUaConfigDb` | `otopcua/sql/configdb-connstr` |
| `ServerHistorian:ApiKey` | `ServerHistorianOptions` | `otopcua/historian/api-key` |

**Delivery.** By default these are delivered as plain environment variables (e.g.
`ServerHistorian__ApiKey=histgw_…`), never committed. To deliver one from the encrypted
store instead, seed it once with the `secret` CLI (from `ZB.MOM.WW.Secrets`), then supply
the token in place of the literal — via env var or a deployment appsettings overlay:

```
secret set otopcua/historian/api-key <value>       # seed the encrypted store first
ServerHistorian__ApiKey='${secret:otopcua/historian/api-key}'
```

Only switch a value to a `${secret:}` token **after** the secret is seeded — an unseeded
token fails the boot fail-closed. Do not commit the KEK (`ZB_SECRETS_MASTER_KEY`) or any
secret value.

For the production posture needed so every clustered node (admin, driver, and any
fused role) resolves the same secrets from the same store, see
[`docs/operations/2026-07-16-secrets-clustered-master-key.md`](operations/2026-07-16-secrets-clustered-master-key.md).

---

## Troubleshooting

### Certificate trust failure

Check `{PkiStoreRoot}/rejected/` for the client's cert. As a FleetAdmin, use the Admin UI Certificates page's **Trust** action to move it into `trusted/` (or copy the `.der` to `trusted/certs/` by hand); the SDK trust list reloads on the next handshake.

### LDAP users can connect but fail authorization

Verify (a) `Security:Ldap:GroupAttribute` (default `memberOf`) returns the user's groups, (b) `Security:Ldap:GroupToRole` maps those groups to the expected roles, and (c) a `NodeAcl` grant exists at some level of the node's scope path that unions to the required permission. The data-plane evaluator is strictly default-deny — there is no fail-open mode to fall back on.

### LDAP bind rejected as "insecure"

Set `Security:Ldap:Transport = "Ldaps"` (or `"StartTls"`) with the matching port (636 for AD `Ldaps`), or temporarily set `Security:Ldap:AllowInsecure = true` in dev. Production Active Directory increasingly refuses plain-LDAP bind under LDAP-signing enforcement.

### Stale ACL trie after a publish

A publish installs a **new generation** into `PermissionTrieCache` via `PermissionTrieBuilder` rather than signalling an invalidation; the evaluator binds each session to a generation. If grants appear stale, confirm the new generation was installed (publish completed) and that sessions re-resolved their auth state — a session past its staleness ceiling fails closed and must re-authenticate. As a last resort `PermissionTrieCache.Invalidate(clusterId)` drops a cluster's cached tries.