From 8c6067b62e1023d5a6bc011b6ebda6429a4e9744 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 15 Oct 2026 06:14:58 -0400 Subject: [PATCH] docs: move subsystem gotchas out of CLAUDE.md into path-scoped .claude/rules Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology, data-plane auth, alarms, historian) only when matching files are touched. --- .claude/rules/alarms.md | 56 ++ .claude/rules/cluster-topology.md | 423 ++++++++++ .claude/rules/data-plane-auth.md | 218 +++++ .claude/rules/drivers.md | 307 +++++++ .claude/rules/historian.md | 275 +++++++ .gitignore | 3 +- CLAUDE.md | 1242 +---------------------------- 7 files changed, 1299 insertions(+), 1225 deletions(-) create mode 100644 .claude/rules/alarms.md create mode 100644 .claude/rules/cluster-topology.md create mode 100644 .claude/rules/data-plane-auth.md create mode 100644 .claude/rules/drivers.md create mode 100644 .claude/rules/historian.md diff --git a/.claude/rules/alarms.md b/.claude/rules/alarms.md new file mode 100644 index 00000000..8c770d7b --- /dev/null +++ b/.claude/rules/alarms.md @@ -0,0 +1,56 @@ +--- +paths: + - "src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/**" + - "docs/ScriptedAlarms.md" + - "docs/AlarmTracking.md" +--- + +# Scripted alarm ack/shelve and Part 9 condition method behaviour + +> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area. +> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file. + +## Scripted Alarm Ack/Shelve + +Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` role — conferred by the session's API-key principal's `NodeAcl` grants (or the `anonymous` principal's), carried by `RoleCarryingUserIdentity`; LDAP is AdminUI-only; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`. + +⚠️ **One condition event per accepted method call is OURS, not the SDK's (Gitea #587).** The SDK's own +report is gated on `AutoReportStateChanges` (left `false`), so before #587 a client Acknowledge emitted **no** +condition event — the old "SDK auto-fires its own event (E2)" comments were wrong. `WireMethodTransitionReporting` +wraps every method's SDK handler (Ack/Confirm/AddComment/Enable/Disable/OneShotShelve/TimedShelve/Unshelve) and, +on Good, reports once via `ReportConditionEvent` — **under `Lock`**, so the engine's/driver's re-projection +(routed by a fire-and-forget Tell *before* the SDK applies state) waits and then hits the delta-gate with no +delta. Things to keep: the SDK's shelve handlers never apply shelving, so the wrapper does; only a real +before/after change is reported (a re-ack is silent and restores the Confirmed/Retain the SDK flips); +`MaxTimeShelved` is set to `int.MaxValue` ms at materialise because `Create()` left it 0 and every TimedShelve +failed. ⚠️ **Native alarms support Acknowledge and AddComment only** — Confirm, the shelve methods and +Enable/Disable return `BadNotSupported` (no upstream path, no pair replication, and `NativeAlarmProjector`'s +Confirmed/Unshelved constants own those fields); a native ack re-asserts `ConfirmedState = true` because the +SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by +`ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`). + +Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a +**no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real +remaining time from `AlarmConditionSnapshot.UnshelveAtUtc`); a **commented** re-ack is stored as an +`AddComment` (durable, replicated) and reports one event, an uncommented re-ack stays silent; a **native +ack is refused `BadInvalidState` off-Primary** via `NativeAlarmAckPrimaryGate` (fed by +`IDriverPrimaryRoleSource`; `DriverHostActor` publishes its Primary verdict at host start and on member +changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT +Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design). + +**Condition `Severity` is carried verbatim, clamped to 1..1000** — `ToPart9Severity`; the node manager used to re-bucket onto +the five `EventSeverity` points (so an MTConnect Fault read 900, scripted Critical 1000 read 900, native Low 200 read 300). +ScadaBridge only clamps, so exact values pass through. ⚠️ **Never select a two-state id with `AddSelectClause(type, "X/Id")`** — +the string becomes ONE QualifiedName named `X/Id`, which no event carries; the Client.CLI/Client.UI filter did this since +`a2883b82` and every alarm read inactive+unacked. Use `new SimpleAttributeOperand(type, [X, Id])` (ScadaBridge already does). + +**Condition children are instance-scoped (#629).** Every child of a materialised condition carries +`{ConditionId}/{browse path}` in the condition's namespace (e.g. `…/temp_hi/ShelvingState/UnshelveTime`), assigned by +the `OtOpcUaNodeManager.New` override — the SDK default kept each child on the TYPE's ns=0 id, so a direct Read hit +the type (or `BadNodeIdUnknown`), translate failed, and children sat outside the ACL (the slash join now puts them +under their condition's scope). ⚠️ The shelve methods resolve by the `AlarmConditionType_ShelvingState_*` ids only +because `New` adopts those (a fixed set) as their `MethodDeclarationId`; calls by the `ShelvedStateMachineType_*` id +are no longer supported (pinned by a test). Condition teardown must forget the whole `PredefinedNodes` subtree +(`ForgetPredefinedSubtree`) or a removed condition's children keep answering. diff --git a/.claude/rules/cluster-topology.md b/.claude/rules/cluster-topology.md new file mode 100644 index 00000000..46d2e9cb --- /dev/null +++ b/.claude/rules/cluster-topology.md @@ -0,0 +1,423 @@ +--- +paths: + - "src/Core/ZB.MOM.WW.OtOpcUa.Cluster/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Host/**" + - "docker-dev/**" + - "docs/Redundancy.md" + - "docs/Configuration.md" + - "docs/Telemetry.md" +--- + +# Cluster topology: redundancy, role authority, per-cluster meshes, MeshTransport, ConfigSource/ConfigServe, telemetry transport, LocalDb, DB-less driver nodes + +> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area. +> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file. + +## Redundancy + +The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide. + +**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.** + +- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is CLOSED** — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (`c50ebcf7`; `docs/plans/2026-07-24-mesh-phase7-failover-drills.md`). +- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**. + +**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". + +**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard". + +Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. + +**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below. + +**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences: + +- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`. +- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh. +- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it. +- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists. +- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip. +- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4). + +See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. + +## Role authority (`EffectiveRoles`) + +**A node's roles are resolved in exactly one place** (archreview R3-04, closing 01/S-12 = 03/C5 and +01/C-9): `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`), with one precedence — +`Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`, else empty — and two-layer normalization: a +plain role (`admin`/`driver`/`dev`) is trimmed and lower-cased, but a cluster role's ClusterId suffix +(`cluster-SITE-A`) is kept verbatim, because lower-casing it would rename the Akka role of a running +deployment. An unknown role from **either** source fails host start. `Program.cs` resolves one +`EffectiveRoles` instance at the top of the composition root and registers it as a singleton; that +SAME instance drives both the conditional host wiring (`hasAdmin`/`hasDriver`) and +`AkkaClusterOptions.Roles` (what Akka gossips) — there is no second derivation for the two to +disagree about, which is why no runtime "do the views agree" assert exists. Before this, `Program.cs` +read `OTOPCUA_ROLES` only, so a node configured solely through `Cluster:Roles` gossiped the right +Akka roles while `Program.cs`'s own wiring mounted none of them — the inverse of the pre-#532 field +incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`). + +- **A role overlay may not itself set `Cluster:Roles`.** Roles are resolved *before* the per-role + appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing + `Cluster:Roles` would leave the bound options disagreeing with the registered instance. + `AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views — a no-op + today since none of the four shipped overlays carries a `Cluster` section. +- **Both sources set to different normalized role sets logs a Warning** at boot (`Cluster:Roles` + still wins outright) — the boot banner also names which source won (`source=Configured` / + `Environment` / `None`). +- **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of + production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through + `EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role + during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:** + `RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, + which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the + library nor with a local case-insensitive pre-filter, which would split the Primary election from + `ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade + window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)"). +- **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to + stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s + — leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException` + (design-time tooling) passes through untouched. +- **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement: + a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads + `OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal — + with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same + English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI + RBAC role). + +See `docs/Configuration.md` §`Roles`, `docs/ServiceHosting.md` §"Role gating", and +`archreview/plans/R3-04-one-role-authority-plan.md`. + +## Mesh command transport (`MeshTransport`) + +**The Akka `ClusterClient` transport described in this section's earlier history is DELETED** +(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md), +Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension, +`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails +validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values) +rather than silently falling back to `Dps`. What remains is exactly two transports, selected by +`MeshTransport:Mode`: + +- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**: + it works only when central and the node share a gossip ring, which after Phase 6 means within one + 2-node mesh. +- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node + carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a + node on anything else. + +Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`, +served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev +rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port +of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is +every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort` +and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and +no rotation: each command goes to the specific node dialled, which is what makes per-node delivery +provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer +applies to either direction. + +Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role +config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than +claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and +stops at the first success. + +Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per +application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets +store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed +on every unresolvable case — constant-time compare, no key material logged. Production keys are +`${secret:OT-MESH-PSK-}`; the docker-dev rig supplies three literal dev keys via env +(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`). + +Things that carried over unchanged from the `ClusterClient` era, because they were never +`ClusterClient`-specific to begin with: + +- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node + and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across + nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery. + The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that + set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't. +- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a + DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`, + `ScriptedAlarmHostActor`) accept a command from either transport identically. +- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a + deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()` + replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient + deletion — it was always about the deploy path, not the transport. + +`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which +makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` +§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) / +[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) / +[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history — +those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md` +is the original Phase 2 design doc and is likewise left as a historical record of the transport this +section now describes as deleted. + +**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: + +- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag. +- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out. +- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist. +- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again. + +## Config source (`ConfigSource` / `ConfigServe`) + +Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its +deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from +central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact +bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in +LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via +`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by +a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`, +fail-closed). Things worth knowing before touching it: + +- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an + admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same + dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport). +- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision + hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the + `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment` + stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch. + **The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** — + `ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and + hashes the result, so the invariant holds unchanged and nothing between the composer and + `DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever + looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so + pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every + decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already + handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after + upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be + upgraded BEFORE central starts emitting compressed blobs.** +- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA + mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, + and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a + `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an + empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**. + **Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and + the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does + **not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches). +- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading + SQL today; `StoreAsync` is idempotent and the cache replicates). +- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with + both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only + ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel + `Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about). +- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState` + ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those + is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts. + +On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays +hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching +key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is +no `OTOPCUA_CONFIG_MODE` env var in the compose file. See +`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and +`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`. + +## Live telemetry transport (`Telemetry` / `TelemetryDial`) + +Per-cluster mesh **Phase 5** (merged `35552a96`; live gate PASSED `f134935f`) added a second +transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and +`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode` +(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's +behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof` +event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts +the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring +ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split +(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before +touching it: + +- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`), + `driver-health` (`DriverHealthChanged`), `driver-resilience-status` + (`DriverResilienceStatusChanged`) — feeding the exact same central sinks + (`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in + either mode. +- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the + delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from + `Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an + observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport + (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not + telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542): + `FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end** + (no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its + `DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just + central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above. +- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits + into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC + server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only + `TelemetryDial:Mode` on central actually switches anything: which of the two always-available + sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what + makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as + [MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve). +- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` == + `TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor` + (`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects + every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture + for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern. +- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from + `ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate + first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event + is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side + stores are not evicted). The node-local hub + replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe + so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and + tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1 + node stream is up) — the same semantics the DPS pill has always had. + +See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md` +§`Telemetry`/`TelemetryDial` for the config keys, and +`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live +gate procedure. + +## LocalDb pair-local store (Phases 1 + 2) + +Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired +LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**: +`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2). + +**Phase 1** caches the +**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a +driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes +the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`, +in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by +`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` + +`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes +never register it.** The cache can optionally replicate to the node's redundant pair peer over the +library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated +on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the +docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is +the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret +exception). + +**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db` +and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm +history no longer takes it to the grave. `SqliteStoreAndForwardSink` became +`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key +`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely +to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy +commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the +sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that +`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new +`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by +design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the +same event accepted on both nodes — which happens in every boot window, since +`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the +docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a +deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue. + +**The library pin is `0.3.4`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and +`.Replication`), bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15). +The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3 +upgrades in place on open, so no rig or volume action was needed. `RegisterReplicated(string)` is +unchanged and everything 0.2.x/0.3.x added is additive, **except** that `RegisterReplicated` (0.3.3) +now throws `LocalDbRegistrationException` for a table that is the parent of a row-mutating ON +DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire. +0.3.0 made the snapshot-resync flag **per-table** (see the ⚠️ below, now corrected); 0.3.2 added +validated config `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, left +unset here) and a public `LocalDbOplogGapException`, neither yet wired to a call site. **Its two +new APIs — `DeregisterReplicated` and `RegisterReplicated(table, baselineExistingRows:)` — are now +both used** (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two +things are worth knowing: + +- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching + **byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized + under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row + cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which + materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps + `16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the + message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" + was corrected with the bump; do not restore that rationale. +- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).** + `LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges` + (via `LocalDbSetup.ReplicationIsConfigured`): + **`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is + not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it + alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` + + `LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only, + silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is + fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has + nothing to capture for. + - ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`: + `ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these + rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no + peer must still drain, so the two predicates are deliberately not the same. + The **passive** half (no `PeerAddress`) cannot name its peer, so it stands down for its **mesh partner** + (the other member of its `cluster-{ClusterId}` role); a legacy passive half with no mesh role still + drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the + published Primary verdict (and the live data-plane gate, which must stay the same decision) **allows** — + for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the + Sparkplug primary-host claim with its Primary peer. + - **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the + same load-bearing DDL→register order. Baselining is what makes enabling replication on a node + that has been running *without* it actually converge: capture is CDC and the snapshot streamer + pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in + used to replicate only writes made after the restart. It seeds the ledger at the LWW floor + (HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote + write of the same key and wins only where the peer has no version at all. It is idempotent, so + it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.4`) the snapshot + flag is **per-table**, not per-database — baselining one table no longer streams every + registered table in both directions. (Prior to `0.3.0`, on the `0.2.1` pin this repo carried + until 2026-09-03, the flag was per-database and baselining anything streamed *every* + registered table in full; the old advice to trim `alarm_sf_events` before enabling replication + no longer applies at the current pin.) + - **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time + self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and + prunes that table's oplog + row-version rows. Merely declining to register would leave every + pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row + inside every write transaction — on precisely the node that has no peer to capture for. The + call returns `true` when it removed something, so the Information line fires **once, and only + when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is + Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where + resolving one from DI would nest a resolution inside a singleton construction; `Program.cs` + assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe + the line rather than infer it. + - ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake + compares registered-table digests **fail-closed**, so a node that deregisters a table its peer + still replicates stops syncing with a schema mismatch. Enabling replication on an established + pair means setting the config and **restarting both nodes** — do it on both, or on neither. + - **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was + enabled from a fresh pair, so the impact of the old gap was low — central and site-b are + default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests` + pins both branches (registered-table set on a configured node, empty set + zero triggers + + the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged. + +See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config +cache), `docs/AlarmHistorian.md`, and the runbook +`docs/operations/2026-07-20-localdb-pair-replication.md`. + +## Phase 4 — cut the driver-side ConfigDb connection + +Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles` +has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs` +gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is +unaffected (it keeps ConfigDb via its admin role). Consequences: + +- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator` + fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused + node may stay `Direct`. +- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not + spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db + factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and + `Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes + **240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture + — where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself + is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less + (driver-only) nodes". +- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore` + (replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces + `EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed + `ScriptedAlarmState` table **was dropped** (Task 9, `3a590a0c`, migration + `20260723183050_DropScriptedAlarmStateTable`); `EfAlarmConditionStateStore` is deleted. +- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on + a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is + the system of record. +- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty + `NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so + `OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only — + the same rows a driver node never received via config either, so this is not a regression. + +Merged to master (`26fad75c`); **live gate PASSED** (`1281aebf` — driver nodes run ConfigDb-free) and +Task 9 landed (`3a590a0c`). See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and +`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4. diff --git a/.claude/rules/data-plane-auth.md b/.claude/rules/data-plane-auth.md new file mode 100644 index 00000000..44492c76 --- /dev/null +++ b/.claude/rules/data-plane-auth.md @@ -0,0 +1,218 @@ +--- +paths: + - "src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/**/*Acl*" + - "docs/security.md" +--- + +# Data-plane authentication (API keys), node ACL enforcement, anonymous principal, AdminUI LDAP + +> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area. +> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file. + +## Authentication: API keys on the data plane, LDAP on AdminUI (#520) + +⚠️ **Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all.** It is machine-to-machine, +so a client presents an OPC UA **UserName** token with the fixed username **`apikey`** and an +`otua__` credential as the password. `ApiKeyOpcUaUserAuthenticator` +(`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/`) implements `IOpcUaUserAuthenticator`: one indexed lookup +on `keyId` plus one peppered HMAC-SHA256, `FixedTimeEquals`, **no network I/O**. The key resolves to a +**principal**; `NodeAcl.Principal` rows grant it permissions. `LdapOpcUaUserAuthenticator` is +**deleted**, along with four `Security:Ldap` keys that existed only for it (`AuthTimeoutMs`, +`MaxConcurrentAuthentications`, `OutageFailureThreshold`, `OutageCooldownSeconds`) — they bounded an +LDAP bind running inline in the SDK's synchronous, `SessionManager`-lock-holding impersonation +callback, and nothing on the remaining LDAP path has that property. + +**LDAP remains, for AdminUI login only** (`Security:Ldap`, `OtOpcUaLdapAuthService` → +`ILdapAuthService`, group→role mapping via `GroupToRole` + `LdapGroupRoleMapping`). Do not "restore" +the data-plane authenticator when you see LDAP wiring in `Program.cs` — it is there for the AdminUI. + +Two things to know before relying on any of this: + +- ⚠️ **Node ACLs are enforced only behind `Security:Acl:Enabled` (default `false`).** With the flag + **off**, a key's grants collapse into the two server-wide role gates (`WriteOperate` on Value writes, + `AlarmAck` on Part 9 methods), **only `Cluster`-scoped grants count** — `ApiKeyInterimRoleMapper` + deliberately under-grants, because honouring a narrower grant's flags while unable to honour its + *scope* would turn "WriteOperate on Equipment X" into server-wide write authority — and **read, + browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous**. With the + flag **on** (`DataPlaneAuthorizer`), scoped grants decide **every** data-plane operation: write and + alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6). + - **The two role gates re-check the session's KEY on every call, flag off or on (Gitea #563).** Roles + are resolved once at activation, so an expired / deployment-disabled key used to keep `WriteOperate` / + `AlarmAck` on its open session until the client disconnected. `DataPlaneAuthorizer.AuthorizeKeyUsable` + runs the same `FindUsableKey` probe against the same snapshot + clock as the ACL path (one dictionary + lookup, no I/O) and denies `BadUserAccessDenied` before the role check. **Only those two gates** — + reads/subscriptions stay ungated with the flag off, by design. Anonymous carries no key and is + unaffected. No flag: a dead key keeping write authority is a defect, not a rollout choice. ⚠️ On a + fused node a key minted after the last deploy authenticates (ConfigDb) but cannot write/ack until a + deployment carries it — the flag-on path always behaved this way. + - ⚠️ **The ACL decision REPLACES the role check — the two are never chained.** Chained, a grant would + have to confer the server-wide role *as well as* pass the scope check, and only `Cluster`-scoped + grants confer a role, so every narrow grant would stay inert and the flag would appear to do + nothing. The load-bearing test is therefore an **allow** — a session carrying *no roles* writing + successfully on the strength of its grant. A deny-only test passes against a build where the ACL is + never consulted, because the role gate denies too. + - **The trie is built inside `DataPlaneAuthorityState.Apply`**, not by a second wiring at the publish + actor, and the state owns its `PermissionTrieCache` rather than being handed one. An uninstalled + trie denies everything, which is indistinguishable from working enforcement until someone checks + that an ALLOW is possible at all. + - **A deploy reaches sessions that are already open**, in both directions: each decision stamps the + *current* generation rather than the one the session activated under, and re-checks the + authenticating key — so revoking a key or a grant stops an open session immediately instead of at + the client's next reconnect. On a driver-only node the artifact is the only revocation channel. + - `UserAuthorizationState` is built **per decision walk, not per session** — and since the hot-path + change it is allocated only on a cache **miss**, so a cache hit never constructs one. Its LDAP-shaped + freshness/staleness predicates are switched off EXPLICITLY (`AuthCacheMaxStaleness = MaxValue`). + ⚠️ Relying on "the stamp is always fresh" was not safe: `MembershipResolvedUtc` comes from + `DataPlaneAuthorizer`'s `TimeProvider` while `IsStale` is evaluated against + `TriePermissionEvaluator`'s, so **any divergence between the two clocks denies every request** with a + verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead. + - **Stage 6's hot seams (read/browse/subscribe) go through a per-`(session, node, operation, realm)` + decision cache**, cleared wholesale when the generation advances. It is **sharded per API key** + (`ApiKeyId ?? Principal` — despite the type name, sessions sharing one key share one shard and one + budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and + not every other key's decisions. ⚠️ It caches the **grant walk only** — key revocation and expiry are + re-checked **before** the cache lookup on every call, because expiry moves with the clock rather than + with a deployment. + - **Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors.** A + grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward + only, so `PermissionTrieNode.BrowseTransitPrincipals` marks the path. Planting a real `Browse` grant + on each ancestor instead would be collected by the additive downward walk and leak `Browse` onto + every sibling subtree beneath that ancestor. ⚠️ **The topmost ancestor is the single shared root + folder** (`s=OtOpcUa`, raw namespace) that BOTH realm trees hang under — its identifier is a display + name, not a path, so `DataPlaneAuthorizer.BuildScope` maps it onto the trie's **zero-segment + realm-root scope** (`V3NodeIds.SharedRootFolderId`, realm-qualified: in the UNS namespace that + identifier is an ordinary Area), and its Browse considers transit in **either** realm because it + fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse + gate denies the whole call when its source node is denied, so clients were stopped at the front door. + - **Revocation reaches live subscriptions** (stage 6): every other operation re-asks per request, but a + subscription is authorized once and then streams — and streaming is how machine consumers actually + read this server. `DataPlaneAuthorityState.Changed` fires `RevalidateMonitoredItems`. ⚠️ A denied + item is queued `BadUserAccessDenied` and then **detached from its `MonitoredNode2` fan-out — NOT set + `MonitoringMode.Disabled`**: disabling discards the item's queued notifications, so the client is + silenced without ever being told why (caught only by the over-the-wire test). Detaching is also + reversible, so restoring the grant resumes the feed with no client action. ⚠️ **`Changed` is no longer + the only trigger (arch-review 03/S19):** `AclRevalidationTimer`, armed by `OtOpcUaServerHostedService` + at `Security:Acl:RevalidationIntervalSeconds` (default 60 s, `0` disables, inert while + `Security:Acl:Enabled` is false), runs the same pass on the clock — because `ExpiresUtc` passes with + the clock, not with a deployment. The pass's verdict memo is **pass-local and must stay that way**; + hoisting it across ticks would serve an expiry that landed between two ticks the earlier tick's allow. + **Event/alarm items ride both triggers too (Gitea #562)** — a second map (`_eventItems`), fed by the + `SubscribeToEvents` override, holds every *direct* notifier subscription; a denied one is detached via + the SDK's own protected `SubscribeToEvents(…, unsubscribe: true)` (what a client delete does) and + re-attached with `unsubscribe: false` on restore — again never `Disabled`. Unlike a data item it is + **not told** (no value slot for a status; the feed goes quiet), and a client delete of a stood-down + item is absorbed by the override (the SDK would otherwise remove an item that is not attached). + **Server-object (`i=2253`) event subscriptions are filtered per event at delivery (Gitea #586).** The + `SubscribeToAllEvents` override attaches them to **nothing** (just records them in `_allEventsItems`); + each condition's own `OnReportEvent` (`OnConditionReportEvent`, fires once per event ⇒ still one + Server-object copy) queues the event to each recorded item whose session holds `Subscribe` on the + condition (RawPath) **or** any notifier folder it reports through — the per-folder path's question. The + root-notifier forward to the core Server object (`OnReportEvent` override) is **dropped**: it was the + ungated path, and it is the only one that served an item created before the alarm was deployed (the + SDK's `AddRootNotifier` subscribes existing all-events items with `unsubscribe: true`). Revalidation is + **implicit** — every event re-asks the authorizer, so a deploy or expiry lands on the next event with no + pass. `ConditionRefresh` on an `i=2253` item is filtered the same way. The dispatcher **must** keep + the SDK's audit guard (`AuditEventState` only while `Server.Auditing` over SignAndEncrypt/https), + because conditions report their Part 9 method audit events through themselves. + `ReconcileAllEventsMonitoring` must keep `AreEventsMonitored` raised on every condition while any + `i=2253` item exists, because `ConditionState` gates its method **audit** reports on + `EventsMonitored()` (method-driven state-change events no longer depend on it — #587, below). The + protected `SubscribeToEvents` override that no-ops all-events items is what + stops the SDK's `AddRootNotifier` (`unsubscribe: true` bug) taking that unit back. Both review + findings (review of `75e559f6`). ⚠️ The materialise path raises only the **new** condition's unit + (`RaiseAllEventsMonitoring`); a full reconcile there made a deploy of N alarms O(N²) under `Lock` with + ScadaBridge connected — keep the full reconcile to remove/rebuild/(un)subscribe. + **Model-change and audit events are gated too (Gitea #588).** The manager's own server-wide events + (`GeneralModelChangeEvent`, the rejected-write `AuditWriteUpdateEvent`) go through + `ReportToServerObject`: flag **off** ⇒ `Server.ReportEvent` as before; flag **on** ⇒ delivered per item to + `_allEventsItems` (same audience), a model change narrowed to the `Affected` nodes the session may + **Browse** (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral + one), an audit write only with `Read` on its source node plus the SDK's audit/encryption guard. Any other + event type fails closed. Auditing is off by default (never configured in this repo). +- **ns=0 standard nodes stay ungated by design, even with `Security:Acl:Enabled`** (#617) — `ServerStatus`, + `NamespaceArray`, `ServiceLevel` etc. are served by the SDK's core node manager outside our authorizer and + clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node, + event subscription and HistoryRead is still gated. `docs/security.md`. +- **Equipment-node writes: `EvaluateEquipmentWriteStructure` is the ONLY type check.** `OnWriteValue` bypasses + the SDK's own type check on every writable equipment node, so `OtOpcUaNodeManager.EvaluateEquipmentWriteStructure` + (plus `EvaluateEquipmentWriteRange`) is the whole gate. Scalars are deliberately lenient (drivers + range-check; ScadaBridge sends Double to a Float node). **Array writes** (#610): a numeric array of another + element type is **converted, checked, element by element** to the node's element type (ScadaBridge writes + CLR `int[]`/`double[]` lists) — any out-of-range, fractional-to-integer or NaN/∞ element refuses the whole + write `BadTypeMismatch` with nothing sent, so a `UInt32` > `Int32.MaxValue` can never reach a Galaxy Int32 + attribute as reinterpreted bits; non-numeric mismatches are refused (`Byte[]` ≈ ByteString); wrong length ⇒ + `BadTypeMismatch`; an array to a **scalar** node is refused only on Modbus/S7 (`EnsureVariable`'s + `refuseArrayOnScalar` — pass-through drivers keep their browse-committed scalar-shaped array tags writable; + a new sink wrapper must forward that argument, and the reflection guard checks only that the call arrives); + `IndexRange` writes ⇒ `BadWriteNotSupported`, `ValueRank > 1` out of scope. An + array node is writable only when the driver can write the array (`DriverArrayWriteSupport`: Modbus bounded + by `maxRegistersPerWrite`/coil cap, S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types **with a + declared `arrayLength`** — an undeclared length stays read-only because a short whole-array write would + resize the Galaxy attribute); `arrayLength >= 1` is required, and the node's `Writable && (!IsArray || + ArrayWritable)` is projected onto the raw node and every UNS mirror. +- **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`, + preserving today's behaviour) controls *admission*; when admitted the session authenticates as the + reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as + the same `RoleCarryingUserIdentity` a key produces. Set it `false` to make keys mandatory. A key may + **not** use the `anonymous` principal, and a failure to read anonymous's grants **denies** rather + than admitting with no roles. +- **Keys reach a driver-only node inside the deployment artifact** (stage 4). `ConfigComposer` snapshots + the `ApiKey` rows — the peppered HMAC only, never a secret, and the pepper is node config that is + deliberately *not* in the artifact — and `DeploymentArtifact.ParseDataPlaneAuthority` decodes them + node-side alongside the cluster's `NodeAcl` rows. `OpcUaPublishActor` installs both on + `IDataPlaneAuthoritySource` on every apply, **before** its `plan.IsEmpty` early-return, so a deployment + that only revokes a key still lands. A node with a ConfigDb keeps `ConfigDbApiKeyStore` (per-activation + read, wrapped — see below); a driver-only node gets `ArtifactApiKeyStore`. ⚠️ **The asymmetry is + ACTIVATION-scoped, not session-scoped — this line used to say "immediate on a ConfigDb node" and that was + wrong.** Clearing `Enabled` in the ConfigDb stops the **next session activation** on an admin/fused node + (bounded by `Security:ApiKeys:StoreCacheSeconds`, default 5 s) and takes until the next deployment on a + driver-only one — but an **already-open** session re-checks its key against the in-memory *artifact + snapshot* (`DataPlaneAuthorizer.Evaluate` → `snapshot.FindUsableKey`), never against `IApiKeyStore`, so on + **both** node kinds an `Enabled` revocation reaches an open session only at the next deployment. + **Expiry is the exception**: `FindUsableKey` compares `ExpiresUtc` against the authorizer's clock on every + call, so it is immediate everywhere — and since arch-review 03/S19 it reaches live subscriptions too, on + the host's `Security:Acl:RevalidationIntervalSeconds` timer (default 60 s; the session is **silenced with + `BadUserAccessDenied`, not closed**, and recovers on re-activation). Direct event/alarm subscriptions + are revalidated on both triggers as well (#562); with the flag **off**, expiry and revocation reach the + two role gates (#563) but not reads or subscriptions. +- ⚠️ **The activation lookup is BOUNDED (arch-review 03/S18).** The SDK raises impersonation synchronously + under a `SessionManager`-wide lock, so a ConfigDb that is *slow* rather than *refusing* stalled **every** + session activation on the server, Anonymous and X.509 included. `BoundedApiKeyStore` wraps the ConfigDb + store — and only it, never `ArtifactApiKeyStore`, which is two in-memory dictionary probes — with + `Security:ApiKeys:StoreTimeoutMs` (2000; exceeding it **denies**, via the authenticator's existing + catch-all) plus a `StoreCacheSeconds` (5) read-through cache. Faults and timeouts are never cached, a + cached hit whose `ExpiresUtc` has passed is re-read rather than served, and a timeout logs a distinct + Warning + `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. Both knobs at `0` restore the old + behaviour exactly. +- ⚠️ **The auth generation is a node-local COUNTER driven by the `RevisionHash`, not derived from its + bits.** `PermissionTrieCache` treats the highest `GenerationId` as current, so a generation taken from + hash bytes would install a *lower* number about half the time and the cache would go on serving the + previous deployment's ACLs with nothing logged. Pinned by + `DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases`. +- Note `OtOpcUaServerHostedService` takes `IOpcUaUserAuthenticator` **and** `IDataPlaneAuthoritySource` + as **required** ctor parameters — the built-in DI container does not fall back to a default value for + an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a + node with neither role) or it does not boot. + +**Grants are authored through a scope picker, not free text (stage 7).** `/clusters/{id}/acls` opens +`AclScopePickerModal` over the cluster's raw + UNS trees, and selecting a node supplies `ScopeKind` +**and** `ScopeId` together — so `RawDevice` + a TagId, a cross-cluster scope, a `Cluster` grant carrying +an id, and the retired `FolderSegment` kind are all unauthorable by construction rather than merely +rejected later. ⚠️ **A `Cluster` row must produce a NULL `ScopeId`** — the deploy gate rejects one that +carries an id (`NodeAclScopeIdUnexpected`), so a picker that helpfully filled in the cluster id would +author rows that cannot be deployed. `IAclScopeService.CheckScopeAsync` mirrors +`DraftValidator.ValidateNodeAclScopes` rule for rule and **blocks the save** when a scope resolves to +nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches +the picker); the list page marks such rows `unresolved`. **UNS tags are deliberately not offered** — a +UNS signal is a `UnsTagReference`, and a `Tag` grant already covers the raw node *and* every UNS +projection of it. The picker is `AclScopeTree`, a separate component from `RawTree`, which already +carries two modes. + +**All eight stages are done** (`d4389f57` enforcement, stage 7 authoring after it), each live-gated on +docker-dev. The rationale for every decision above is in +`docs/plans/2026-07-31-apikey-acl-enforcement-design.md`. See `docs/security.md` for the full guide. + +Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.) diff --git a/.claude/rules/drivers.md b/.claude/rules/drivers.md new file mode 100644 index 00000000..bf69b55e --- /dev/null +++ b/.claude/rules/drivers.md @@ -0,0 +1,307 @@ +--- +paths: + - "src/Drivers/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/**" + - "src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/**" + - "tests/Drivers/**" + - "docs/DriverLifecycle.md" + - "docs/drivers/**" +--- + +# Drivers: rediscovery signal, comms-loss watchdog + quality contract, AB native transport, per-driver notes + +> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area. +> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file. + +### Change Detection + +`DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys. + +**The signal is consumed as an operator prompt, not as an address-space rebuild** (§8.2, 2026-07-27). +`DriverInstanceActor` subscribes in `PreStart` and unsubscribes in `PostStop` — the detach is +load-bearing, because the `IDriver` instance outlives the actor when the host respawns a child around +the same driver object. A raise is recorded and rides the driver-health snapshot +(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`) to the AdminUI `/hosts` page as a +**"re-browse" chip**. + +⚠️ **It is advisory: a Galaxy redeploy still does NOT change the served address space.** v3 authors raw +tags explicitly through the `/raw` browse-commit flow, so an operator must re-browse the device and +commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact +records. + +Two gotchas if you touch this: + +- `PublishHealthSnapshot` dedups on a health fingerprint. The rediscovery timestamp **must** stay in + that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the + operator. Pinned by `DriverInstanceActorRediscoverySignalTests`. +- The shared test `StubDriver.GetHealth()` returns `DateTime.UtcNow`, so its fingerprint differs on + every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests + use a stub with stable health for exactly this reason. + +**#507 is closed as superseded, and the injection path is deleted.** Its retained code read +`EquipmentNode.DriverInstanceId` and `EquipmentTags`, both *structurally empty* in v3 +(`AddressSpaceComposer.cs`, `DeploymentArtifact.cs`), so removing the guard would have changed a log +line and injected nothing. Gone with it: `HandleDiscoveredNodes`, `PartitionDiscoveredByDeviceHost`, +`ApplyDiscoveredPlansForDriver`, the redeploy re-inject tail, `DiscoveredNodeMapper`, +`AddressSpaceApplier.MaterialiseDiscoveredNodes`, the connect-time discovery loop +(`StartDiscovery`/`RediscoverTick`/`DiscoveredNodesReady`/`TriggerRediscovery`), and the 18 +`DiscoveryInjectionDormantV3` tests — v2 characterization tests asserting an equipment-rooted graft +(`EquipmentRootNodeId == "EQ-1"`) that v3 cannot produce. `ITagDiscovery` itself stays: the `/raw` +browse picker drives it through `Commons/Browsing/DiscoveryDriverBrowser`, which never went through +the actor. + +**`IRediscoverable` is the ONLY tag-change signal, and four drivers do not raise it.** A periodic +tag-set drift check briefly existed (shipped `e77c8a35`, **removed** by Gitea **#523**): every +`SupportsOnlineDiscovery` driver was re-browsed on a 5-minute timer and the result diffed against its +authored tags. It was taken back out because it generated recurring, **unconfigurable** (interval was a +constructor parameter with no appsettings key), **never-live-gated** outbound traffic to real +PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the +design treated it as an implementation detail. + +⚠️ **Consequence to know before you rely on rediscovery:** **AbCip and FOCAS browse a live backend but +implement no `IRediscoverable`, so they have no tag-change signal at all** — a PLC program re-download +or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT, +MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an +event-driven signal, or one that runs when something *else* already indicates change (a reconnect, a +deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`TagSetDriftDetector`, +`DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` / +`DiscoveryStreamIncludesAuthoredTags`. + +**`IHostConnectivityProbe` is consumed (Gitea #521, closed).** `DriverInstanceActor` subscribes +`OnHostStatusChanged`, pulls `GetHostStatuses()` fresh, and the host set rides `DriverHealthChanged` to the +AdminUI `/hosts` page — no SQL: the `DriverHostStatus` table is dropped, and so is +`DriverInstanceResilienceStatus` (#524; the live data goes over the `driver-resilience-status` telemetry +channel only). **Multi-host dispatch is wired too:** `DriverInstanceActor` resolves the breaker/host key +per call via `IPerCallHostResolver` (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip). + +### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed) + +Until #527, **`DriverInstanceActor.DisconnectObserved` had no production sender.** The only paths into +`Reconnecting` were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected +and then lost its device stayed `Connected` indefinitely — publishing Bad for every tag, never +re-initialising, and **not recovering when the device returned** (live-confirmed: a Modbus driver against a +stopped fixture needed a node restart). `PublishHealthSnapshot` read `GetHealth()` for *reporting only*. + +The `Connected` health tick now runs a watchdog that self-sends `DisconnectObserved` when a driver reports a +non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. Things to know before touching it: + +- ⚠️ **The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal.** + `Degraded` means *connected but some data is bad*, and several drivers write health **per tag inside the + read loop** (`ModbusDriver.ReadAsync`), so one unreadable address latches `Degraded` forever against a + perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely. + `LastSuccessfulRead` is the honest evidence — preserved across `Degraded`/`Faulted` writes, advanced by any + success. Each conjunct has a test that goes red when the *other* is deleted. +- **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across + all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect + supervisor, OpcUaClient via a bad session keep-alive (#670 — it used to stay `Healthy` forever; its SDK + `SessionReconnectHandler` is deleted, a lost session publishes `BadNoCommunication` once per outage and the + watchdog's re-init is the recovery; good keep-alives + notifications advance `LastSuccessfulRead`). `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has + no transport. Re-check this if a new driver lands. +- **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.** + `ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable` + field inside `InitializeAsync` without disposing the old one, so re-initialising without `ShutdownAsync` + leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose + first: `ModbusDriver.InitializeCoreAsync` releases `_transport` **and** the `_probeCts`/`_reprobeCts` + probe loops before rebuilding them (`c3970a2e` + the lifecycle gate in `409a58b6`), deliberately *not* + via the shared `TeardownAsync()` because that would also drop `_poll` and every active subscription with + nothing to re-drive them; `MqttDriver.InitializeCoreAsync` calls its (idempotent) `TeardownAsync()` first, + which reaches `_connection` and the primary-host connection (`2b4b3fcb`). The path is no longer latent — + the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new + `IAsyncDisposable` field on these two, must repeat the dispose-before-overwrite. +- The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` / + `DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy + system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped + reading. +- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises + `ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient: + keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed + connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the + SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only + while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with + the watchdog's conjunction. `docs/DriverLifecycle.md`. + +- **What comms loss now does downstream — one contract, six seams (archreview R3-02).** "A value consumer + never reports a quality better than its source's" is stated once in `docs/DriverLifecycle.md` §"The + quality contract" and implemented by `Core.Abstractions.InputQuality` (`WorstOf`/`IsUsable`). Three + things changed with it, all client-visible: + - ⚠️ **Value nodes of a comms-lost driver now go Bad.** `DriverHostActor.FanCommsLossToValueNodes` + stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained + server-side**, via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host + holds no value cache, so only the node can keep the value). ⚠️ Retention is **not client-observable**: + the SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read/sample, so clients see + `Value = null` + the Bad status; the retained value is what the next Good re-publish and the + `(value, status)` dedup compare against. #481 had wired comms loss to the mux and to Part 9 + conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop + the tag" (ScadaBridge) will see more Bads — that is the fix working. + - **Nothing Good is written on reconnect** — recovery is the driver's own re-publish (the poll engine's + `forceRaise`, MQTT's new initial push). Instead the host **audits**: refs still latched 30 s after a + reconnect log a Warning and increment `otopcua.driver.comms_latched_refs`. Non-zero on that counter + means a driver reconnects without re-publishing. + - **Every dedup on the recovery path is keyed on (value, status), not value.** A device that comes back + usually comes back holding the same reading, so a value-only dedup swallows the recovery and the node + stays Bad forever — the v3-gate trap, now closed in the Calculation driver, the virtual-tag actor and + (via a connection-loss stale) MQTT's plain-mode suppression. **Any new consumer of driver values must + repeat this**; a regression test for it must replay the IDENTICAL value or it proves nothing. + - Consequences per seam: MQTT/Sparkplug stale on connection loss and push cached snapshots on every + (re)subscribe (`docs/drivers/Mqtt.md`); the Calculation driver and virtual tags **hold** while any + input is Bad and inherit the worst input's sub-code, evaluate through Uncertain, and hand scripts the + input's real `StatusCode`/`SourceTimestampUtc` (`docs/drivers/Calculation.md`, `docs/ScriptEditor.md`) + — so a script that relied on receiving `null` for a dead input no longer runs at all while it is Bad. + +See `docs/DriverLifecycle.md` §"The comms-loss watchdog" and §"The quality contract". + +## AB drivers: native transport (`Transport`) + +AbCip and AbLegacy have a second, managed wire transport (`Driver.Ab.Transport`) beside libplctag, chosen by +the driver-config key `transport` = `Native` | `Libplctag` (string; takes effect when the driver is +**constructed** — every process start, or a deploy that recreates it, not `ReinitializeAsync`; AdminUI +"Transport" dropdown; CLIs `--transport`). "Default" is encoded by omitting the key; the AdminUI must never +re-emit a concrete default. The defaults differ per driver (final review, 2026-10-07): + +- **AbCip: an absent key resolves to the node default, `Native` unless pinned.** Precedence (one place, + `AbCipTransportResolver`, shared by driver, probe and CLI): explicit per-driver key → node-level appsetting + **`Drivers:AbCip:DefaultTransport`** (`Native` | `Libplctag`, env `Drivers__AbCip__DefaultTransport`, + `AbCipDriverDefaultsOptions`; a bad value **fails host start**) → compiled `Native`. Bound unconditionally + because the driver factory AND the admin-node Test Connect probe both read it. ⚠️ So **the upgrade restart + alone moves no-key AbCip drivers to Native**; to stay on libplctag, set the node key — NOT a per-driver key + through the old AdminUI, whose typed form drops unknown keys (#576). Evidence: AbCip reads proven on a real + CompactLogix (Z-35162); scalar native writes (DINT/BOOL/REAL) proven on it once (2026-10-09, user-approved one-off, restored) — and native UDT member writes (DINT/BIT/REAL, #611 gate, same day) and a native STRING write (hw5, same day, restored byte-exact); and a native array-element write (`EM01_PowerUnit._SEQ_EP[31]`, user-approved, an element no alias targets, restored; neighbours unchanged); libplctag member and fragmented writes are `ab_server`-only. #566: a real Logix drops TCP on a connection idled past its timeout (no 0x0107/0x0203); the faulted session reopens on the next request. +- **AbLegacy: an absent key resolves to `Libplctag` for every family; `Native` is opt-in** (no node key). On + a real MicroLogix 1400 (Z-33323, idle, 2026-10-09) native read S/N/F/B/T/C/R/ST/O/I incl. `S:0`, T/C/R + sub-elements and status bits, and ST byte order, all correct, and wrote N/F8/B-bit (restored). libplctag (the default) + used to fail `S:`/`O:`/`I:` and `R` status bits there (incl. the default probe `S:0`) — its `pccc.c` needs explicit + file numbers and has no R status-bit mnemonics; since #659 `LibplctagLegacyName` translates (`S2:`/`O0:`/`I1:`, + `R6:0.CON/13`) and both transports read identically on that press. The authored grammar is unchanged. Read-side + only: a libplctag WRITE to any address the translation changes (`S:`/`O:`/`I:` incl. bit writes, R `EN…FD`, `C…UA`) + is refused `BadNotSupported` before any runtime is created (none could reach the PLC before — no write-surface + widening on the default transport); libplctag T/C status-bit writes it parses itself are unchanged. + L/A files, function files, SLC, PLC-5 and `logixpccc` remain unverified natively. Explicit Native on PLC-5/`LogixPccc` logs a once-per-instance Warning. + AbLegacy native refuses status-bit sub-element writes (`T4:0.DN`…, `BadNotSupported`). + +Refusals are not comms faults: an Unsupported refusal (`BadNotSupported`) and an AbCip native **type +mismatch** (`BadTypeMismatch`, where libplctag returned Good garbage for a same-width mis-authored type) both +keep the tag handle, leave driver health alone (no `Degraded`, nothing for the comms-loss watchdog) and warn +once per tag — only genuine transport failures evict and degrade (a runtime `NotSupportedException` too, #611 — +it used to latch `Degraded`). AbCip width-checks every scalar **write** against the controller's bytes before +sending, on both transports (#611; libplctag used to patch part of its buffer and send it all). Controller-discovered +UDT members are writable once browse-committed and authored `ReadWrite` (BIT members included — the controller +applies the bit) — native member writes incl. a BIT member are live-gated on Z35162 (2026-10-09); sibling-bit preservation and the libplctag path are ab_server-only; whole-UDT writes are refused `BadNotSupported` by decision (`docs/drivers/AbCip.md` §Member writes). `DeviceSessionPool.Shared` logs through the +host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT factories' `Register(registry, ILoggerFactory?)` take one too, #602 CONV-3). The AbCip `browsePageDelayMs` knob has an AdminUI field (`AbCipDriverForm` "Browse page delay (ms)"). libplctag is confined to `Libplctag*.cs` adapters by +`LibplctagConfinementGuardTests`; do not reference it elsewhere. See `docs/drivers/AbCip.md` §Transport, +`docs/drivers/AbLegacy.md` §Transport + §Wire evidence, `docs/Configuration.md` §`Drivers:AbCip`, and +`docs/plans/2026-10-07-native-ab-transport-gate-results.md`. + +## Other driver notes + +- **S7:** connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe + tick); a write-side access-denied (PUT/GET disabled) latches a **sticky Faulted** that outranks later read + passes; driver-side array writes cover non-Bool narrow/`Int32`/`Float32` types only — Bool arrays are + refused (a one-byte-per-element encode would clobber neighbour bits). +- **MQTT:** Plain `writeTopic` writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug + primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types + `BadNotSupported`). +- **No driver-side history (#655):** `IHistoryProvider`, `DriverCapability.HistoryRead`, `HistoryReadStatusException` and + the OpcUaClient history passthrough are deleted — nothing ever dispatched HistoryRead to a driver; it is served only by the + server-wide `IHistorianDataSource`. `isHistorized` on an OpcUaClient tag means OtOpcUa records it, not that upstream + history is proxied. A stored `ResilienceConfig` still carrying a `HistoryRead` key parses (diagnostic only). +- **FOCAS / AbLegacy:** background loops are joined (≤ 5 s) on shutdown. +- **FOCAS (#257/#266):** `cnc_statinfo` flags are a `Status/` fixed-tree subtree (`FIXED:Status/Run`…, capability-probed); + adjacent PMC tags in one poll batch coalesce into one `pmc_rdpmcrng` (`PmcCoalesce` config, default on) — a refused range + falls back to per-tag reads, a transport failure fails all members without retry. Parts count stays `PARAM:6711` (docs only). +- **FOCAS alarms (#647):** an `ALARM:` address (`*`, ``, `/`, `/-`, `T`, `OTHER`) + + the `alarm` object = ONE condition at the RawPath (either alone is skipped + reported). Overlapping selectors all + fire; `OTHER` = what no type selector claims. ⚠️ A failed `cnc_rdalmmsg2` THROWS (it used to return `[]` ⇒ mass + clear + un-acked re-raise) — and annotates its conditions `BadNoCommunication` once per outage (quality-only) and + degrades driver health with `LastSuccessfulRead` frozen, so an alarm-only driver trips the #527 watchdog. ⚠️ A CNC + REFUSAL (non-OK rc, transport fine) is `BadNotSupported` + one Warning and does NOT touch health — else the + watchdog reconnect-loops a production control. Health = worst of the value and alarm paths, computed together. The told + record lives on the driver INSTANCE (reconnect / in-place re-init re-raise nothing); a deploy RESPAWNS the driver, + and its Raise for a still-active alarm is a host-side re-statement (below). Type names are the 16i-era table only + (30i codes unconfirmed, not invented — use `T` there). Read-only; ack server-side. `AlarmProjection.Enabled` + is a no-op. E2E: `Runtime.Tests/FocasNativeAlarmEndToEndTests` (incl. a respawn). `docs/drivers/FOCAS.md`. +- **A Raise on an already-active condition with the SAME message is a RE-STATEMENT (#647 review):** `NativeAlarmProjector` keeps its ack and + `ForwardNativeAlarm` writes no second "Activated" `/alerts` row — every native driver's told-record dies with its + instance, and a deploy respawns the driver. Only `Retrigger` (or inactive → active) un-acks, so a source whose + UPSTREAM is authoritative for the ack sends Retrigger for an unacked re-statement: OpcUaClient on first sight of a + condition (no prior told), Galaxy for an unacked active snapshot entry. **Those two also flag + `AlarmEventArgs.IsRestatement` (#656)** — history-only: a flagged Retrigger on an Active+unacked condition with the + same message AND severity, or a flagged Clear on an inactive one, writes no `/alerts` row (every Galaxy reconnect used + to duplicate them); a flagged `Unspecified` on an already-active condition (AbCip ALMA governing-limit drop-out) + moves the node's text/severity, keeps active/acked and writes no row. Active+acked is the atomic **`AlarmTransitionKind.ActiveAcknowledged`** (Galaxy snapshot + `ActiveAcked`, OpcUaClient inactive→acked-active) — never Acknowledge (kept a cold host's `Active=false`: the alarm was + invisible) nor Raise+Acknowledge (an unacked intermediate). Its mirror **`InactiveUnacknowledged`** (#657, OpcUaClient) + replaces Raise/Retrigger+Clear (an Active flicker; rows only for what changed). OpcUaClient's first sight of an inactive + condition is a flagged InactiveUnacknowledged, or flagged Clear+Acknowledge for inactive+acked (a flagged Acknowledge on an + acked condition writes no row) — never `Unspecified`, which wrote "Activated" for an inactive alarm and left a + retained-Active condition Active. `NativeAlarmProjector` returns the rows to write; unflagged + sources (FOCAS/MTConnect, live Galaxy) are unchanged. E2E: `Runtime.Tests/NativeAlarmRestatementEndToEndTests`. +- **Re-Initialize without teardown (#630):** FOCAS, AbLegacy, AbCip, TwinCAT, S7 and OpcUaClient `InitializeAsync` + first release the prior device/session state (cancel, ≤ 5 s join, dispose — incl. the AbCip alarm projection; + FOCAS's is PAUSED, its told record kept, #647) — the device-state half of `ShutdownAsync` only; the poll engine is kept and the actor re-subscribes + after every successful init. AbCip publishes fresh tag tables by reference swap (a second init used to throw + "tag name collision"); S7 swaps `Plc` under `_gate`. +- **Data and alarm subscriptions are independent in `DriverInstanceActor` (#662):** a data re-subscribe (every + deploy's bulk push to a kept driver) detaches DATA only (`DetachDataSubscription`) — it used to unhook + `OnAlarmEvent` too, silently killing the native-alarm feed until a respawn. The health-poll reconcile + re-attaches/re-subscribes alarms as well as data, so a failed alarm subscribe is retried every tick. +- **A config change is a respawn — there is NO in-place path (#665).** The `ApplyDelta`/`ApplyResult`/`DeltaApplied` + seam (host `ApplyChildDelta`, `DriverSpawnPlan.ToApplyDelta`, the actor's Connected reinit and mid-connect + `AdoptConfigDuringInit`) was unreachable since #516 and is deleted; #663/#664 were latent bugs in it. `IDriver.ReinitializeAsync` + stays as a per-driver contract the runtime never calls. Every driver's `InitializeAsync` still serves the config it is + handed (Calculation adopts, Galaxy refuses a non-equivalent one — #664), and still releases prior state (#630). + `docs/DriverLifecycle.md`. +- **S7 (#632/#638):** STRING/WSTRING/CHAR/WCHAR and DTL/DATE/TOD/TIME/S5TIME codecs (`S7ValueCodecs`; TOD/TIME/S5TIME + surface as Int32 ms); an undecodable buffer or unrepresentable write is `BadOutOfRange` with no health impact. + Optional `localTsap`/`remoteTsap`/`connectionType` override the slot TSAP (`remoteTsap` ⊕ `connectionType`). Test + Connect reads 2 bytes of `MB0` so a PUT/GET-disabled CPU fails the probe. The negotiated PDU size is logged, on + `S7Driver.GetDiagnostics()`, and (#640) on `/hosts` via the optional `IDriverDiagnostics` bag (string map on + `DriverHealthChanged.Diagnostics` / proto `has_diagnostics`+`diagnostics`; its digest is in the publish fingerprint). +- **TwinCAT (#637):** TIME/TOD publish Float64 ms, DATE/DT a UTC DateTime; a tag authored `UDInt` keeps the raw counter + (and keeps historizing — DateTime is not historized). +- **TwinCAT (#645, #310/#314):** opt-in ADS sum-read (`useSumRead`, `sumReadMaxItems` default 200 / ceiling 500, 32 KiB payload + budget). It reads raw bytes and decodes as the declared type, so only a scalar passing the strict width+kind check + (`CheckNotification`) rides a batch; value-converted pairs (DInt over INT, Real over LREAL/DINT) read singly. Only overall + ADS 1793/1794 latch the device to single reads; any other non-comms overall code (1792/1795/1797/1798...) reads that chunk singly for the poll without latching, with a once-per-device Warning; a comms failure neither. + Opt-in `diagnosticsEnabled` (default off) puts ADS state/app/online-change/cycle/exec/`HeadroomMs` on the `/hosts` + `IDriverDiagnostics` bag (≤ 4 extra reads per device per probe tick once settled; TC2 falls back to `.SYSTEMINFO` / + `.SYSTEMTASKINFOARR[1]`, costing 8 on the first tick and after each reconnect). Live-gated read-only 2026-10-08 on + Z35474 (TC3) + Z34834 (TC2, which answers sum commands) via a harness driving `TwinCATDriver` — `/hosts` end-to-end + not run (a stock host has no AMS router); the gate found ⚠️ Beckhoff's `SumRead` ctor throws for a `List<>` of entities (only an array survives + its internal cast) — `AdsTwinCATClient.CreateSumRead` owns it. Flat browse stays (VirtualTree is a strict subset + on TC3). Sum-write not ported. +- **AbLegacy (#636):** `AbLegacyAddress` (in `AbLegacy.Contracts`) is family-aware: PLC-5 `I:`/`O:` indices are OCTAL + and are emitted to libplctag as decimal (libplctag's `pccc.c` parses decimal only). MicroLogix function files parse + but read `BadNotSupported`. Indirect `N7:[N7:0]` (#646) is resolved by the driver at read/write time (index re-read + every call, never cached; bad index = `BadOutOfRange`; indirect writes are not atomic). One check (`AbLegacyAddress.Validate`) backs the + editor (family-independent) and the deploy gate (family from the tag's `DeviceConfig.PlcFamily`); an unparseable + address reads `BadNodeIdUnknown` for that tag only. +- **OpcUaClient (#635):** OperationLimits are honoured by the SDK session (`SessionClientBatched`) — do not re-add + driver-side chunking. Subscription tuning keys are range-checked at `ParseOptions`. `discoveryUrl` endpoints go + AFTER configured ones; more than one discovered server without `discoveryServerUri` fails the connect. Session + events are the OTel counter `otopcua.driver.opcuaclient.events`. `HistoryAggregateType` carries the full Part 13 + set; an aggregate the gateway can't serve is `BadAggregateNotSupported` per node. +- **MTConnect (#627/#628):** TIME_SERIES reads as a variable-length `double[]` (Float64 only); a count disagreeing + with `sampleCount` is `BadDecodingError`. One MTConnect driver = one agent client = at most one `/raw` Device + (`MTConnectMultipleDevices` deploy error; `RawTreeService.CreateDeviceAsync` refuses a second). +- **MTConnect CONDITION alarms (#626):** a CONDITION raw tag is a native Part 9 condition only when its `TagConfig` + carries the standard `alarm` object (opt-in — alarm tags leave the value subscription, so auto-promotion would break + existing String tags). Normal→inactive, Warning→500, Fault→800; one condition per DataItem showing the most severe + active nativeCode (ties: ordinal nativeCode, then type); ack is server-side only. ⚠️ UNAVAILABLE never clears: an + active condition stays active+acked with a quality-only `BadNoCommunication` (same as #481 transport loss). The + driver dedups against what the server was last TOLD (level+nativeCode+nativeSeverity+qualifier — not text), records + only while a handler is attached, and drops the record of only the conditions that LEFT the set when the subscribed + alarm SET changes (#641) — never on a reconnect of the same set; a replayed `/current` must never re-raise (un-ack). + `AlarmEventArgs` gained optional `OpcUaSeverity` (exact Part 9 severity) and `QualityStatusCode`; a quality-only + event goes out as `AlarmQualityUpdate` per node and never through the snapshot projection. +- **Native condition state survives a full rebuild (#641):** `RebuildAddressSpace` captures each NATIVE condition's + live node state (client acks included), the same-apply re-materialise restores it, and `CompleteAddressSpaceApply` + (called by `OpcUaPublishActor` at the end of every apply) reports ONE event per restored off-normal condition and + discards unclaimed captures. `NativeAlarmProjector` is pruned per apply, not cleared. Native routing is an exact + `ConditionId == RawPath` match — AbCip's `#active` suffix was fixed; FOCAS routes since #647 (below). +- **Driver removal (#609):** removing a driver releases its capability-invoker pipelines and resilience + tracker, *then* evicts its telemetry snapshots. ⚠️ Both are **tombstoned**, not just cleared: the removed + child stops asynchronously and its in-flight subscribe/connect still completes through the invoker (and + emits Health) on the thread pool *after* `StopChild`, which used to resurrect the state — so a removed + driver's counters were republished forever. `Release` → tracker `Retire` and hub `Evict` ignore every later + write for that id until it is spawned again (`Create` → `Reinstate`, `SpawnChild` → `Readmit`). A + Watch/`Terminated` release would not work — the late writes land after the actor terminates. diff --git a/.claude/rules/historian.md b/.claude/rules/historian.md new file mode 100644 index 00000000..5a2f9421 --- /dev/null +++ b/.claude/rules/historian.md @@ -0,0 +1,275 @@ +--- +paths: + - "src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.*/**" + - "src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/**" + - "src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/**" + - "src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/**" + - "tests/**/*Historian*/**" + - "docs/Historian.md" +--- + +# Historian / HistoryRead (Gateway, SqlServer, QuestDb backends; alarm history; continuous historization; provisioning) + +> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area. +> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file. + +## Historian / HistoryRead + +**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default), `SqlServer` or `QuestDb`** (see the subsection below). +As of the gateway-integration cutover, the **Gateway** backend's historian read, alarm-write, and +continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed +**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin +`IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke +Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were +retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`, +now a retired stub). + +A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the +raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default +historian tagname, which is the tag's driver `FullName`. + +### Pluggable backends, SQL Server and HistoryUpdate (2026-10) + +- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`. +- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. +- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". +- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. +- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads". +- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates. +- **Raw `ReturnBounds` is honoured on every backend (#691, client-visible):** Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by `HistoryBounds` from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 `ReverseRawReader` walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward `NumValuesPerNode`, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ `Bad_BoundNotFound` (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ **Breaking for SDK-default clients:** the .NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true` — a client that cleared only `IsReadModified` now gets bounds, and one that set neither is a modified read with bounds ⇒ `Bad_InvalidArgument` (§6.5.3.3); set both explicitly (Client.Shared does; tests building details must too). No data AND no bound found ⇒ `GoodNoData`, no values (the pre-bounds answer). On Gateway the at-or-after lookup ends at now + 1 day (never a year-9999 end to AVEVA). Kill switch `ServerHistorian:HonorReturnBounds` (default `true`; `false` = pre-#691). `docs/Historian.md` §Bounding values. +- ⚠️ **Raw pages are full (#692, client-visible, every backend):** each raw page asks the backend for `N + skip + 1` rows (the boundary ties it will trim + one lookahead row) — a resumed page used to ask only N, trim the ties and come back short; at N=1 that was an EMPTY page + continuation point that then fell into the `MaxTieClusterOverfetch + 1` over-fetch (live: 8 values = 17 pages / 25 reads / 524k rows ⇒ now 8/8/≤24). No lookahead row ⇒ last page, no CP (an exact multiple no longer ends on an empty page). Exception: an oversized tie cluster — its last slice is `min(N, remaining)` with a CP, and stepping past one at the end of data can give one empty final page. The over-fetch fires only when every row past the trimmed ties sits on the boundary timestamp; a cursor inside such a cluster is marked `WithinTieCluster` (next page goes straight to the over-fetch). `SliceTieCluster` resumes only strictly before EndTime (truncated to the backend resolution) — it used to resume AT the exclusive EndTime and return samples there. Bound placeholders carry an explicit marker (`HistoryBounds.IsPlaceholder` no longer matches a stored `Bad_BoundNotFound`/null row); the N=1 bounds probe carries its walk hint + resolved end bound into the cursor. Reviewer fuzz harnesses (paged vs unpaged over random tie series) are the right test for any paging change — repo tests passed while master mismatched 1-in-160. +- ⚠️ **Strict-ceiling paging (#694):** `IHistorianDataSource.MaxRawValuesPerRead` = L = learned gateway ceiling − 16 (0 on SQL backends/Null, guard-tested; 0 until a refusal names the ceiling). `RawPageLimit.Plan(want, k, L)` sizes every raw page so page + boundary ties + lookahead ≤ L (room = L − k − 1, signed; first page L−1, resumed L−k−1; uncapped `N = 0` reads page too) — a capped read above the ceiling used to be `BadTooManyOperations`, now a ceiling-sized page + CP. One re-plan when the read taught a new limit or was refused for crowding (`HistoryReadLimitExceededException.PreWindowRows`, set only above the 16-row allowance; the page shrinks by it). No room on a resumed page ⇒ the tie-cluster over-fetch, bounded min(M, L−1); the slice size is never 0. The page path passes `retryOnLearnedLimit: false` to `ReverseRawReader` (the walk re-running at the old N cost 200–3,000 reads cold); bounds lookups keep it. ⚠️ A Gateway window starting at/before the Unix epoch counts AVEVA's synthetic row as pre-window — it used to set `reachedEnd`, so a ceiling-truncated read came back Good with no CP (forward-from-1970 = first page only; end-only past the data = the OLDEST rows as "latest"; pre-existing). The #687 crowding re-read now doubles its room up to 8 re-reads, never past a known ceiling, and its room is capped at `MaxTieClusterOverfetch` (passed to the Gateway source by the factory; past it ⇒ `BadTooManyOperations` with `PreWindowRows`). A raised ceiling is not re-probed by raw paging (pages stay small until another read re-probes or restart). +- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. +- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. +- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key). +- **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history. + +### Read path (`ServerHistorian` section) + +The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the +`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default +(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value **null** + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (`AllowStreamingBeyondCeiling=false`) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (`GatewayRowCeiling`: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (`NumValuesPerNode = 0`) read of a small window is served; a ceiling-bounded stream that is not provably complete is `BadTooManyOperations`, never a truncated answer), Processed +(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes (⚠️ #693, client-visible: Gateway AtTime is evaluated from RAW reads by the shared `Sql.Common` `AtTimeEvaluator` — exact stored value at a stored time, null + Bad at/after a stored NULL, Uncertain hold before a Bad neighbour, linear + Interpolated bit, `BadNoData` before the first sample — never AVEVA's own interpolation (which answered 107 Good at a NULL, null+Bad at a Good sample after it, 99.99999999944 for 100, no Interpolated bit); `GatewayAtTimeReader`: one ≤1,024-row window read for the batch (jumps to far-apart times use 64-row reads) + a walk-back for the first time's prev (gallop ×4, whole-remainder read after two empties, then real BISECTION seeded by any full read's last row — a dense tag stopped a year ago ≈ 25–31 reads; the first cut regrew segments after a full read and gave up at 65) + an 18-row probe for the last time's next — 10 dense times = 1 read (exact) / 3 (interpolated), 10 hourly = 12; read cap 64 + 16 per distinct time (a runaway guard: a time in a gap of a bursty tag needs ≈ 6 reads; 4/time refused a 1 h/day tag asked daily ×30) and a strict ceiling < 4 rows ⇒ `BadTooManyOperations` with a message naming the limit; DateTime hints stepped, non-finite/out-of-range values under an integer hint unusable; all request bounds whole ms, rows counted only strictly inside a request; the node manager passes the node's scalar type (`ReadAtTimeAsync(…, TypeCode? valueType, …)` default-interface overload, ignored by SqlServer/QuestDB/Null) and the Gateway converts raw rows to it before evaluating — Boolean stepped, integers interpolated + rounded half-away and returned as the node's integer type, floats Double; no hint ⇒ Double; ⚠️ the shared evaluator used to box interpolated Int64 as Double (`cond ? (long)x : double` widens) — fixed for SQL backends too, and Shouldly `ShouldBe` hides numeric-type bugs (assert `ShouldBeOfType`); kill switch `ServerHistorian:GatewayAtTimeFromRaw` (default `true`); `docs/Historian.md` §Gateway AtTime); Events over +alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node); +authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization. + +`ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`): + +| Key | Default | Notes | +|---|---|---| +| `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` | +| `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) | +| `ApiKey` | `""` | Peppered-HMAC key `histgw__` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** | +| `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme | +| `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) | +| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store | +| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read | +| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) | +| `HonorReturnBounds` | `true` | #691 kill switch: `false` ignores HistoryRead-Raw `ReturnBounds` as before (no bounding values, no modified-read `Bad_InvalidArgument`) | +| `GatewayAtTimeFromRaw` | `true` | #693 kill switch (Gateway only): `false` = AVEVA's own AtTime interpolation passed through, as before | +| `EventReadHorizon` | `00:00:00` | #694, opt-in and LOSSY: when > 0, HistoryReadEvents never reads events older than now − horizon (fresh windows clamped, both directions, all backends; a window wholly older ⇒ `GoodNoData` unsent). Bounds the deep "rest is empty" event probe (~3.8 s live, 30 s gateway SQL timeout unfiltered). No per-source earliest-event cache by design — late S&F drains would make it stale | +| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential | +| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 | +| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway | +| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded | + +### Alarm-history path (`AlarmHistorian` section) + +Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind +the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink` +default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the +replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to +the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node +holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The +`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`, +`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection +(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the +target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround). + +### Continuous historization (`ContinuousHistorization` section) + +When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a +durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed +`IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The +recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the +crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**) +with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries +only the recorder + outbox knobs: + +| Key | Default | Notes | +|---|---|---| +| `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder | +| `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage | +| `CommitMode` | `PerEntry` | `PerEntry` = fsync before the append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs`. ⚠️ `PerEntry` now means **one fsync per ingest burst, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands | +| `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` | +| `DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs `tags × latency` — while 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, letting the outbox fall permanently behind during any gateway hiccup | +| `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) | +| `DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential `tags × latency` pass becomes `⌈tags / concurrency⌉ × latency` with outstanding round-trips bounded. `1` = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass" | +| `Capacity` | `500000` | Max un-acked outbox entries before drop-oldest; `0` = unbounded. Non-zero **by default since #538** — the disk backstop for a historian refusing *every* write, the one case the per-tag quarantine deliberately cannot resolve | +| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance | +| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass | +| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff | +| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 | + +### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook) + +`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` +hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's +`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues` +land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a +deploy. **Gateway backend:** non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not +provisioned), the recorder likewise drops + meters non-numeric values, and continuous historization is +**numeric-analog only** (`UInt16→UInt4` is a documented fallback). The **SqlServer** backend provisions and +records every type (see "Pluggable backends" above). + +The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`), +built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the +read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier. +When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not +run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with +`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR +#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.) +**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists). +`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the +historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable +stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are +supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see +the "Boolean historization maps to Int2" section in `docs/Historian.md`. + +### Gateway-side prerequisites + +The target HistorianGateway OtOpcUa points at **must** run with: + +- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization). +- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads). +- `RuntimeDb:EventReadMaxRows=0` (its default, unlimited) — or larger than the reverse-events walk's biggest read + (`2·MaxTieClusterOverfetch + 4`). The cap truncates **silently**, so a walked-back reverse events read (#691.4) would + lose the newest events of a segment it cut. +- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**. + +### Migration note (deployments upgrading from the Wonderware backend) + +The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env: + +| Old (Wonderware) key | New (gateway) key | +|---|---| +| `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) | +| `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) | +| `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) | + +The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`) +were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from +`ServerHistorian`. See `docs/Historian.md` for the full guide. + +### Live-validation gate — PASSED (was "KNOWN LIMITATION 1") + +**No longer a limitation.** The cutover was live-validated against the real gateway on +`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed +protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot +populate `Source_Object` — not a fixable bug). Evidence: +`docs/plans/2026-06-27-otopcua-historian-followups.md`. + +Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to +`wonder-sql-vd03`, gateway running the prerequisites above): + +```bash +export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway URI; absent ⇒ all live tests skip +export HISTGW_GATEWAY_APIKEY=histgw__ # must carry historian:read + historian:write (+ tags:write) scopes +export HISTGW_TEST_TAG= # read round-trip +export HISTGW_WRITE_SANDBOX_TAG= # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write) +export HISTGW_BOOL_SANDBOX_TAG= # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441) +export HISTGW_ALARM_SOURCE= # alarm SendEvent → ReadEvents round-trip +dotnet test --filter "Category=LiveIntegration" +``` + +The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the +gate the operator runs on the VPN before trusting the cutover. + +### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04) + +The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer + +meters). It is spawned with an **empty *initial* ref set** (`Array.Empty()` in +`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built +later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the +`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it +through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the +recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the +deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe +`Tell`; a faulting feed can never break a deploy). + +**The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a +production-fatal defect** — see `docs/plans/2026-08-01-continuous-historization-live-gate.md`. Three things +to know: + +- ⚠️ **`ConfigureAwait(false)` in an Akka handler destroys `Self`.** `OnValueChangedAsync` touched `Self` + after awaiting `IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false)`, which discards Akka's + `ActorTaskScheduler` synchronization context — so `ActorBase.Context` was gone and it threw + `NotSupportedException` on the FIRST value. The throw reached supervision, the actor **restarted**, and + because production seeds an **empty** ref set the restart discarded every historized ref: one value + captured, then silence. Fixed by capturing `IActorRef self = Self` before the await (the idiom the + `GetStatus` handler already used for `Sender`). +- ⚠️ **A fake that completes synchronously hides this entire class of bug**, and 13 tests passed because + of one. `ValueTask.CompletedTask` never yields, so the continuation ran inline with the context intact. + Worse, the obvious fix to the fake — `await Task.Yield()` — is *also* vacuous: `Yield()` posts back to + the captured context, so the caller still resumes on the actor thread. The fake must complete on a + **pool** thread (`await Task.Delay(1).ConfigureAwait(false)`), as a real per-entry fsync does. Only + reverting the production fix exposed that the first regression test proved nothing. +- **Both pair nodes historized the same value** — the recorder had **no Primary gate**, unlike the alarm + store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as + **#537** and **fixed** — see the paragraph below. + +**Gate COMPLETE — #491 CLOSED 2026-08-04.** Outbox append, drain, `WriteLiveValues`, +**restart-convergence**, `EnsureTags` provisioning (`ensured=2 failed=0`) and **read-back** all ✅. The +read-back leg ran against the licensed `wonder-sql-vd03` with the VPN up: two registers carrying +*distinct non-zero* constants (`8224`, `120`) each read back as its own value — a `0`-valued source +would have proved nothing (the #536 trap). + +**Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison +tag); evidence in the same plan doc. Two things to know before touching either.** + +- **#538 — the discriminator is "did another tag write this pass?", NOT a failure count.** `DrainOnceAsync` + writes every per-tag group and acks **all-or-nothing** (deliberate: `RemoveAsync` truncates *through* an + id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned + the FIFO head while every *healthy* tag was re-submitted on every pass, forever — duplicate samples, an + outbox that never truncated (observed 8.3 MB; signature = the gateway's `ok` **and** `error` counters + climbing together). The drain now quarantines a tag after `MaxWriteAttempts` consecutive failures, but + ⚠️ **only counts a failure on a pass where some other tag succeeded** — otherwise an ordinary gateway + restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug + into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so + the load-bearing test is the **falsification control** (`A_backend_that_fails_every_tag_abandons_nothing`). + Abandonment is also deferred until *every* failing tag is over budget, so nothing is discarded on a pass + that could not have advanced the head anyway. +- **#537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink.** The alarm + queue is in LocalDb and *replicates*, so the Secondary holds the Primary's rows and gating the drain + loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its + drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of + stale duplicates on promotion). ⚠️ **And the peer that matters is one recording the SAME refs, not one + holding the same rows** — a node stands down only for a Primary in its own per-cluster mesh, and a node + with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is + usually running another cluster's configuration, and deferring to it ends this cluster's historization + *silently* (the drain reports success on an empty outbox forever) — the same scope mismatch that once + suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on + `RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes + (NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the + #511 registration accident). + - **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary + stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the + redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and + *nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`) + subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role + source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change + re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the + failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable` + (default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT + adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay + put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address + clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the + gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real + partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each + node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates. + +Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for +newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType) +provisioned nothing (`2026-06-27-historian-gateway-integration-issues.md` Issue 5). It now also fires for those +changed raw tags. diff --git a/.gitignore b/.gitignore index 6213b63c..0a7b712b 100644 --- a/.gitignore +++ b/.gitignore @@ -29,7 +29,8 @@ packages/ *.pem # Claude Code (per-developer settings, runtime lock files, agent transcripts) -.claude/ +.claude/* +!.claude/rules/ .local/ diff --git a/CLAUDE.md b/CLAUDE.md index 9d70fe72..c5e8ff56 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,6 +90,24 @@ stranded on a feature branch, cherry-pick them onto `main` rather than merging t > into the SQL Servers. Deploy steps that need a command on the box are blocked; see the umbrella > [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md) for the current hold. +## Subsystem guides (lazy-loaded — read before touching the area) + +Detailed gotchas live in path-scoped files under `.claude/rules/`, which load automatically when you work on +matching files. **If you are changing behaviour in one of these areas and the file has not loaded, read it first.** + +- `.claude/rules/drivers.md` — Galaxy change detection / `IRediscoverable`, the comms-loss watchdog and quality + contract, AB native transport, and per-driver notes (S7, MQTT, FOCAS, TwinCAT, AbLegacy, OpcUaClient, MTConnect, removal). +- `.claude/rules/cluster-topology.md` — Redundancy, `EffectiveRoles`, per-cluster meshes, `MeshTransport`, + `ConfigSource`/`ConfigServe`, telemetry transport, LocalDb pair store, DB-less driver nodes (Phase 4). +- `.claude/rules/data-plane-auth.md` — API keys on the OPC UA data plane, `Security:Acl` enforcement, anonymous + principal, equipment-node write checks, ACL scope picker; LDAP is AdminUI-only. +- `.claude/rules/alarms.md` — scripted alarm ack/shelve, Part 9 condition methods, condition severity / children. +- `.claude/rules/historian.md` — HistoryRead/HistoryUpdate across the Gateway/SqlServer/QuestDb backends, alarm + history, continuous historization (#491/#537/#538/#672), provisioning, gateway prerequisites, live gate recipe. + +**Never commit secrets.** API keys, PSKs and historian keys come from env vars or `${secret:}` (e.g. +`ServerHistorian__ApiKey`); the docker-dev rig's literal dev keys are the only committed exception. + ## Architecture Overview ### Data Flow @@ -218,161 +236,12 @@ S7. There is no alias machinery, no `SystemPlatform` namespace kind, and no relay→alias converter. See `docs/plans/2026-06-12-galaxy-standard-driver-design.md` for the full design. -### Data Type Mapping - -Galaxy `mx_data_type` values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`. - -### Change Detection - -`DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys. - -**The signal is consumed as an operator prompt, not as an address-space rebuild** (§8.2, 2026-07-27). -`DriverInstanceActor` subscribes in `PreStart` and unsubscribes in `PostStop` — the detach is -load-bearing, because the `IDriver` instance outlives the actor when the host respawns a child around -the same driver object. A raise is recorded and rides the driver-health snapshot -(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`) to the AdminUI `/hosts` page as a -**"re-browse" chip**. - -⚠️ **It is advisory: a Galaxy redeploy still does NOT change the served address space.** v3 authors raw -tags explicitly through the `/raw` browse-commit flow, so an operator must re-browse the device and -commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact -records. - -Two gotchas if you touch this: - -- `PublishHealthSnapshot` dedups on a health fingerprint. The rediscovery timestamp **must** stay in - that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the - operator. Pinned by `DriverInstanceActorRediscoverySignalTests`. -- The shared test `StubDriver.GetHealth()` returns `DateTime.UtcNow`, so its fingerprint differs on - every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests - use a stub with stable health for exactly this reason. - -**#507 is closed as superseded, and the injection path is deleted.** Its retained code read -`EquipmentNode.DriverInstanceId` and `EquipmentTags`, both *structurally empty* in v3 -(`AddressSpaceComposer.cs`, `DeploymentArtifact.cs`), so removing the guard would have changed a log -line and injected nothing. Gone with it: `HandleDiscoveredNodes`, `PartitionDiscoveredByDeviceHost`, -`ApplyDiscoveredPlansForDriver`, the redeploy re-inject tail, `DiscoveredNodeMapper`, -`AddressSpaceApplier.MaterialiseDiscoveredNodes`, the connect-time discovery loop -(`StartDiscovery`/`RediscoverTick`/`DiscoveredNodesReady`/`TriggerRediscovery`), and the 18 -`DiscoveryInjectionDormantV3` tests — v2 characterization tests asserting an equipment-rooted graft -(`EquipmentRootNodeId == "EQ-1"`) that v3 cannot produce. `ITagDiscovery` itself stays: the `/raw` -browse picker drives it through `Commons/Browsing/DiscoveryDriverBrowser`, which never went through -the actor. - -**`IRediscoverable` is the ONLY tag-change signal, and four drivers do not raise it.** A periodic -tag-set drift check briefly existed (shipped `e77c8a35`, **removed** by Gitea **#523**): every -`SupportsOnlineDiscovery` driver was re-browsed on a 5-minute timer and the result diffed against its -authored tags. It was taken back out because it generated recurring, **unconfigurable** (interval was a -constructor parameter with no appsettings key), **never-live-gated** outbound traffic to real -PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the -design treated it as an implementation detail. - -⚠️ **Consequence to know before you rely on rediscovery:** **AbCip and FOCAS browse a live backend but -implement no `IRediscoverable`, so they have no tag-change signal at all** — a PLC program re-download -or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT, -MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an -event-driven signal, or one that runs when something *else* already indicates change (a reconnect, a -deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`TagSetDriftDetector`, -`DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` / -`DiscoveryStreamIncludesAuthoredTags`. - -**`IHostConnectivityProbe` is consumed (Gitea #521, closed).** `DriverInstanceActor` subscribes -`OnHostStatusChanged`, pulls `GetHostStatuses()` fresh, and the host set rides `DriverHealthChanged` to the -AdminUI `/hosts` page — no SQL: the `DriverHostStatus` table is dropped, and so is -`DriverInstanceResilienceStatus` (#524; the live data goes over the `driver-resilience-status` telemetry -channel only). **Multi-host dispatch is wired too:** `DriverInstanceActor` resolves the breaker/host key -per call via `IPerCallHostResolver` (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip). - -### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed) - -Until #527, **`DriverInstanceActor.DisconnectObserved` had no production sender.** The only paths into -`Reconnecting` were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected -and then lost its device stayed `Connected` indefinitely — publishing Bad for every tag, never -re-initialising, and **not recovering when the device returned** (live-confirmed: a Modbus driver against a -stopped fixture needed a node restart). `PublishHealthSnapshot` read `GetHealth()` for *reporting only*. - -The `Connected` health tick now runs a watchdog that self-sends `DisconnectObserved` when a driver reports a -non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. Things to know before touching it: - -- ⚠️ **The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal.** - `Degraded` means *connected but some data is bad*, and several drivers write health **per tag inside the - read loop** (`ModbusDriver.ReadAsync`), so one unreadable address latches `Degraded` forever against a - perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely. - `LastSuccessfulRead` is the honest evidence — preserved across `Degraded`/`Faulted` writes, advanced by any - success. Each conjunct has a test that goes red when the *other* is deleted. -- **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across - all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect - supervisor, OpcUaClient via a bad session keep-alive (#670 — it used to stay `Healthy` forever; its SDK - `SessionReconnectHandler` is deleted, a lost session publishes `BadNoCommunication` once per outage and the - watchdog's re-init is the recovery; good keep-alives + notifications advance `LastSuccessfulRead`). `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has - no transport. Re-check this if a new driver lands. -- **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.** - `ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable` - field inside `InitializeAsync` without disposing the old one, so re-initialising without `ShutdownAsync` - leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose - first: `ModbusDriver.InitializeCoreAsync` releases `_transport` **and** the `_probeCts`/`_reprobeCts` - probe loops before rebuilding them (`c3970a2e` + the lifecycle gate in `409a58b6`), deliberately *not* - via the shared `TeardownAsync()` because that would also drop `_poll` and every active subscription with - nothing to re-drive them; `MqttDriver.InitializeCoreAsync` calls its (idempotent) `TeardownAsync()` first, - which reaches `_connection` and the primary-host connection (`2b4b3fcb`). The path is no longer latent — - the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new - `IAsyncDisposable` field on these two, must repeat the dispose-before-overwrite. -- The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` / - `DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy - system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped - reading. -- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises - `ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient: - keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed - connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the - SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only - while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with - the watchdog's conjunction. `docs/DriverLifecycle.md`. - -- **What comms loss now does downstream — one contract, six seams (archreview R3-02).** "A value consumer - never reports a quality better than its source's" is stated once in `docs/DriverLifecycle.md` §"The - quality contract" and implemented by `Core.Abstractions.InputQuality` (`WorstOf`/`IsUsable`). Three - things changed with it, all client-visible: - - ⚠️ **Value nodes of a comms-lost driver now go Bad.** `DriverHostActor.FanCommsLossToValueNodes` - stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained - server-side**, via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host - holds no value cache, so only the node can keep the value). ⚠️ Retention is **not client-observable**: - the SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read/sample, so clients see - `Value = null` + the Bad status; the retained value is what the next Good re-publish and the - `(value, status)` dedup compare against. #481 had wired comms loss to the mux and to Part 9 - conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop - the tag" (ScadaBridge) will see more Bads — that is the fix working. - - **Nothing Good is written on reconnect** — recovery is the driver's own re-publish (the poll engine's - `forceRaise`, MQTT's new initial push). Instead the host **audits**: refs still latched 30 s after a - reconnect log a Warning and increment `otopcua.driver.comms_latched_refs`. Non-zero on that counter - means a driver reconnects without re-publishing. - - **Every dedup on the recovery path is keyed on (value, status), not value.** A device that comes back - usually comes back holding the same reading, so a value-only dedup swallows the recovery and the node - stays Bad forever — the v3-gate trap, now closed in the Calculation driver, the virtual-tag actor and - (via a connection-loss stale) MQTT's plain-mode suppression. **Any new consumer of driver values must - repeat this**; a regression test for it must replay the IDENTICAL value or it proves nothing. - - Consequences per seam: MQTT/Sparkplug stale on connection loss and push cached snapshots on every - (re)subscribe (`docs/drivers/Mqtt.md`); the Calculation driver and virtual tags **hold** while any - input is Bad and inherit the worst input's sub-code, evaluate through Uncertain, and hand scripts the - input's real `StatusCode`/`SourceTimestampUtc` (`docs/drivers/Calculation.md`, `docs/ScriptEditor.md`) - — so a script that relied on receiving `null` for a dead input no longer runs at all while it is Bad. - -See `docs/DriverLifecycle.md` §"The comms-loss watchdog" and §"The quality contract". - ## mxaccessgw The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3. ## Build Commands -```bash -dotnet restore ZB.MOM.WW.OtOpcUa.slnx -dotnet build ZB.MOM.WW.OtOpcUa.slnx -dotnet test ZB.MOM.WW.OtOpcUa.slnx # all tests -dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests # a single test project -dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod" # a single test -``` - `Category=SlowUnit` (#606) tags the 16 wall-clock-bound Driver.Mqtt unit tests (blackhole connect-timeout / reconnect-backoff windows, ~1 min of the project's 73 s). CI runs them (its filter excludes only E2E + LiveIntegration); for a fast local loop add `--filter "Category!=SlowUnit"`. @@ -471,778 +340,9 @@ See `docs/v2/dev-environment.md` for the full inventory and rationale. The server supports configurable OPC UA transport security via the `OpcUa:EnabledSecurityProfiles` list in `appsettings.json`. Phase 1 profiles (the `OpcUaSecurityProfile` enum members): `None` (default), `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`. Security policies are built from the enabled profiles by `BuildSecurityPolicies` at startup. The server certificate is always created even for `None`-only deployments because `UserName` token encryption depends on it. See `docs/security.md` for the full guide. -## Redundancy - -The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide. - -**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.** - -- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is CLOSED** — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (`c50ebcf7`; `docs/plans/2026-07-24-mesh-phase7-failover-drills.md`). -- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**. - -**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". - -**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard". - -Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. - -**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below. - -**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences: - -- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`. -- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh. -- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it. -- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists. -- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip. -- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4). - -See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. - -## Role authority (`EffectiveRoles`) - -**A node's roles are resolved in exactly one place** (archreview R3-04, closing 01/S-12 = 03/C5 and -01/C-9): `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`), with one precedence — -`Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`, else empty — and two-layer normalization: a -plain role (`admin`/`driver`/`dev`) is trimmed and lower-cased, but a cluster role's ClusterId suffix -(`cluster-SITE-A`) is kept verbatim, because lower-casing it would rename the Akka role of a running -deployment. An unknown role from **either** source fails host start. `Program.cs` resolves one -`EffectiveRoles` instance at the top of the composition root and registers it as a singleton; that -SAME instance drives both the conditional host wiring (`hasAdmin`/`hasDriver`) and -`AkkaClusterOptions.Roles` (what Akka gossips) — there is no second derivation for the two to -disagree about, which is why no runtime "do the views agree" assert exists. Before this, `Program.cs` -read `OTOPCUA_ROLES` only, so a node configured solely through `Cluster:Roles` gossiped the right -Akka roles while `Program.cs`'s own wiring mounted none of them — the inverse of the pre-#532 field -incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`). - -- **A role overlay may not itself set `Cluster:Roles`.** Roles are resolved *before* the per-role - appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing - `Cluster:Roles` would leave the bound options disagreeing with the registered instance. - `AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views — a no-op - today since none of the four shipped overlays carries a `Cluster` section. -- **Both sources set to different normalized role sets logs a Warning** at boot (`Cluster:Roles` - still wins outright) — the boot banner also names which source won (`source=Configured` / - `Environment` / `None`). -- **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of - production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through - `EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role - during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:** - `RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, - which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the - library nor with a local case-insensitive pre-filter, which would split the Primary election from - `ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade - window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)"). -- **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to - stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s - — leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException` - (design-time tooling) passes through untouched. -- **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement: - a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads - `OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal — - with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same - English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI - RBAC role). - -See `docs/Configuration.md` §`Roles`, `docs/ServiceHosting.md` §"Role gating", and -`archreview/plans/R3-04-one-role-authority-plan.md`. - -## Mesh command transport (`MeshTransport`) - -**The Akka `ClusterClient` transport described in this section's earlier history is DELETED** -(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md), -Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension, -`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails -validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values) -rather than silently falling back to `Dps`. What remains is exactly two transports, selected by -`MeshTransport:Mode`: - -- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**: - it works only when central and the node share a gossip ring, which after Phase 6 means within one - 2-node mesh. -- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node - carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a - node on anything else. - -Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`, -served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev -rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port -of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is -every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort` -and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and -no rotation: each command goes to the specific node dialled, which is what makes per-node delivery -provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer -applies to either direction. - -Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role -config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than -claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and -stops at the first success. - -Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per -application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets -store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed -on every unresolvable case — constant-time compare, no key material logged. Production keys are -`${secret:OT-MESH-PSK-}`; the docker-dev rig supplies three literal dev keys via env -(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`). - -Things that carried over unchanged from the `ClusterClient` era, because they were never -`ClusterClient`-specific to begin with: - -- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node - and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across - nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery. - The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that - set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't. -- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a - DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`, - `ScriptedAlarmHostActor`) accept a command from either transport identically. -- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a - deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()` - replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient - deletion — it was always about the deploy path, not the transport. - -`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which -makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` -§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) / -[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) / -[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history — -those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md` -is the original Phase 2 design doc and is likewise left as a historical record of the transport this -section now describes as deleted. - -**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: - -- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag. -- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out. -- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist. -- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again. - -## Config source (`ConfigSource` / `ConfigServe`) - -Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its -deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from -central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact -bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in -LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via -`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by -a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`, -fail-closed). Things worth knowing before touching it: - -- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an - admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same - dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport). -- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision - hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the - `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment` - stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch. - **The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** — - `ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and - hashes the result, so the invariant holds unchanged and nothing between the composer and - `DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever - looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so - pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every - decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already - handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after - upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be - upgraded BEFORE central starts emitting compressed blobs.** -- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA - mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, - and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a - `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an - empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**. - **Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and - the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does - **not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches). -- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading - SQL today; `StoreAsync` is idempotent and the cache replicates). -- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with - both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only - ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel - `Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about). -- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState` - ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those - is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts. - -On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays -hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching -key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is -no `OTOPCUA_CONFIG_MODE` env var in the compose file. See -`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and -`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`. - -## Live telemetry transport (`Telemetry` / `TelemetryDial`) - -Per-cluster mesh **Phase 5** (merged `35552a96`; live gate PASSED `f134935f`) added a second -transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and -`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode` -(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's -behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof` -event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts -the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring -ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split -(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before -touching it: - -- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`), - `driver-health` (`DriverHealthChanged`), `driver-resilience-status` - (`DriverResilienceStatusChanged`) — feeding the exact same central sinks - (`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in - either mode. -- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the - delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from - `Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an - observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport - (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not - telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542): - `FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end** - (no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its - `DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just - central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above. -- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits - into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC - server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only - `TelemetryDial:Mode` on central actually switches anything: which of the two always-available - sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what - makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as - [MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve). -- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` == - `TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor` - (`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects - every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture - for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern. -- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from - `ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate - first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event - is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side - stores are not evicted). The node-local hub - replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe - so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and - tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1 - node stream is up) — the same semantics the DPS pill has always had. - -See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md` -§`Telemetry`/`TelemetryDial` for the config keys, and -`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live -gate procedure. - -## LocalDb pair-local store (Phases 1 + 2) - -Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired -LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**: -`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2). - -**Phase 1** caches the -**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a -driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes -the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`, -in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by -`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` + -`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes -never register it.** The cache can optionally replicate to the node's redundant pair peer over the -library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated -on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the -docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is -the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret -exception). - -**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db` -and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm -history no longer takes it to the grave. `SqliteStoreAndForwardSink` became -`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key -`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely -to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy -commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the -sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that -`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new -`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by -design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the -same event accepted on both nodes — which happens in every boot window, since -`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the -docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a -deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue. - -**The library pin is `0.3.4`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and -`.Replication`), bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15). -The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3 -upgrades in place on open, so no rig or volume action was needed. `RegisterReplicated(string)` is -unchanged and everything 0.2.x/0.3.x added is additive, **except** that `RegisterReplicated` (0.3.3) -now throws `LocalDbRegistrationException` for a table that is the parent of a row-mutating ON -DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire. -0.3.0 made the snapshot-resync flag **per-table** (see the ⚠️ below, now corrected); 0.3.2 added -validated config `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, left -unset here) and a public `LocalDbOplogGapException`, neither yet wired to a call site. **Its two -new APIs — `DeregisterReplicated` and `RegisterReplicated(table, baselineExistingRows:)` — are now -both used** (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two -things are worth knowing: - -- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching - **byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized - under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row - cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which - materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps - `16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the - message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" - was corrected with the bump; do not restore that rationale. -- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).** - `LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges` - (via `LocalDbSetup.ReplicationIsConfigured`): - **`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is - not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it - alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` + - `LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only, - silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is - fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has - nothing to capture for. - - ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`: - `ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these - rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no - peer must still drain, so the two predicates are deliberately not the same. - The **passive** half (no `PeerAddress`) cannot name its peer, so it stands down for its **mesh partner** - (the other member of its `cluster-{ClusterId}` role); a legacy passive half with no mesh role still - drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the - published Primary verdict (and the live data-plane gate, which must stay the same decision) **allows** — - for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the - Sparkplug primary-host claim with its Primary peer. - - **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the - same load-bearing DDL→register order. Baselining is what makes enabling replication on a node - that has been running *without* it actually converge: capture is CDC and the snapshot streamer - pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in - used to replicate only writes made after the restart. It seeds the ledger at the LWW floor - (HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote - write of the same key and wins only where the peer has no version at all. It is idempotent, so - it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.4`) the snapshot - flag is **per-table**, not per-database — baselining one table no longer streams every - registered table in both directions. (Prior to `0.3.0`, on the `0.2.1` pin this repo carried - until 2026-09-03, the flag was per-database and baselining anything streamed *every* - registered table in full; the old advice to trim `alarm_sf_events` before enabling replication - no longer applies at the current pin.) - - **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time - self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and - prunes that table's oplog + row-version rows. Merely declining to register would leave every - pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row - inside every write transaction — on precisely the node that has no peer to capture for. The - call returns `true` when it removed something, so the Information line fires **once, and only - when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is - Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where - resolving one from DI would nest a resolution inside a singleton construction; `Program.cs` - assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe - the line rather than infer it. - - ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake - compares registered-table digests **fail-closed**, so a node that deregisters a table its peer - still replicates stops syncing with a schema mismatch. Enabling replication on an established - pair means setting the config and **restarting both nodes** — do it on both, or on neither. - - **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was - enabled from a fresh pair, so the impact of the old gap was low — central and site-b are - default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests` - pins both branches (registered-table set on a configured node, empty set + zero triggers + - the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged. - -See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config -cache), `docs/AlarmHistorian.md`, and the runbook -`docs/operations/2026-07-20-localdb-pair-replication.md`. - -## Phase 4 — cut the driver-side ConfigDb connection - -Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles` -has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs` -gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is -unaffected (it keeps ConfigDb via its admin role). Consequences: - -- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator` - fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused - node may stay `Direct`. -- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not - spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db - factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and - `Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes - **240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture - — where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself - is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less - (driver-only) nodes". -- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore` - (replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces - `EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed - `ScriptedAlarmState` table **was dropped** (Task 9, `3a590a0c`, migration - `20260723183050_DropScriptedAlarmStateTable`); `EfAlarmConditionStateStore` is deleted. -- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on - a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is - the system of record. -- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty - `NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so - `OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only — - the same rows a driver node never received via config either, so this is not a regression. - -Merged to master (`26fad75c`); **live gate PASSED** (`1281aebf` — driver nodes run ConfigDb-free) and -Task 9 landed (`3a590a0c`). See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and -`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4. - -## Authentication: API keys on the data plane, LDAP on AdminUI (#520) - -⚠️ **Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all.** It is machine-to-machine, -so a client presents an OPC UA **UserName** token with the fixed username **`apikey`** and an -`otua__` credential as the password. `ApiKeyOpcUaUserAuthenticator` -(`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/`) implements `IOpcUaUserAuthenticator`: one indexed lookup -on `keyId` plus one peppered HMAC-SHA256, `FixedTimeEquals`, **no network I/O**. The key resolves to a -**principal**; `NodeAcl.Principal` rows grant it permissions. `LdapOpcUaUserAuthenticator` is -**deleted**, along with four `Security:Ldap` keys that existed only for it (`AuthTimeoutMs`, -`MaxConcurrentAuthentications`, `OutageFailureThreshold`, `OutageCooldownSeconds`) — they bounded an -LDAP bind running inline in the SDK's synchronous, `SessionManager`-lock-holding impersonation -callback, and nothing on the remaining LDAP path has that property. - -**LDAP remains, for AdminUI login only** (`Security:Ldap`, `OtOpcUaLdapAuthService` → -`ILdapAuthService`, group→role mapping via `GroupToRole` + `LdapGroupRoleMapping`). Do not "restore" -the data-plane authenticator when you see LDAP wiring in `Program.cs` — it is there for the AdminUI. - -Two things to know before relying on any of this: - -- ⚠️ **Node ACLs are enforced only behind `Security:Acl:Enabled` (default `false`).** With the flag - **off**, a key's grants collapse into the two server-wide role gates (`WriteOperate` on Value writes, - `AlarmAck` on Part 9 methods), **only `Cluster`-scoped grants count** — `ApiKeyInterimRoleMapper` - deliberately under-grants, because honouring a narrower grant's flags while unable to honour its - *scope* would turn "WriteOperate on Equipment X" into server-wide write authority — and **read, - browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous**. With the - flag **on** (`DataPlaneAuthorizer`), scoped grants decide **every** data-plane operation: write and - alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6). - - **The two role gates re-check the session's KEY on every call, flag off or on (Gitea #563).** Roles - are resolved once at activation, so an expired / deployment-disabled key used to keep `WriteOperate` / - `AlarmAck` on its open session until the client disconnected. `DataPlaneAuthorizer.AuthorizeKeyUsable` - runs the same `FindUsableKey` probe against the same snapshot + clock as the ACL path (one dictionary - lookup, no I/O) and denies `BadUserAccessDenied` before the role check. **Only those two gates** — - reads/subscriptions stay ungated with the flag off, by design. Anonymous carries no key and is - unaffected. No flag: a dead key keeping write authority is a defect, not a rollout choice. ⚠️ On a - fused node a key minted after the last deploy authenticates (ConfigDb) but cannot write/ack until a - deployment carries it — the flag-on path always behaved this way. - - ⚠️ **The ACL decision REPLACES the role check — the two are never chained.** Chained, a grant would - have to confer the server-wide role *as well as* pass the scope check, and only `Cluster`-scoped - grants confer a role, so every narrow grant would stay inert and the flag would appear to do - nothing. The load-bearing test is therefore an **allow** — a session carrying *no roles* writing - successfully on the strength of its grant. A deny-only test passes against a build where the ACL is - never consulted, because the role gate denies too. - - **The trie is built inside `DataPlaneAuthorityState.Apply`**, not by a second wiring at the publish - actor, and the state owns its `PermissionTrieCache` rather than being handed one. An uninstalled - trie denies everything, which is indistinguishable from working enforcement until someone checks - that an ALLOW is possible at all. - - **A deploy reaches sessions that are already open**, in both directions: each decision stamps the - *current* generation rather than the one the session activated under, and re-checks the - authenticating key — so revoking a key or a grant stops an open session immediately instead of at - the client's next reconnect. On a driver-only node the artifact is the only revocation channel. - - `UserAuthorizationState` is built **per decision walk, not per session** — and since the hot-path - change it is allocated only on a cache **miss**, so a cache hit never constructs one. Its LDAP-shaped - freshness/staleness predicates are switched off EXPLICITLY (`AuthCacheMaxStaleness = MaxValue`). - ⚠️ Relying on "the stamp is always fresh" was not safe: `MembershipResolvedUtc` comes from - `DataPlaneAuthorizer`'s `TimeProvider` while `IsStale` is evaluated against - `TriePermissionEvaluator`'s, so **any divergence between the two clocks denies every request** with a - verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead. - - **Stage 6's hot seams (read/browse/subscribe) go through a per-`(session, node, operation, realm)` - decision cache**, cleared wholesale when the generation advances. It is **sharded per API key** - (`ApiKeyId ?? Principal` — despite the type name, sessions sharing one key share one shard and one - budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and - not every other key's decisions. ⚠️ It caches the **grant walk only** — key revocation and expiry are - re-checked **before** the cache lookup on every call, because expiry moves with the clock rather than - with a deployment. - - **Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors.** A - grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward - only, so `PermissionTrieNode.BrowseTransitPrincipals` marks the path. Planting a real `Browse` grant - on each ancestor instead would be collected by the additive downward walk and leak `Browse` onto - every sibling subtree beneath that ancestor. ⚠️ **The topmost ancestor is the single shared root - folder** (`s=OtOpcUa`, raw namespace) that BOTH realm trees hang under — its identifier is a display - name, not a path, so `DataPlaneAuthorizer.BuildScope` maps it onto the trie's **zero-segment - realm-root scope** (`V3NodeIds.SharedRootFolderId`, realm-qualified: in the UNS namespace that - identifier is an ordinary Area), and its Browse considers transit in **either** realm because it - fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse - gate denies the whole call when its source node is denied, so clients were stopped at the front door. - - **Revocation reaches live subscriptions** (stage 6): every other operation re-asks per request, but a - subscription is authorized once and then streams — and streaming is how machine consumers actually - read this server. `DataPlaneAuthorityState.Changed` fires `RevalidateMonitoredItems`. ⚠️ A denied - item is queued `BadUserAccessDenied` and then **detached from its `MonitoredNode2` fan-out — NOT set - `MonitoringMode.Disabled`**: disabling discards the item's queued notifications, so the client is - silenced without ever being told why (caught only by the over-the-wire test). Detaching is also - reversible, so restoring the grant resumes the feed with no client action. ⚠️ **`Changed` is no longer - the only trigger (arch-review 03/S19):** `AclRevalidationTimer`, armed by `OtOpcUaServerHostedService` - at `Security:Acl:RevalidationIntervalSeconds` (default 60 s, `0` disables, inert while - `Security:Acl:Enabled` is false), runs the same pass on the clock — because `ExpiresUtc` passes with - the clock, not with a deployment. The pass's verdict memo is **pass-local and must stay that way**; - hoisting it across ticks would serve an expiry that landed between two ticks the earlier tick's allow. - **Event/alarm items ride both triggers too (Gitea #562)** — a second map (`_eventItems`), fed by the - `SubscribeToEvents` override, holds every *direct* notifier subscription; a denied one is detached via - the SDK's own protected `SubscribeToEvents(…, unsubscribe: true)` (what a client delete does) and - re-attached with `unsubscribe: false` on restore — again never `Disabled`. Unlike a data item it is - **not told** (no value slot for a status; the feed goes quiet), and a client delete of a stood-down - item is absorbed by the override (the SDK would otherwise remove an item that is not attached). - **Server-object (`i=2253`) event subscriptions are filtered per event at delivery (Gitea #586).** The - `SubscribeToAllEvents` override attaches them to **nothing** (just records them in `_allEventsItems`); - each condition's own `OnReportEvent` (`OnConditionReportEvent`, fires once per event ⇒ still one - Server-object copy) queues the event to each recorded item whose session holds `Subscribe` on the - condition (RawPath) **or** any notifier folder it reports through — the per-folder path's question. The - root-notifier forward to the core Server object (`OnReportEvent` override) is **dropped**: it was the - ungated path, and it is the only one that served an item created before the alarm was deployed (the - SDK's `AddRootNotifier` subscribes existing all-events items with `unsubscribe: true`). Revalidation is - **implicit** — every event re-asks the authorizer, so a deploy or expiry lands on the next event with no - pass. `ConditionRefresh` on an `i=2253` item is filtered the same way. The dispatcher **must** keep - the SDK's audit guard (`AuditEventState` only while `Server.Auditing` over SignAndEncrypt/https), - because conditions report their Part 9 method audit events through themselves. - `ReconcileAllEventsMonitoring` must keep `AreEventsMonitored` raised on every condition while any - `i=2253` item exists, because `ConditionState` gates its method **audit** reports on - `EventsMonitored()` (method-driven state-change events no longer depend on it — #587, below). The - protected `SubscribeToEvents` override that no-ops all-events items is what - stops the SDK's `AddRootNotifier` (`unsubscribe: true` bug) taking that unit back. Both review - findings (review of `75e559f6`). ⚠️ The materialise path raises only the **new** condition's unit - (`RaiseAllEventsMonitoring`); a full reconcile there made a deploy of N alarms O(N²) under `Lock` with - ScadaBridge connected — keep the full reconcile to remove/rebuild/(un)subscribe. - **Model-change and audit events are gated too (Gitea #588).** The manager's own server-wide events - (`GeneralModelChangeEvent`, the rejected-write `AuditWriteUpdateEvent`) go through - `ReportToServerObject`: flag **off** ⇒ `Server.ReportEvent` as before; flag **on** ⇒ delivered per item to - `_allEventsItems` (same audience), a model change narrowed to the `Affected` nodes the session may - **Browse** (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral - one), an audit write only with `Read` on its source node plus the SDK's audit/encryption guard. Any other - event type fails closed. Auditing is off by default (never configured in this repo). -- **ns=0 standard nodes stay ungated by design, even with `Security:Acl:Enabled`** (#617) — `ServerStatus`, - `NamespaceArray`, `ServiceLevel` etc. are served by the SDK's core node manager outside our authorizer and - clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node, - event subscription and HistoryRead is still gated. `docs/security.md`. -- **Equipment-node writes: `EvaluateEquipmentWriteStructure` is the ONLY type check.** `OnWriteValue` bypasses - the SDK's own type check on every writable equipment node, so `OtOpcUaNodeManager.EvaluateEquipmentWriteStructure` - (plus `EvaluateEquipmentWriteRange`) is the whole gate. Scalars are deliberately lenient (drivers - range-check; ScadaBridge sends Double to a Float node). **Array writes** (#610): a numeric array of another - element type is **converted, checked, element by element** to the node's element type (ScadaBridge writes - CLR `int[]`/`double[]` lists) — any out-of-range, fractional-to-integer or NaN/∞ element refuses the whole - write `BadTypeMismatch` with nothing sent, so a `UInt32` > `Int32.MaxValue` can never reach a Galaxy Int32 - attribute as reinterpreted bits; non-numeric mismatches are refused (`Byte[]` ≈ ByteString); wrong length ⇒ - `BadTypeMismatch`; an array to a **scalar** node is refused only on Modbus/S7 (`EnsureVariable`'s - `refuseArrayOnScalar` — pass-through drivers keep their browse-committed scalar-shaped array tags writable; - a new sink wrapper must forward that argument, and the reflection guard checks only that the call arrives); - `IndexRange` writes ⇒ `BadWriteNotSupported`, `ValueRank > 1` out of scope. An - array node is writable only when the driver can write the array (`DriverArrayWriteSupport`: Modbus bounded - by `maxRegistersPerWrite`/coil cap, S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types **with a - declared `arrayLength`** — an undeclared length stays read-only because a short whole-array write would - resize the Galaxy attribute); `arrayLength >= 1` is required, and the node's `Writable && (!IsArray || - ArrayWritable)` is projected onto the raw node and every UNS mirror. -- **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`, - preserving today's behaviour) controls *admission*; when admitted the session authenticates as the - reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as - the same `RoleCarryingUserIdentity` a key produces. Set it `false` to make keys mandatory. A key may - **not** use the `anonymous` principal, and a failure to read anonymous's grants **denies** rather - than admitting with no roles. -- **Keys reach a driver-only node inside the deployment artifact** (stage 4). `ConfigComposer` snapshots - the `ApiKey` rows — the peppered HMAC only, never a secret, and the pepper is node config that is - deliberately *not* in the artifact — and `DeploymentArtifact.ParseDataPlaneAuthority` decodes them - node-side alongside the cluster's `NodeAcl` rows. `OpcUaPublishActor` installs both on - `IDataPlaneAuthoritySource` on every apply, **before** its `plan.IsEmpty` early-return, so a deployment - that only revokes a key still lands. A node with a ConfigDb keeps `ConfigDbApiKeyStore` (per-activation - read, wrapped — see below); a driver-only node gets `ArtifactApiKeyStore`. ⚠️ **The asymmetry is - ACTIVATION-scoped, not session-scoped — this line used to say "immediate on a ConfigDb node" and that was - wrong.** Clearing `Enabled` in the ConfigDb stops the **next session activation** on an admin/fused node - (bounded by `Security:ApiKeys:StoreCacheSeconds`, default 5 s) and takes until the next deployment on a - driver-only one — but an **already-open** session re-checks its key against the in-memory *artifact - snapshot* (`DataPlaneAuthorizer.Evaluate` → `snapshot.FindUsableKey`), never against `IApiKeyStore`, so on - **both** node kinds an `Enabled` revocation reaches an open session only at the next deployment. - **Expiry is the exception**: `FindUsableKey` compares `ExpiresUtc` against the authorizer's clock on every - call, so it is immediate everywhere — and since arch-review 03/S19 it reaches live subscriptions too, on - the host's `Security:Acl:RevalidationIntervalSeconds` timer (default 60 s; the session is **silenced with - `BadUserAccessDenied`, not closed**, and recovers on re-activation). Direct event/alarm subscriptions - are revalidated on both triggers as well (#562); with the flag **off**, expiry and revocation reach the - two role gates (#563) but not reads or subscriptions. -- ⚠️ **The activation lookup is BOUNDED (arch-review 03/S18).** The SDK raises impersonation synchronously - under a `SessionManager`-wide lock, so a ConfigDb that is *slow* rather than *refusing* stalled **every** - session activation on the server, Anonymous and X.509 included. `BoundedApiKeyStore` wraps the ConfigDb - store — and only it, never `ArtifactApiKeyStore`, which is two in-memory dictionary probes — with - `Security:ApiKeys:StoreTimeoutMs` (2000; exceeding it **denies**, via the authenticator's existing - catch-all) plus a `StoreCacheSeconds` (5) read-through cache. Faults and timeouts are never cached, a - cached hit whose `ExpiresUtc` has passed is re-read rather than served, and a timeout logs a distinct - Warning + `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. Both knobs at `0` restore the old - behaviour exactly. -- ⚠️ **The auth generation is a node-local COUNTER driven by the `RevisionHash`, not derived from its - bits.** `PermissionTrieCache` treats the highest `GenerationId` as current, so a generation taken from - hash bytes would install a *lower* number about half the time and the cache would go on serving the - previous deployment's ACLs with nothing logged. Pinned by - `DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases`. -- Note `OtOpcUaServerHostedService` takes `IOpcUaUserAuthenticator` **and** `IDataPlaneAuthoritySource` - as **required** ctor parameters — the built-in DI container does not fall back to a default value for - an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a - node with neither role) or it does not boot. - -**Grants are authored through a scope picker, not free text (stage 7).** `/clusters/{id}/acls` opens -`AclScopePickerModal` over the cluster's raw + UNS trees, and selecting a node supplies `ScopeKind` -**and** `ScopeId` together — so `RawDevice` + a TagId, a cross-cluster scope, a `Cluster` grant carrying -an id, and the retired `FolderSegment` kind are all unauthorable by construction rather than merely -rejected later. ⚠️ **A `Cluster` row must produce a NULL `ScopeId`** — the deploy gate rejects one that -carries an id (`NodeAclScopeIdUnexpected`), so a picker that helpfully filled in the cluster id would -author rows that cannot be deployed. `IAclScopeService.CheckScopeAsync` mirrors -`DraftValidator.ValidateNodeAclScopes` rule for rule and **blocks the save** when a scope resolves to -nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches -the picker); the list page marks such rows `unresolved`. **UNS tags are deliberately not offered** — a -UNS signal is a `UnsTagReference`, and a `Tag` grant already covers the raw node *and* every UNS -projection of it. The picker is `AclScopeTree`, a separate component from `RawTree`, which already -carries two modes. - -**All eight stages are done** (`d4389f57` enforcement, stage 7 authoring after it), each live-gated on -docker-dev. The rationale for every decision above is in -`docs/plans/2026-07-31-apikey-acl-enforcement-design.md`. See `docs/security.md` for the full guide. - -Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.) - -## AB drivers: native transport (`Transport`) - -AbCip and AbLegacy have a second, managed wire transport (`Driver.Ab.Transport`) beside libplctag, chosen by -the driver-config key `transport` = `Native` | `Libplctag` (string; takes effect when the driver is -**constructed** — every process start, or a deploy that recreates it, not `ReinitializeAsync`; AdminUI -"Transport" dropdown; CLIs `--transport`). "Default" is encoded by omitting the key; the AdminUI must never -re-emit a concrete default. The defaults differ per driver (final review, 2026-10-07): - -- **AbCip: an absent key resolves to the node default, `Native` unless pinned.** Precedence (one place, - `AbCipTransportResolver`, shared by driver, probe and CLI): explicit per-driver key → node-level appsetting - **`Drivers:AbCip:DefaultTransport`** (`Native` | `Libplctag`, env `Drivers__AbCip__DefaultTransport`, - `AbCipDriverDefaultsOptions`; a bad value **fails host start**) → compiled `Native`. Bound unconditionally - because the driver factory AND the admin-node Test Connect probe both read it. ⚠️ So **the upgrade restart - alone moves no-key AbCip drivers to Native**; to stay on libplctag, set the node key — NOT a per-driver key - through the old AdminUI, whose typed form drops unknown keys (#576). Evidence: AbCip reads proven on a real - CompactLogix (Z-35162); scalar native writes (DINT/BOOL/REAL) proven on it once (2026-10-09, user-approved one-off, restored) — and native UDT member writes (DINT/BIT/REAL, #611 gate, same day) and a native STRING write (hw5, same day, restored byte-exact); and a native array-element write (`EM01_PowerUnit._SEQ_EP[31]`, user-approved, an element no alias targets, restored; neighbours unchanged); libplctag member and fragmented writes are `ab_server`-only. #566: a real Logix drops TCP on a connection idled past its timeout (no 0x0107/0x0203); the faulted session reopens on the next request. -- **AbLegacy: an absent key resolves to `Libplctag` for every family; `Native` is opt-in** (no node key). On - a real MicroLogix 1400 (Z-33323, idle, 2026-10-09) native read S/N/F/B/T/C/R/ST/O/I incl. `S:0`, T/C/R - sub-elements and status bits, and ST byte order, all correct, and wrote N/F8/B-bit (restored). libplctag (the default) - used to fail `S:`/`O:`/`I:` and `R` status bits there (incl. the default probe `S:0`) — its `pccc.c` needs explicit - file numbers and has no R status-bit mnemonics; since #659 `LibplctagLegacyName` translates (`S2:`/`O0:`/`I1:`, - `R6:0.CON/13`) and both transports read identically on that press. The authored grammar is unchanged. Read-side - only: a libplctag WRITE to any address the translation changes (`S:`/`O:`/`I:` incl. bit writes, R `EN…FD`, `C…UA`) - is refused `BadNotSupported` before any runtime is created (none could reach the PLC before — no write-surface - widening on the default transport); libplctag T/C status-bit writes it parses itself are unchanged. - L/A files, function files, SLC, PLC-5 and `logixpccc` remain unverified natively. Explicit Native on PLC-5/`LogixPccc` logs a once-per-instance Warning. - AbLegacy native refuses status-bit sub-element writes (`T4:0.DN`…, `BadNotSupported`). - -Refusals are not comms faults: an Unsupported refusal (`BadNotSupported`) and an AbCip native **type -mismatch** (`BadTypeMismatch`, where libplctag returned Good garbage for a same-width mis-authored type) both -keep the tag handle, leave driver health alone (no `Degraded`, nothing for the comms-loss watchdog) and warn -once per tag — only genuine transport failures evict and degrade (a runtime `NotSupportedException` too, #611 — -it used to latch `Degraded`). AbCip width-checks every scalar **write** against the controller's bytes before -sending, on both transports (#611; libplctag used to patch part of its buffer and send it all). Controller-discovered -UDT members are writable once browse-committed and authored `ReadWrite` (BIT members included — the controller -applies the bit) — native member writes incl. a BIT member are live-gated on Z35162 (2026-10-09); sibling-bit preservation and the libplctag path are ab_server-only; whole-UDT writes are refused `BadNotSupported` by decision (`docs/drivers/AbCip.md` §Member writes). `DeviceSessionPool.Shared` logs through the -host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT factories' `Register(registry, ILoggerFactory?)` take one too, #602 CONV-3). The AbCip `browsePageDelayMs` knob has an AdminUI field (`AbCipDriverForm` "Browse page delay (ms)"). libplctag is confined to `Libplctag*.cs` adapters by -`LibplctagConfinementGuardTests`; do not reference it elsewhere. See `docs/drivers/AbCip.md` §Transport, -`docs/drivers/AbLegacy.md` §Transport + §Wire evidence, `docs/Configuration.md` §`Drivers:AbCip`, and -`docs/plans/2026-10-07-native-ab-transport-gate-results.md`. - -## Other driver notes - -- **S7:** connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe - tick); a write-side access-denied (PUT/GET disabled) latches a **sticky Faulted** that outranks later read - passes; driver-side array writes cover non-Bool narrow/`Int32`/`Float32` types only — Bool arrays are - refused (a one-byte-per-element encode would clobber neighbour bits). -- **MQTT:** Plain `writeTopic` writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug - primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types - `BadNotSupported`). -- **No driver-side history (#655):** `IHistoryProvider`, `DriverCapability.HistoryRead`, `HistoryReadStatusException` and - the OpcUaClient history passthrough are deleted — nothing ever dispatched HistoryRead to a driver; it is served only by the - server-wide `IHistorianDataSource`. `isHistorized` on an OpcUaClient tag means OtOpcUa records it, not that upstream - history is proxied. A stored `ResilienceConfig` still carrying a `HistoryRead` key parses (diagnostic only). -- **FOCAS / AbLegacy:** background loops are joined (≤ 5 s) on shutdown. -- **FOCAS (#257/#266):** `cnc_statinfo` flags are a `Status/` fixed-tree subtree (`FIXED:Status/Run`…, capability-probed); - adjacent PMC tags in one poll batch coalesce into one `pmc_rdpmcrng` (`PmcCoalesce` config, default on) — a refused range - falls back to per-tag reads, a transport failure fails all members without retry. Parts count stays `PARAM:6711` (docs only). -- **FOCAS alarms (#647):** an `ALARM:` address (`*`, ``, `/`, `/-`, `T`, `OTHER`) + - the `alarm` object = ONE condition at the RawPath (either alone is skipped + reported). Overlapping selectors all - fire; `OTHER` = what no type selector claims. ⚠️ A failed `cnc_rdalmmsg2` THROWS (it used to return `[]` ⇒ mass - clear + un-acked re-raise) — and annotates its conditions `BadNoCommunication` once per outage (quality-only) and - degrades driver health with `LastSuccessfulRead` frozen, so an alarm-only driver trips the #527 watchdog. ⚠️ A CNC - REFUSAL (non-OK rc, transport fine) is `BadNotSupported` + one Warning and does NOT touch health — else the - watchdog reconnect-loops a production control. Health = worst of the value and alarm paths, computed together. The told - record lives on the driver INSTANCE (reconnect / in-place re-init re-raise nothing); a deploy RESPAWNS the driver, - and its Raise for a still-active alarm is a host-side re-statement (below). Type names are the 16i-era table only - (30i codes unconfirmed, not invented — use `T` there). Read-only; ack server-side. `AlarmProjection.Enabled` - is a no-op. E2E: `Runtime.Tests/FocasNativeAlarmEndToEndTests` (incl. a respawn). `docs/drivers/FOCAS.md`. -- **A Raise on an already-active condition with the SAME message is a RE-STATEMENT (#647 review):** `NativeAlarmProjector` keeps its ack and - `ForwardNativeAlarm` writes no second "Activated" `/alerts` row — every native driver's told-record dies with its - instance, and a deploy respawns the driver. Only `Retrigger` (or inactive → active) un-acks, so a source whose - UPSTREAM is authoritative for the ack sends Retrigger for an unacked re-statement: OpcUaClient on first sight of a - condition (no prior told), Galaxy for an unacked active snapshot entry. **Those two also flag - `AlarmEventArgs.IsRestatement` (#656)** — history-only: a flagged Retrigger on an Active+unacked condition with the - same message AND severity, or a flagged Clear on an inactive one, writes no `/alerts` row (every Galaxy reconnect used - to duplicate them); a flagged `Unspecified` on an already-active condition (AbCip ALMA governing-limit drop-out) - moves the node's text/severity, keeps active/acked and writes no row. Active+acked is the atomic **`AlarmTransitionKind.ActiveAcknowledged`** (Galaxy snapshot - `ActiveAcked`, OpcUaClient inactive→acked-active) — never Acknowledge (kept a cold host's `Active=false`: the alarm was - invisible) nor Raise+Acknowledge (an unacked intermediate). Its mirror **`InactiveUnacknowledged`** (#657, OpcUaClient) - replaces Raise/Retrigger+Clear (an Active flicker; rows only for what changed). OpcUaClient's first sight of an inactive - condition is a flagged InactiveUnacknowledged, or flagged Clear+Acknowledge for inactive+acked (a flagged Acknowledge on an - acked condition writes no row) — never `Unspecified`, which wrote "Activated" for an inactive alarm and left a - retained-Active condition Active. `NativeAlarmProjector` returns the rows to write; unflagged - sources (FOCAS/MTConnect, live Galaxy) are unchanged. E2E: `Runtime.Tests/NativeAlarmRestatementEndToEndTests`. -- **Re-Initialize without teardown (#630):** FOCAS, AbLegacy, AbCip, TwinCAT, S7 and OpcUaClient `InitializeAsync` - first release the prior device/session state (cancel, ≤ 5 s join, dispose — incl. the AbCip alarm projection; - FOCAS's is PAUSED, its told record kept, #647) — the device-state half of `ShutdownAsync` only; the poll engine is kept and the actor re-subscribes - after every successful init. AbCip publishes fresh tag tables by reference swap (a second init used to throw - "tag name collision"); S7 swaps `Plc` under `_gate`. -- **Data and alarm subscriptions are independent in `DriverInstanceActor` (#662):** a data re-subscribe (every - deploy's bulk push to a kept driver) detaches DATA only (`DetachDataSubscription`) — it used to unhook - `OnAlarmEvent` too, silently killing the native-alarm feed until a respawn. The health-poll reconcile - re-attaches/re-subscribes alarms as well as data, so a failed alarm subscribe is retried every tick. -- **A config change is a respawn — there is NO in-place path (#665).** The `ApplyDelta`/`ApplyResult`/`DeltaApplied` - seam (host `ApplyChildDelta`, `DriverSpawnPlan.ToApplyDelta`, the actor's Connected reinit and mid-connect - `AdoptConfigDuringInit`) was unreachable since #516 and is deleted; #663/#664 were latent bugs in it. `IDriver.ReinitializeAsync` - stays as a per-driver contract the runtime never calls. Every driver's `InitializeAsync` still serves the config it is - handed (Calculation adopts, Galaxy refuses a non-equivalent one — #664), and still releases prior state (#630). - `docs/DriverLifecycle.md`. -- **S7 (#632/#638):** STRING/WSTRING/CHAR/WCHAR and DTL/DATE/TOD/TIME/S5TIME codecs (`S7ValueCodecs`; TOD/TIME/S5TIME - surface as Int32 ms); an undecodable buffer or unrepresentable write is `BadOutOfRange` with no health impact. - Optional `localTsap`/`remoteTsap`/`connectionType` override the slot TSAP (`remoteTsap` ⊕ `connectionType`). Test - Connect reads 2 bytes of `MB0` so a PUT/GET-disabled CPU fails the probe. The negotiated PDU size is logged, on - `S7Driver.GetDiagnostics()`, and (#640) on `/hosts` via the optional `IDriverDiagnostics` bag (string map on - `DriverHealthChanged.Diagnostics` / proto `has_diagnostics`+`diagnostics`; its digest is in the publish fingerprint). -- **TwinCAT (#637):** TIME/TOD publish Float64 ms, DATE/DT a UTC DateTime; a tag authored `UDInt` keeps the raw counter - (and keeps historizing — DateTime is not historized). -- **TwinCAT (#645, #310/#314):** opt-in ADS sum-read (`useSumRead`, `sumReadMaxItems` default 200 / ceiling 500, 32 KiB payload - budget). It reads raw bytes and decodes as the declared type, so only a scalar passing the strict width+kind check - (`CheckNotification`) rides a batch; value-converted pairs (DInt over INT, Real over LREAL/DINT) read singly. Only overall - ADS 1793/1794 latch the device to single reads; any other non-comms overall code (1792/1795/1797/1798...) reads that chunk singly for the poll without latching, with a once-per-device Warning; a comms failure neither. - Opt-in `diagnosticsEnabled` (default off) puts ADS state/app/online-change/cycle/exec/`HeadroomMs` on the `/hosts` - `IDriverDiagnostics` bag (≤ 4 extra reads per device per probe tick once settled; TC2 falls back to `.SYSTEMINFO` / - `.SYSTEMTASKINFOARR[1]`, costing 8 on the first tick and after each reconnect). Live-gated read-only 2026-10-08 on - Z35474 (TC3) + Z34834 (TC2, which answers sum commands) via a harness driving `TwinCATDriver` — `/hosts` end-to-end - not run (a stock host has no AMS router); the gate found ⚠️ Beckhoff's `SumRead` ctor throws for a `List<>` of entities (only an array survives - its internal cast) — `AdsTwinCATClient.CreateSumRead` owns it. Flat browse stays (VirtualTree is a strict subset - on TC3). Sum-write not ported. -- **AbLegacy (#636):** `AbLegacyAddress` (in `AbLegacy.Contracts`) is family-aware: PLC-5 `I:`/`O:` indices are OCTAL - and are emitted to libplctag as decimal (libplctag's `pccc.c` parses decimal only). MicroLogix function files parse - but read `BadNotSupported`. Indirect `N7:[N7:0]` (#646) is resolved by the driver at read/write time (index re-read - every call, never cached; bad index = `BadOutOfRange`; indirect writes are not atomic). One check (`AbLegacyAddress.Validate`) backs the - editor (family-independent) and the deploy gate (family from the tag's `DeviceConfig.PlcFamily`); an unparseable - address reads `BadNodeIdUnknown` for that tag only. -- **OpcUaClient (#635):** OperationLimits are honoured by the SDK session (`SessionClientBatched`) — do not re-add - driver-side chunking. Subscription tuning keys are range-checked at `ParseOptions`. `discoveryUrl` endpoints go - AFTER configured ones; more than one discovered server without `discoveryServerUri` fails the connect. Session - events are the OTel counter `otopcua.driver.opcuaclient.events`. `HistoryAggregateType` carries the full Part 13 - set; an aggregate the gateway can't serve is `BadAggregateNotSupported` per node. -- **MTConnect (#627/#628):** TIME_SERIES reads as a variable-length `double[]` (Float64 only); a count disagreeing - with `sampleCount` is `BadDecodingError`. One MTConnect driver = one agent client = at most one `/raw` Device - (`MTConnectMultipleDevices` deploy error; `RawTreeService.CreateDeviceAsync` refuses a second). -- **MTConnect CONDITION alarms (#626):** a CONDITION raw tag is a native Part 9 condition only when its `TagConfig` - carries the standard `alarm` object (opt-in — alarm tags leave the value subscription, so auto-promotion would break - existing String tags). Normal→inactive, Warning→500, Fault→800; one condition per DataItem showing the most severe - active nativeCode (ties: ordinal nativeCode, then type); ack is server-side only. ⚠️ UNAVAILABLE never clears: an - active condition stays active+acked with a quality-only `BadNoCommunication` (same as #481 transport loss). The - driver dedups against what the server was last TOLD (level+nativeCode+nativeSeverity+qualifier — not text), records - only while a handler is attached, and drops the record of only the conditions that LEFT the set when the subscribed - alarm SET changes (#641) — never on a reconnect of the same set; a replayed `/current` must never re-raise (un-ack). - `AlarmEventArgs` gained optional `OpcUaSeverity` (exact Part 9 severity) and `QualityStatusCode`; a quality-only - event goes out as `AlarmQualityUpdate` per node and never through the snapshot projection. -- **Native condition state survives a full rebuild (#641):** `RebuildAddressSpace` captures each NATIVE condition's - live node state (client acks included), the same-apply re-materialise restores it, and `CompleteAddressSpaceApply` - (called by `OpcUaPublishActor` at the end of every apply) reports ONE event per restored off-normal condition and - discards unclaimed captures. `NativeAlarmProjector` is pruned per apply, not cleared. Native routing is an exact - `ConditionId == RawPath` match — AbCip's `#active` suffix was fixed; FOCAS routes since #647 (below). -- **Driver removal (#609):** removing a driver releases its capability-invoker pipelines and resilience - tracker, *then* evicts its telemetry snapshots. ⚠️ Both are **tombstoned**, not just cleared: the removed - child stops asynchronously and its in-flight subscribe/connect still completes through the invoker (and - emits Health) on the thread pool *after* `StopChild`, which used to resurrect the state — so a removed - driver's counters were republished forever. `Release` → tracker `Retire` and hub `Evict` ignore every later - write for that id until it is spawned again (`Create` → `Reinstate`, `SpawnChild` → `Readmit`). A - Watch/`Terminated` release would not work — the late writes land after the actor terminates. - ## Library Preferences -- **Logging**: Serilog with rolling daily file sink -- **Unit tests**: xUnit + Shouldly for assertions - **Service hosting (Server, Admin)**: .NET generic host with `AddWindowsService` (decision #30 — replaced TopShelf in v2; see `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs`) -- **OPC UA**: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: `OPCFoundation.NetStandard.Opc.Ua.Server` ## OPC UA .NET Standard Documentation @@ -1252,13 +352,6 @@ Use the DeepWiki MCP (`mcp__deepwiki`) to query documentation for the OPC UA .NE Use the Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/` for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See `docs/Client.CLI.md` for full documentation. -```bash -dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840 -dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3 -dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500 -``` - **AdminUI interactive pages must wrap their body in ``** — `MainLayout` is static, so it cannot catch an interactive page's render failure. Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`. @@ -1288,302 +381,3 @@ endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy `ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`). `Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted. See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`. - -## Scripted Alarm Ack/Shelve - -Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` role — conferred by the session's API-key principal's `NodeAcl` grants (or the `anonymous` principal's), carried by `RoleCarryingUserIdentity`; LDAP is AdminUI-only; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`. - -⚠️ **One condition event per accepted method call is OURS, not the SDK's (Gitea #587).** The SDK's own -report is gated on `AutoReportStateChanges` (left `false`), so before #587 a client Acknowledge emitted **no** -condition event — the old "SDK auto-fires its own event (E2)" comments were wrong. `WireMethodTransitionReporting` -wraps every method's SDK handler (Ack/Confirm/AddComment/Enable/Disable/OneShotShelve/TimedShelve/Unshelve) and, -on Good, reports once via `ReportConditionEvent` — **under `Lock`**, so the engine's/driver's re-projection -(routed by a fire-and-forget Tell *before* the SDK applies state) waits and then hits the delta-gate with no -delta. Things to keep: the SDK's shelve handlers never apply shelving, so the wrapper does; only a real -before/after change is reported (a re-ack is silent and restores the Confirmed/Retain the SDK flips); -`MaxTimeShelved` is set to `int.MaxValue` ms at materialise because `Create()` left it 0 and every TimedShelve -failed. ⚠️ **Native alarms support Acknowledge and AddComment only** — Confirm, the shelve methods and -Enable/Disable return `BadNotSupported` (no upstream path, no pair replication, and `NativeAlarmProjector`'s -Confirmed/Unshelved constants own those fields); a native ack re-asserts `ConfirmedState = true` because the -SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by -`ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`). - -Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a -**no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real -remaining time from `AlarmConditionSnapshot.UnshelveAtUtc`); a **commented** re-ack is stored as an -`AddComment` (durable, replicated) and reports one event, an uncommented re-ack stays silent; a **native -ack is refused `BadInvalidState` off-Primary** via `NativeAlarmAckPrimaryGate` (fed by -`IDriverPrimaryRoleSource`; `DriverHostActor` publishes its Primary verdict at host start and on member -changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT -Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design). - -**Condition `Severity` is carried verbatim, clamped to 1..1000** — `ToPart9Severity`; the node manager used to re-bucket onto -the five `EventSeverity` points (so an MTConnect Fault read 900, scripted Critical 1000 read 900, native Low 200 read 300). -ScadaBridge only clamps, so exact values pass through. ⚠️ **Never select a two-state id with `AddSelectClause(type, "X/Id")`** — -the string becomes ONE QualifiedName named `X/Id`, which no event carries; the Client.CLI/Client.UI filter did this since -`a2883b82` and every alarm read inactive+unacked. Use `new SimpleAttributeOperand(type, [X, Id])` (ScadaBridge already does). - -**Condition children are instance-scoped (#629).** Every child of a materialised condition carries -`{ConditionId}/{browse path}` in the condition's namespace (e.g. `…/temp_hi/ShelvingState/UnshelveTime`), assigned by -the `OtOpcUaNodeManager.New` override — the SDK default kept each child on the TYPE's ns=0 id, so a direct Read hit -the type (or `BadNodeIdUnknown`), translate failed, and children sat outside the ACL (the slash join now puts them -under their condition's scope). ⚠️ The shelve methods resolve by the `AlarmConditionType_ShelvingState_*` ids only -because `New` adopts those (a fixed set) as their `MethodDeclarationId`; calls by the `ShelvedStateMachineType_*` id -are no longer supported (pinned by a test). Condition teardown must forget the whole `PredefinedNodes` subtree -(`ForgetPredefinedSubtree`) or a removed condition's children keep answering. - -## Historian / HistoryRead - -**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default), `SqlServer` or `QuestDb`** (see the subsection below). -As of the gateway-integration cutover, the **Gateway** backend's historian read, alarm-write, and -continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed -**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin -`IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke -Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were -retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`, -now a retired stub). - -A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the -raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default -historian tagname, which is the tag's driver `FullName`. - -### Pluggable backends, SQL Server and HistoryUpdate (2026-10) - -- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`. -- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. -- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". -- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. -- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads". -- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates. -- **Raw `ReturnBounds` is honoured on every backend (#691, client-visible):** Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by `HistoryBounds` from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 `ReverseRawReader` walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward `NumValuesPerNode`, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ `Bad_BoundNotFound` (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ **Breaking for SDK-default clients:** the .NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true` — a client that cleared only `IsReadModified` now gets bounds, and one that set neither is a modified read with bounds ⇒ `Bad_InvalidArgument` (§6.5.3.3); set both explicitly (Client.Shared does; tests building details must too). No data AND no bound found ⇒ `GoodNoData`, no values (the pre-bounds answer). On Gateway the at-or-after lookup ends at now + 1 day (never a year-9999 end to AVEVA). Kill switch `ServerHistorian:HonorReturnBounds` (default `true`; `false` = pre-#691). `docs/Historian.md` §Bounding values. -- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. -- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. -- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key). -- **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history. - -### Read path (`ServerHistorian` section) - -The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the -`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default -(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value **null** + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (`AllowStreamingBeyondCeiling=false`) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (`GatewayRowCeiling`: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (`NumValuesPerNode = 0`) read of a small window is served; a ceiling-bounded stream that is not provably complete is `BadTooManyOperations`, never a truncated answer), Processed -(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes; Events over -alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node); -authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization. - -`ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`): - -| Key | Default | Notes | -|---|---|---| -| `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` | -| `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) | -| `ApiKey` | `""` | Peppered-HMAC key `histgw__` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** | -| `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme | -| `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) | -| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store | -| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read | -| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) | -| `HonorReturnBounds` | `true` | #691 kill switch: `false` ignores HistoryRead-Raw `ReturnBounds` as before (no bounding values, no modified-read `Bad_InvalidArgument`) | -| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential | -| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 | -| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway | -| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded | - -### Alarm-history path (`AlarmHistorian` section) - -Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind -the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink` -default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the -replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to -the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node -holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The -`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`, -`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection -(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the -target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround). - -### Continuous historization (`ContinuousHistorization` section) - -When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a -durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed -`IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The -recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the -crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**) -with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries -only the recorder + outbox knobs: - -| Key | Default | Notes | -|---|---|---| -| `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder | -| `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage | -| `CommitMode` | `PerEntry` | `PerEntry` = fsync before the append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs`. ⚠️ `PerEntry` now means **one fsync per ingest burst, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands | -| `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` | -| `DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs `tags × latency` — while 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, letting the outbox fall permanently behind during any gateway hiccup | -| `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) | -| `DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential `tags × latency` pass becomes `⌈tags / concurrency⌉ × latency` with outstanding round-trips bounded. `1` = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass" | -| `Capacity` | `500000` | Max un-acked outbox entries before drop-oldest; `0` = unbounded. Non-zero **by default since #538** — the disk backstop for a historian refusing *every* write, the one case the per-tag quarantine deliberately cannot resolve | -| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance | -| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass | -| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff | -| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 | - -### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook) - -`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` -hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's -`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues` -land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a -deploy. **Gateway backend:** non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not -provisioned), the recorder likewise drops + meters non-numeric values, and continuous historization is -**numeric-analog only** (`UInt16→UInt4` is a documented fallback). The **SqlServer** backend provisions and -records every type (see "Pluggable backends" above). - -The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`), -built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the -read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier. -When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not -run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with -`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR -#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.) -**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists). -`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the -historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable -stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are -supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see -the "Boolean historization maps to Int2" section in `docs/Historian.md`. - -### Gateway-side prerequisites - -The target HistorianGateway OtOpcUa points at **must** run with: - -- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization). -- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads). -- `RuntimeDb:EventReadMaxRows=0` (its default, unlimited) — or larger than the reverse-events walk's biggest read - (`2·MaxTieClusterOverfetch + 4`). The cap truncates **silently**, so a walked-back reverse events read (#691.4) would - lose the newest events of a segment it cut. -- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**. - -### Migration note (deployments upgrading from the Wonderware backend) - -The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env: - -| Old (Wonderware) key | New (gateway) key | -|---|---| -| `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) | -| `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) | -| `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) | - -The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`) -were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from -`ServerHistorian`. See `docs/Historian.md` for the full guide. - -### Live-validation gate — PASSED (was "KNOWN LIMITATION 1") - -**No longer a limitation.** The cutover was live-validated against the real gateway on -`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed -protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot -populate `Source_Object` — not a fixable bug). Evidence: -`docs/plans/2026-06-27-otopcua-historian-followups.md`. - -Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to -`wonder-sql-vd03`, gateway running the prerequisites above): - -```bash -export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway URI; absent ⇒ all live tests skip -export HISTGW_GATEWAY_APIKEY=histgw__ # must carry historian:read + historian:write (+ tags:write) scopes -export HISTGW_TEST_TAG= # read round-trip -export HISTGW_WRITE_SANDBOX_TAG= # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write) -export HISTGW_BOOL_SANDBOX_TAG= # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441) -export HISTGW_ALARM_SOURCE= # alarm SendEvent → ReadEvents round-trip -dotnet test --filter "Category=LiveIntegration" -``` - -The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the -gate the operator runs on the VPN before trusting the cutover. - -### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04) - -The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer + -meters). It is spawned with an **empty *initial* ref set** (`Array.Empty()` in -`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built -later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the -`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it -through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the -recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the -deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe -`Tell`; a faulting feed can never break a deploy). - -**The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a -production-fatal defect** — see `docs/plans/2026-08-01-continuous-historization-live-gate.md`. Three things -to know: - -- ⚠️ **`ConfigureAwait(false)` in an Akka handler destroys `Self`.** `OnValueChangedAsync` touched `Self` - after awaiting `IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false)`, which discards Akka's - `ActorTaskScheduler` synchronization context — so `ActorBase.Context` was gone and it threw - `NotSupportedException` on the FIRST value. The throw reached supervision, the actor **restarted**, and - because production seeds an **empty** ref set the restart discarded every historized ref: one value - captured, then silence. Fixed by capturing `IActorRef self = Self` before the await (the idiom the - `GetStatus` handler already used for `Sender`). -- ⚠️ **A fake that completes synchronously hides this entire class of bug**, and 13 tests passed because - of one. `ValueTask.CompletedTask` never yields, so the continuation ran inline with the context intact. - Worse, the obvious fix to the fake — `await Task.Yield()` — is *also* vacuous: `Yield()` posts back to - the captured context, so the caller still resumes on the actor thread. The fake must complete on a - **pool** thread (`await Task.Delay(1).ConfigureAwait(false)`), as a real per-entry fsync does. Only - reverting the production fix exposed that the first regression test proved nothing. -- **Both pair nodes historized the same value** — the recorder had **no Primary gate**, unlike the alarm - store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as - **#537** and **fixed** — see the paragraph below. - -**Gate COMPLETE — #491 CLOSED 2026-08-04.** Outbox append, drain, `WriteLiveValues`, -**restart-convergence**, `EnsureTags` provisioning (`ensured=2 failed=0`) and **read-back** all ✅. The -read-back leg ran against the licensed `wonder-sql-vd03` with the VPN up: two registers carrying -*distinct non-zero* constants (`8224`, `120`) each read back as its own value — a `0`-valued source -would have proved nothing (the #536 trap). - -**Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison -tag); evidence in the same plan doc. Two things to know before touching either.** - -- **#538 — the discriminator is "did another tag write this pass?", NOT a failure count.** `DrainOnceAsync` - writes every per-tag group and acks **all-or-nothing** (deliberate: `RemoveAsync` truncates *through* an - id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned - the FIFO head while every *healthy* tag was re-submitted on every pass, forever — duplicate samples, an - outbox that never truncated (observed 8.3 MB; signature = the gateway's `ok` **and** `error` counters - climbing together). The drain now quarantines a tag after `MaxWriteAttempts` consecutive failures, but - ⚠️ **only counts a failure on a pass where some other tag succeeded** — otherwise an ordinary gateway - restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug - into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so - the load-bearing test is the **falsification control** (`A_backend_that_fails_every_tag_abandons_nothing`). - Abandonment is also deferred until *every* failing tag is over budget, so nothing is discarded on a pass - that could not have advanced the head anyway. -- **#537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink.** The alarm - queue is in LocalDb and *replicates*, so the Secondary holds the Primary's rows and gating the drain - loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its - drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of - stale duplicates on promotion). ⚠️ **And the peer that matters is one recording the SAME refs, not one - holding the same rows** — a node stands down only for a Primary in its own per-cluster mesh, and a node - with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is - usually running another cluster's configuration, and deferring to it ends this cluster's historization - *silently* (the drain reports success on an empty outbox forever) — the same scope mismatch that once - suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on - `RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes - (NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the - #511 registration accident). - - **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary - stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the - redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and - *nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`) - subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role - source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change - re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the - failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable` - (default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT - adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay - put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address - clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the - gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real - partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each - node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates. - -Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for -newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType) -provisioned nothing (`2026-06-27-historian-gateway-integration-issues.md` Issue 5). It now also fires for those -changed raw tags.