From 90bfcbe47a9cbb946256e33c009c0ce2a6ea5269 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 8 Oct 2026 08:13:14 -0400 Subject: [PATCH 1/4] docs: rewrite IncrementalSync + AddressSpace for v3, fix driver-specs/driver-stability tier claims, update Secrets runbook for pair replication + KEK rotation (#623) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- docs/AddressSpace.md | 142 ++++++++---------- docs/IncrementalSync.md | 135 +++++++---------- docs/drivers/Modbus.md | 5 +- docs/drivers/README.md | 6 +- ...2026-07-16-secrets-clustered-master-key.md | 139 +++++++++++------ docs/v2/driver-specs.md | 18 ++- docs/v2/driver-stability.md | 4 + 7 files changed, 239 insertions(+), 210 deletions(-) diff --git a/docs/AddressSpace.md b/docs/AddressSpace.md index fd208f0f..4dd9bd89 100644 --- a/docs/AddressSpace.md +++ b/docs/AddressSpace.md @@ -1,96 +1,84 @@ # Address Space -> ⚠️ **Accuracy warning (audited 2026-07-27) — this page is v2-era and contradicts the shipped v3 -> address space.** Corrections, in order of how badly they mislead: -> -> - **There is no "single custom namespace".** v3 exposes **two**: -> `https://zb.com/otopcua/raw` (`s=`) and `https://zb.com/otopcua/uns` -> (`s=///`), registered together at -> `OtOpcUaNodeManager.cs:45`. Identity authority is `V3NodeIds` + `AddressSpaceRealm`. The retired -> `https://zb.com/otopcua/ns` survives only in a comment marking it retired (`V3NodeIds.cs:17`). -> - **`Phase7Applier` / `Phase7Composer` / `GalaxyTagPlan` do not exist**, nor do the file paths cited -> for them. Renamed by `40e8a23e` to `AddressSpaceApplier` / `AddressSpaceComposer` -> (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`); the composition result type is -> `AddressSpaceComposition` (`AddressSpaceComposer.cs:13`). -> - **The `SystemPlatform` namespace kind is retired** — Galaxy is a standard Equipment-kind driver. -> - **`GenericDriverNodeManager` is test scaffolding**, not a production dispatch path -> (`GenericDriverNodeManager.cs:71`). -> - **The §Rediscovery section is wrong**: nothing consumes `OnRediscoveryNeeded` -> ([#518](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/518)), and the driver list is stale -> (MQTT and MTConnect now raise it too). -> -> For the current design see the "v3 OPC UA Address Space (Batch 4)" section of `CLAUDE.md`, -> `docs/Raw.md`, and `docs/Uns.md`. +The server serves **two OPC UA namespaces** under one shared root folder. Identity is owned by +`V3NodeIds` (`src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/V3NodeIds.cs`) and `AddressSpaceRealm`; the tree is +built from the deploy artifact (see `docs/IncrementalSync.md`), not by driver discovery at runtime. -Address-space construction is a two-layer system. The **driver-facing layer** is the streaming builder: a driver implements `ITagDiscovery.DiscoverAsync` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ITagDiscovery.cs`) and emits `Folder` / `Variable` / `AddProperty` calls into an `IAddressSpaceBuilder` as it walks its backend — no buffering of the whole tree. `GenericDriverNodeManager` (`src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/GenericDriverNodeManager.cs`) wraps that builder to capture alarm-condition sinks and routes alarm events from the driver to them. The **SDK materialization layer** turns the resulting node descriptions into live OPC UA nodes: `OpcUaPublishActor` drives the write-only `IOpcUaAddressSpaceSink`, whose production binding `SdkAddressSpaceSink` (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs`) forwards to `OtOpcUaNodeManager` (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`), a `CustomNodeManager2` subclass that owns the `FolderState` / `BaseDataVariableState` instances. The same code path serves Galaxy object hierarchies, Modbus PLC registers, AB CIP tags, TwinCAT symbols, FOCAS CNC parameters, and OPC UA Client aggregations — Galaxy is one driver of seven, not the driver. +| Realm | Namespace URI | Subtree | NodeId `s=` identifier | +|---|---|---|---| +| `Raw` | `https://zb.com/otopcua/raw` | `/raw` device tree: Folder → Driver → Device → TagGroup → Tag | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) | +| `Uns` | `https://zb.com/otopcua/uns` | UNS tree: Area → Line → Equipment → signal | slash-joined `Area/Line/Equipment[/EffectiveName]` | + +`OtOpcUaNodeManager` registers both URIs (raw is index `NamespaceIndexes[0]`, UNS `[1]`). The retired +single namespace `https://zb.com/otopcua/ns` and the `EquipmentNodeIds` scheme no longer exist. The realm +travels beside the identifier at every sink seam (`IOpcUaAddressSpaceSink`); it is never parsed out of the +id string. Authoring and semantics: `docs/Raw.md`, `docs/Uns.md`. ## Root folder -`OtOpcUaNodeManager.CreateAddressSpace` creates a single shared root `FolderState` (`NodeId = OtOpcUa`, `BrowseName = OtOpcUa`, `EventNotifier = None`) under the standard OPC UA `Objects` folder, wired with an `Organizes` reference. Every driver's folders and variables hang beneath this one root; the server is published under a single `ApplicationUri = urn:OtOpcUa` (the `OpcUaApplicationHostOptions.ApplicationUri` default) and all nodes live in the server's single custom namespace, not a per-driver `urn:OtOpcUa:{DriverInstanceId}`. The UNS Area → Line → Equipment folder skeleton under the root is materialised by `Phase7Applier.MaterialiseHierarchy` (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Phase7Applier.cs`); SystemPlatform (Galaxy) tags are materialised by `Phase7Applier.MaterialiseGalaxyTags`. - -## IAddressSpaceBuilder surface - -`IAddressSpaceBuilder` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IAddressSpaceBuilder.cs`) offers three calls: - -- `Folder(browseName, displayName)` — creates a child `FolderState` and returns a child builder scoped to it. -- `Variable(browseName, displayName, DriverAttributeInfo attributeInfo)` — creates a `BaseDataVariableState` and returns an `IVariableHandle` the driver keeps for alarm wiring. -- `AddProperty(browseName, DriverDataType, value)` — attaches a `PropertyState` for static metadata (e.g. equipment identification fields). - -Drivers drive ordering. Typical pattern: root → folder per equipment → variables per tag. `GenericDriverNodeManager.BuildAddressSpaceAsync` calls `DiscoverAsync` once on startup and once per rediscovery cycle, tearing down the previous alarm subscription and clearing its sink registry before each re-walk so a redeploy doesn't double-fire alarm events. - -## DriverAttributeInfo → OPC UA variable - -Each variable carries a `DriverAttributeInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverAttributeInfo.cs`): - -| Field | Role | -|---|---| -| `FullName` | driver-side full reference used as the lookup key for Read/Write/Subscribe; also seeds the variable's string `NodeId` | -| `DriverDataType` | resolved to a built-in `DataTypeIds.*` NodeId at materialization time — `OtOpcUaNodeManager.ResolveBuiltInDataType` maps the data-type name string; each driver first maps its native type into `DriverDataType` (e.g. Galaxy via `Browse/DataTypeMap.Map`) | -| `IsArray` / `ArrayDim` | declared 1-D-array length carried as metadata; the Galaxy discoverer sets `ArrayDim` only when the gateway reports a positive dimension | -| `SecurityClass` | write-authorization tier (`SecurityClassification`); enforced server-side by the `NodePermissions` ACL evaluator (`TriePermissionEvaluator`) mapping each `OpcUaOperation` to a required permission bit. The Galaxy driver also caches it per full reference (`_securityByFullRef`) to answer `GetSecurityClassification` | -| `IsHistorized` | marks the attribute as feeding historian / HistoryRead | -| `IsAlarm` | drives the `MarkAsAlarmCondition` pass (see below) | -| `WriteIdempotent` | when true the attribute's writes are safe to replay, so the capability invoker may apply Polly retry; defaults false so pulses / acks / counters aren't auto-retried | -| `Source` | `NodeSourceKind` discriminator (`Driver` / `Virtual` / `ScriptedAlarm`) that decides which subsystem dispatches the node's Read/Write/Subscribe | - -The variable is created with `StatusCode = BadWaitingForInitialData` and a null value until the first Read or `ISubscribable.OnDataChange` push lands. Note the production SDK sink (`OtOpcUaNodeManager.EnsureVariable`) currently materialises every variable as `ValueRank = Scalar`, read-only `AccessLevel`, and `Historizing = false` — the `IsArray`/`IsHistorized` intent lives in `DriverAttributeInfo` but is not yet projected onto the SDK node. - -## CapturingBuilder + alarm sink registration - -`GenericDriverNodeManager.BuildAddressSpaceAsync` wraps the supplied builder in a `CapturingBuilder` before calling `DiscoverAsync`. The wrapper observes every `Variable()` call: when a returned `IVariableHandle.MarkAsAlarmCondition(AlarmConditionInfo)` fires, the sink is registered in the manager's `_alarmSinks` dictionary keyed by the variable's `FullReference`. Subsequent `IAlarmSource.OnAlarmEvent` pushes are routed to the matching sink by `SourceNodeId`. This keeps the alarm-wiring protocol declarative — drivers just flag `DriverAttributeInfo.IsAlarm = true` and the materialization of the OPC UA `AlarmConditionState` node is handled by the server layer. See `docs/AlarmTracking.md`. +`OtOpcUaNodeManager.CreateAddressSpace` creates **one** shared root `FolderState` — identifier and browse +name `OtOpcUa` (`V3NodeIds.SharedRootFolderId`), minted in the **raw** namespace — under the standard +`Objects` folder with an `Organizes` reference. Top-level Raw folders and top-level UNS Areas both parent +under it. The identifier is a display name, not a path segment; the data-plane authorizer maps it onto the +trie's zero-segment realm-root scope (`docs/security.md`). The application URI is +`OpcUaApplicationHostOptions.ApplicationUri` (default `urn:OtOpcUa`); there is no per-driver namespace. ## NodeId scheme -All nodes share the server's single custom namespace (`NamespaceIndex`); NodeIds are string identifiers, not numeric. The string values come from the source rows / driver references — there is no per-driver namespace prefix: +Segments are validated by `RawPaths.ValidateSegment` (no embedded `/`, no leading/trailing whitespace; +ordinal, case-sensitive). `V3NodeIds.Raw(rawPath)` is a pass-through; `V3NodeIds.Uns(...)` / +`UnsChild(parent, segment)` build UNS ids. -| Node type | NodeId (string identifier) | Example | -|---|---|---| -| Shared root | `OtOpcUa` | `OtOpcUa` | -| UNS Area / Line / Equipment folder | the Config-DB `UnsAreaId` / `UnsLineId` / `EquipmentId` | `EQ_Press_07` | -| Galaxy tag variable | the MXAccess reference (`Phase7Applier` uses `GalaxyTagPlan.MxAccessRef`) | `DelmiaReceiver_001.DownloadPath` | -| Equipment tag variable | the driver full reference from `DriverAttributeInfo.FullName` | driver-specific | +| Node | Realm | `s=` identifier | Example | +|---|---|---|---| +| Shared root | Raw | `OtOpcUa` | `OtOpcUa` | +| Raw folder / driver / device / tag group / tag | Raw | its own RawPath | `Plant/Modbus/dev1/Speed` | +| UNS Area / Line / Equipment folder | Uns | `Area`, `Area/Line`, `Area/Line/Equipment` | `packaging/line-1/press-07` | +| UNS reference variable (`UnsTagReference`), virtual tag, scripted alarm | Uns | `Area/Line/Equipment/EffectiveName` | `packaging/line-1/press-07/speed` | -For Galaxy the variable `FullName` is the `tag_name.AttributeName` MXAccess reference; AB CIP uses `tag.Name` or `tag.Name.member` for UDT members; the shape is the driver's choice. Browse-path resolution (OPC UA `TranslateBrowsePathsToNodeIds`) is the canonical way clients map a browse path to one of these flat NodeIds. +Area, Line and Equipment names must match `^[a-z0-9-]{1,32}$` (`docs/Uns.md`). EffectiveNames are unique per +equipment across references, virtual tags and scripted alarms. -## Per-driver hierarchy examples +## One source, fanned to both namespaces -- **Galaxy**: `GalaxyDriver.DiscoverAsync` delegates to `GalaxyDiscoverer` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/GalaxyDiscoverer.cs`), which walks the hierarchy from `IGalaxyHierarchySource` — one folder per Galaxy object (browse name = `contained_name`, falling back to `tag_name`), one variable per dynamic attribute (full reference = `tag_name.AttributeName`). It copies the gateway-supplied `IsAlarm` flag through to `DriverAttributeInfo` and, for alarm-bearing attributes, calls `MarkAsAlarmCondition` with the five sub-attribute refs built by `AlarmRefBuilder`. -- **Modbus**: streams one folder per device, one variable per register range from `ModbusDriverOptions`. No alarm surface. -- **AB CIP**: `AbCipDriver.DiscoverAsync` emits an `AbCip` root, then a folder per configured device. Pre-declared tags become variables under the device folder; UDT (`Structure`) tags fan out into a sub-folder with one variable per member; when controller browse is enabled, `IAbCipTagEnumerator` adds discovered tags under a `Discovered/` sub-folder. (`AbCipTemplateCache` caches UDT layouts for the libplctag enumerator.) -- **OPC UA Client**: re-exposes a remote server's address space — `OpcUaClientDriver.DiscoverAsync` browses the upstream from `BrowseRoot` into a `Remote` folder (pass 1), then batch-reads DataType/AccessLevel/ValueRank/Historizing per variable before registering them (pass 2). +Every device value has exactly one source: the raw tag's node, keyed by RawPath. A `UnsTagReference` +projects that raw tag into an equipment as a UNS variable that carries an `Organizes` reference to its raw +node and mirrors it. `DriverHostActor` fans one driver publish for a RawPath to the raw NodeId **and** every +referencing UNS NodeId with identical value, quality and timestamp. Writes via either NodeId route to the +same driver reference under the same realm-qualified `WriteOperate` gate. Native alarms materialise once at +the raw tag (`ConditionId = RawPath`) and fan through notifiers to the raw device folder and every +referencing equipment folder. Both NodeIds register the same historian tagname. -See `docs/v2/driver-specs.md` for the per-driver discovery contracts. +## Materialisation -## Rediscovery +`AddressSpaceComposer` turns the deployment artifact into an `AddressSpaceComposition`; +`AddressSpaceApplier` (both in `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`) walks it (`MaterialiseHierarchy`, +`MaterialiseRawSubtree`, `MaterialiseUnsReferences`, `MaterialiseEquipmentVirtualTags`, +`MaterialiseScriptedAlarms`, ...) into the write-only `IOpcUaAddressSpaceSink`, whose production binding +`SdkAddressSpaceSink` forwards to `OtOpcUaNodeManager`, the `CustomNodeManager2` that owns the +`FolderState` / `BaseDataVariableState` instances. Variables are created through `EnsureVariable`: +`ValueRank` is `OneDimension` with `ArrayDimensions=[arrayLength]` for array tags and `Scalar` otherwise; +`AccessLevel` carries the write bit only for writable nodes (array nodes only when the driver can write the +array); `Historizing` plus the `HistoryRead` bit are set when a historian tagname is supplied. -Drivers that implement `IRediscoverable` fire `OnRediscoveryNeeded` when their backend signals a change. Galaxy's `DeployWatcher` raises it when the observed `time_of_last_deploy` advances; TwinCAT raises it on the ADS symbol-version-changed signal (`DeviceSymbolVersionInvalid`, error 1809). Core re-runs `DiscoverAsync` and diffs — see `docs/IncrementalSync.md`. Drivers that don't implement `IRediscoverable` (Modbus, S7, OPC UA Client) only change their address space when a new generation is published from the Config DB. +The nodes start `BadWaitingForInitialData` until the first read or subscription push. Datatype names resolve +through `OtOpcUaNodeManager.ResolveBuiltInDataType`; each driver first maps its native type into +`DriverDataType` (Galaxy: `Driver.Galaxy/Browse/DataTypeMap.cs`). + +## Driver discovery is the browse picker, not the address space + +`ITagDiscovery.DiscoverAsync` (streaming into `IAddressSpaceBuilder`: `Folder` / `Variable` / +`AddProperty`, each variable described by a `DriverAttributeInfo`) feeds the `/raw` browse-commit flow via +`Commons/Browsing/DiscoveryDriverBrowser`: the operator browses a device, picks leaves and commits them as +raw tags, which then reach the server as an ordinary deployment. Drivers that signal backend change via +`IRediscoverable` only raise an operator "re-browse" prompt — the served tree never changes on its own. +See `docs/IncrementalSync.md`. `GenericDriverNodeManager` (`Core/OpcUa/`) is leftover scaffolding with no +production caller. ## Key source files -- `src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/GenericDriverNodeManager.cs` — orchestration + `CapturingBuilder` -- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`, `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs` — OPC UA materialization (write-only sink fed by the actor system) -- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Phase7Applier.cs` — materialises the UNS folder hierarchy + Galaxy tags into the sink -- `src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/EquipmentNodeWalker.cs` — walks Config-DB Equipment-namespace rows into the builder -- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IAddressSpaceBuilder.cs` — builder contract -- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ITagDiscovery.cs` — driver discovery capability -- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverAttributeInfo.cs` — per-attribute descriptor +- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/V3NodeIds.cs`, `AddressSpaceRealm.cs` — identity authority +- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Types/RawPaths.cs` — RawPath/segment rules +- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`, `SdkAddressSpaceSink.cs` — SDK materialisation +- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceComposer.cs`, `AddressSpaceApplier.cs` — compose and apply +- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ITagDiscovery.cs`, `IAddressSpaceBuilder.cs`, `DriverAttributeInfo.cs` — discovery-side contracts diff --git a/docs/IncrementalSync.md b/docs/IncrementalSync.md index 6b3e4606..deb6be2f 100644 --- a/docs/IncrementalSync.md +++ b/docs/IncrementalSync.md @@ -1,90 +1,71 @@ # Incremental Sync -> ⚠️ **Accuracy warning (audited 2026-07-27) — most of this page describes machinery that is retired -> or was never wired.** Read it as v2-era design history, not as current behaviour. -> -> - **"Rebuild flow" (below) is not a production path.** It is built on `GenericDriverNodeManager`, -> which is Core test scaffolding with **zero production references** -> (`GenericDriverNodeManager.cs:71`). Its `_variablesByFullRef` / `_sourceByFullRef` fields do not -> exist in `src/`. -> - **Driver-backend rediscovery has no consumer.** Nothing in `src/Server/` or `src/Core/` -> subscribes to `IRediscoverable.OnRediscoveryNeeded`; `DriverHostActor.HandleDiscoveredNodes` -> (`DriverHostActor.cs:986-1002`) short-circuits unconditionally. The `ScopeHint` is read by -> nothing. Drivers that raise it today: Galaxy, TwinCAT, MQTT/Sparkplug, MTConnect -> ([#518](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/518), -> [#507](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/507)). -> - **The generation-publish path is retired.** `sp_PublishGeneration` and -> `sp_ComputeGenerationDiff` are stubs that `RAISERROR` -> (`20260715230637_V3Initial.StoredProcedures.cs:194`); `DraftRevisionToken` does not exist. -> -> **The one live rebuild path in v3** is the deploy artifact: `ConfigComposer` → -> `Deployment.ArtifactBlob` → `DriverHostActor` apply → `AddressSpaceComposer` / -> `AddressSpaceApplier` diff-apply. +How a change reaches the running server's address space in v3. There is exactly one path that +changes the served address space — the **deploy artifact** — and one advisory signal +(`IRediscoverable`) that never changes it. -Two distinct change-detection paths feed the running server: driver-backend rediscovery (Galaxy's `time_of_last_deploy`, TwinCAT's symbol-version-changed) and generation-level config publishes from the Admin UI. Both flow into re-runs of `ITagDiscovery.DiscoverAsync`, but they originate differently. +## The deploy-artifact path -## Driver-backend rediscovery — IRediscoverable +1. **Compose.** `ConfigComposer` (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs`) + snapshots the whole Config DB into one gzip-compressed JSON blob (`Deployment.ArtifactBlob`) whose + `RevisionHash` is the SHA-256 of those exact bytes. A deployment is fleet-wide; each node + self-scopes the artifact (`DeploymentArtifact.ResolveClusterScope`). +2. **Dispatch.** `ConfigPublishCoordinator` (`.../ControlPlane/Coordinators/`) sends each enabled + `ClusterNode` a payload-free dispatch and waits for acks until the apply deadline. A node obtains the + bytes either straight from central SQL (`ConfigSource:Mode=Direct`) or over gRPC with a SHA-256 + check (`FetchAndCache`) — see `docs/Configuration.md` §`ConfigSource`. +3. **Apply.** `DriverHostActor` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/`) opens the artifact and + reconciles in two halves: + - **Drivers** — `DriverSpawnPlanner.Compute` yields `ToStop` / `ToApplyDelta` / `ToSpawn`. A changed + driver config reaches the live driver as `IDriver.ReinitializeAsync` on its `DriverInstanceActor`; + a removed driver is stopped. An unreadable or empty blob is "no answer": the apply fails and the + running children are left alone (never an empty configuration). + - **Address space** — `AddressSpaceComposer` builds an `AddressSpaceComposition` from the artifact; + `AddressSpacePlan` diffs it against the last applied composition; `AddressSpaceChangeClassifier` + picks the delta class; `AddressSpaceApplier` + (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`) mutates `OtOpcUaNodeManager` through the sink. +4. **Classes** (`AddressSpaceChangeKind`): `Empty`, `AttributeOnly`, `PureAdd`, `PureRemove`, + `AddRemoveMix` — all applied surgically, so unrelated MonitoredItems survive — and `Rebuild`, the + default-closed fallback for anything the classifier does not whitelist. Any surgical `false`/throw + also ratchets to a safe rebuild, and a degraded non-rebuild apply does not advance the last-applied + composition (the next deploy re-diffs and re-ensures). The whitelist, the UNS-mirror repair rule and + the add/remove announcements are described in the "surgical deploys" paragraph of `CLAUDE.md` + and `docs/plans/2026-08-15-surgical-deploys.md`. -Drivers whose backend has a native change signal implement `IRediscoverable` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs`): +Virtual tags and scripted alarms ride the same artifact and the same plan; a script edit is just another +changed record in the composition (see `docs/VirtualTags.md`, `docs/ScriptedAlarms.md`). -```csharp -public interface IRediscoverable -{ - event EventHandler? OnRediscoveryNeeded; -} -public sealed record RediscoveryEventArgs(string Reason, string? ScopeHint); -``` +## Driver-backend rediscovery is advisory -The driver fires the event with a reason string (for the diagnostic log) and an optional scope hint — a non-null hint lets Core scope the rebuild surgically to that subtree; null means "the whole address space may have changed". +Drivers whose backend signals a tag-set change implement `IRediscoverable` +(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs`; `OnRediscoveryNeeded` carries a +`Reason` and an optional `ScopeHint`). Raisers today: **Galaxy** (`DeployWatcher`, new +`time_of_last_deploy`), **TwinCAT** (ADS symbol-version-changed), **MQTT/Sparkplug** and **MTConnect**. +AbCip, AbLegacy, FOCAS, Modbus, S7 and the rest raise nothing. -Drivers that implement the capability today: +`DriverInstanceActor` is the only production consumer: it subscribes in `PreStart`, detaches in +`PostStop`, and folds the raise into the driver-health snapshot +(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`), which the AdminUI `/hosts` page shows +as a **re-browse** chip. **It never rebuilds or edits the served address space** — v3 raw tags are authored +explicitly in `/raw`, so an operator re-browses the device and commits, and the result reaches the +server as an ordinary deployment. `ScopeHint` is not read by the actor. (The older runtime +injection path was removed, #507 closed as superseded.) -- **Galaxy** — `DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) subscribes to the mxaccessgw gRPC stream (`IGalaxyDeployWatchSource.WatchAsync`) and fires on a new `time_of_last_deploy` value. The gateway polls the Galaxy repository DB internally; the driver side is event-driven. -- **TwinCAT** — observes ADS symbol-version-changed notifications (ADS error `DeviceSymbolVersionInvalid`, decimal 1809 / `0x0711`). Note: legacy Beckhoff documentation sometimes cites `0x0702` (`DeviceInvalidGroup`) — that is a transcription error; the correct code is `0x0711` per `TwinCATStatusMapper.AdsSymbolVersionChanged` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATStatusMapper.cs:35`). +## Not part of the v3 path -Static drivers (Modbus, S7, AB CIP, AB Legacy, FOCAS) do not implement `IRediscoverable` — their tags only change when a new generation is published from the Config DB. Core sees absence of the interface and skips change-detection wiring for those drivers (decision #54). - -## Config-DB generation publishes - -Tag-set changes authored in the Admin UI (UNS edits, CSV imports, driver-config edits) accumulate in a draft generation and commit via `sp_PublishGeneration`. The delta between the currently-published generation and the proposed next one is computed by `sp_ComputeGenerationDiff`, which drives: - -- The publish-preview surface in the Admin UI (`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Deployments.razor`, backed by `AdminOperationsClient`) so operators can preview what will change before clicking Publish. -- The 409-on-stale-draft flow (decision #161) — a UNS drag-reorder preview carries a `DraftRevisionToken` so Confirm returns `409 Conflict / refresh-required` if the draft advanced between preview and commit. - -After publish, the server's generation applier invokes `IDriver.ReinitializeAsync(driverConfigJson, ct)` on every driver whose `DriverInstance.DriverConfig` row changed in the new generation. Reinitialize is the in-process recovery path for Tier A/B drivers; if it fails the driver is marked `DriverState.Faulted` and its nodes go Bad quality — but the server process stays running. See `docs/v2/driver-stability.md`. - -Drivers whose discovery depends on Config DB state (Modbus register maps, S7 DBs, AB CIP tag lists) re-run their discovery inside `ReinitializeAsync`; Core then diffs the new node set against the current address space. - -## Rebuild flow - -When a rediscovery is triggered (by either source), `GenericDriverNodeManager` re-runs `ITagDiscovery.DiscoverAsync` into the same `CapturingBuilder` it used at first build. The new node set is diffed against the current: - -1. **Diff** — full-name comparison of the new `DriverAttributeInfo` set against the existing `_variablesByFullRef` map. Added / removed / modified references are partitioned. -2. **Snapshot subscriptions** — before teardown, Core captures the current monitored-item ref-counts for every affected reference so subscriptions can be replayed after rebuild. -3. **Teardown** — removed / modified variable nodes are deleted via `CustomNodeManager2.DeleteNode`. Driver-side subscriptions for those references are unwound via `ISubscribable.UnsubscribeAsync`. -4. **Rebuild** — added / modified references get fresh `BaseDataVariableState` nodes via the standard `IAddressSpaceBuilder.Variable(...)` path. Alarm-flagged references re-register their `IAlarmConditionSink` through `CapturingBuilder`. -5. **Restore subscriptions** — for every captured reference that still exists after rebuild, Core re-opens the driver subscription and restores the original ref-count. - -Exceptions during teardown are swallowed per decision #12 — a driver throw must not leave the node tree half-deleted. - -## Scope hint - -When `RediscoveryEventArgs.ScopeHint` is non-null (e.g. a folder path), Core restricts the diff to that subtree. This matters for Galaxy Platform-scoped deployments where a `time_of_last_deploy` advance may only affect one platform's subtree. Null scope falls back to a full-tree diff. - -## Virtual tags in the rebuild - -Per [ADR-002](v2/implementation/adr-002-driver-vs-virtual-dispatch.md), virtual (scripted) tags live in the same address space as driver tags and flow through the same rebuild. `EquipmentNodeWalker` (`src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/EquipmentNodeWalker.cs`) emits virtual-tag children alongside driver-tag children with `DriverAttributeInfo.Source = NodeSourceKind.Virtual`, and `DriverNodeManager` registers each variable's source in `_sourceByFullRef` so the dispatch branches correctly after rebuild. Virtual-tag script changes published from the Admin UI land through the same generation-publish path — the `VirtualTagEngine` recompiles its script bundle when its config row changes and `DriverNodeManager` re-registers any added/removed virtual variables through the standard diff path. Subscription restoration after rebuild runs through each source's `ISubscribable` — either the driver's or `VirtualTagSource` — without special-casing. - -## Active subscriptions survive rebuild - -Subscriptions for unchanged references stay live across rebuilds — their ref-count map is not disturbed. Clients monitoring a stable tag never see a data-change gap during a deploy, only clients monitoring a tag that was genuinely removed see the subscription drop. +- `sp_PublishGeneration` / `sp_ComputeGenerationDiff` are stubs that `RAISERROR` in the v3 baseline + migration; there is no generation-publish flow and no `DraftRevisionToken`. +- `GenericDriverNodeManager` (`Core/OpcUa/`) is retained scaffolding with no production caller; the + diff / teardown / subscription-replay "rebuild flow" described by earlier versions of this page does not + exist in `src/`. ## Key source files -- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs` — backend-change capability -- `src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/GenericDriverNodeManager.cs` — discovery orchestration -- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriver.cs` — `ReinitializeAsync` contract -- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` — publish-flow driver -- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs` — cluster singleton invoked by the Admin UI's `AdminOperationsClient` -- `docs/v2/config-db-schema.md` — `sp_PublishGeneration` + `sp_ComputeGenerationDiff` -- `docs/v2/admin-ui.md` — DiffViewer + draft-revision-token flow +- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs` — artifact composition + hash +- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` — dispatch + ack set +- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — apply + driver reconcile +- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs` — node-side artifact parsing +- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceComposer.cs`, `AddressSpacePlan.cs`, + `AddressSpaceChangeClassifier.cs`, `AddressSpaceApplier.cs` — compose, diff, classify, apply +- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs` — `IRediscoverable` consumer +- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/ISurgicalAddressSpaceSink.cs` — in-place update seam diff --git a/docs/drivers/Modbus.md b/docs/drivers/Modbus.md index 1f9e7262..7fbb7185 100644 --- a/docs/drivers/Modbus.md +++ b/docs/drivers/Modbus.md @@ -96,14 +96,15 @@ to the actual offending register(s). Per-tag escape hatch: from the driver's `DriverConfig` JSON. Key fields: - **Endpoint** — `Host`, `Port` (default 502), `UnitId`, `Timeout`. Per-tag `UnitId` overrides drive multi-slave gateway topology. +- **`Transport`** — `Tcp` (default; MBAP header + transaction id) or `RtuOverTcp` (RTU `[addr][PDU][CRC-16]` framing tunnelled over the socket to a serial-to-Ethernet gateway); string-valued, absent = `Tcp`. The Test Connect probe follows the authored mode. - **`Tags`** — the pre-declared `ModbusTagDefinition` list; this *is* the address space. - **`Probe`** — connectivity-probe interval / timeout / probe register (default register 0). - **Read/write caps** — `MaxRegistersPerRead` (125), `MaxRegistersPerWrite` (123), `MaxCoilsPerRead` (2000), plus `MaxReadGap` and `AutoProhibitReprobeInterval` for coalescing. - **Function-code overrides** — `UseFC15ForSingleCoilWrites`, `UseFC16ForSingleRegisterWrites` for PLCs that only accept multi-write codes. - **Resilience** — `AutoReconnect`, `KeepAlive`, `IdleDisconnectTimeout`, `Reconnect` backoff, and `WriteOnChangeOnly` redundant-write suppression. -Full per-field descriptions live in `ModbusDriverOptions.cs`. The JSON skeleton -is reproduced in [docs/v2/driver-specs.md §2](../v2/driver-specs.md). +Full per-field descriptions live in `ModbusDriverOptions.cs`. The v2-era JSON skeleton +in [docs/v2/driver-specs.md §2](../v2/driver-specs.md) predates the `Transport` key and the v3 raw-tag model. ## Testing diff --git a/docs/drivers/README.md b/docs/drivers/README.md index 9982505f..3a708051 100644 --- a/docs/drivers/README.md +++ b/docs/drivers/README.md @@ -103,7 +103,7 @@ Three properties worth knowing: - **Historian.Gateway** (server-side historian backend, not a tag driver) is documented in the main guide: - [../Historian.md](../Historian.md) — HistorianGateway backend: read-path registration, HistoryRead dispatch, alarm store-and-forward (`SendEvent`), continuous historization (`WriteLiveValues`), `EnsureTags` provisioning, config keys, deployment prerequisites. (The retired Wonderware sidecar backend it replaced: [Historian.Wonderware.md](Historian.Wonderware.md).) -- The full per-field spec (capability surface, config schema, addressing, data-type maps, connection settings, quirks for every driver) lives in [docs/v2/driver-specs.md](../v2/driver-specs.md). The overview pages above are the short path; that file is the authoritative per-driver reference. +- The full per-field spec (capability surface, config schema, addressing, data-type maps, connection settings, quirks for every driver) lives in [docs/v2/driver-specs.md](../v2/driver-specs.md). The overview pages above are the short path and are the current reference; that file is the v2-era per-field spec (see its status banner for which drivers it covers). ## Test-fixture coverage maps @@ -114,7 +114,7 @@ Each driver has a dedicated fixture doc that lays out what the integration / uni - [Siemens S7](S7-Test-Fixture.md) — Dockerized `python-snap7` server; DB/MB read + write round-trip verified end-to-end on `:1102` - [AB Legacy](AbLegacy-Test-Fixture.md) — Dockerized `ab_server` PCCC mode across SLC500 / MicroLogix / PLC-5 profiles (task #224); N/F/L-file round-trip verified end-to-end. `/1,0` cip-path required for the Docker fixture; real hardware uses empty. Residual gap: bit-file writes (`B3:0/5`) still surface BadState — real HW / RSEmulate 500 for those - [TwinCAT](TwinCAT-Test-Fixture.md) — XAR-VM integration scaffolding (task #221); three smoke tests skip when VM unreachable. Unit via `FakeTwinCATClient` with native-notification harness -- [FOCAS](FOCAS-Test-Fixture.md) — no integration fixture, unit-only via `FakeFocasClient`; Tier C out-of-process isolation scoped but not shipped +- [FOCAS](FOCAS-Test-Fixture.md) — no integration fixture, unit-only via `FakeFocasClient`; FOCAS runs in-process (Tier A) — the Tier C out-of-process isolation once scoped was never built - [OPC UA Client](OpcUaClient-Test-Fixture.md) — Dockerized `opc-plc` integration suite (task #215): real Secure Channel + Session, read + subscribe verified end-to-end; write not yet exercised in the integration suite; exhaustive capability matrix (reconnect, failover, cert-auth, history, alarms) via unit suite with mocked `Session` - [MQTT](Mqtt-Test-Fixture.md) — Dockerized `eclipse-mosquitto` with **TLS + real auth on both listeners (no anonymous fallback)** + a JSON-publisher sidecar; env-gated live suite (**15 tests**) — 7 plain (TLS/auth connect, CA pinning both ways, subscribe→value, retained seeding, wrong-password rejection) + 8 Sparkplug against a **project-owned C# edge-node simulator** on the `sparkplug` compose profile (birth→alias bind, DDATA, NDEATH→STALE, rebirth NCMD round-trip). Births are never retained, so `docker restart otopcua-sparkplug-sim` is how you force a fresh one - [Galaxy](../v1/drivers/Galaxy-Test-Fixture.md) — richest harness: gateway E2E + ZB SQL live-smoke + MXAccess opt-in (v1 archive) @@ -125,5 +125,5 @@ Each driver has a dedicated fixture doc that lays out what the integration / uni - [HistoricalDataAccess.md](../v1/HistoricalDataAccess.md) — `IHistoryProvider` dispatch, aggregate mapping, continuation points. The OPC UA Client driver is the only driver that implements driver-side `IHistoryProvider` (it forwards HistoryRead to the upstream server); the AVEVA Historian path is served server-side by the HistorianGateway-backed `IHistorianDataSource` instead. Other drivers do not implement the interface and return `BadHistoryOperationUnsupported`. - [AlarmTracking.md](../AlarmTracking.md) — `IAlarmSource` event model and filtering. Implemented by Galaxy (native MxAccess alarms, working end-to-end), OPC UA Client, AB CIP, and FOCAS; AB Legacy, Modbus, S7, TwinCAT, and MQTT have no alarm source. - [Subscriptions.md](../v1/Subscriptions.md) — how the Server multiplexes subscriptions onto `ISubscribable.OnDataChange`. -- [docs/v2/driver-stability.md](../v2/driver-stability.md) — tier system (A / B / C), shared `CapabilityPolicy` defaults per tier × capability, `MemoryTracking` hybrid formula, and process-level recycle rules. +- [docs/v2/driver-stability.md](../v2/driver-stability.md) — historical design record of the A / B / C tier model. Only `DriverTier` + the `CapabilityPolicy` defaults per tier × capability (`DriverResilienceOptions.GetTierDefaults`) are live; its `MemoryTracking` / recycle / out-of-process isolation machinery was deleted (#522) and every driver runs Tier A. - [docs/v2/plan.md](../v2/plan.md) — authoritative vision, architecture decisions, migration strategy. diff --git a/docs/operations/2026-07-16-secrets-clustered-master-key.md b/docs/operations/2026-07-16-secrets-clustered-master-key.md index 5b1d9e9f..c62343d0 100644 --- a/docs/operations/2026-07-16-secrets-clustered-master-key.md +++ b/docs/operations/2026-07-16-secrets-clustered-master-key.md @@ -31,35 +31,90 @@ plaintext secret on every node, no matter its role: combination — must unwrap the store with the exact same master key. A per-node KEK (e.g. a per-box DPAPI-protected key) would make each node decrypt every *other* node's ciphertext rows to garbage. -2. **Identical store rows on every node.** All nodes must read the same SQLite - database (same file, or a replicated/shared copy with the same rows) — not - independently-seeded stores that happen to share a KEK. +2. **The same store rows on every node of a pair.** Nodes must read the same rows — + not independently-seeded stores that merely share a KEK. Either mount one SQLite file + (see "Alternative: shared volume" below) or, preferred, turn on Akka secret replication + so each pair converges its own rows. -`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator` -— there is no built-in cross-node replication today. Meeting both requirements in -production is a deployment concern, covered below. +The base `ZB.MOM.WW.Secrets` store is a plain local SQLite `ISecretStore` whose default +replicator is `NoOpSecretReplicator`: **with `Secrets:Replication:Enabled=false` (the +default) nothing replicates and each node's store is private.** That is what this page +used to describe as the only option. It is no longer: pair replication exists +(`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`, lib pin `0.6.2` in `Directory.Packages.props`; +gated by `SecretsRegistration`, see below). -## Recommended interim posture (G-5) +## Recommended posture: pair-local Akka replication -Until real replication exists (G-7, below), the recommended production posture is: +Opt in with `Secrets:Replication:Enabled=true` on **every node** (default `false`, so absent +config means a plain local SQLite store). `SecretsRegistration.AddOtOpcUaSecrets` +(`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`) then registers +`AddZbSecretsAkkaReplication` **instead of** `AddZbSecrets` (the two are alternatives, never +additive) and a `SecretReplicationStarter` hosted service resolves the store at boot so a +node that never reads a secret still joins anti-entropy. Keys: `Secrets:Replication:Enabled`, +`:AnnounceInterval` (default `00:00:30`; each node publishes a name/revision/timestamp/tombstone +manifest per interval, no ciphertext), `:ActorName` (`zb-secret-replication`). -- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only - key file that is identical on every node of every role** — a base64-encoded - 32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed - to each node's filesystem/secret-mount by the deployment tooling, and **never - committed** to the repo. Treat it with the same discipline as any other - production secret (restrictive file ACLs, no logging, rotated via a future - KEK-rotation runbook — not yet built). -- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume** - that every node mounts (admin, driver, and dev/fused alike), so every node's - migrator opens and reads the same rows at boot. +- **Scope is PAIR-LOCAL.** Since mesh Phase 6 each application cluster is its own 2-node Akka + mesh and replication rides that mesh's DistributedPubSub, so a secret converges **within its + pair and nowhere else**. There is no fleet-wide sync (site nodes have no SQL hub). Seed each + pair independently: central via `/admin/secrets`, driver-only site pairs (no AdminUI) via the + secrets CLI against one node's store, or a ciphertext-only bundle import. See + `docs/Redundancy.md` §"Secrets replication is pair-local". +- **Both nodes of a pair must share the same KEK** — rows replicate verbatim (ciphertext, wrapped + DEK and `kek_id` untouched), never re-encrypted. +- **Deletes replicate** as soft tombstones (last-writer-wins on `updated_utc`/`revision`). +- **Re-wraps do not replicate** (a re-wrap preserves `revision`/`updated_utc`, so it is invisible to + last-writer-wins). KEK rotation therefore runs against each node's store — see the next section. +- The store should sit on a **persistent per-node path** (`Secrets:SqlitePath`, e.g. the node's + `/app/data` volume in containers); the relative default lands inside the image layer and is lost on + recreate. +- Evidence: live-gated PASS 4/4 on the three-mesh docker-dev rig (convergence <= 7 s, no cross-pair + leakage in any direction, delete < 1 s, clean restart without resurrection) — + `docs/plans/2026-08-07-secrets-pair-local-live-gate.md`. Replication stays default-off in the product. -Writes to the store are rare and human-driven — an operator using the -`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while -reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle, -regardless of role. The access pattern is read-mostly / effectively -single-writer, which is what makes a shared SQLite volume viable as an interim -posture (see caveat below). +Whichever store layout is used, the KEK is delivered per node as before: +**`Secrets:MasterKey:Source = File`** with `FilePath` pointing at a read-only base64 32-byte key +(`openssl rand -base64 32`), distributed identically to every node of the pair by deployment tooling and +**never committed**. Writes are rare and human-driven (`/admin/secrets` on admin nodes, or the +CLI); reads happen at every node's boot and each `ResolveCacheTtl` refresh. + +## Rotating the KEK + +Rotation re-wraps each row's per-secret DEK under the new KEK (`wrapped_dek`, `wrap_nonce`, +`wrap_tag`, `kek_id` change; the sealed body, `revision` and `updated_utc` do not). The +authoritative operator procedure and troubleshooting table live with the library: +`scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`, driven by the `secret rewrap-all` +CLI verb (in that repo; not packed). The OtOpcUa-specific points: + +1. **Back up every store first** (copy each node's SQLite file) — the backup is the complete undo. +2. **Stage the new KEK on every node of the pair** (same file path / env var as `Secrets:MasterKey` + resolves) but do not restart yet. Never pass key values on a command line; use + `--old-key-env VAR` / `--old-key-file PATH` and, if the new key is not the node's configured + provider, `--new-key-file`. +3. **Run `secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY` once per store, i.e. on EACH node.** + Because re-wraps do not replicate, rotating one node leaves its partner's rows on the old + `kek_id`. With replication on, do not leave the pair half-rotated: a node whose rows carry a KEK + it cannot unwrap fails closed (`Row wrapped by unknown KEK`) on resolve. +4. **Quiesce first**: stop (or drain) the node while its store is rewrapped. Rotating a pair + node-by-node while the other node keeps serving works for resolves, but do not author secrets + anywhere in the pair mid-rotation — a fresh write is wrapped under whichever KEK the writing node + currently uses and replicates verbatim to a partner that may not yet hold that KEK. +5. **Cut over**: confirm `Secrets:MasterKey` resolves the new key on both nodes, start them, and + confirm a `${secret:}` value resolves (the pre-host expander fails the boot loudly if not). +6. **Destroy the old key** only after both stores resolve cleanly. Re-running `rewrap-all` is + idempotent (`alreadyCurrent`), so an interrupted pass is resumed by running it again. +7. Rollback: restore the step-1 backups on both nodes and revert the KEK source. + +Pair rotation across **separate pairs** (central, site-a, site-b) is independent per pair; there is no +fleet-wide order. + +## Alternative: shared SQLite volume (no replication) + +The earlier interim posture — every node of a pair mounting one `Secrets:SqlitePath` on a shared +or replicated volume — still works for a deployment that keeps `Secrets:Replication:Enabled=false`: +reads dominate and writes are effectively single-writer. See the caveat below; it is not safe with +concurrent writers, and it is not available to DB-less driver-only site nodes that have no shared +volume with their partner. ## How it's delivered (do NOT commit these values) @@ -107,14 +162,14 @@ or key is missing. SQLite's file-locking model does not tolerate concurrent multi-writer access well over network filesystems (SMB/NFS locking is unreliable, and even on a clustered -block volume only one writer should be active at a time). The interim posture +block volume only one writer should be active at a time). The shared-volume alternative above is acceptable because: - Reads dominate (every node's boot + cache-refresh cycle, across every role). - Writes are rare, human-initiated, and effectively single-writer in practice (an operator runs the CLI/UI against one admin node at a time). -It is **not** a substitute for real replication, and it is not safe if multiple +It is **not** a substitute for real replication (Akka pair replication, above), and it is not safe if multiple nodes attempt concurrent writes. Do not build automation that writes secrets from more than one node simultaneously. @@ -128,31 +183,15 @@ ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryptio (KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption work — doing so risks invalidating active sessions for an unrelated reason. -It is, however, the model for where the *next* iteration of secret storage -should go — see the G-7 hand-off below. +## The G-7 hand-off (status) -## The G-7 hand-off - -The posture above is an interim, ops-only workaround. The long-term shape, -tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a -ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for -the Data Protection key ring: - -```csharp -services.AddDataProtection() - .PersistKeysToDbContext() - .SetApplicationName("OtOpcUa"); -``` - -(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`). -`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused -alike — a single MS SQL-backed source of truth for the Data Protection key ring; -the secret store is the natural next tenant of that same shared database instead -of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library -code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet, -overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not -built as part of this cut. This runbook's shared-SQLite-volume posture is the -bridge until G-7 lands. +G-7 (`scadaproj/components/secrets/GAPS.md`) asked for real cross-node replication. It was +built in two flavours in the library (`ZB.MOM.WW.Secrets.Replicator.SqlServer` and +`.AkkaDotNet`, plus a gRPC replicator package). **OtOpcUa production uses the Akka pair-local +option** (the decision recorded in scadaproj#3), wired by `SecretsRegistration` as described above; +the ConfigDb-backed store this page used to anticipate is not what shipped here. Driver-only nodes +have no ConfigDb since mesh Phase 4, which is a further reason a ConfigDb secret store would not +reach them. ## Dev/test/default posture (unchanged) diff --git a/docs/v2/driver-specs.md b/docs/v2/driver-specs.md index a56789d8..25c887f1 100644 --- a/docs/v2/driver-specs.md +++ b/docs/v2/driver-specs.md @@ -1,6 +1,22 @@ # OtOpcUa v2 — Driver Implementation Specifications -> **Status**: DRAFT — reference document for plan.md +> **Status (2026-10)**: v2-era DRAFT, **not authoritative**. It covers 8 of the 12 shipped drivers and +> several of its architecture claims are stale. The current per-driver references are the pages in +> [`docs/drivers/`](../drivers/README.md) (the README there lists all 12 and is the accurate overview). +> +> - **Not covered here at all:** MQTT/Sparkplug ([Mqtt.md](../drivers/Mqtt.md)), MTConnect +> ([MTConnect.md](../drivers/MTConnect.md)), Sql ([Sql.md](../drivers/Sql.md)), Calculation +> ([Calculation.md](../drivers/Calculation.md)). +> - **Hosting/tier claims are stale:** every driver runs in-process as Tier A (Galaxy reaches MXAccess over +> gRPC to `mxaccessgw`; FOCAS is a pure-managed in-process `FocasWireClient`, not a `Focas.Host` pipe +> service) — see `docs/v2/driver-stability.md`'s banner. The "SystemPlatform namespace kind" is retired: +> Galaxy is a standard Equipment-kind driver, and the address space is the dual Raw/UNS namespace +> ([../AddressSpace.md](../AddressSpace.md)). +> - **Modbus:** the driver has its own socket layer (no NModbus dependency) and a driver-level +> `transport` key — `Tcp` (default, MBAP framing) or `RtuOverTcp` (RTU `[addr][PDU][CRC-16]` framing tunnelled +> to a serial-to-Ethernet gateway); see [Modbus.md](../drivers/Modbus.md). +> - **AB CIP / AB Legacy:** a second, managed `Transport` (`Native` | `Libplctag`) exists — `CLAUDE.md` +> §"AB drivers: native transport" and the driver pages. > > **Created**: 2026-04-16 diff --git a/docs/v2/driver-stability.md b/docs/v2/driver-stability.md index c5be713d..cab9791e 100644 --- a/docs/v2/driver-stability.md +++ b/docs/v2/driver-stability.md @@ -49,6 +49,10 @@ Driver stability needs to be a first-class architectural concern, not a per-driv Every driver is assigned to one of three tiers based on the trust level of its dependency stack: +> **Design intent, not the shipped assignment.** The per-driver tier tables below record the proposed +> assignment (Galaxy/FOCAS in Tier C, the Ab/S7/TwinCAT drivers in Tier B). As shipped every driver is Tier A — +> see the banner above and `docs/drivers/README.md`, which is the accurate reference. + ### Tier A — Pure Managed Drivers whose entire dependency chain is verifiable .NET. Standard exception handling and Polly are sufficient. Run in-process in the main server. From f383996127536216eaf4bec6b407727338a809d6 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 8 Oct 2026 08:13:18 -0400 Subject: [PATCH 2/4] docs: soften shared-volume remark in secrets runbook (#623) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- docs/operations/2026-07-16-secrets-clustered-master-key.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/operations/2026-07-16-secrets-clustered-master-key.md b/docs/operations/2026-07-16-secrets-clustered-master-key.md index c62343d0..df2347f8 100644 --- a/docs/operations/2026-07-16-secrets-clustered-master-key.md +++ b/docs/operations/2026-07-16-secrets-clustered-master-key.md @@ -113,8 +113,7 @@ fleet-wide order. The earlier interim posture — every node of a pair mounting one `Secrets:SqlitePath` on a shared or replicated volume — still works for a deployment that keeps `Secrets:Replication:Enabled=false`: reads dominate and writes are effectively single-writer. See the caveat below; it is not safe with -concurrent writers, and it is not available to DB-less driver-only site nodes that have no shared -volume with their partner. +concurrent writers, and it is impractical for site pairs whose nodes share no volume. ## How it's delivered (do NOT commit these values) From 2befe9f9ba7959c71ac5e10bbfc1bec6e8ee6434 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 8 Oct 2026 08:16:43 -0400 Subject: [PATCH 3/4] docs: correct stale LDAP-role, host-probe, resilience-table, Bad-value claims; fix tasks.json (#624) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- deferment.md | 8 +++++--- docs/AlarmTracking.md | 8 +++++--- docs/Client.CLI.md | 2 +- docs/DriverLifecycle.md | 10 ++++++++-- docs/OpcUaServer.md | 5 +++-- docs/ScriptedAlarms.md | 2 +- docs/Telemetry.md | 6 +++--- docs/Uns.md | 5 +++-- docs/drivers/Calculation.md | 3 ++- docs/drivers/MTConnect.md | 9 +++++---- docs/drivers/Mqtt.md | 2 +- .../2026-08-15-perf-roadmap-remainder.md.tasks.json | 8 ++++---- docs/plans/2026-08-17-perf-followups-r3.md.tasks.json | 2 +- 13 files changed, 42 insertions(+), 28 deletions(-) diff --git a/deferment.md b/deferment.md index 42af16c7..93b16382 100644 --- a/deferment.md +++ b/deferment.md @@ -41,6 +41,8 @@ Nothing here is a data-loss or outage defect. The sharpest edge is §3 — silen ### 0.1 What is actually left (re-verified at `4bdce910`, 2026-08-04) +> **Refreshed 2026-10-08.** The re-verification below is as of `4bdce910` (2026-08-04) and is *not* current. Since then the #562–#591 issue sweep closed on master (`3baa332d`), and the work it deferred was filed as **#592–#625** — those are the live tracker, not "0 open". Re-derive counts from the Gitea API before relying on any figure in this subsection; the §4.1 rows for telemetry eviction, the `TODO(v3 WaveC)` set and the `/raw` Browse-device placeholder are resolved (struck through below). + > **Read this subsection, not the table below it.** The 2026-07-30 table that follows is preserved as the > record of what the remediation left behind; every tracker item in it has since closed. The live picture > is here. @@ -362,11 +364,11 @@ commented-out DI registrations.** Real deferred work lives in prose doc-comments | Item | Evidence | |---|---| -| **Telemetry snapshot cache never evicted** — a decommissioned driver replays last-known Health/Resilience to new subscribers forever | `TelemetryLocalHub.cs:40` `TODO(mesh-phase5+)` — self-declared, bounded, "pre-production, accepted" | -| **Device host recovered by string-matching instead of `DeviceConfig`** — 7 of the repo's 9 TODOs are this one item, across 4 drivers | `FocasDriver.cs:137,287`; `TwinCATDriver.cs:775`; `AbLegacyDriver.cs:557`; `AbCipDriver.cs:89,1272` + `AbCipDriverOptions.cs:130` — all `TODO(v3 WaveC)` | +| ~~**Telemetry snapshot cache never evicted** — a decommissioned driver replays last-known Health/Resilience to new subscribers forever~~ ✅ **FIXED** — `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`DriverHostActor.StopChild` via `plan.ToStop`, #609); central-side stores are not evicted | `TelemetryLocalHub.cs:40` `TODO(mesh-phase5+)` — self-declared, bounded, "pre-production, accepted" | +| ~~**Device host recovered by string-matching instead of `DeviceConfig`** — 7 of the repo's 9 TODOs are this one item, across 4 drivers~~ ✅ **RESOLVED (#608)** — no `TODO(v3 WaveC)` remains in `src/` (re-verified 2026-10-08) | `FocasDriver.cs:137,287`; `TwinCATDriver.cs:775`; `AbLegacyDriver.cs:557`; `AbCipDriver.cs:89,1272` + `AbCipDriverOptions.cs:130` — all `TODO(v3 WaveC)` | | **Galaxy writes can never confirm a COM commit** — empty status array ⇒ provisional Good, metered `galaxy.writes.unconfirmed` | `GatewayGalaxyDataWriter.cs:44,328,354` — **cross-repo**, blocked on mxaccessgw | | **Subscription liveness undetectable** — `ISubscriptionHandle` is opaque | `DriverInstanceActor.cs:527` (#488) — well-reasoned, deferred "until a real occurrence" | -| **`/raw` "Browse device" is still a placeholder** despite the universal discovery browser landing | `RawTree.razor:15,426` — **verify**; may be reachable now via `DiscoveryDriverBrowser` | +| ~~**`/raw` "Browse device" is still a placeholder** despite the universal discovery browser landing~~ ✅ **RESOLVED** — `RawTree.razor` wires `OnBrowseDevice` to the discovery modal | `RawTree.razor:15,426` — **verify**; may be reachable now via `DiscoveryDriverBrowser` | ### 4.2 Per-driver diff --git a/docs/AlarmTracking.md b/docs/AlarmTracking.md index 445037d0..afc82cc2 100644 --- a/docs/AlarmTracking.md +++ b/docs/AlarmTracking.md @@ -398,9 +398,11 @@ Scripted alarms use a separate inbound path that converges on the `OtOpcUaNodeManager` wires the Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) on each scripted-alarm `AlarmConditionState` node. Every call is **gated on the -`AlarmAck` LDAP role** — fail-closed: sessions with no role or without -`AlarmAck` group membership receive `BadUserAccessDenied` immediately. -The LDAP-resolved role set is carried past `OpcUaApplicationHost` by +`AlarmAck` role** — fail-closed: sessions with no role or without +an `AlarmAck`-conferring grant receive `BadUserAccessDenied` immediately. +The session's roles come from its API-key principal's `NodeAcl` grants (or the +reserved `anonymous` principal's; LDAP is AdminUI-only — `docs/security.md`) and are carried past +`OpcUaApplicationHost` by `RoleCarryingUserIdentity` (a `UserIdentity` subclass), making it readable inside the method handler at dispatch time. diff --git a/docs/Client.CLI.md b/docs/Client.CLI.md index 61b73e2a..e72d6403 100644 --- a/docs/Client.CLI.md +++ b/docs/Client.CLI.md @@ -250,7 +250,7 @@ otopcua-cli alarms -u opc.tcp://localhost:4840/OtOpcUa \ ### ack -Acknowledges an active alarm condition via the OPC UA Part 9 `Acknowledge` method. The server gates this on the **`AlarmAck`** LDAP role (fail-closed — a session without `AlarmAck` group membership gets `BadUserAccessDenied`; see [ScriptedAlarms.md](ScriptedAlarms.md) §"AlarmAck gate"). The `--event-id` is the hex-encoded `EventId` taken from a prior `alarms` subscription notification for the same condition. +Acknowledges an active alarm condition via the OPC UA Part 9 `Acknowledge` method. The server gates this on the **`AlarmAck`** role (fail-closed — a session whose API-key principal holds no `AlarmAck`-conferring `NodeAcl` grant gets `BadUserAccessDenied`; the data plane does not use LDAP; see [ScriptedAlarms.md](ScriptedAlarms.md) §"AlarmAck gate"). The `--event-id` is the hex-encoded `EventId` taken from a prior `alarms` subscription notification for the same condition. ```bash otopcua-cli ack -u opc.tcp://localhost:4840/OtOpcUa \ diff --git a/docs/DriverLifecycle.md b/docs/DriverLifecycle.md index 60320bc8..c1651178 100644 --- a/docs/DriverLifecycle.md +++ b/docs/DriverLifecycle.md @@ -311,12 +311,18 @@ shutdown paths deliberately keep the single trailing write. > **A value consumer never reports a quality better than its source's.** A node derived from a driver > value — the raw node, every UNS mirror, a Calculation output, a virtual tag, a scripted condition's > quality — carries, at worst, the worst status among its inputs: a Bad input makes the derived node Bad -> **with that input's sub-code and the last-known value**; an Uncertain input makes it Uncertain; a source +> **with that input's sub-code** (the last-known value is kept server-side, but see the note below: a client never sees it beside a Bad status); an Uncertain input makes it Uncertain; a source > that has stopped communicating is a Bad input (`BadNoCommunication`) for everything downstream of it, > **including the external OPC UA data plane**. Recovery is a *publish*, never an assumption: the first > post-recovery value must land even when it equals the value held before the outage, so every dedup on > the path — driver caches, the node's own status, a consumer's last-published memory — must be keyed on > **(value, status)**, not value alone. +> +> **"Value retained" is server-side state, not wire-observable.** The node keeps its last value under the +> Bad status (that is what lets the next Good re-publish carry it and what the `(value, status)` dedup +> compares against), but the OPC UA SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read +> and monitored-item sample, so a client sees `Value = null` with the Bad status and never the last-known +> value. Pinned server-side by `OtOpcUaNodeManagerWriteStatusesTests`; observation: `archreview/plans/R3-02-stale-good-family-plan.md`. This is one rule with one implementation of its comparison, `ZB.MOM.WW.OtOpcUa.Core.Abstractions.InputQuality` (`WorstOf` / `IsUsable`), used by every derived-value @@ -325,7 +331,7 @@ a fifth (archreview round 3, theme "the stale-Good family"). The seams that impl | Seam | What it does | Where | |---|---|---| -| Driver comms loss → **value nodes** | one status-only batch stamps `BadNoCommunication` on the driver's raw NodeIds and every referencing UNS NodeId, **value retained** | `DriverHostActor.FanCommsLossToValueNodes` (below) | +| Driver comms loss → **value nodes** | one status-only batch stamps `BadNoCommunication` on the driver's raw NodeIds and every referencing UNS NodeId, **value retained server-side** (clients see a null value with the Bad status) | `DriverHostActor.FanCommsLossToValueNodes` (below) | | Driver comms loss → dependency mux | synthetic Bad per ref, so scripts and alarms see the source die | `DriverHostActor.FanCommsLossToDependencyMux` | | Driver comms loss → Part 9 conditions | condition quality annotated out of band | `DriverHostActor.OnDriverConnectivityChanged` | | MQTT / Sparkplug | connection loss stales observed refs; every (re)subscribe pushes the cached snapshot | [`docs/drivers/Mqtt.md`](drivers/Mqtt.md#quality-on-connection-loss-and-re-subscribe) | diff --git a/docs/OpcUaServer.md b/docs/OpcUaServer.md index 1c17345c..b9f7f6f5 100644 --- a/docs/OpcUaServer.md +++ b/docs/OpcUaServer.md @@ -56,8 +56,9 @@ The server binds a TCP endpoint at `opc.tcp://{PublicHostname}:{OpcUaPort}/OtOpc `OpcUaApplicationHost` subscribes to `SessionManager.ImpersonateUser` after `ApplicationInstance.Start`. The handler (`HandleImpersonation`) deals with the token types as follows: -- `UserNameIdentityToken` → the password is decrypted, then `IOpcUaUserAuthenticator.AuthenticateUserNameAsync` validates the credential (`LdapOpcUaUserAuthenticator` in production, a stub in tests). On success a `UserIdentity` carrying the token is attached and the LDAP-derived roles are logged; on failure `ImpersonateEventArgs.IdentityValidationError` is set to `BadIdentityTokenRejected`. -- `AnonymousIdentityToken` and X.509 tokens → the handler returns without intervening, so the SDK's default validation stands. +- `UserNameIdentityToken` → the password is decrypted, then `IOpcUaUserAuthenticator.AuthenticateUserNameAsync` validates the credential (`ApiKeyOpcUaUserAuthenticator` in production — fixed username `apikey`, `otua__` password, no network I/O — and a stub in tests). On success a `RoleCarryingUserIdentity` carrying the token and the key's resolved roles is attached; on failure `ImpersonateEventArgs.IdentityValidationError` is set to `BadIdentityTokenRejected`. +- `AnonymousIdentityToken` → admitted only when `OpcUa:AllowAnonymous` (default `true`); the session then authenticates as the reserved `anonymous` principal and carries whatever roles `NodeAcl` rows naming it grant (`HandleAnonymousImpersonation`). A failure to read those grants denies. +- X.509 tokens → the handler returns without intervening, so the SDK's default validation stands. Decryption failures and authenticator exceptions also map to `BadIdentityTokenRejected`. diff --git a/docs/ScriptedAlarms.md b/docs/ScriptedAlarms.md index b7b61e4f..eeebc949 100644 --- a/docs/ScriptedAlarms.md +++ b/docs/ScriptedAlarms.md @@ -347,7 +347,7 @@ Operators interact with active scripted alarms through two surfaces — both con ### AlarmAck gate (OPC UA method path) -`OtOpcUaNodeManager` wires the OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve / Enable / Disable) on each `AlarmConditionState` node. Every method call is gated on the `AlarmAck` LDAP role — fail-closed: a session with no resolved roles or no `AlarmAck` group membership receives `BadUserAccessDenied` immediately without reaching the engine. The role is carried on the session by `RoleCarryingUserIdentity` (a `UserIdentity` subclass that preserves the LDAP-resolved role set past `OpcUaApplicationHost`). +`OtOpcUaNodeManager` wires the OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve / Enable / Disable) on each `AlarmConditionState` node. Every method call is gated on the `AlarmAck` role — fail-closed: a session with no resolved roles or no `AlarmAck`-conferring grant on its API-key principal (`NodeAcl`; LDAP is AdminUI-only, see `docs/security.md`) receives `BadUserAccessDenied` immediately without reaching the engine. The role is carried on the session by `RoleCarryingUserIdentity` (a `UserIdentity` subclass that preserves the API-key-resolved role set past `OpcUaApplicationHost`). On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` (containing command kind, condition id, comment, and operator principal) onto the `alarm-commands` DPS topic. The node manager itself stays Akka-free: the dispatch action is a settable `Action` injected at boot by the hosted service. diff --git a/docs/Telemetry.md b/docs/Telemetry.md index b4430173..f653a071 100644 --- a/docs/Telemetry.md +++ b/docs/Telemetry.md @@ -140,9 +140,9 @@ Three things to know before touching it: The digest is a flattened, host-name-ordered **string** for the converse reason: a tuple holding an `IReadOnlyList` compares by reference and would re-publish on every 30 s heartbeat forever. -⚠️ **`DriverInstanceResilienceStatus` is the same defect, still open.** That table also has no writer -and no reader — the only reference in the repo is its DbSet declaration — while the live data reaches -the AdminUI over the `driver-resilience-status` channel above. Keep-or-delete is Gitea **#524**. +**`DriverInstanceResilienceStatus` was dropped (Gitea #524, migration +`DropDriverInstanceResilienceStatusTable`).** The table had no writer and no reader; the live data +reaches the AdminUI over the `driver-resilience-status` channel above, which is its only home. ## The three non-migrated channels (NOT migrated in Phase 5 — do not read this as "seven done") diff --git a/docs/Uns.md b/docs/Uns.md index 15a1277a..575fe19e 100644 --- a/docs/Uns.md +++ b/docs/Uns.md @@ -391,7 +391,8 @@ See the individual driver docs under `docs/drivers/` for per-driver implementati An equipment tag is **writable by default** — the materialised OPC UA node is created `CurrentReadWrite` and inbound client writes are routed to the device (subject to the -`WriteOperate` LDAP role gate). Setting the optional `"writable": false` key in a tag's +`WriteOperate` gate — a role the session's API-key principal earns from its `NodeAcl` grants, see +[security.md](security.md)). Setting the optional `"writable": false` key in a tag's `TagConfig` makes the node **read-only** at the driver seam. The six typed tag editors (Modbus, S7, AB CIP, AB Legacy, TwinCAT, FOCAS) expose this as a **Writable** checkbox (checked by default); it can also be hand-authored in the raw-JSON textarea. @@ -405,7 +406,7 @@ An equipment tag is **writable by default** — the materialised OPC UA node is The key is honoured by Modbus, S7, AB CIP, AB Legacy, and TwinCAT. **FOCAS forces read-only regardless** — FOCAS PMC writes are unsupported in the wire client, so its editor shows the checkbox disabled/unchecked and any authored `writable:true` is ignored -(and flagged at the deploy gate below). Node-level authz (LDAP `WriteOperate`) remains the +(and flagged at the deploy gate below). Node-level authz (`WriteOperate` via the API-key principal's grants, not LDAP) remains the effective write gate above this flag. ## Deploy-time TagConfig validation (`Deployment:TagConfigValidationMode`) diff --git a/docs/drivers/Calculation.md b/docs/drivers/Calculation.md index e808467a..db7f6f7f 100644 --- a/docs/drivers/Calculation.md +++ b/docs/drivers/Calculation.md @@ -121,7 +121,8 @@ Notes worth knowing before authoring against this: ## Error semantics An evaluator failure publishes **Bad quality carrying the last-known value** — the value is not -zeroed, because a stale-but-flagged reading is more useful to an operator than a fabricated one. It +zeroed, so the node holds it server-side for the next Good re-publish and the `(value, status)` dedup. (Over the wire the +OPC UA SDK nulls the value of a Bad-status read, so a client sees only the Bad status — see `docs/DriverLifecycle.md`.) It publishes: - **once per Good→Bad transition**, not on every failed evaluation (a permanently broken script does diff --git a/docs/drivers/MTConnect.md b/docs/drivers/MTConnect.md index 3768a4c1..1068984b 100644 --- a/docs/drivers/MTConnect.md +++ b/docs/drivers/MTConnect.md @@ -378,10 +378,11 @@ for anything beyond values-and-conditions. — it would materialise nodes nobody approved and no deployment artifact records. - **`IHostConnectivityProbe` is still unconsumed** — `GetHostStatuses()` has - no production call site and nothing writes a `DriverHostStatus` row. That - half remains a fleet-wide gap affecting every driver that implements it - (Gitea #521). + **`IHostConnectivityProbe` is consumed** (Gitea #521, closed) — + `DriverInstanceActor` subscribes `OnHostStatusChanged`, pulls + `GetHostStatuses()` fresh, and the result rides `DriverHealthChanged` to + the AdminUI `/hosts` page (no SQL; the `DriverHostStatus` table was + dropped). See `docs/Telemetry.md`. 2. **`RawTagEntry.DeviceName` is accepted on the wire shape but ignored for routing.** One driver instance owns exactly one Agent client scoped by `MTConnectDriverOptions.DeviceName` (the top-level config key); the diff --git a/docs/drivers/Mqtt.md b/docs/drivers/Mqtt.md index 5a064550..bfbb3dc1 100644 --- a/docs/drivers/Mqtt.md +++ b/docs/drivers/Mqtt.md @@ -428,7 +428,7 @@ which is archreview 05/STAB-18. - **Connection loss stales, keeping the value.** On the `Connected → Reconnecting` edge `MqttConnection.ConnectionLost` fires, and both ingest paths mark every **already-observed** - reference `BadNoCommunication` with its **last value retained** and publish the change. A + reference `BadNoCommunication` with its **last value retained** (server-side; a client reads the Bad status with a null value) and publish the change. A never-observed reference (still `BadWaitingForInitialData`) and an already-Bad one are left alone. The event does **not** fire for a failed *initial* connect or during dispose — neither is an outage. - **Every (re)subscribe pushes the cached snapshots.** `SubscribeAsync` republishes each observed diff --git a/docs/plans/2026-08-15-perf-roadmap-remainder.md.tasks.json b/docs/plans/2026-08-15-perf-roadmap-remainder.md.tasks.json index b07c6c1d..6765c402 100644 --- a/docs/plans/2026-08-15-perf-roadmap-remainder.md.tasks.json +++ b/docs/plans/2026-08-15-perf-roadmap-remainder.md.tasks.json @@ -1,19 +1,19 @@ { "planPath": "docs/plans/2026-08-15-perf-roadmap-remainder.md", "tasks": [ - {"id": 1, "subject": "Task 1: Outbox batch append with group commit", "status": "in_progress", "note": "6ff64ca0 + fix 73205865 (incremental bookkeeping, second-reader durability pin); re-review in flight"}, + {"id": 1, "subject": "Task 1: Outbox batch append with group commit", "status": "completed", "note": "6ff64ca0 + fix 73205865 (incremental bookkeeping, second-reader durability pin); merged to master (verified 2026-10-08)"}, {"id": 2, "subject": "Task 2: Batched mux→subscriber delivery, single-eval virtual tags", "status": "completed", "note": "content inside 140bf0c1 (shared-index sweep; message belongs to Task 22 — needs an outcomes note); spec + code approved; code review's two Importants (yielding-fake batch test, mid-batch exception policy) folded into Task 4's brief"}, {"id": 3, "subject": "Task 3: Parallel drain with bounded concurrency", "status": "completed", "note": "a76b4e07; spec + code APPROVED; follow-up (inherited, pre-existing): bare catch around WriteLiveValuesAsync swallows OperationCanceledException at shutdown → spurious _tagWriteFailures increments when succeeded>0; recommend rethrowing genuine ct cancellations"}, {"id": 4, "subject": "Task 4: Recorder batched ingest — one fsync per burst", "status": "completed", "blockedBy": [1, 2, 3], "note": "384568b4 + follow-up 451273c8 (append-failure branch pinned via 3 independent mutations, loss-accounting docs corrected, recorder 64→1024 literals aligned); all reviews APPROVED; register: #491 falsification nondeterministic under load, IngestFaults not surfaced in docs/gauges, latch re-arms on any non-throwing item"}, {"id": 5, "subject": "Task 5: Historian defaults, config drift, docs", "status": "completed", "blockedBy": [3, 4], "note": "0298a0b9 (new Historian.md key table, cross-repo batch-RPC follow-up recorded); code APPROVED; recorder 64→1024 literals aligned in 451273c8; minor: pinning-test doc comment overclaims appsettings-drift coverage"}, {"id": 6, "subject": "Task 6: Per-session decision-cache shards", "status": "completed", "note": "debd44c2; follow-ups: _globalCount drift under races, concurrency stress test, bound-ratio comment"}, - {"id": 7, "subject": "Task 7: Authorizer hot path — lookup first, no closures, lazy deny reasons", "status": "in_progress", "blockedBy": [6]}, + {"id": 7, "subject": "Task 7: Authorizer hot path — lookup first, no closures, lazy deny reasons", "status": "completed", "blockedBy": [6], "note": "d8a17f2e; merged to master (verified 2026-10-08)"}, {"id": 8, "subject": "Task 8: Browse and TranslateBrowsePath compaction", "status": "completed", "note": "541a223b + follow-up test aad7583b (survivor-order pin, falsified red, review APPROVED); minor: FanOutTargetIdList fake couples to SDK single-Add contract — comment if pattern reused"}, - {"id": 9, "subject": "Task 9: RevalidateMonitoredItems dedup", "status": "in_progress", "blockedBy": [8], "note": "fbe8aaad; realm-qualified 3-tuple key (safer than spec); spec+code reviews in flight; watch: its clock-count observable couples to DataPlaneAuthorizer reading the clock once — Task 7 edits that file"}, + {"id": 9, "subject": "Task 9: RevalidateMonitoredItems dedup", "status": "completed", "blockedBy": [8], "note": "fbe8aaad; realm-qualified 3-tuple key (safer than spec); merged to master (verified 2026-10-08); watch: its clock-count observable couples to DataPlaneAuthorizer reading the clock once — Task 7 edits that file"}, {"id": 10, "subject": "Task 10: Modbus InitializeAsync transport leak", "status": "completed", "note": "c3970a2e + 409a58b6"}, {"id": 11, "subject": "Task 11: PollGroupEngine fixed-rate pacing, overrun counter, async unsubscribe", "status": "completed", "blockedBy": [10, 12, 13, 14], "note": "9cf400f1 + tolerance fix 485f4ea1 (fast-reader test: 200ms interval, <=1 blip); spec ✅ + code APPROVED; follow-ups: NoteOverrun conflates slow-success/slow-failure (doc caveat), backoffCap param doc note for future callers"}, {"id": 12, "subject": "Task 12: S7 narrow-scalar span codec", "status": "completed", "note": "8d91833f; SByte spec mention was a plan error"}, - {"id": 13, "subject": "Task 13: S7 span planner + grouped ReadAsync", "status": "in_progress", "blockedBy": [12], "note": "39f67a0f; spec passed (demotion rule + factory-binding 6th file accepted); code review in flight; follow-up: live gate vs s7_1500 fixture before trusting 16/220 defaults"}, + {"id": 13, "subject": "Task 13: S7 span planner + grouped ReadAsync", "status": "completed", "blockedBy": [12], "note": "39f67a0f; spec passed (demotion rule + factory-binding 6th file accepted); merged to master (verified 2026-10-08); follow-up: live gate vs s7_1500 fixture before trusting 16/220 defaults"}, {"id": 14, "subject": "Task 14: AbCip bounded-parallel reads", "status": "completed", "note": "d6a4fbd6 + fix d5f573b0 (per-device ConnectGate single-flight); re-review APPROVED; minor follow-ups: stale EvictRuntime-race comment ~L1161, test timing margin on slow CI"}, {"id": 15, "subject": "Task 15: Dirty-set coalescing in ScriptedAlarmEngine", "status": "completed", "note": "5df793e7; follow-ups: TrackBackgroundTask registration-ordering gap (pre-existing), stale line-334 comment, non-volatile _disposed"}, {"id": 16, "subject": "Task 16: Deployments list projection", "status": "completed", "note": "dbc9a132"}, diff --git a/docs/plans/2026-08-17-perf-followups-r3.md.tasks.json b/docs/plans/2026-08-17-perf-followups-r3.md.tasks.json index 10258dd7..4cbd7357 100644 --- a/docs/plans/2026-08-17-perf-followups-r3.md.tasks.json +++ b/docs/plans/2026-08-17-perf-followups-r3.md.tasks.json @@ -8,7 +8,7 @@ {"id": 5, "subject": "Task 5: Live gate — FetchAndCache legs", "status": "completed", "note": "GATE PASSED, no commit. All 6 legs: fetch+SHA-verify on all 4 site nodes (1584 bytes from central-1:4055), Served×4 central-side, applied-FETCHED + cached on site-a-1 (acks=6), boot-from-cache after restart (restored rev matches what it cached 52s earlier), absence proof (0 Stale/ConfigDb-back/Direct-form/ERR/WRN over full lifetime), post-restart browse proves both namespaces materialised (ns=2 raw + ns=3 UNS). Method: same-revision deploys short-circuit the fetch path (immediate-ACK at HandleDispatchFromSteady) — gate forced a temp UnsArea row then reverted; rig config restored byte-identical (rev 315de861). Register: central-2 serve path unexercised (first-success endpoint walk always hit central-1)"}, {"id": 6, "subject": "Task 6: Live gate — alarm-bearing surgical ratchet", "status": "completed", "note": "GATE PASSED, no commit. Four deploys, all routings correct: Writable flip ⇒ kind=AttributeOnly surgicalRawTags=1 rebuild=False; alarm ADD ⇒ kind=Rebuild rebuild=True; alarm REMOVE ⇒ rebuild=True (previous-side disqualifies); restore ⇒ surgical again (ratchet per-deploy, not latched). All sealed 6/6 acks. Authoring: direct UPDATE on the rig's OWN sql container (Database=OtOpcUa — NOT 10.100.0.35; sqlcmd needs -I for QUOTED_IDENTIFIER). Findings: only cluster-MAIN nodes emit applier lines (site plans empty via ResolveClusterScope, apply early-returns); artifact hash can never return to baseline after any UPDATE — ConfigComposer snapshots SQL rowversion (verified by gunzip+diff: only the RowVersion differs). Rig config semantically restored; rig UP"}, {"id": 7, "subject": "Task 7: Live gate — ACL-enabled enforcement on site-a", "status": "completed", "note": "GATE PASSED, no commit. TOPOLOGY ADAPTATION: gated node = central-1 via scratchpad-only compose overlay (site-a serves NO value nodes — the rig's one tag is cluster MAIN; central-2 became the ungated A/B control). ALLOW leg genuinely load-bearing: ungated baseline captured first — same key+call DENIED 0x801F0000 with ACL off (Tag grant confers no role), SUCCEEDED with ACL on ⇒ ACL decision REPLACES the role check, live. Deny×3 (anonymous write, out-of-scope browse, ungated control reads fine); revocation split into grant-removal (gen 1→2, primed per-key shard discarded — exercises this round's CAS-loop path) and key-disable (pre-cache re-check). Authoring: DB fallback (Chrome extension not connected) — ApiKeySecret hashing reproduced from source, pepper verified live. Restore byte-identical (rev a3a9061b), zero residue. FINDINGS for register: (1) browse transit stops below the realm root — grant holder cannot browse DOWN to its own tag from ns=2;s=OtOpcUa (fails closed; file as issue); (2) ns=0 core nodes are ungated (CoreNodeManager bypasses DataPlaneAuthorizer) — boot warning overstates 'see nothing'"}, - {"id": 8, "subject": "Task 8: Close-out — sweep + Outcomes + register annotations", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 9, 10], "note": "c4d6cba7 (Outcomes + sweep record + r2 register annotations, +383/−0) + the coordinator's flake-register append. Rig torn down BEFORE the sweep (rig-up skews skips: Host.IntegrationTests 229/0/2 up vs 226/0/5 down). Build 0 errors / 904 warnings — IDENTICAL to r2 (note: incremental builds under-report at 896). Sweep 43/43 green: 10,377 / 0 / 19 — +7 reconciled exactly (OpcUaServer +1 T1, S7 +6 T2); skips identical project-for-project; AdminUI 1540/0-skipped + Configuration 190/0-skipped ran vs 10.100.0.35,14330. Host.IntegrationTests (outside the 43-project baseline) +9 vs merge-base (LocalDbSetupTests 7→15, WiringTests 6→7). One flake isolated: Client.CLI LoggerLifecycleTests — pre-existing global-Serilog race, no r3 commit touches it, registered"} + {"id": 8, "subject": "Task 8: Close-out — sweep + Outcomes + register annotations", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 9, 10], "note": "c4d6cba7 (Outcomes + sweep record + r2 register annotations, +383/−0) + the coordinator's flake-register append. Rig torn down BEFORE the sweep (rig-up skews skips: Host.IntegrationTests 229/0/2 up vs 226/0/5 down). Build 0 errors / 904 warnings — IDENTICAL to r2 (note: incremental builds under-report at 896). Sweep 43/43 green: 10,377 / 0 / 19 — +7 reconciled exactly (OpcUaServer +1 T1, S7 +6 T2); skips identical project-for-project; AdminUI 1540/0-skipped + Configuration 190/0-skipped ran vs 10.100.0.35,14330. Host.IntegrationTests (outside the 43-project baseline) +9 vs merge-base (LocalDbSetupTests 7→15, WiringTests 6→7). One flake isolated: Client.CLI LoggerLifecycleTests — pre-existing global-Serilog race, no r3 commit touches it, registered"}, {"id": 9, "subject": "Task 9: Roslyn evaluator racing test — remove timing assumption (ScadaBridge row 38)", "status": "completed", "note": "34829ea0 + 837d8a17 (review round). Option (b): renamed to Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure; three-bucket classification (leaked=fail-first, exhausted + starved each bounded max(evals/10,8)); duration-bounded loops (10s OR 400 iters) — the REAL load failure was the test's own 45s WaitAsync budget under recompile pressure (old shape RED 4/4 at 4-way concurrency, new GREEN 4/4 + 8/8); reviewer-found starvation-timeout flake vector closed (TimedScriptEvaluator's 5s clock starts at enqueue), falsified via 12ms-budget probe (old classification RED, new GREEN at 0.11%); literals promoted to shared ScriptEvalFailureReasons in Commons (both evaluators + test). Honest disclosure: disposed race does not reproduce on this arm64 box at all. Spec ✅ + code CHANGES REQUESTED → delta re-APPROVED. Host.IntegrationTests 229/0/2, Calculation 53/53"}, {"id": 10, "subject": "Task 10: LocalDb conditional registration + baseline + deregister self-heal (ScadaBridge row 39)", "status": "completed", "note": "3e62f161 + df533916 (review round: content-based data-survival pin, mid-loop loud-fail pin via exhaustive delegating fake, Log.Logger assumption comment). Spec ✅ + code APPROVED + delta re-APPROVED. Predicate = PeerAddress OR ApiKey (SyncListenPort deliberately out — passive endpoint fail-closed on key); 0.2.1 APIs verified by reflection+probe (baseline seeds hlc=0; DeregisterReplicated returns bool) — NO pin bump; 0.3.1 now published (ScadaBridge heads-up) — bump recorded as deliberate both-nodes-together future change (bookkeeping v2→v3 one-way). Surfaced+fixed two vacuous-fixture bugs (PairHarness/AlarmSfLegacyMigrator two-config split). scadaproj index updated + pushed (1145df6). Suites: LocalDbSetupTests 15/15, Host.IntegrationTests 226/0/5"} ], From 0a5d03da951dd02c604fc30c5e90a68f8c43079d Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 8 Oct 2026 08:19:30 -0400 Subject: [PATCH 4/4] docs: apply CLAUDE-FACTS to CLAUDE.md, fix AlarmAck LDAP wording, retitle KNOWN LIMITATION 2 (#624) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- CLAUDE.md | 115 ++++++++++++++++++++++++++-------- docker-dev/docker-compose.yml | 8 ++- 2 files changed, 93 insertions(+), 30 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 3d65298b..63feeb58 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -186,10 +186,13 @@ is still default-closed; what changed is which deltas it whitelists. bounded resync). A **rebuild** still advances even with failures. - **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is what makes every whitelist above safe. -- ⚠️ **Known gap, no longer neutral:** `AnnounceAddedNodes` is Uns-only, so surgically-added - **raw** containers/tags are materialised but never announced — previously invisible (the - rebuild forced a re-browse anyway), now genuinely undiscoverable until a client re-browses. - Added UNS reference variables ARE announced. +- **Announcements:** `AnnounceAddedNodes` announces surgically-added **Raw** containers/tags + (`ComputeRawAddAnnouncements`, #599 — the former "raw adds are undiscoverable until a re-browse" gap is + closed) as well as added UNS reference variables. +- **Array writability rides the same whitelist:** `ArrayWritable` joins the surgical raw-tag attributes and + `UnsReferenceVariable` carries `BackingArrayWritable` (see "Equipment-node writes" under Authentication). Removing + an equipment that has references removes the references before the equipment subtree, and + `RemoveEquipmentSubtree` severs the inverse `Organizes` edges. See `docs/plans/2026-08-15-surgical-deploys.md`, `docs/Uns.md` and `docs/Raw.md`. @@ -268,10 +271,12 @@ deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`Tag `DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` / `DiscoveryStreamIncludesAuthoredTags`. -⚠️ **`IHostConnectivityProbe` is still unconsumed.** `GetHostStatuses()` has no production call site and -nothing ever writes a `DriverHostStatus` row (the table was re-created in the v3 initial migration, so -confirm intent before deleting). The separable `OnHostStatusChanged` event is the useful half. Tracked -as Gitea **#521**. +**`IHostConnectivityProbe` is consumed (Gitea #521, closed).** `DriverInstanceActor` subscribes +`OnHostStatusChanged`, pulls `GetHostStatuses()` fresh, and the host set rides `DriverHealthChanged` to the +AdminUI `/hosts` page — no SQL: the `DriverHostStatus` table is dropped, and so is +`DriverInstanceResilienceStatus` (#524; the live data goes over the `driver-resilience-status` telemetry +channel only). **Multi-host dispatch is wired too:** `DriverInstanceActor` resolves the breaker/host key +per call via `IPerCallHostResolver` (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip). ### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed) @@ -315,9 +320,12 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T quality contract" and implemented by `Core.Abstractions.InputQuality` (`WorstOf`/`IsUsable`). Three things changed with it, all client-visible: - ⚠️ **Value nodes of a comms-lost driver now go Bad.** `DriverHostActor.FanCommsLossToValueNodes` - stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained**, - via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host holds no value - cache, so only the node can keep the value). #481 had wired comms loss to the mux and to Part 9 + stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained + server-side**, via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host + holds no value cache, so only the node can keep the value). ⚠️ Retention is **not client-observable**: + the SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read/sample, so clients see + `Value = null` + the Bad status; the retained value is what the next Good re-publish and the + `(value, status)` dedup compare against. #481 had wired comms loss to the mux and to Part 9 conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop the tag" (ScadaBridge) will see more Bads — that is the fix working. - **Nothing Good is written on reconnect** — recovery is the driver's own re-publish (the poll engine's @@ -500,13 +508,16 @@ incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`). - **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through `EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role - during a rolling upgrade. ⚠️ **Known residual, not closed by R3-04:** - `RedundancyStateActor.SelectOldestUpMemberOfRole`/`SelectDriverPrimary` delegate to - `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, which matches via Akka's OWN ordinal - `Member.HasRole` — not this repo's to patch, the same class of "not ours to change" carve-out as - `ClusterSingletonOptions.Role`. A pair where every member is on a fixed build can no longer gossip - a miscased role at all (normalization happens before gossip), so this only bites a genuinely mixed - fixed/pre-fix rolling-upgrade window. + during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:** + `RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, + which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the + library nor with a local case-insensitive pre-filter, which would split the Primary election from + `ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade + window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)"). +- **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to + stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s + — leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException` + (design-time tooling) passes through untouched. - **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement: a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads `OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal — @@ -621,6 +632,9 @@ fail-closed). Things worth knowing before touching it: and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**. + **Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and + the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does + **not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches). - **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading SQL today; `StoreAsync` is idempotent and the cache replicates). - **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with @@ -681,7 +695,8 @@ touching it: - **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from `ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event - is ignored, and it never permanently gives up (observability, not data plane). The node-local hub + is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side + stores are not evicted). The node-local hub replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1 @@ -751,7 +766,8 @@ things are worth knowing: message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" was corrected with the bump; do not restore that rationale. - **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).** - `LocalDbSetup.OnReady` reads one predicate, `LocalDbSetup.ReplicationIsConfigured`: + `LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges` + (via `LocalDbSetup.ReplicationIsConfigured`): **`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` + @@ -759,6 +775,10 @@ things are worth knowing: silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has nothing to capture for. + - ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`: + `ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these + rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no + peer must still drain, so the two predicates are deliberately not the same. - **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the same load-bearing DDL→register order. Baselining is what makes enabling replication on a node that has been running *without* it actually converge: capture is CDC and the snapshot streamer @@ -952,6 +972,21 @@ Two things to know before relying on any of this: **Browse** (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral one), an audit write only with `Read` on its source node plus the SDK's audit/encryption guard. Any other event type fails closed. Auditing is off by default (never configured in this repo). +- **ns=0 standard nodes stay ungated by design, even with `Security:Acl:Enabled`** (#617) — `ServerStatus`, + `NamespaceArray`, `ServiceLevel` etc. are served by the SDK's core node manager outside our authorizer and + clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node, + event subscription and HistoryRead is still gated. `docs/security.md`. +- **Equipment-node writes: `EvaluateEquipmentWriteStructure` is the ONLY type check.** `OnWriteValue` bypasses + the SDK's own type check on every writable equipment node, so `OtOpcUaNodeManager.EvaluateEquipmentWriteStructure` + (plus `EvaluateEquipmentWriteRange`) is the whole gate. Scalars are deliberately lenient (drivers + range-check; ScadaBridge sends Double to a Float node). **Array writes** (#610): exact element-type match + (`Byte[]` ≈ ByteString) else `BadTypeMismatch` with nothing sent, wrong length ⇒ `BadTypeMismatch`, an array + to a scalar node is refused, `IndexRange` writes ⇒ `BadWriteNotSupported`, `ValueRank > 1` out of scope. An + array node is writable only when the driver can write the array (`DriverArrayWriteSupport`: Modbus bounded + by `maxRegistersPerWrite`/coil cap, S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types **with a + declared `arrayLength`** — an undeclared length stays read-only because a short whole-array write would + resize the Galaxy attribute); `arrayLength >= 1` is required, and the node's `Writable && (!IsArray || + ArrayWritable)` is projected onto the raw node and every UNS mirror. - **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`, preserving today's behaviour) controls *admission*; when admitted the session authenticates as the reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as @@ -1043,11 +1078,24 @@ Refusals are not comms faults: an Unsupported refusal (`BadNotSupported`) and an mismatch** (`BadTypeMismatch`, where libplctag returned Good garbage for a same-width mis-authored type) both keep the tag handle, leave driver health alone (no `Degraded`, nothing for the comms-loss watchdog) and warn once per tag — only genuine transport failures evict and degrade. `DeviceSessionPool.Shared` logs through the -host logger factory (wired in `DriverFactoryBootstrap`). libplctag is confined to `Libplctag*.cs` adapters by +host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT factories' `Register(registry, ILoggerFactory?)` take one too, #602 CONV-3). The AbCip `browsePageDelayMs` knob has an AdminUI field (`AbCipDriverForm` "Browse page delay (ms)"). libplctag is confined to `Libplctag*.cs` adapters by `LibplctagConfinementGuardTests`; do not reference it elsewhere. See `docs/drivers/AbCip.md` §Transport, `docs/drivers/AbLegacy.md` §Transport + §Wire evidence, `docs/Configuration.md` §`Drivers:AbCip`, and `docs/plans/2026-10-07-native-ab-transport-gate-results.md`. +## Other driver notes + +- **S7:** connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe + tick); a write-side access-denied (PUT/GET disabled) latches a **sticky Faulted** that outranks later read + passes; driver-side array writes cover non-Bool narrow/`Int32`/`Float32` types only — Bool arrays are + refused (a one-byte-per-element encode would clobber neighbour bits). +- **MQTT:** Plain `writeTopic` writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug + primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types + `BadNotSupported`). +- **FOCAS / AbLegacy:** background loops are joined (≤ 5 s) on shutdown. +- **Driver removal (#609):** removing a driver releases its capability-invoker pipelines and resilience + tracker, *then* evicts its telemetry snapshots. + ## Library Preferences - **Logging**: Serilog with rolling daily file sink @@ -1070,6 +1118,8 @@ dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp: dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500 ``` +**AdminUI interactive pages must wrap their body in ``** — `MainLayout` is static, so it cannot catch an interactive page's render failure. + Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`. The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`. @@ -1095,11 +1145,12 @@ exactly. The backend lives in endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy (Administrator or Designer) via `RequireAuthorization` in `ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`). +`Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted. See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`. ## Scripted Alarm Ack/Shelve -Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` LDAP role via `RoleCarryingUserIdentity`; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`. +Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` role — conferred by the session's API-key principal's `NodeAcl` grants (or the `anonymous` principal's), carried by `RoleCarryingUserIdentity`; LDAP is AdminUI-only; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`. ⚠️ **One condition event per accepted method call is OURS, not the SDK's (Gitea #587).** The SDK's own report is gated on `AutoReportStateChanges` (left `false`), so before #587 a client Acknowledge emitted **no** @@ -1116,6 +1167,15 @@ Confirmed/Unshelved constants own those fields); a native ack re-asserts `Confir SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by `ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`). +Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a +**no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real +remaining time from `AlarmConditionSnapshot.UnshelveAtUtc`); a **commented** re-ack is stored as an +`AddComment` (durable, replicated) and reports one event, an uncommented re-ack stays silent; a **native +ack is refused `BadInvalidState` off-Primary** via `NativeAlarmAckPrimaryGate` (fed by +`IDriverPrimaryRoleSource`; `DriverHostActor` publishes its Primary verdict at host start and on member +changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT +Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design). + ## Historian / HistoryRead **Backend: HistorianGateway (sole historian backend).** As of the gateway-integration cutover, the @@ -1196,7 +1256,7 @@ only the recorder + outbox knobs: ### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook) `AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` -hook for added historized value tags — the gateway-backed `GatewayTagProvisioner` calls the gateway's +hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's `EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues` land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a deploy. Non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not provisioned); the @@ -1263,7 +1323,7 @@ dotnet test --filter "Category=LiveIntegration" The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the gate the operator runs on the VPN before trusting the cutover. -### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491) +### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04) The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer + meters). It is spawned with an **empty *initial* ref set** (`Array.Empty()` in @@ -1331,6 +1391,7 @@ tag); evidence in the same plan doc. Two things to know before touching either.* (NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the #511 registration accident). -Also re-confirmed live: **`EnsureTags` provisioning fires only for newly-ADDED historized tags** — -flipping an existing tag's `historianTagname` produces no provisioning at all -(`2026-06-27-historian-gateway-integration-issues.md` Issue 5). +Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for +newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType) +provisioned nothing (`2026-06-27-historian-gateway-integration-issues.md` Issue 5). It now also fires for those +changed raw tags. diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index e1d5542f..395601a4 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -104,9 +104,11 @@ # mesh gRPC call on a login-enabled admin node) survived seven phases of prior live gates. To drive # this rig production-faithfully: # OTOPCUA_DISABLE_LOGIN=false docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2 -# Sign in as `alice` / `password` (or `bob`) against the shared GLAuth (10.100.0.35:3893, -# dc=zb,dc=local) — the same LDAP users every other node in this file already authenticates OPC UA -# sessions against. Restore the default with `docker compose ... up -d central-1 central-2` (no env +# Sign in as `admin` / `password` (OtOpcUa-Admins, AdminUI Administrator) or `multi-role` / `password` +# (member of every group) against the shared GLAuth (10.100.0.35:3893, dc=zb,dc=local). The old +# `alice`/`bob` users no longer exist there. NOTE: the serviceaccount bind password below is the +# pre-rotation value (rotated out-of-band, SEC-36, scadaproj/infra/glauth/config.toml); a real-login run +# needs the current one supplied via the environment. The data plane no longer uses LDAP at all (API keys). Restore the default with `docker compose ... up -d central-1 central-2` (no env # var). See docs/security.md "Mesh gRPC ingress is AllowAnonymous by design" for what this actually # exercises. #