Merge docs pass: #623 stale docs rewritten (IncrementalSync, AddressSpace, driver-specs banner, Secrets runbook replication), #624 stale claims + perf-followups-r3 tasks.json valid, CLAUDE.md updated with this batch's facts, #619 compose comment
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
@@ -186,10 +186,13 @@ is still default-closed; what changed is which deltas it whitelists.
|
||||
bounded resync). A **rebuild** still advances even with failures.
|
||||
- **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is
|
||||
what makes every whitelist above safe.
|
||||
- ⚠️ **Known gap, no longer neutral:** `AnnounceAddedNodes` is Uns-only, so surgically-added
|
||||
**raw** containers/tags are materialised but never announced — previously invisible (the
|
||||
rebuild forced a re-browse anyway), now genuinely undiscoverable until a client re-browses.
|
||||
Added UNS reference variables ARE announced.
|
||||
- **Announcements:** `AnnounceAddedNodes` announces surgically-added **Raw** containers/tags
|
||||
(`ComputeRawAddAnnouncements`, #599 — the former "raw adds are undiscoverable until a re-browse" gap is
|
||||
closed) as well as added UNS reference variables.
|
||||
- **Array writability rides the same whitelist:** `ArrayWritable` joins the surgical raw-tag attributes and
|
||||
`UnsReferenceVariable` carries `BackingArrayWritable` (see "Equipment-node writes" under Authentication). Removing
|
||||
an equipment that has references removes the references before the equipment subtree, and
|
||||
`RemoveEquipmentSubtree` severs the inverse `Organizes` edges.
|
||||
|
||||
See `docs/plans/2026-08-15-surgical-deploys.md`, `docs/Uns.md` and `docs/Raw.md`.
|
||||
|
||||
@@ -268,10 +271,12 @@ deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`Tag
|
||||
`DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` /
|
||||
`DiscoveryStreamIncludesAuthoredTags`.
|
||||
|
||||
⚠️ **`IHostConnectivityProbe` is still unconsumed.** `GetHostStatuses()` has no production call site and
|
||||
nothing ever writes a `DriverHostStatus` row (the table was re-created in the v3 initial migration, so
|
||||
confirm intent before deleting). The separable `OnHostStatusChanged` event is the useful half. Tracked
|
||||
as Gitea **#521**.
|
||||
**`IHostConnectivityProbe` is consumed (Gitea #521, closed).** `DriverInstanceActor` subscribes
|
||||
`OnHostStatusChanged`, pulls `GetHostStatuses()` fresh, and the host set rides `DriverHealthChanged` to the
|
||||
AdminUI `/hosts` page — no SQL: the `DriverHostStatus` table is dropped, and so is
|
||||
`DriverInstanceResilienceStatus` (#524; the live data goes over the `driver-resilience-status` telemetry
|
||||
channel only). **Multi-host dispatch is wired too:** `DriverInstanceActor` resolves the breaker/host key
|
||||
per call via `IPerCallHostResolver` (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip).
|
||||
|
||||
### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed)
|
||||
|
||||
@@ -315,9 +320,12 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T
|
||||
quality contract" and implemented by `Core.Abstractions.InputQuality` (`WorstOf`/`IsUsable`). Three
|
||||
things changed with it, all client-visible:
|
||||
- ⚠️ **Value nodes of a comms-lost driver now go Bad.** `DriverHostActor.FanCommsLossToValueNodes`
|
||||
stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained**,
|
||||
via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host holds no value
|
||||
cache, so only the node can keep the value). #481 had wired comms loss to the mux and to Part 9
|
||||
stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained
|
||||
server-side**, via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host
|
||||
holds no value cache, so only the node can keep the value). ⚠️ Retention is **not client-observable**:
|
||||
the SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read/sample, so clients see
|
||||
`Value = null` + the Bad status; the retained value is what the next Good re-publish and the
|
||||
`(value, status)` dedup compare against. #481 had wired comms loss to the mux and to Part 9
|
||||
conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop
|
||||
the tag" (ScadaBridge) will see more Bads — that is the fix working.
|
||||
- **Nothing Good is written on reconnect** — recovery is the driver's own re-publish (the poll engine's
|
||||
@@ -500,13 +508,16 @@ incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`).
|
||||
- **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of
|
||||
production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through
|
||||
`EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role
|
||||
during a rolling upgrade. ⚠️ **Known residual, not closed by R3-04:**
|
||||
`RedundancyStateActor.SelectOldestUpMemberOfRole`/`SelectDriverPrimary` delegate to
|
||||
`ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, which matches via Akka's OWN ordinal
|
||||
`Member.HasRole` — not this repo's to patch, the same class of "not ours to change" carve-out as
|
||||
`ClusterSingletonOptions.Role`. A pair where every member is on a fixed build can no longer gossip
|
||||
a miscased role at all (normalization happens before gossip), so this only bites a genuinely mixed
|
||||
fixed/pre-fix rolling-upgrade window.
|
||||
during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:**
|
||||
`RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`,
|
||||
which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the
|
||||
library nor with a local case-insensitive pre-filter, which would split the Primary election from
|
||||
`ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade
|
||||
window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)").
|
||||
- **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to
|
||||
stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s
|
||||
— leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException`
|
||||
(design-time tooling) passes through untouched.
|
||||
- **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement:
|
||||
a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads
|
||||
`OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal —
|
||||
@@ -621,6 +632,9 @@ fail-closed). Things worth knowing before touching it:
|
||||
and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
|
||||
`FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
|
||||
empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
|
||||
**Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and
|
||||
the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does
|
||||
**not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches).
|
||||
- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
|
||||
SQL today; `StoreAsync` is idempotent and the cache replicates).
|
||||
- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
|
||||
@@ -681,7 +695,8 @@ touching it:
|
||||
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
|
||||
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
|
||||
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
|
||||
is ignored, and it never permanently gives up (observability, not data plane). The node-local hub
|
||||
is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side
|
||||
stores are not evicted). The node-local hub
|
||||
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
|
||||
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
|
||||
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
|
||||
@@ -751,7 +766,8 @@ things are worth knowing:
|
||||
message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap"
|
||||
was corrected with the bump; do not restore that rationale.
|
||||
- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).**
|
||||
`LocalDbSetup.OnReady` reads one predicate, `LocalDbSetup.ReplicationIsConfigured`:
|
||||
`LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges`
|
||||
(via `LocalDbSetup.ReplicationIsConfigured`):
|
||||
**`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is
|
||||
not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it
|
||||
alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` +
|
||||
@@ -759,6 +775,10 @@ things are worth knowing:
|
||||
silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is
|
||||
fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has
|
||||
nothing to capture for.
|
||||
- ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`:
|
||||
`ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these
|
||||
rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no
|
||||
peer must still drain, so the two predicates are deliberately not the same.
|
||||
- **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the
|
||||
same load-bearing DDL→register order. Baselining is what makes enabling replication on a node
|
||||
that has been running *without* it actually converge: capture is CDC and the snapshot streamer
|
||||
@@ -952,6 +972,21 @@ Two things to know before relying on any of this:
|
||||
**Browse** (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral
|
||||
one), an audit write only with `Read` on its source node plus the SDK's audit/encryption guard. Any other
|
||||
event type fails closed. Auditing is off by default (never configured in this repo).
|
||||
- **ns=0 standard nodes stay ungated by design, even with `Security:Acl:Enabled`** (#617) — `ServerStatus`,
|
||||
`NamespaceArray`, `ServiceLevel` etc. are served by the SDK's core node manager outside our authorizer and
|
||||
clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node,
|
||||
event subscription and HistoryRead is still gated. `docs/security.md`.
|
||||
- **Equipment-node writes: `EvaluateEquipmentWriteStructure` is the ONLY type check.** `OnWriteValue` bypasses
|
||||
the SDK's own type check on every writable equipment node, so `OtOpcUaNodeManager.EvaluateEquipmentWriteStructure`
|
||||
(plus `EvaluateEquipmentWriteRange`) is the whole gate. Scalars are deliberately lenient (drivers
|
||||
range-check; ScadaBridge sends Double to a Float node). **Array writes** (#610): exact element-type match
|
||||
(`Byte[]` ≈ ByteString) else `BadTypeMismatch` with nothing sent, wrong length ⇒ `BadTypeMismatch`, an array
|
||||
to a scalar node is refused, `IndexRange` writes ⇒ `BadWriteNotSupported`, `ValueRank > 1` out of scope. An
|
||||
array node is writable only when the driver can write the array (`DriverArrayWriteSupport`: Modbus bounded
|
||||
by `maxRegistersPerWrite`/coil cap, S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types **with a
|
||||
declared `arrayLength`** — an undeclared length stays read-only because a short whole-array write would
|
||||
resize the Galaxy attribute); `arrayLength >= 1` is required, and the node's `Writable && (!IsArray ||
|
||||
ArrayWritable)` is projected onto the raw node and every UNS mirror.
|
||||
- **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`,
|
||||
preserving today's behaviour) controls *admission*; when admitted the session authenticates as the
|
||||
reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as
|
||||
@@ -1043,11 +1078,24 @@ Refusals are not comms faults: an Unsupported refusal (`BadNotSupported`) and an
|
||||
mismatch** (`BadTypeMismatch`, where libplctag returned Good garbage for a same-width mis-authored type) both
|
||||
keep the tag handle, leave driver health alone (no `Degraded`, nothing for the comms-loss watchdog) and warn
|
||||
once per tag — only genuine transport failures evict and degrade. `DeviceSessionPool.Shared` logs through the
|
||||
host logger factory (wired in `DriverFactoryBootstrap`). libplctag is confined to `Libplctag*.cs` adapters by
|
||||
host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT factories' `Register(registry, ILoggerFactory?)` take one too, #602 CONV-3). The AbCip `browsePageDelayMs` knob has an AdminUI field (`AbCipDriverForm` "Browse page delay (ms)"). libplctag is confined to `Libplctag*.cs` adapters by
|
||||
`LibplctagConfinementGuardTests`; do not reference it elsewhere. See `docs/drivers/AbCip.md` §Transport,
|
||||
`docs/drivers/AbLegacy.md` §Transport + §Wire evidence, `docs/Configuration.md` §`Drivers:AbCip`, and
|
||||
`docs/plans/2026-10-07-native-ab-transport-gate-results.md`.
|
||||
|
||||
## Other driver notes
|
||||
|
||||
- **S7:** connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe
|
||||
tick); a write-side access-denied (PUT/GET disabled) latches a **sticky Faulted** that outranks later read
|
||||
passes; driver-side array writes cover non-Bool narrow/`Int32`/`Float32` types only — Bool arrays are
|
||||
refused (a one-byte-per-element encode would clobber neighbour bits).
|
||||
- **MQTT:** Plain `writeTopic` writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug
|
||||
primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types
|
||||
`BadNotSupported`).
|
||||
- **FOCAS / AbLegacy:** background loops are joined (≤ 5 s) on shutdown.
|
||||
- **Driver removal (#609):** removing a driver releases its capability-invoker pipelines and resilience
|
||||
tracker, *then* evicts its telemetry snapshots.
|
||||
|
||||
## Library Preferences
|
||||
|
||||
- **Logging**: Serilog with rolling daily file sink
|
||||
@@ -1070,6 +1118,8 @@ dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp:
|
||||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500
|
||||
```
|
||||
|
||||
**AdminUI interactive pages must wrap their body in `<PageErrorBoundary>`** — `MainLayout` is static, so it cannot catch an interactive page's render failure.
|
||||
|
||||
Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`.
|
||||
|
||||
The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`.
|
||||
@@ -1095,11 +1145,12 @@ exactly. The backend lives in
|
||||
endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
|
||||
(Administrator or Designer) via `RequireAuthorization` in
|
||||
`ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`).
|
||||
`Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted.
|
||||
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.
|
||||
|
||||
## Scripted Alarm Ack/Shelve
|
||||
|
||||
Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` LDAP role via `RoleCarryingUserIdentity`; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`.
|
||||
Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` role — conferred by the session's API-key principal's `NodeAcl` grants (or the `anonymous` principal's), carried by `RoleCarryingUserIdentity`; LDAP is AdminUI-only; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`.
|
||||
|
||||
⚠️ **One condition event per accepted method call is OURS, not the SDK's (Gitea #587).** The SDK's own
|
||||
report is gated on `AutoReportStateChanges` (left `false`), so before #587 a client Acknowledge emitted **no**
|
||||
@@ -1116,6 +1167,15 @@ Confirmed/Unshelved constants own those fields); a native ack re-asserts `Confir
|
||||
SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by
|
||||
`ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`).
|
||||
|
||||
Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a
|
||||
**no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real
|
||||
remaining time from `AlarmConditionSnapshot.UnshelveAtUtc`); a **commented** re-ack is stored as an
|
||||
`AddComment` (durable, replicated) and reports one event, an uncommented re-ack stays silent; a **native
|
||||
ack is refused `BadInvalidState` off-Primary** via `NativeAlarmAckPrimaryGate` (fed by
|
||||
`IDriverPrimaryRoleSource`; `DriverHostActor` publishes its Primary verdict at host start and on member
|
||||
changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT
|
||||
Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design).
|
||||
|
||||
## Historian / HistoryRead
|
||||
|
||||
**Backend: HistorianGateway (sole historian backend).** As of the gateway-integration cutover, the
|
||||
@@ -1196,7 +1256,7 @@ only the recorder + outbox knobs:
|
||||
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
|
||||
|
||||
`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync`
|
||||
hook for added historized value tags — the gateway-backed `GatewayTagProvisioner` calls the gateway's
|
||||
hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's
|
||||
`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues`
|
||||
land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a
|
||||
deploy. Non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not provisioned); the
|
||||
@@ -1263,7 +1323,7 @@ dotnet test --filter "Category=LiveIntegration"
|
||||
The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
|
||||
gate the operator runs on the VPN before trusting the cutover.
|
||||
|
||||
### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)
|
||||
### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04)
|
||||
|
||||
The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
|
||||
meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
|
||||
@@ -1331,6 +1391,7 @@ tag); evidence in the same plan doc. Two things to know before touching either.*
|
||||
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
|
||||
#511 registration accident).
|
||||
|
||||
Also re-confirmed live: **`EnsureTags` provisioning fires only for newly-ADDED historized tags** —
|
||||
flipping an existing tag's `historianTagname` produces no provisioning at all
|
||||
(`2026-06-27-historian-gateway-integration-issues.md` Issue 5).
|
||||
Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for
|
||||
newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType)
|
||||
provisioned nothing (`2026-06-27-historian-gateway-integration-issues.md` Issue 5). It now also fires for those
|
||||
changed raw tags.
|
||||
|
||||
+5
-3
@@ -41,6 +41,8 @@ Nothing here is a data-loss or outage defect. The sharpest edge is §3 — silen
|
||||
|
||||
### 0.1 What is actually left (re-verified at `4bdce910`, 2026-08-04)
|
||||
|
||||
> **Refreshed 2026-10-08.** The re-verification below is as of `4bdce910` (2026-08-04) and is *not* current. Since then the #562–#591 issue sweep closed on master (`3baa332d`), and the work it deferred was filed as **#592–#625** — those are the live tracker, not "0 open". Re-derive counts from the Gitea API before relying on any figure in this subsection; the §4.1 rows for telemetry eviction, the `TODO(v3 WaveC)` set and the `/raw` Browse-device placeholder are resolved (struck through below).
|
||||
|
||||
> **Read this subsection, not the table below it.** The 2026-07-30 table that follows is preserved as the
|
||||
> record of what the remediation left behind; every tracker item in it has since closed. The live picture
|
||||
> is here.
|
||||
@@ -362,11 +364,11 @@ commented-out DI registrations.** Real deferred work lives in prose doc-comments
|
||||
|
||||
| Item | Evidence |
|
||||
|---|---|
|
||||
| **Telemetry snapshot cache never evicted** — a decommissioned driver replays last-known Health/Resilience to new subscribers forever | `TelemetryLocalHub.cs:40` `TODO(mesh-phase5+)` — self-declared, bounded, "pre-production, accepted" |
|
||||
| **Device host recovered by string-matching instead of `DeviceConfig`** — 7 of the repo's 9 TODOs are this one item, across 4 drivers | `FocasDriver.cs:137,287`; `TwinCATDriver.cs:775`; `AbLegacyDriver.cs:557`; `AbCipDriver.cs:89,1272` + `AbCipDriverOptions.cs:130` — all `TODO(v3 WaveC)` |
|
||||
| ~~**Telemetry snapshot cache never evicted** — a decommissioned driver replays last-known Health/Resilience to new subscribers forever~~ ✅ **FIXED** — `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`DriverHostActor.StopChild` via `plan.ToStop`, #609); central-side stores are not evicted | `TelemetryLocalHub.cs:40` `TODO(mesh-phase5+)` — self-declared, bounded, "pre-production, accepted" |
|
||||
| ~~**Device host recovered by string-matching instead of `DeviceConfig`** — 7 of the repo's 9 TODOs are this one item, across 4 drivers~~ ✅ **RESOLVED (#608)** — no `TODO(v3 WaveC)` remains in `src/` (re-verified 2026-10-08) | `FocasDriver.cs:137,287`; `TwinCATDriver.cs:775`; `AbLegacyDriver.cs:557`; `AbCipDriver.cs:89,1272` + `AbCipDriverOptions.cs:130` — all `TODO(v3 WaveC)` |
|
||||
| **Galaxy writes can never confirm a COM commit** — empty status array ⇒ provisional Good, metered `galaxy.writes.unconfirmed` | `GatewayGalaxyDataWriter.cs:44,328,354` — **cross-repo**, blocked on mxaccessgw |
|
||||
| **Subscription liveness undetectable** — `ISubscriptionHandle` is opaque | `DriverInstanceActor.cs:527` (#488) — well-reasoned, deferred "until a real occurrence" |
|
||||
| **`/raw` "Browse device" is still a placeholder** despite the universal discovery browser landing | `RawTree.razor:15,426` — **verify**; may be reachable now via `DiscoveryDriverBrowser` |
|
||||
| ~~**`/raw` "Browse device" is still a placeholder** despite the universal discovery browser landing~~ ✅ **RESOLVED** — `RawTree.razor` wires `OnBrowseDevice` to the discovery modal | `RawTree.razor:15,426` — **verify**; may be reachable now via `DiscoveryDriverBrowser` |
|
||||
|
||||
### 4.2 Per-driver
|
||||
|
||||
|
||||
@@ -104,9 +104,11 @@
|
||||
# mesh gRPC call on a login-enabled admin node) survived seven phases of prior live gates. To drive
|
||||
# this rig production-faithfully:
|
||||
# OTOPCUA_DISABLE_LOGIN=false docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2
|
||||
# Sign in as `alice` / `password` (or `bob`) against the shared GLAuth (10.100.0.35:3893,
|
||||
# dc=zb,dc=local) — the same LDAP users every other node in this file already authenticates OPC UA
|
||||
# sessions against. Restore the default with `docker compose ... up -d central-1 central-2` (no env
|
||||
# Sign in as `admin` / `password` (OtOpcUa-Admins, AdminUI Administrator) or `multi-role` / `password`
|
||||
# (member of every group) against the shared GLAuth (10.100.0.35:3893, dc=zb,dc=local). The old
|
||||
# `alice`/`bob` users no longer exist there. NOTE: the serviceaccount bind password below is the
|
||||
# pre-rotation value (rotated out-of-band, SEC-36, scadaproj/infra/glauth/config.toml); a real-login run
|
||||
# needs the current one supplied via the environment. The data plane no longer uses LDAP at all (API keys). Restore the default with `docker compose ... up -d central-1 central-2` (no env
|
||||
# var). See docs/security.md "Mesh gRPC ingress is AllowAnonymous by design" for what this actually
|
||||
# exercises.
|
||||
#
|
||||
|
||||
+65
-77
@@ -1,96 +1,84 @@
|
||||
# Address Space
|
||||
|
||||
> ⚠️ **Accuracy warning (audited 2026-07-27) — this page is v2-era and contradicts the shipped v3
|
||||
> address space.** Corrections, in order of how badly they mislead:
|
||||
>
|
||||
> - **There is no "single custom namespace".** v3 exposes **two**:
|
||||
> `https://zb.com/otopcua/raw` (`s=<RawPath>`) and `https://zb.com/otopcua/uns`
|
||||
> (`s=<Area>/<Line>/<Equipment>/<EffectiveName>`), registered together at
|
||||
> `OtOpcUaNodeManager.cs:45`. Identity authority is `V3NodeIds` + `AddressSpaceRealm`. The retired
|
||||
> `https://zb.com/otopcua/ns` survives only in a comment marking it retired (`V3NodeIds.cs:17`).
|
||||
> - **`Phase7Applier` / `Phase7Composer` / `GalaxyTagPlan` do not exist**, nor do the file paths cited
|
||||
> for them. Renamed by `40e8a23e` to `AddressSpaceApplier` / `AddressSpaceComposer`
|
||||
> (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`); the composition result type is
|
||||
> `AddressSpaceComposition` (`AddressSpaceComposer.cs:13`).
|
||||
> - **The `SystemPlatform` namespace kind is retired** — Galaxy is a standard Equipment-kind driver.
|
||||
> - **`GenericDriverNodeManager` is test scaffolding**, not a production dispatch path
|
||||
> (`GenericDriverNodeManager.cs:71`).
|
||||
> - **The §Rediscovery section is wrong**: nothing consumes `OnRediscoveryNeeded`
|
||||
> ([#518](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/518)), and the driver list is stale
|
||||
> (MQTT and MTConnect now raise it too).
|
||||
>
|
||||
> For the current design see the "v3 OPC UA Address Space (Batch 4)" section of `CLAUDE.md`,
|
||||
> `docs/Raw.md`, and `docs/Uns.md`.
|
||||
The server serves **two OPC UA namespaces** under one shared root folder. Identity is owned by
|
||||
`V3NodeIds` (`src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/V3NodeIds.cs`) and `AddressSpaceRealm`; the tree is
|
||||
built from the deploy artifact (see `docs/IncrementalSync.md`), not by driver discovery at runtime.
|
||||
|
||||
Address-space construction is a two-layer system. The **driver-facing layer** is the streaming builder: a driver implements `ITagDiscovery.DiscoverAsync` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ITagDiscovery.cs`) and emits `Folder` / `Variable` / `AddProperty` calls into an `IAddressSpaceBuilder` as it walks its backend — no buffering of the whole tree. `GenericDriverNodeManager` (`src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/GenericDriverNodeManager.cs`) wraps that builder to capture alarm-condition sinks and routes alarm events from the driver to them. The **SDK materialization layer** turns the resulting node descriptions into live OPC UA nodes: `OpcUaPublishActor` drives the write-only `IOpcUaAddressSpaceSink`, whose production binding `SdkAddressSpaceSink` (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs`) forwards to `OtOpcUaNodeManager` (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`), a `CustomNodeManager2` subclass that owns the `FolderState` / `BaseDataVariableState` instances. The same code path serves Galaxy object hierarchies, Modbus PLC registers, AB CIP tags, TwinCAT symbols, FOCAS CNC parameters, and OPC UA Client aggregations — Galaxy is one driver of seven, not the driver.
|
||||
| Realm | Namespace URI | Subtree | NodeId `s=` identifier |
|
||||
|---|---|---|---|
|
||||
| `Raw` | `https://zb.com/otopcua/raw` | `/raw` device tree: Folder → Driver → Device → TagGroup → Tag | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) |
|
||||
| `Uns` | `https://zb.com/otopcua/uns` | UNS tree: Area → Line → Equipment → signal | slash-joined `Area/Line/Equipment[/EffectiveName]` |
|
||||
|
||||
`OtOpcUaNodeManager` registers both URIs (raw is index `NamespaceIndexes[0]`, UNS `[1]`). The retired
|
||||
single namespace `https://zb.com/otopcua/ns` and the `EquipmentNodeIds` scheme no longer exist. The realm
|
||||
travels beside the identifier at every sink seam (`IOpcUaAddressSpaceSink`); it is never parsed out of the
|
||||
id string. Authoring and semantics: `docs/Raw.md`, `docs/Uns.md`.
|
||||
|
||||
## Root folder
|
||||
|
||||
`OtOpcUaNodeManager.CreateAddressSpace` creates a single shared root `FolderState` (`NodeId = OtOpcUa`, `BrowseName = OtOpcUa`, `EventNotifier = None`) under the standard OPC UA `Objects` folder, wired with an `Organizes` reference. Every driver's folders and variables hang beneath this one root; the server is published under a single `ApplicationUri = urn:OtOpcUa` (the `OpcUaApplicationHostOptions.ApplicationUri` default) and all nodes live in the server's single custom namespace, not a per-driver `urn:OtOpcUa:{DriverInstanceId}`. The UNS Area → Line → Equipment folder skeleton under the root is materialised by `Phase7Applier.MaterialiseHierarchy` (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Phase7Applier.cs`); SystemPlatform (Galaxy) tags are materialised by `Phase7Applier.MaterialiseGalaxyTags`.
|
||||
|
||||
## IAddressSpaceBuilder surface
|
||||
|
||||
`IAddressSpaceBuilder` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IAddressSpaceBuilder.cs`) offers three calls:
|
||||
|
||||
- `Folder(browseName, displayName)` — creates a child `FolderState` and returns a child builder scoped to it.
|
||||
- `Variable(browseName, displayName, DriverAttributeInfo attributeInfo)` — creates a `BaseDataVariableState` and returns an `IVariableHandle` the driver keeps for alarm wiring.
|
||||
- `AddProperty(browseName, DriverDataType, value)` — attaches a `PropertyState` for static metadata (e.g. equipment identification fields).
|
||||
|
||||
Drivers drive ordering. Typical pattern: root → folder per equipment → variables per tag. `GenericDriverNodeManager.BuildAddressSpaceAsync` calls `DiscoverAsync` once on startup and once per rediscovery cycle, tearing down the previous alarm subscription and clearing its sink registry before each re-walk so a redeploy doesn't double-fire alarm events.
|
||||
|
||||
## DriverAttributeInfo → OPC UA variable
|
||||
|
||||
Each variable carries a `DriverAttributeInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverAttributeInfo.cs`):
|
||||
|
||||
| Field | Role |
|
||||
|---|---|
|
||||
| `FullName` | driver-side full reference used as the lookup key for Read/Write/Subscribe; also seeds the variable's string `NodeId` |
|
||||
| `DriverDataType` | resolved to a built-in `DataTypeIds.*` NodeId at materialization time — `OtOpcUaNodeManager.ResolveBuiltInDataType` maps the data-type name string; each driver first maps its native type into `DriverDataType` (e.g. Galaxy via `Browse/DataTypeMap.Map`) |
|
||||
| `IsArray` / `ArrayDim` | declared 1-D-array length carried as metadata; the Galaxy discoverer sets `ArrayDim` only when the gateway reports a positive dimension |
|
||||
| `SecurityClass` | write-authorization tier (`SecurityClassification`); enforced server-side by the `NodePermissions` ACL evaluator (`TriePermissionEvaluator`) mapping each `OpcUaOperation` to a required permission bit. The Galaxy driver also caches it per full reference (`_securityByFullRef`) to answer `GetSecurityClassification` |
|
||||
| `IsHistorized` | marks the attribute as feeding historian / HistoryRead |
|
||||
| `IsAlarm` | drives the `MarkAsAlarmCondition` pass (see below) |
|
||||
| `WriteIdempotent` | when true the attribute's writes are safe to replay, so the capability invoker may apply Polly retry; defaults false so pulses / acks / counters aren't auto-retried |
|
||||
| `Source` | `NodeSourceKind` discriminator (`Driver` / `Virtual` / `ScriptedAlarm`) that decides which subsystem dispatches the node's Read/Write/Subscribe |
|
||||
|
||||
The variable is created with `StatusCode = BadWaitingForInitialData` and a null value until the first Read or `ISubscribable.OnDataChange` push lands. Note the production SDK sink (`OtOpcUaNodeManager.EnsureVariable`) currently materialises every variable as `ValueRank = Scalar`, read-only `AccessLevel`, and `Historizing = false` — the `IsArray`/`IsHistorized` intent lives in `DriverAttributeInfo` but is not yet projected onto the SDK node.
|
||||
|
||||
## CapturingBuilder + alarm sink registration
|
||||
|
||||
`GenericDriverNodeManager.BuildAddressSpaceAsync` wraps the supplied builder in a `CapturingBuilder` before calling `DiscoverAsync`. The wrapper observes every `Variable()` call: when a returned `IVariableHandle.MarkAsAlarmCondition(AlarmConditionInfo)` fires, the sink is registered in the manager's `_alarmSinks` dictionary keyed by the variable's `FullReference`. Subsequent `IAlarmSource.OnAlarmEvent` pushes are routed to the matching sink by `SourceNodeId`. This keeps the alarm-wiring protocol declarative — drivers just flag `DriverAttributeInfo.IsAlarm = true` and the materialization of the OPC UA `AlarmConditionState` node is handled by the server layer. See `docs/AlarmTracking.md`.
|
||||
`OtOpcUaNodeManager.CreateAddressSpace` creates **one** shared root `FolderState` — identifier and browse
|
||||
name `OtOpcUa` (`V3NodeIds.SharedRootFolderId`), minted in the **raw** namespace — under the standard
|
||||
`Objects` folder with an `Organizes` reference. Top-level Raw folders and top-level UNS Areas both parent
|
||||
under it. The identifier is a display name, not a path segment; the data-plane authorizer maps it onto the
|
||||
trie's zero-segment realm-root scope (`docs/security.md`). The application URI is
|
||||
`OpcUaApplicationHostOptions.ApplicationUri` (default `urn:OtOpcUa`); there is no per-driver namespace.
|
||||
|
||||
## NodeId scheme
|
||||
|
||||
All nodes share the server's single custom namespace (`NamespaceIndex`); NodeIds are string identifiers, not numeric. The string values come from the source rows / driver references — there is no per-driver namespace prefix:
|
||||
Segments are validated by `RawPaths.ValidateSegment` (no embedded `/`, no leading/trailing whitespace;
|
||||
ordinal, case-sensitive). `V3NodeIds.Raw(rawPath)` is a pass-through; `V3NodeIds.Uns(...)` /
|
||||
`UnsChild(parent, segment)` build UNS ids.
|
||||
|
||||
| Node type | NodeId (string identifier) | Example |
|
||||
|---|---|---|
|
||||
| Shared root | `OtOpcUa` | `OtOpcUa` |
|
||||
| UNS Area / Line / Equipment folder | the Config-DB `UnsAreaId` / `UnsLineId` / `EquipmentId` | `EQ_Press_07` |
|
||||
| Galaxy tag variable | the MXAccess reference (`Phase7Applier` uses `GalaxyTagPlan.MxAccessRef`) | `DelmiaReceiver_001.DownloadPath` |
|
||||
| Equipment tag variable | the driver full reference from `DriverAttributeInfo.FullName` | driver-specific |
|
||||
| Node | Realm | `s=` identifier | Example |
|
||||
|---|---|---|---|
|
||||
| Shared root | Raw | `OtOpcUa` | `OtOpcUa` |
|
||||
| Raw folder / driver / device / tag group / tag | Raw | its own RawPath | `Plant/Modbus/dev1/Speed` |
|
||||
| UNS Area / Line / Equipment folder | Uns | `Area`, `Area/Line`, `Area/Line/Equipment` | `packaging/line-1/press-07` |
|
||||
| UNS reference variable (`UnsTagReference`), virtual tag, scripted alarm | Uns | `Area/Line/Equipment/EffectiveName` | `packaging/line-1/press-07/speed` |
|
||||
|
||||
For Galaxy the variable `FullName` is the `tag_name.AttributeName` MXAccess reference; AB CIP uses `tag.Name` or `tag.Name.member` for UDT members; the shape is the driver's choice. Browse-path resolution (OPC UA `TranslateBrowsePathsToNodeIds`) is the canonical way clients map a browse path to one of these flat NodeIds.
|
||||
Area, Line and Equipment names must match `^[a-z0-9-]{1,32}$` (`docs/Uns.md`). EffectiveNames are unique per
|
||||
equipment across references, virtual tags and scripted alarms.
|
||||
|
||||
## Per-driver hierarchy examples
|
||||
## One source, fanned to both namespaces
|
||||
|
||||
- **Galaxy**: `GalaxyDriver.DiscoverAsync` delegates to `GalaxyDiscoverer` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/GalaxyDiscoverer.cs`), which walks the hierarchy from `IGalaxyHierarchySource` — one folder per Galaxy object (browse name = `contained_name`, falling back to `tag_name`), one variable per dynamic attribute (full reference = `tag_name.AttributeName`). It copies the gateway-supplied `IsAlarm` flag through to `DriverAttributeInfo` and, for alarm-bearing attributes, calls `MarkAsAlarmCondition` with the five sub-attribute refs built by `AlarmRefBuilder`.
|
||||
- **Modbus**: streams one folder per device, one variable per register range from `ModbusDriverOptions`. No alarm surface.
|
||||
- **AB CIP**: `AbCipDriver.DiscoverAsync` emits an `AbCip` root, then a folder per configured device. Pre-declared tags become variables under the device folder; UDT (`Structure`) tags fan out into a sub-folder with one variable per member; when controller browse is enabled, `IAbCipTagEnumerator` adds discovered tags under a `Discovered/` sub-folder. (`AbCipTemplateCache` caches UDT layouts for the libplctag enumerator.)
|
||||
- **OPC UA Client**: re-exposes a remote server's address space — `OpcUaClientDriver.DiscoverAsync` browses the upstream from `BrowseRoot` into a `Remote` folder (pass 1), then batch-reads DataType/AccessLevel/ValueRank/Historizing per variable before registering them (pass 2).
|
||||
Every device value has exactly one source: the raw tag's node, keyed by RawPath. A `UnsTagReference`
|
||||
projects that raw tag into an equipment as a UNS variable that carries an `Organizes` reference to its raw
|
||||
node and mirrors it. `DriverHostActor` fans one driver publish for a RawPath to the raw NodeId **and** every
|
||||
referencing UNS NodeId with identical value, quality and timestamp. Writes via either NodeId route to the
|
||||
same driver reference under the same realm-qualified `WriteOperate` gate. Native alarms materialise once at
|
||||
the raw tag (`ConditionId = RawPath`) and fan through notifiers to the raw device folder and every
|
||||
referencing equipment folder. Both NodeIds register the same historian tagname.
|
||||
|
||||
See `docs/v2/driver-specs.md` for the per-driver discovery contracts.
|
||||
## Materialisation
|
||||
|
||||
## Rediscovery
|
||||
`AddressSpaceComposer` turns the deployment artifact into an `AddressSpaceComposition`;
|
||||
`AddressSpaceApplier` (both in `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`) walks it (`MaterialiseHierarchy`,
|
||||
`MaterialiseRawSubtree`, `MaterialiseUnsReferences`, `MaterialiseEquipmentVirtualTags`,
|
||||
`MaterialiseScriptedAlarms`, ...) into the write-only `IOpcUaAddressSpaceSink`, whose production binding
|
||||
`SdkAddressSpaceSink` forwards to `OtOpcUaNodeManager`, the `CustomNodeManager2` that owns the
|
||||
`FolderState` / `BaseDataVariableState` instances. Variables are created through `EnsureVariable`:
|
||||
`ValueRank` is `OneDimension` with `ArrayDimensions=[arrayLength]` for array tags and `Scalar` otherwise;
|
||||
`AccessLevel` carries the write bit only for writable nodes (array nodes only when the driver can write the
|
||||
array); `Historizing` plus the `HistoryRead` bit are set when a historian tagname is supplied.
|
||||
|
||||
Drivers that implement `IRediscoverable` fire `OnRediscoveryNeeded` when their backend signals a change. Galaxy's `DeployWatcher` raises it when the observed `time_of_last_deploy` advances; TwinCAT raises it on the ADS symbol-version-changed signal (`DeviceSymbolVersionInvalid`, error 1809). Core re-runs `DiscoverAsync` and diffs — see `docs/IncrementalSync.md`. Drivers that don't implement `IRediscoverable` (Modbus, S7, OPC UA Client) only change their address space when a new generation is published from the Config DB.
|
||||
The nodes start `BadWaitingForInitialData` until the first read or subscription push. Datatype names resolve
|
||||
through `OtOpcUaNodeManager.ResolveBuiltInDataType`; each driver first maps its native type into
|
||||
`DriverDataType` (Galaxy: `Driver.Galaxy/Browse/DataTypeMap.cs`).
|
||||
|
||||
## Driver discovery is the browse picker, not the address space
|
||||
|
||||
`ITagDiscovery.DiscoverAsync` (streaming into `IAddressSpaceBuilder`: `Folder` / `Variable` /
|
||||
`AddProperty`, each variable described by a `DriverAttributeInfo`) feeds the `/raw` browse-commit flow via
|
||||
`Commons/Browsing/DiscoveryDriverBrowser`: the operator browses a device, picks leaves and commits them as
|
||||
raw tags, which then reach the server as an ordinary deployment. Drivers that signal backend change via
|
||||
`IRediscoverable` only raise an operator "re-browse" prompt — the served tree never changes on its own.
|
||||
See `docs/IncrementalSync.md`. `GenericDriverNodeManager` (`Core/OpcUa/`) is leftover scaffolding with no
|
||||
production caller.
|
||||
|
||||
## Key source files
|
||||
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/GenericDriverNodeManager.cs` — orchestration + `CapturingBuilder`
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`, `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs` — OPC UA materialization (write-only sink fed by the actor system)
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Phase7Applier.cs` — materialises the UNS folder hierarchy + Galaxy tags into the sink
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/EquipmentNodeWalker.cs` — walks Config-DB Equipment-namespace rows into the builder
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IAddressSpaceBuilder.cs` — builder contract
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ITagDiscovery.cs` — driver discovery capability
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverAttributeInfo.cs` — per-attribute descriptor
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/V3NodeIds.cs`, `AddressSpaceRealm.cs` — identity authority
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Types/RawPaths.cs` — RawPath/segment rules
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`, `SdkAddressSpaceSink.cs` — SDK materialisation
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceComposer.cs`, `AddressSpaceApplier.cs` — compose and apply
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ITagDiscovery.cs`, `IAddressSpaceBuilder.cs`, `DriverAttributeInfo.cs` — discovery-side contracts
|
||||
|
||||
@@ -398,9 +398,11 @@ Scripted alarms use a separate inbound path that converges on the
|
||||
`OtOpcUaNodeManager` wires the Part 9 condition methods (Acknowledge /
|
||||
Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) on each
|
||||
scripted-alarm `AlarmConditionState` node. Every call is **gated on the
|
||||
`AlarmAck` LDAP role** — fail-closed: sessions with no role or without
|
||||
`AlarmAck` group membership receive `BadUserAccessDenied` immediately.
|
||||
The LDAP-resolved role set is carried past `OpcUaApplicationHost` by
|
||||
`AlarmAck` role** — fail-closed: sessions with no role or without
|
||||
an `AlarmAck`-conferring grant receive `BadUserAccessDenied` immediately.
|
||||
The session's roles come from its API-key principal's `NodeAcl` grants (or the
|
||||
reserved `anonymous` principal's; LDAP is AdminUI-only — `docs/security.md`) and are carried past
|
||||
`OpcUaApplicationHost` by
|
||||
`RoleCarryingUserIdentity` (a `UserIdentity` subclass), making it
|
||||
readable inside the method handler at dispatch time.
|
||||
|
||||
|
||||
+1
-1
@@ -250,7 +250,7 @@ otopcua-cli alarms -u opc.tcp://localhost:4840/OtOpcUa \
|
||||
|
||||
### ack
|
||||
|
||||
Acknowledges an active alarm condition via the OPC UA Part 9 `Acknowledge` method. The server gates this on the **`AlarmAck`** LDAP role (fail-closed — a session without `AlarmAck` group membership gets `BadUserAccessDenied`; see [ScriptedAlarms.md](ScriptedAlarms.md) §"AlarmAck gate"). The `--event-id` is the hex-encoded `EventId` taken from a prior `alarms` subscription notification for the same condition.
|
||||
Acknowledges an active alarm condition via the OPC UA Part 9 `Acknowledge` method. The server gates this on the **`AlarmAck`** role (fail-closed — a session whose API-key principal holds no `AlarmAck`-conferring `NodeAcl` grant gets `BadUserAccessDenied`; the data plane does not use LDAP; see [ScriptedAlarms.md](ScriptedAlarms.md) §"AlarmAck gate"). The `--event-id` is the hex-encoded `EventId` taken from a prior `alarms` subscription notification for the same condition.
|
||||
|
||||
```bash
|
||||
otopcua-cli ack -u opc.tcp://localhost:4840/OtOpcUa \
|
||||
|
||||
@@ -311,12 +311,18 @@ shutdown paths deliberately keep the single trailing write.
|
||||
> **A value consumer never reports a quality better than its source's.** A node derived from a driver
|
||||
> value — the raw node, every UNS mirror, a Calculation output, a virtual tag, a scripted condition's
|
||||
> quality — carries, at worst, the worst status among its inputs: a Bad input makes the derived node Bad
|
||||
> **with that input's sub-code and the last-known value**; an Uncertain input makes it Uncertain; a source
|
||||
> **with that input's sub-code** (the last-known value is kept server-side, but see the note below: a client never sees it beside a Bad status); an Uncertain input makes it Uncertain; a source
|
||||
> that has stopped communicating is a Bad input (`BadNoCommunication`) for everything downstream of it,
|
||||
> **including the external OPC UA data plane**. Recovery is a *publish*, never an assumption: the first
|
||||
> post-recovery value must land even when it equals the value held before the outage, so every dedup on
|
||||
> the path — driver caches, the node's own status, a consumer's last-published memory — must be keyed on
|
||||
> **(value, status)**, not value alone.
|
||||
>
|
||||
> **"Value retained" is server-side state, not wire-observable.** The node keeps its last value under the
|
||||
> Bad status (that is what lets the next Good re-publish carry it and what the `(value, status)` dedup
|
||||
> compares against), but the OPC UA SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read
|
||||
> and monitored-item sample, so a client sees `Value = null` with the Bad status and never the last-known
|
||||
> value. Pinned server-side by `OtOpcUaNodeManagerWriteStatusesTests`; observation: `archreview/plans/R3-02-stale-good-family-plan.md`.
|
||||
|
||||
This is one rule with one implementation of its comparison,
|
||||
`ZB.MOM.WW.OtOpcUa.Core.Abstractions.InputQuality` (`WorstOf` / `IsUsable`), used by every derived-value
|
||||
@@ -325,7 +331,7 @@ a fifth (archreview round 3, theme "the stale-Good family"). The seams that impl
|
||||
|
||||
| Seam | What it does | Where |
|
||||
|---|---|---|
|
||||
| Driver comms loss → **value nodes** | one status-only batch stamps `BadNoCommunication` on the driver's raw NodeIds and every referencing UNS NodeId, **value retained** | `DriverHostActor.FanCommsLossToValueNodes` (below) |
|
||||
| Driver comms loss → **value nodes** | one status-only batch stamps `BadNoCommunication` on the driver's raw NodeIds and every referencing UNS NodeId, **value retained server-side** (clients see a null value with the Bad status) | `DriverHostActor.FanCommsLossToValueNodes` (below) |
|
||||
| Driver comms loss → dependency mux | synthetic Bad per ref, so scripts and alarms see the source die | `DriverHostActor.FanCommsLossToDependencyMux` |
|
||||
| Driver comms loss → Part 9 conditions | condition quality annotated out of band | `DriverHostActor.OnDriverConnectivityChanged` |
|
||||
| MQTT / Sparkplug | connection loss stales observed refs; every (re)subscribe pushes the cached snapshot | [`docs/drivers/Mqtt.md`](drivers/Mqtt.md#quality-on-connection-loss-and-re-subscribe) |
|
||||
|
||||
+58
-77
@@ -1,90 +1,71 @@
|
||||
# Incremental Sync
|
||||
|
||||
> ⚠️ **Accuracy warning (audited 2026-07-27) — most of this page describes machinery that is retired
|
||||
> or was never wired.** Read it as v2-era design history, not as current behaviour.
|
||||
>
|
||||
> - **"Rebuild flow" (below) is not a production path.** It is built on `GenericDriverNodeManager`,
|
||||
> which is Core test scaffolding with **zero production references**
|
||||
> (`GenericDriverNodeManager.cs:71`). Its `_variablesByFullRef` / `_sourceByFullRef` fields do not
|
||||
> exist in `src/`.
|
||||
> - **Driver-backend rediscovery has no consumer.** Nothing in `src/Server/` or `src/Core/`
|
||||
> subscribes to `IRediscoverable.OnRediscoveryNeeded`; `DriverHostActor.HandleDiscoveredNodes`
|
||||
> (`DriverHostActor.cs:986-1002`) short-circuits unconditionally. The `ScopeHint` is read by
|
||||
> nothing. Drivers that raise it today: Galaxy, TwinCAT, MQTT/Sparkplug, MTConnect
|
||||
> ([#518](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/518),
|
||||
> [#507](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/507)).
|
||||
> - **The generation-publish path is retired.** `sp_PublishGeneration` and
|
||||
> `sp_ComputeGenerationDiff` are stubs that `RAISERROR`
|
||||
> (`20260715230637_V3Initial.StoredProcedures.cs:194`); `DraftRevisionToken` does not exist.
|
||||
>
|
||||
> **The one live rebuild path in v3** is the deploy artifact: `ConfigComposer` →
|
||||
> `Deployment.ArtifactBlob` → `DriverHostActor` apply → `AddressSpaceComposer` /
|
||||
> `AddressSpaceApplier` diff-apply.
|
||||
How a change reaches the running server's address space in v3. There is exactly one path that
|
||||
changes the served address space — the **deploy artifact** — and one advisory signal
|
||||
(`IRediscoverable`) that never changes it.
|
||||
|
||||
Two distinct change-detection paths feed the running server: driver-backend rediscovery (Galaxy's `time_of_last_deploy`, TwinCAT's symbol-version-changed) and generation-level config publishes from the Admin UI. Both flow into re-runs of `ITagDiscovery.DiscoverAsync`, but they originate differently.
|
||||
## The deploy-artifact path
|
||||
|
||||
## Driver-backend rediscovery — IRediscoverable
|
||||
1. **Compose.** `ConfigComposer` (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs`)
|
||||
snapshots the whole Config DB into one gzip-compressed JSON blob (`Deployment.ArtifactBlob`) whose
|
||||
`RevisionHash` is the SHA-256 of those exact bytes. A deployment is fleet-wide; each node
|
||||
self-scopes the artifact (`DeploymentArtifact.ResolveClusterScope`).
|
||||
2. **Dispatch.** `ConfigPublishCoordinator` (`.../ControlPlane/Coordinators/`) sends each enabled
|
||||
`ClusterNode` a payload-free dispatch and waits for acks until the apply deadline. A node obtains the
|
||||
bytes either straight from central SQL (`ConfigSource:Mode=Direct`) or over gRPC with a SHA-256
|
||||
check (`FetchAndCache`) — see `docs/Configuration.md` §`ConfigSource`.
|
||||
3. **Apply.** `DriverHostActor` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/`) opens the artifact and
|
||||
reconciles in two halves:
|
||||
- **Drivers** — `DriverSpawnPlanner.Compute` yields `ToStop` / `ToApplyDelta` / `ToSpawn`. A changed
|
||||
driver config reaches the live driver as `IDriver.ReinitializeAsync` on its `DriverInstanceActor`;
|
||||
a removed driver is stopped. An unreadable or empty blob is "no answer": the apply fails and the
|
||||
running children are left alone (never an empty configuration).
|
||||
- **Address space** — `AddressSpaceComposer` builds an `AddressSpaceComposition` from the artifact;
|
||||
`AddressSpacePlan` diffs it against the last applied composition; `AddressSpaceChangeClassifier`
|
||||
picks the delta class; `AddressSpaceApplier`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`) mutates `OtOpcUaNodeManager` through the sink.
|
||||
4. **Classes** (`AddressSpaceChangeKind`): `Empty`, `AttributeOnly`, `PureAdd`, `PureRemove`,
|
||||
`AddRemoveMix` — all applied surgically, so unrelated MonitoredItems survive — and `Rebuild`, the
|
||||
default-closed fallback for anything the classifier does not whitelist. Any surgical `false`/throw
|
||||
also ratchets to a safe rebuild, and a degraded non-rebuild apply does not advance the last-applied
|
||||
composition (the next deploy re-diffs and re-ensures). The whitelist, the UNS-mirror repair rule and
|
||||
the add/remove announcements are described in the "surgical deploys" paragraph of `CLAUDE.md`
|
||||
and `docs/plans/2026-08-15-surgical-deploys.md`.
|
||||
|
||||
Drivers whose backend has a native change signal implement `IRediscoverable` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs`):
|
||||
Virtual tags and scripted alarms ride the same artifact and the same plan; a script edit is just another
|
||||
changed record in the composition (see `docs/VirtualTags.md`, `docs/ScriptedAlarms.md`).
|
||||
|
||||
```csharp
|
||||
public interface IRediscoverable
|
||||
{
|
||||
event EventHandler<RediscoveryEventArgs>? OnRediscoveryNeeded;
|
||||
}
|
||||
public sealed record RediscoveryEventArgs(string Reason, string? ScopeHint);
|
||||
```
|
||||
## Driver-backend rediscovery is advisory
|
||||
|
||||
The driver fires the event with a reason string (for the diagnostic log) and an optional scope hint — a non-null hint lets Core scope the rebuild surgically to that subtree; null means "the whole address space may have changed".
|
||||
Drivers whose backend signals a tag-set change implement `IRediscoverable`
|
||||
(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs`; `OnRediscoveryNeeded` carries a
|
||||
`Reason` and an optional `ScopeHint`). Raisers today: **Galaxy** (`DeployWatcher`, new
|
||||
`time_of_last_deploy`), **TwinCAT** (ADS symbol-version-changed), **MQTT/Sparkplug** and **MTConnect**.
|
||||
AbCip, AbLegacy, FOCAS, Modbus, S7 and the rest raise nothing.
|
||||
|
||||
Drivers that implement the capability today:
|
||||
`DriverInstanceActor` is the only production consumer: it subscribes in `PreStart`, detaches in
|
||||
`PostStop`, and folds the raise into the driver-health snapshot
|
||||
(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`), which the AdminUI `/hosts` page shows
|
||||
as a **re-browse** chip. **It never rebuilds or edits the served address space** — v3 raw tags are authored
|
||||
explicitly in `/raw`, so an operator re-browses the device and commits, and the result reaches the
|
||||
server as an ordinary deployment. `ScopeHint` is not read by the actor. (The older runtime
|
||||
injection path was removed, #507 closed as superseded.)
|
||||
|
||||
- **Galaxy** — `DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) subscribes to the mxaccessgw gRPC stream (`IGalaxyDeployWatchSource.WatchAsync`) and fires on a new `time_of_last_deploy` value. The gateway polls the Galaxy repository DB internally; the driver side is event-driven.
|
||||
- **TwinCAT** — observes ADS symbol-version-changed notifications (ADS error `DeviceSymbolVersionInvalid`, decimal 1809 / `0x0711`). Note: legacy Beckhoff documentation sometimes cites `0x0702` (`DeviceInvalidGroup`) — that is a transcription error; the correct code is `0x0711` per `TwinCATStatusMapper.AdsSymbolVersionChanged` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATStatusMapper.cs:35`).
|
||||
## Not part of the v3 path
|
||||
|
||||
Static drivers (Modbus, S7, AB CIP, AB Legacy, FOCAS) do not implement `IRediscoverable` — their tags only change when a new generation is published from the Config DB. Core sees absence of the interface and skips change-detection wiring for those drivers (decision #54).
|
||||
|
||||
## Config-DB generation publishes
|
||||
|
||||
Tag-set changes authored in the Admin UI (UNS edits, CSV imports, driver-config edits) accumulate in a draft generation and commit via `sp_PublishGeneration`. The delta between the currently-published generation and the proposed next one is computed by `sp_ComputeGenerationDiff`, which drives:
|
||||
|
||||
- The publish-preview surface in the Admin UI (`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Deployments.razor`, backed by `AdminOperationsClient`) so operators can preview what will change before clicking Publish.
|
||||
- The 409-on-stale-draft flow (decision #161) — a UNS drag-reorder preview carries a `DraftRevisionToken` so Confirm returns `409 Conflict / refresh-required` if the draft advanced between preview and commit.
|
||||
|
||||
After publish, the server's generation applier invokes `IDriver.ReinitializeAsync(driverConfigJson, ct)` on every driver whose `DriverInstance.DriverConfig` row changed in the new generation. Reinitialize is the in-process recovery path for Tier A/B drivers; if it fails the driver is marked `DriverState.Faulted` and its nodes go Bad quality — but the server process stays running. See `docs/v2/driver-stability.md`.
|
||||
|
||||
Drivers whose discovery depends on Config DB state (Modbus register maps, S7 DBs, AB CIP tag lists) re-run their discovery inside `ReinitializeAsync`; Core then diffs the new node set against the current address space.
|
||||
|
||||
## Rebuild flow
|
||||
|
||||
When a rediscovery is triggered (by either source), `GenericDriverNodeManager` re-runs `ITagDiscovery.DiscoverAsync` into the same `CapturingBuilder` it used at first build. The new node set is diffed against the current:
|
||||
|
||||
1. **Diff** — full-name comparison of the new `DriverAttributeInfo` set against the existing `_variablesByFullRef` map. Added / removed / modified references are partitioned.
|
||||
2. **Snapshot subscriptions** — before teardown, Core captures the current monitored-item ref-counts for every affected reference so subscriptions can be replayed after rebuild.
|
||||
3. **Teardown** — removed / modified variable nodes are deleted via `CustomNodeManager2.DeleteNode`. Driver-side subscriptions for those references are unwound via `ISubscribable.UnsubscribeAsync`.
|
||||
4. **Rebuild** — added / modified references get fresh `BaseDataVariableState` nodes via the standard `IAddressSpaceBuilder.Variable(...)` path. Alarm-flagged references re-register their `IAlarmConditionSink` through `CapturingBuilder`.
|
||||
5. **Restore subscriptions** — for every captured reference that still exists after rebuild, Core re-opens the driver subscription and restores the original ref-count.
|
||||
|
||||
Exceptions during teardown are swallowed per decision #12 — a driver throw must not leave the node tree half-deleted.
|
||||
|
||||
## Scope hint
|
||||
|
||||
When `RediscoveryEventArgs.ScopeHint` is non-null (e.g. a folder path), Core restricts the diff to that subtree. This matters for Galaxy Platform-scoped deployments where a `time_of_last_deploy` advance may only affect one platform's subtree. Null scope falls back to a full-tree diff.
|
||||
|
||||
## Virtual tags in the rebuild
|
||||
|
||||
Per [ADR-002](v2/implementation/adr-002-driver-vs-virtual-dispatch.md), virtual (scripted) tags live in the same address space as driver tags and flow through the same rebuild. `EquipmentNodeWalker` (`src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/EquipmentNodeWalker.cs`) emits virtual-tag children alongside driver-tag children with `DriverAttributeInfo.Source = NodeSourceKind.Virtual`, and `DriverNodeManager` registers each variable's source in `_sourceByFullRef` so the dispatch branches correctly after rebuild. Virtual-tag script changes published from the Admin UI land through the same generation-publish path — the `VirtualTagEngine` recompiles its script bundle when its config row changes and `DriverNodeManager` re-registers any added/removed virtual variables through the standard diff path. Subscription restoration after rebuild runs through each source's `ISubscribable` — either the driver's or `VirtualTagSource` — without special-casing.
|
||||
|
||||
## Active subscriptions survive rebuild
|
||||
|
||||
Subscriptions for unchanged references stay live across rebuilds — their ref-count map is not disturbed. Clients monitoring a stable tag never see a data-change gap during a deploy, only clients monitoring a tag that was genuinely removed see the subscription drop.
|
||||
- `sp_PublishGeneration` / `sp_ComputeGenerationDiff` are stubs that `RAISERROR` in the v3 baseline
|
||||
migration; there is no generation-publish flow and no `DraftRevisionToken`.
|
||||
- `GenericDriverNodeManager` (`Core/OpcUa/`) is retained scaffolding with no production caller; the
|
||||
diff / teardown / subscription-replay "rebuild flow" described by earlier versions of this page does not
|
||||
exist in `src/`.
|
||||
|
||||
## Key source files
|
||||
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs` — backend-change capability
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/GenericDriverNodeManager.cs` — discovery orchestration
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriver.cs` — `ReinitializeAsync` contract
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` — publish-flow driver
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs` — cluster singleton invoked by the Admin UI's `AdminOperationsClient`
|
||||
- `docs/v2/config-db-schema.md` — `sp_PublishGeneration` + `sp_ComputeGenerationDiff`
|
||||
- `docs/v2/admin-ui.md` — DiffViewer + draft-revision-token flow
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs` — artifact composition + hash
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` — dispatch + ack set
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — apply + driver reconcile
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs` — node-side artifact parsing
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceComposer.cs`, `AddressSpacePlan.cs`,
|
||||
`AddressSpaceChangeClassifier.cs`, `AddressSpaceApplier.cs` — compose, diff, classify, apply
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs` — `IRediscoverable` consumer
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/ISurgicalAddressSpaceSink.cs` — in-place update seam
|
||||
|
||||
+3
-2
@@ -56,8 +56,9 @@ The server binds a TCP endpoint at `opc.tcp://{PublicHostname}:{OpcUaPort}/OtOpc
|
||||
|
||||
`OpcUaApplicationHost` subscribes to `SessionManager.ImpersonateUser` after `ApplicationInstance.Start`. The handler (`HandleImpersonation`) deals with the token types as follows:
|
||||
|
||||
- `UserNameIdentityToken` → the password is decrypted, then `IOpcUaUserAuthenticator.AuthenticateUserNameAsync` validates the credential (`LdapOpcUaUserAuthenticator` in production, a stub in tests). On success a `UserIdentity` carrying the token is attached and the LDAP-derived roles are logged; on failure `ImpersonateEventArgs.IdentityValidationError` is set to `BadIdentityTokenRejected`.
|
||||
- `AnonymousIdentityToken` and X.509 tokens → the handler returns without intervening, so the SDK's default validation stands.
|
||||
- `UserNameIdentityToken` → the password is decrypted, then `IOpcUaUserAuthenticator.AuthenticateUserNameAsync` validates the credential (`ApiKeyOpcUaUserAuthenticator` in production — fixed username `apikey`, `otua_<keyId>_<secret>` password, no network I/O — and a stub in tests). On success a `RoleCarryingUserIdentity` carrying the token and the key's resolved roles is attached; on failure `ImpersonateEventArgs.IdentityValidationError` is set to `BadIdentityTokenRejected`.
|
||||
- `AnonymousIdentityToken` → admitted only when `OpcUa:AllowAnonymous` (default `true`); the session then authenticates as the reserved `anonymous` principal and carries whatever roles `NodeAcl` rows naming it grant (`HandleAnonymousImpersonation`). A failure to read those grants denies.
|
||||
- X.509 tokens → the handler returns without intervening, so the SDK's default validation stands.
|
||||
|
||||
Decryption failures and authenticator exceptions also map to `BadIdentityTokenRejected`.
|
||||
|
||||
|
||||
@@ -347,7 +347,7 @@ Operators interact with active scripted alarms through two surfaces — both con
|
||||
|
||||
### AlarmAck gate (OPC UA method path)
|
||||
|
||||
`OtOpcUaNodeManager` wires the OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve / Enable / Disable) on each `AlarmConditionState` node. Every method call is gated on the `AlarmAck` LDAP role — fail-closed: a session with no resolved roles or no `AlarmAck` group membership receives `BadUserAccessDenied` immediately without reaching the engine. The role is carried on the session by `RoleCarryingUserIdentity` (a `UserIdentity` subclass that preserves the LDAP-resolved role set past `OpcUaApplicationHost`).
|
||||
`OtOpcUaNodeManager` wires the OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve / Enable / Disable) on each `AlarmConditionState` node. Every method call is gated on the `AlarmAck` role — fail-closed: a session with no resolved roles or no `AlarmAck`-conferring grant on its API-key principal (`NodeAcl`; LDAP is AdminUI-only, see `docs/security.md`) receives `BadUserAccessDenied` immediately without reaching the engine. The role is carried on the session by `RoleCarryingUserIdentity` (a `UserIdentity` subclass that preserves the API-key-resolved role set past `OpcUaApplicationHost`).
|
||||
|
||||
On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` (containing command kind, condition id, comment, and operator principal) onto the `alarm-commands` DPS topic. The node manager itself stays Akka-free: the dispatch action is a settable `Action<AlarmCommand>` injected at boot by the hosted service.
|
||||
|
||||
|
||||
+3
-3
@@ -140,9 +140,9 @@ Three things to know before touching it:
|
||||
The digest is a flattened, host-name-ordered **string** for the converse reason: a tuple holding an
|
||||
`IReadOnlyList` compares by reference and would re-publish on every 30 s heartbeat forever.
|
||||
|
||||
⚠️ **`DriverInstanceResilienceStatus` is the same defect, still open.** That table also has no writer
|
||||
and no reader — the only reference in the repo is its DbSet declaration — while the live data reaches
|
||||
the AdminUI over the `driver-resilience-status` channel above. Keep-or-delete is Gitea **#524**.
|
||||
**`DriverInstanceResilienceStatus` was dropped (Gitea #524, migration
|
||||
`DropDriverInstanceResilienceStatusTable`).** The table had no writer and no reader; the live data
|
||||
reaches the AdminUI over the `driver-resilience-status` channel above, which is its only home.
|
||||
|
||||
## The three non-migrated channels (NOT migrated in Phase 5 — do not read this as "seven done")
|
||||
|
||||
|
||||
+3
-2
@@ -391,7 +391,8 @@ See the individual driver docs under `docs/drivers/` for per-driver implementati
|
||||
|
||||
An equipment tag is **writable by default** — the materialised OPC UA node is created
|
||||
`CurrentReadWrite` and inbound client writes are routed to the device (subject to the
|
||||
`WriteOperate` LDAP role gate). Setting the optional `"writable": false` key in a tag's
|
||||
`WriteOperate` gate — a role the session's API-key principal earns from its `NodeAcl` grants, see
|
||||
[security.md](security.md)). Setting the optional `"writable": false` key in a tag's
|
||||
`TagConfig` makes the node **read-only** at the driver seam. The six typed tag editors
|
||||
(Modbus, S7, AB CIP, AB Legacy, TwinCAT, FOCAS) expose this as a **Writable** checkbox
|
||||
(checked by default); it can also be hand-authored in the raw-JSON textarea.
|
||||
@@ -405,7 +406,7 @@ An equipment tag is **writable by default** — the materialised OPC UA node is
|
||||
The key is honoured by Modbus, S7, AB CIP, AB Legacy, and TwinCAT. **FOCAS forces
|
||||
read-only regardless** — FOCAS PMC writes are unsupported in the wire client, so its
|
||||
editor shows the checkbox disabled/unchecked and any authored `writable:true` is ignored
|
||||
(and flagged at the deploy gate below). Node-level authz (LDAP `WriteOperate`) remains the
|
||||
(and flagged at the deploy gate below). Node-level authz (`WriteOperate` via the API-key principal's grants, not LDAP) remains the
|
||||
effective write gate above this flag.
|
||||
|
||||
## Deploy-time TagConfig validation (`Deployment:TagConfigValidationMode`)
|
||||
|
||||
@@ -121,7 +121,8 @@ Notes worth knowing before authoring against this:
|
||||
## Error semantics
|
||||
|
||||
An evaluator failure publishes **Bad quality carrying the last-known value** — the value is not
|
||||
zeroed, because a stale-but-flagged reading is more useful to an operator than a fabricated one. It
|
||||
zeroed, so the node holds it server-side for the next Good re-publish and the `(value, status)` dedup. (Over the wire the
|
||||
OPC UA SDK nulls the value of a Bad-status read, so a client sees only the Bad status — see `docs/DriverLifecycle.md`.) It
|
||||
publishes:
|
||||
|
||||
- **once per Good→Bad transition**, not on every failed evaluation (a permanently broken script does
|
||||
|
||||
@@ -378,10 +378,11 @@ for anything beyond values-and-conditions.
|
||||
— it would materialise nodes nobody approved and no deployment artifact
|
||||
records.
|
||||
|
||||
**`IHostConnectivityProbe` is still unconsumed** — `GetHostStatuses()` has
|
||||
no production call site and nothing writes a `DriverHostStatus` row. That
|
||||
half remains a fleet-wide gap affecting every driver that implements it
|
||||
(Gitea #521).
|
||||
**`IHostConnectivityProbe` is consumed** (Gitea #521, closed) —
|
||||
`DriverInstanceActor` subscribes `OnHostStatusChanged`, pulls
|
||||
`GetHostStatuses()` fresh, and the result rides `DriverHealthChanged` to
|
||||
the AdminUI `/hosts` page (no SQL; the `DriverHostStatus` table was
|
||||
dropped). See `docs/Telemetry.md`.
|
||||
2. **`RawTagEntry.DeviceName` is accepted on the wire shape but ignored for
|
||||
routing.** One driver instance owns exactly one Agent client scoped by
|
||||
`MTConnectDriverOptions.DeviceName` (the top-level config key); the
|
||||
|
||||
@@ -96,14 +96,15 @@ to the actual offending register(s). Per-tag escape hatch:
|
||||
from the driver's `DriverConfig` JSON. Key fields:
|
||||
|
||||
- **Endpoint** — `Host`, `Port` (default 502), `UnitId`, `Timeout`. Per-tag `UnitId` overrides drive multi-slave gateway topology.
|
||||
- **`Transport`** — `Tcp` (default; MBAP header + transaction id) or `RtuOverTcp` (RTU `[addr][PDU][CRC-16]` framing tunnelled over the socket to a serial-to-Ethernet gateway); string-valued, absent = `Tcp`. The Test Connect probe follows the authored mode.
|
||||
- **`Tags`** — the pre-declared `ModbusTagDefinition` list; this *is* the address space.
|
||||
- **`Probe`** — connectivity-probe interval / timeout / probe register (default register 0).
|
||||
- **Read/write caps** — `MaxRegistersPerRead` (125), `MaxRegistersPerWrite` (123), `MaxCoilsPerRead` (2000), plus `MaxReadGap` and `AutoProhibitReprobeInterval` for coalescing.
|
||||
- **Function-code overrides** — `UseFC15ForSingleCoilWrites`, `UseFC16ForSingleRegisterWrites` for PLCs that only accept multi-write codes.
|
||||
- **Resilience** — `AutoReconnect`, `KeepAlive`, `IdleDisconnectTimeout`, `Reconnect` backoff, and `WriteOnChangeOnly` redundant-write suppression.
|
||||
|
||||
Full per-field descriptions live in `ModbusDriverOptions.cs`. The JSON skeleton
|
||||
is reproduced in [docs/v2/driver-specs.md §2](../v2/driver-specs.md).
|
||||
Full per-field descriptions live in `ModbusDriverOptions.cs`. The v2-era JSON skeleton
|
||||
in [docs/v2/driver-specs.md §2](../v2/driver-specs.md) predates the `Transport` key and the v3 raw-tag model.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@@ -428,7 +428,7 @@ which is archreview 05/STAB-18.
|
||||
|
||||
- **Connection loss stales, keeping the value.** On the `Connected → Reconnecting` edge
|
||||
`MqttConnection.ConnectionLost` fires, and both ingest paths mark every **already-observed**
|
||||
reference `BadNoCommunication` with its **last value retained** and publish the change. A
|
||||
reference `BadNoCommunication` with its **last value retained** (server-side; a client reads the Bad status with a null value) and publish the change. A
|
||||
never-observed reference (still `BadWaitingForInitialData`) and an already-Bad one are left alone.
|
||||
The event does **not** fire for a failed *initial* connect or during dispose — neither is an outage.
|
||||
- **Every (re)subscribe pushes the cached snapshots.** `SubscribeAsync` republishes each observed
|
||||
|
||||
@@ -103,7 +103,7 @@ Three properties worth knowing:
|
||||
- **Historian.Gateway** (server-side historian backend, not a tag driver) is documented in the main guide:
|
||||
- [../Historian.md](../Historian.md) — HistorianGateway backend: read-path registration, HistoryRead dispatch, alarm store-and-forward (`SendEvent`), continuous historization (`WriteLiveValues`), `EnsureTags` provisioning, config keys, deployment prerequisites. (The retired Wonderware sidecar backend it replaced: [Historian.Wonderware.md](Historian.Wonderware.md).)
|
||||
|
||||
- The full per-field spec (capability surface, config schema, addressing, data-type maps, connection settings, quirks for every driver) lives in [docs/v2/driver-specs.md](../v2/driver-specs.md). The overview pages above are the short path; that file is the authoritative per-driver reference.
|
||||
- The full per-field spec (capability surface, config schema, addressing, data-type maps, connection settings, quirks for every driver) lives in [docs/v2/driver-specs.md](../v2/driver-specs.md). The overview pages above are the short path and are the current reference; that file is the v2-era per-field spec (see its status banner for which drivers it covers).
|
||||
|
||||
## Test-fixture coverage maps
|
||||
|
||||
@@ -114,7 +114,7 @@ Each driver has a dedicated fixture doc that lays out what the integration / uni
|
||||
- [Siemens S7](S7-Test-Fixture.md) — Dockerized `python-snap7` server; DB/MB read + write round-trip verified end-to-end on `:1102`
|
||||
- [AB Legacy](AbLegacy-Test-Fixture.md) — Dockerized `ab_server` PCCC mode across SLC500 / MicroLogix / PLC-5 profiles (task #224); N/F/L-file round-trip verified end-to-end. `/1,0` cip-path required for the Docker fixture; real hardware uses empty. Residual gap: bit-file writes (`B3:0/5`) still surface BadState — real HW / RSEmulate 500 for those
|
||||
- [TwinCAT](TwinCAT-Test-Fixture.md) — XAR-VM integration scaffolding (task #221); three smoke tests skip when VM unreachable. Unit via `FakeTwinCATClient` with native-notification harness
|
||||
- [FOCAS](FOCAS-Test-Fixture.md) — no integration fixture, unit-only via `FakeFocasClient`; Tier C out-of-process isolation scoped but not shipped
|
||||
- [FOCAS](FOCAS-Test-Fixture.md) — no integration fixture, unit-only via `FakeFocasClient`; FOCAS runs in-process (Tier A) — the Tier C out-of-process isolation once scoped was never built
|
||||
- [OPC UA Client](OpcUaClient-Test-Fixture.md) — Dockerized `opc-plc` integration suite (task #215): real Secure Channel + Session, read + subscribe verified end-to-end; write not yet exercised in the integration suite; exhaustive capability matrix (reconnect, failover, cert-auth, history, alarms) via unit suite with mocked `Session`
|
||||
- [MQTT](Mqtt-Test-Fixture.md) — Dockerized `eclipse-mosquitto` with **TLS + real auth on both listeners (no anonymous fallback)** + a JSON-publisher sidecar; env-gated live suite (**15 tests**) — 7 plain (TLS/auth connect, CA pinning both ways, subscribe→value, retained seeding, wrong-password rejection) + 8 Sparkplug against a **project-owned C# edge-node simulator** on the `sparkplug` compose profile (birth→alias bind, DDATA, NDEATH→STALE, rebirth NCMD round-trip). Births are never retained, so `docker restart otopcua-sparkplug-sim` is how you force a fresh one
|
||||
- [Galaxy](../v1/drivers/Galaxy-Test-Fixture.md) — richest harness: gateway E2E + ZB SQL live-smoke + MXAccess opt-in (v1 archive)
|
||||
@@ -125,5 +125,5 @@ Each driver has a dedicated fixture doc that lays out what the integration / uni
|
||||
- [HistoricalDataAccess.md](../v1/HistoricalDataAccess.md) — `IHistoryProvider` dispatch, aggregate mapping, continuation points. The OPC UA Client driver is the only driver that implements driver-side `IHistoryProvider` (it forwards HistoryRead to the upstream server); the AVEVA Historian path is served server-side by the HistorianGateway-backed `IHistorianDataSource` instead. Other drivers do not implement the interface and return `BadHistoryOperationUnsupported`.
|
||||
- [AlarmTracking.md](../AlarmTracking.md) — `IAlarmSource` event model and filtering. Implemented by Galaxy (native MxAccess alarms, working end-to-end), OPC UA Client, AB CIP, and FOCAS; AB Legacy, Modbus, S7, TwinCAT, and MQTT have no alarm source.
|
||||
- [Subscriptions.md](../v1/Subscriptions.md) — how the Server multiplexes subscriptions onto `ISubscribable.OnDataChange`.
|
||||
- [docs/v2/driver-stability.md](../v2/driver-stability.md) — tier system (A / B / C), shared `CapabilityPolicy` defaults per tier × capability, `MemoryTracking` hybrid formula, and process-level recycle rules.
|
||||
- [docs/v2/driver-stability.md](../v2/driver-stability.md) — historical design record of the A / B / C tier model. Only `DriverTier` + the `CapabilityPolicy` defaults per tier × capability (`DriverResilienceOptions.GetTierDefaults`) are live; its `MemoryTracking` / recycle / out-of-process isolation machinery was deleted (#522) and every driver runs Tier A.
|
||||
- [docs/v2/plan.md](../v2/plan.md) — authoritative vision, architecture decisions, migration strategy.
|
||||
|
||||
@@ -31,35 +31,89 @@ plaintext secret on every node, no matter its role:
|
||||
combination — must unwrap the store with the exact same master key. A
|
||||
per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
|
||||
decrypt every *other* node's ciphertext rows to garbage.
|
||||
2. **Identical store rows on every node.** All nodes must read the same SQLite
|
||||
database (same file, or a replicated/shared copy with the same rows) — not
|
||||
independently-seeded stores that happen to share a KEK.
|
||||
2. **The same store rows on every node of a pair.** Nodes must read the same rows —
|
||||
not independently-seeded stores that merely share a KEK. Either mount one SQLite file
|
||||
(see "Alternative: shared volume" below) or, preferred, turn on Akka secret replication
|
||||
so each pair converges its own rows.
|
||||
|
||||
`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
|
||||
— there is no built-in cross-node replication today. Meeting both requirements in
|
||||
production is a deployment concern, covered below.
|
||||
The base `ZB.MOM.WW.Secrets` store is a plain local SQLite `ISecretStore` whose default
|
||||
replicator is `NoOpSecretReplicator`: **with `Secrets:Replication:Enabled=false` (the
|
||||
default) nothing replicates and each node's store is private.** That is what this page
|
||||
used to describe as the only option. It is no longer: pair replication exists
|
||||
(`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`, lib pin `0.6.2` in `Directory.Packages.props`;
|
||||
gated by `SecretsRegistration`, see below).
|
||||
|
||||
## Recommended interim posture (G-5)
|
||||
## Recommended posture: pair-local Akka replication
|
||||
|
||||
Until real replication exists (G-7, below), the recommended production posture is:
|
||||
Opt in with `Secrets:Replication:Enabled=true` on **every node** (default `false`, so absent
|
||||
config means a plain local SQLite store). `SecretsRegistration.AddOtOpcUaSecrets`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`) then registers
|
||||
`AddZbSecretsAkkaReplication` **instead of** `AddZbSecrets` (the two are alternatives, never
|
||||
additive) and a `SecretReplicationStarter` hosted service resolves the store at boot so a
|
||||
node that never reads a secret still joins anti-entropy. Keys: `Secrets:Replication:Enabled`,
|
||||
`:AnnounceInterval` (default `00:00:30`; each node publishes a name/revision/timestamp/tombstone
|
||||
manifest per interval, no ciphertext), `:ActorName` (`zb-secret-replication`).
|
||||
|
||||
- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
|
||||
key file that is identical on every node of every role** — a base64-encoded
|
||||
32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed
|
||||
to each node's filesystem/secret-mount by the deployment tooling, and **never
|
||||
committed** to the repo. Treat it with the same discipline as any other
|
||||
production secret (restrictive file ACLs, no logging, rotated via a future
|
||||
KEK-rotation runbook — not yet built).
|
||||
- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
|
||||
that every node mounts (admin, driver, and dev/fused alike), so every node's
|
||||
migrator opens and reads the same rows at boot.
|
||||
- **Scope is PAIR-LOCAL.** Since mesh Phase 6 each application cluster is its own 2-node Akka
|
||||
mesh and replication rides that mesh's DistributedPubSub, so a secret converges **within its
|
||||
pair and nowhere else**. There is no fleet-wide sync (site nodes have no SQL hub). Seed each
|
||||
pair independently: central via `/admin/secrets`, driver-only site pairs (no AdminUI) via the
|
||||
secrets CLI against one node's store, or a ciphertext-only bundle import. See
|
||||
`docs/Redundancy.md` §"Secrets replication is pair-local".
|
||||
- **Both nodes of a pair must share the same KEK** — rows replicate verbatim (ciphertext, wrapped
|
||||
DEK and `kek_id` untouched), never re-encrypted.
|
||||
- **Deletes replicate** as soft tombstones (last-writer-wins on `updated_utc`/`revision`).
|
||||
- **Re-wraps do not replicate** (a re-wrap preserves `revision`/`updated_utc`, so it is invisible to
|
||||
last-writer-wins). KEK rotation therefore runs against each node's store — see the next section.
|
||||
- The store should sit on a **persistent per-node path** (`Secrets:SqlitePath`, e.g. the node's
|
||||
`/app/data` volume in containers); the relative default lands inside the image layer and is lost on
|
||||
recreate.
|
||||
- Evidence: live-gated PASS 4/4 on the three-mesh docker-dev rig (convergence <= 7 s, no cross-pair
|
||||
leakage in any direction, delete < 1 s, clean restart without resurrection) —
|
||||
`docs/plans/2026-08-07-secrets-pair-local-live-gate.md`. Replication stays default-off in the product.
|
||||
|
||||
Writes to the store are rare and human-driven — an operator using the
|
||||
`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while
|
||||
reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle,
|
||||
regardless of role. The access pattern is read-mostly / effectively
|
||||
single-writer, which is what makes a shared SQLite volume viable as an interim
|
||||
posture (see caveat below).
|
||||
Whichever store layout is used, the KEK is delivered per node as before:
|
||||
**`Secrets:MasterKey:Source = File`** with `FilePath` pointing at a read-only base64 32-byte key
|
||||
(`openssl rand -base64 32`), distributed identically to every node of the pair by deployment tooling and
|
||||
**never committed**. Writes are rare and human-driven (`/admin/secrets` on admin nodes, or the
|
||||
CLI); reads happen at every node's boot and each `ResolveCacheTtl` refresh.
|
||||
|
||||
## Rotating the KEK
|
||||
|
||||
Rotation re-wraps each row's per-secret DEK under the new KEK (`wrapped_dek`, `wrap_nonce`,
|
||||
`wrap_tag`, `kek_id` change; the sealed body, `revision` and `updated_utc` do not). The
|
||||
authoritative operator procedure and troubleshooting table live with the library:
|
||||
`scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`, driven by the `secret rewrap-all`
|
||||
CLI verb (in that repo; not packed). The OtOpcUa-specific points:
|
||||
|
||||
1. **Back up every store first** (copy each node's SQLite file) — the backup is the complete undo.
|
||||
2. **Stage the new KEK on every node of the pair** (same file path / env var as `Secrets:MasterKey`
|
||||
resolves) but do not restart yet. Never pass key values on a command line; use
|
||||
`--old-key-env VAR` / `--old-key-file PATH` and, if the new key is not the node's configured
|
||||
provider, `--new-key-file`.
|
||||
3. **Run `secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY` once per store, i.e. on EACH node.**
|
||||
Because re-wraps do not replicate, rotating one node leaves its partner's rows on the old
|
||||
`kek_id`. With replication on, do not leave the pair half-rotated: a node whose rows carry a KEK
|
||||
it cannot unwrap fails closed (`Row wrapped by unknown KEK`) on resolve.
|
||||
4. **Quiesce first**: stop (or drain) the node while its store is rewrapped. Rotating a pair
|
||||
node-by-node while the other node keeps serving works for resolves, but do not author secrets
|
||||
anywhere in the pair mid-rotation — a fresh write is wrapped under whichever KEK the writing node
|
||||
currently uses and replicates verbatim to a partner that may not yet hold that KEK.
|
||||
5. **Cut over**: confirm `Secrets:MasterKey` resolves the new key on both nodes, start them, and
|
||||
confirm a `${secret:}` value resolves (the pre-host expander fails the boot loudly if not).
|
||||
6. **Destroy the old key** only after both stores resolve cleanly. Re-running `rewrap-all` is
|
||||
idempotent (`alreadyCurrent`), so an interrupted pass is resumed by running it again.
|
||||
7. Rollback: restore the step-1 backups on both nodes and revert the KEK source.
|
||||
|
||||
Pair rotation across **separate pairs** (central, site-a, site-b) is independent per pair; there is no
|
||||
fleet-wide order.
|
||||
|
||||
## Alternative: shared SQLite volume (no replication)
|
||||
|
||||
The earlier interim posture — every node of a pair mounting one `Secrets:SqlitePath` on a shared
|
||||
or replicated volume — still works for a deployment that keeps `Secrets:Replication:Enabled=false`:
|
||||
reads dominate and writes are effectively single-writer. See the caveat below; it is not safe with
|
||||
concurrent writers, and it is impractical for site pairs whose nodes share no volume.
|
||||
|
||||
## How it's delivered (do NOT commit these values)
|
||||
|
||||
@@ -107,14 +161,14 @@ or key is missing.
|
||||
|
||||
SQLite's file-locking model does not tolerate concurrent multi-writer access well
|
||||
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
|
||||
block volume only one writer should be active at a time). The interim posture
|
||||
block volume only one writer should be active at a time). The shared-volume alternative
|
||||
above is acceptable because:
|
||||
|
||||
- Reads dominate (every node's boot + cache-refresh cycle, across every role).
|
||||
- Writes are rare, human-initiated, and effectively single-writer in practice
|
||||
(an operator runs the CLI/UI against one admin node at a time).
|
||||
|
||||
It is **not** a substitute for real replication, and it is not safe if multiple
|
||||
It is **not** a substitute for real replication (Akka pair replication, above), and it is not safe if multiple
|
||||
nodes attempt concurrent writes. Do not build automation that writes secrets
|
||||
from more than one node simultaneously.
|
||||
|
||||
@@ -128,31 +182,15 @@ ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryptio
|
||||
(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
|
||||
work — doing so risks invalidating active sessions for an unrelated reason.
|
||||
|
||||
It is, however, the model for where the *next* iteration of secret storage
|
||||
should go — see the G-7 hand-off below.
|
||||
## The G-7 hand-off (status)
|
||||
|
||||
## The G-7 hand-off
|
||||
|
||||
The posture above is an interim, ops-only workaround. The long-term shape,
|
||||
tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a
|
||||
ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for
|
||||
the Data Protection key ring:
|
||||
|
||||
```csharp
|
||||
services.AddDataProtection()
|
||||
.PersistKeysToDbContext<OtOpcUaConfigDbContext>()
|
||||
.SetApplicationName("OtOpcUa");
|
||||
```
|
||||
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`).
|
||||
`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused
|
||||
alike — a single MS SQL-backed source of truth for the Data Protection key ring;
|
||||
the secret store is the natural next tenant of that same shared database instead
|
||||
of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library
|
||||
code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet,
|
||||
overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not
|
||||
built as part of this cut. This runbook's shared-SQLite-volume posture is the
|
||||
bridge until G-7 lands.
|
||||
G-7 (`scadaproj/components/secrets/GAPS.md`) asked for real cross-node replication. It was
|
||||
built in two flavours in the library (`ZB.MOM.WW.Secrets.Replicator.SqlServer` and
|
||||
`.AkkaDotNet`, plus a gRPC replicator package). **OtOpcUa production uses the Akka pair-local
|
||||
option** (the decision recorded in scadaproj#3), wired by `SecretsRegistration` as described above;
|
||||
the ConfigDb-backed store this page used to anticipate is not what shipped here. Driver-only nodes
|
||||
have no ConfigDb since mesh Phase 4, which is a further reason a ConfigDb secret store would not
|
||||
reach them.
|
||||
|
||||
## Dev/test/default posture (unchanged)
|
||||
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-08-15-perf-roadmap-remainder.md",
|
||||
"tasks": [
|
||||
{"id": 1, "subject": "Task 1: Outbox batch append with group commit", "status": "in_progress", "note": "6ff64ca0 + fix 73205865 (incremental bookkeeping, second-reader durability pin); re-review in flight"},
|
||||
{"id": 1, "subject": "Task 1: Outbox batch append with group commit", "status": "completed", "note": "6ff64ca0 + fix 73205865 (incremental bookkeeping, second-reader durability pin); merged to master (verified 2026-10-08)"},
|
||||
{"id": 2, "subject": "Task 2: Batched mux→subscriber delivery, single-eval virtual tags", "status": "completed", "note": "content inside 140bf0c1 (shared-index sweep; message belongs to Task 22 — needs an outcomes note); spec + code approved; code review's two Importants (yielding-fake batch test, mid-batch exception policy) folded into Task 4's brief"},
|
||||
{"id": 3, "subject": "Task 3: Parallel drain with bounded concurrency", "status": "completed", "note": "a76b4e07; spec + code APPROVED; follow-up (inherited, pre-existing): bare catch around WriteLiveValuesAsync swallows OperationCanceledException at shutdown → spurious _tagWriteFailures increments when succeeded>0; recommend rethrowing genuine ct cancellations"},
|
||||
{"id": 4, "subject": "Task 4: Recorder batched ingest — one fsync per burst", "status": "completed", "blockedBy": [1, 2, 3], "note": "384568b4 + follow-up 451273c8 (append-failure branch pinned via 3 independent mutations, loss-accounting docs corrected, recorder 64→1024 literals aligned); all reviews APPROVED; register: #491 falsification nondeterministic under load, IngestFaults not surfaced in docs/gauges, latch re-arms on any non-throwing item"},
|
||||
{"id": 5, "subject": "Task 5: Historian defaults, config drift, docs", "status": "completed", "blockedBy": [3, 4], "note": "0298a0b9 (new Historian.md key table, cross-repo batch-RPC follow-up recorded); code APPROVED; recorder 64→1024 literals aligned in 451273c8; minor: pinning-test doc comment overclaims appsettings-drift coverage"},
|
||||
{"id": 6, "subject": "Task 6: Per-session decision-cache shards", "status": "completed", "note": "debd44c2; follow-ups: _globalCount drift under races, concurrency stress test, bound-ratio comment"},
|
||||
{"id": 7, "subject": "Task 7: Authorizer hot path — lookup first, no closures, lazy deny reasons", "status": "in_progress", "blockedBy": [6]},
|
||||
{"id": 7, "subject": "Task 7: Authorizer hot path — lookup first, no closures, lazy deny reasons", "status": "completed", "blockedBy": [6], "note": "d8a17f2e; merged to master (verified 2026-10-08)"},
|
||||
{"id": 8, "subject": "Task 8: Browse and TranslateBrowsePath compaction", "status": "completed", "note": "541a223b + follow-up test aad7583b (survivor-order pin, falsified red, review APPROVED); minor: FanOutTargetIdList fake couples to SDK single-Add contract — comment if pattern reused"},
|
||||
{"id": 9, "subject": "Task 9: RevalidateMonitoredItems dedup", "status": "in_progress", "blockedBy": [8], "note": "fbe8aaad; realm-qualified 3-tuple key (safer than spec); spec+code reviews in flight; watch: its clock-count observable couples to DataPlaneAuthorizer reading the clock once — Task 7 edits that file"},
|
||||
{"id": 9, "subject": "Task 9: RevalidateMonitoredItems dedup", "status": "completed", "blockedBy": [8], "note": "fbe8aaad; realm-qualified 3-tuple key (safer than spec); merged to master (verified 2026-10-08); watch: its clock-count observable couples to DataPlaneAuthorizer reading the clock once — Task 7 edits that file"},
|
||||
{"id": 10, "subject": "Task 10: Modbus InitializeAsync transport leak", "status": "completed", "note": "c3970a2e + 409a58b6"},
|
||||
{"id": 11, "subject": "Task 11: PollGroupEngine fixed-rate pacing, overrun counter, async unsubscribe", "status": "completed", "blockedBy": [10, 12, 13, 14], "note": "9cf400f1 + tolerance fix 485f4ea1 (fast-reader test: 200ms interval, <=1 blip); spec ✅ + code APPROVED; follow-ups: NoteOverrun conflates slow-success/slow-failure (doc caveat), backoffCap param doc note for future callers"},
|
||||
{"id": 12, "subject": "Task 12: S7 narrow-scalar span codec", "status": "completed", "note": "8d91833f; SByte spec mention was a plan error"},
|
||||
{"id": 13, "subject": "Task 13: S7 span planner + grouped ReadAsync", "status": "in_progress", "blockedBy": [12], "note": "39f67a0f; spec passed (demotion rule + factory-binding 6th file accepted); code review in flight; follow-up: live gate vs s7_1500 fixture before trusting 16/220 defaults"},
|
||||
{"id": 13, "subject": "Task 13: S7 span planner + grouped ReadAsync", "status": "completed", "blockedBy": [12], "note": "39f67a0f; spec passed (demotion rule + factory-binding 6th file accepted); merged to master (verified 2026-10-08); follow-up: live gate vs s7_1500 fixture before trusting 16/220 defaults"},
|
||||
{"id": 14, "subject": "Task 14: AbCip bounded-parallel reads", "status": "completed", "note": "d6a4fbd6 + fix d5f573b0 (per-device ConnectGate single-flight); re-review APPROVED; minor follow-ups: stale EvictRuntime-race comment ~L1161, test timing margin on slow CI"},
|
||||
{"id": 15, "subject": "Task 15: Dirty-set coalescing in ScriptedAlarmEngine", "status": "completed", "note": "5df793e7; follow-ups: TrackBackgroundTask registration-ordering gap (pre-existing), stale line-334 comment, non-volatile _disposed"},
|
||||
{"id": 16, "subject": "Task 16: Deployments list projection", "status": "completed", "note": "dbc9a132"},
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
{"id": 5, "subject": "Task 5: Live gate — FetchAndCache legs", "status": "completed", "note": "GATE PASSED, no commit. All 6 legs: fetch+SHA-verify on all 4 site nodes (1584 bytes from central-1:4055), Served×4 central-side, applied-FETCHED + cached on site-a-1 (acks=6), boot-from-cache after restart (restored rev matches what it cached 52s earlier), absence proof (0 Stale/ConfigDb-back/Direct-form/ERR/WRN over full lifetime), post-restart browse proves both namespaces materialised (ns=2 raw + ns=3 UNS). Method: same-revision deploys short-circuit the fetch path (immediate-ACK at HandleDispatchFromSteady) — gate forced a temp UnsArea row then reverted; rig config restored byte-identical (rev 315de861). Register: central-2 serve path unexercised (first-success endpoint walk always hit central-1)"},
|
||||
{"id": 6, "subject": "Task 6: Live gate — alarm-bearing surgical ratchet", "status": "completed", "note": "GATE PASSED, no commit. Four deploys, all routings correct: Writable flip ⇒ kind=AttributeOnly surgicalRawTags=1 rebuild=False; alarm ADD ⇒ kind=Rebuild rebuild=True; alarm REMOVE ⇒ rebuild=True (previous-side disqualifies); restore ⇒ surgical again (ratchet per-deploy, not latched). All sealed 6/6 acks. Authoring: direct UPDATE on the rig's OWN sql container (Database=OtOpcUa — NOT 10.100.0.35; sqlcmd needs -I for QUOTED_IDENTIFIER). Findings: only cluster-MAIN nodes emit applier lines (site plans empty via ResolveClusterScope, apply early-returns); artifact hash can never return to baseline after any UPDATE — ConfigComposer snapshots SQL rowversion (verified by gunzip+diff: only the RowVersion differs). Rig config semantically restored; rig UP"},
|
||||
{"id": 7, "subject": "Task 7: Live gate — ACL-enabled enforcement on site-a", "status": "completed", "note": "GATE PASSED, no commit. TOPOLOGY ADAPTATION: gated node = central-1 via scratchpad-only compose overlay (site-a serves NO value nodes — the rig's one tag is cluster MAIN; central-2 became the ungated A/B control). ALLOW leg genuinely load-bearing: ungated baseline captured first — same key+call DENIED 0x801F0000 with ACL off (Tag grant confers no role), SUCCEEDED with ACL on ⇒ ACL decision REPLACES the role check, live. Deny×3 (anonymous write, out-of-scope browse, ungated control reads fine); revocation split into grant-removal (gen 1→2, primed per-key shard discarded — exercises this round's CAS-loop path) and key-disable (pre-cache re-check). Authoring: DB fallback (Chrome extension not connected) — ApiKeySecret hashing reproduced from source, pepper verified live. Restore byte-identical (rev a3a9061b), zero residue. FINDINGS for register: (1) browse transit stops below the realm root — grant holder cannot browse DOWN to its own tag from ns=2;s=OtOpcUa (fails closed; file as issue); (2) ns=0 core nodes are ungated (CoreNodeManager bypasses DataPlaneAuthorizer) — boot warning overstates 'see nothing'"},
|
||||
{"id": 8, "subject": "Task 8: Close-out — sweep + Outcomes + register annotations", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 9, 10], "note": "c4d6cba7 (Outcomes + sweep record + r2 register annotations, +383/−0) + the coordinator's flake-register append. Rig torn down BEFORE the sweep (rig-up skews skips: Host.IntegrationTests 229/0/2 up vs 226/0/5 down). Build 0 errors / 904 warnings — IDENTICAL to r2 (note: incremental builds under-report at 896). Sweep 43/43 green: 10,377 / 0 / 19 — +7 reconciled exactly (OpcUaServer +1 T1, S7 +6 T2); skips identical project-for-project; AdminUI 1540/0-skipped + Configuration 190/0-skipped ran vs 10.100.0.35,14330. Host.IntegrationTests (outside the 43-project baseline) +9 vs merge-base (LocalDbSetupTests 7→15, WiringTests 6→7). One flake isolated: Client.CLI LoggerLifecycleTests — pre-existing global-Serilog race, no r3 commit touches it, registered"}
|
||||
{"id": 8, "subject": "Task 8: Close-out — sweep + Outcomes + register annotations", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 9, 10], "note": "c4d6cba7 (Outcomes + sweep record + r2 register annotations, +383/−0) + the coordinator's flake-register append. Rig torn down BEFORE the sweep (rig-up skews skips: Host.IntegrationTests 229/0/2 up vs 226/0/5 down). Build 0 errors / 904 warnings — IDENTICAL to r2 (note: incremental builds under-report at 896). Sweep 43/43 green: 10,377 / 0 / 19 — +7 reconciled exactly (OpcUaServer +1 T1, S7 +6 T2); skips identical project-for-project; AdminUI 1540/0-skipped + Configuration 190/0-skipped ran vs 10.100.0.35,14330. Host.IntegrationTests (outside the 43-project baseline) +9 vs merge-base (LocalDbSetupTests 7→15, WiringTests 6→7). One flake isolated: Client.CLI LoggerLifecycleTests — pre-existing global-Serilog race, no r3 commit touches it, registered"},
|
||||
{"id": 9, "subject": "Task 9: Roslyn evaluator racing test — remove timing assumption (ScadaBridge row 38)", "status": "completed", "note": "34829ea0 + 837d8a17 (review round). Option (b): renamed to Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure; three-bucket classification (leaked=fail-first, exhausted + starved each bounded max(evals/10,8)); duration-bounded loops (10s OR 400 iters) — the REAL load failure was the test's own 45s WaitAsync budget under recompile pressure (old shape RED 4/4 at 4-way concurrency, new GREEN 4/4 + 8/8); reviewer-found starvation-timeout flake vector closed (TimedScriptEvaluator's 5s clock starts at enqueue), falsified via 12ms-budget probe (old classification RED, new GREEN at 0.11%); literals promoted to shared ScriptEvalFailureReasons in Commons (both evaluators + test). Honest disclosure: disposed race does not reproduce on this arm64 box at all. Spec ✅ + code CHANGES REQUESTED → delta re-APPROVED. Host.IntegrationTests 229/0/2, Calculation 53/53"},
|
||||
{"id": 10, "subject": "Task 10: LocalDb conditional registration + baseline + deregister self-heal (ScadaBridge row 39)", "status": "completed", "note": "3e62f161 + df533916 (review round: content-based data-survival pin, mid-loop loud-fail pin via exhaustive delegating fake, Log.Logger assumption comment). Spec ✅ + code APPROVED + delta re-APPROVED. Predicate = PeerAddress OR ApiKey (SyncListenPort deliberately out — passive endpoint fail-closed on key); 0.2.1 APIs verified by reflection+probe (baseline seeds hlc=0; DeregisterReplicated returns bool) — NO pin bump; 0.3.1 now published (ScadaBridge heads-up) — bump recorded as deliberate both-nodes-together future change (bookkeeping v2→v3 one-way). Surfaced+fixed two vacuous-fixture bugs (PairHarness/AlarmSfLegacyMigrator two-config split). scadaproj index updated + pushed (1145df6). Suites: LocalDbSetupTests 15/15, Host.IntegrationTests 226/0/5"}
|
||||
],
|
||||
|
||||
+17
-1
@@ -1,6 +1,22 @@
|
||||
# OtOpcUa v2 — Driver Implementation Specifications
|
||||
|
||||
> **Status**: DRAFT — reference document for plan.md
|
||||
> **Status (2026-10)**: v2-era DRAFT, **not authoritative**. It covers 8 of the 12 shipped drivers and
|
||||
> several of its architecture claims are stale. The current per-driver references are the pages in
|
||||
> [`docs/drivers/`](../drivers/README.md) (the README there lists all 12 and is the accurate overview).
|
||||
>
|
||||
> - **Not covered here at all:** MQTT/Sparkplug ([Mqtt.md](../drivers/Mqtt.md)), MTConnect
|
||||
> ([MTConnect.md](../drivers/MTConnect.md)), Sql ([Sql.md](../drivers/Sql.md)), Calculation
|
||||
> ([Calculation.md](../drivers/Calculation.md)).
|
||||
> - **Hosting/tier claims are stale:** every driver runs in-process as Tier A (Galaxy reaches MXAccess over
|
||||
> gRPC to `mxaccessgw`; FOCAS is a pure-managed in-process `FocasWireClient`, not a `Focas.Host` pipe
|
||||
> service) — see `docs/v2/driver-stability.md`'s banner. The "SystemPlatform namespace kind" is retired:
|
||||
> Galaxy is a standard Equipment-kind driver, and the address space is the dual Raw/UNS namespace
|
||||
> ([../AddressSpace.md](../AddressSpace.md)).
|
||||
> - **Modbus:** the driver has its own socket layer (no NModbus dependency) and a driver-level
|
||||
> `transport` key — `Tcp` (default, MBAP framing) or `RtuOverTcp` (RTU `[addr][PDU][CRC-16]` framing tunnelled
|
||||
> to a serial-to-Ethernet gateway); see [Modbus.md](../drivers/Modbus.md).
|
||||
> - **AB CIP / AB Legacy:** a second, managed `Transport` (`Native` | `Libplctag`) exists — `CLAUDE.md`
|
||||
> §"AB drivers: native transport" and the driver pages.
|
||||
>
|
||||
> **Created**: 2026-04-16
|
||||
|
||||
|
||||
@@ -49,6 +49,10 @@ Driver stability needs to be a first-class architectural concern, not a per-driv
|
||||
|
||||
Every driver is assigned to one of three tiers based on the trust level of its dependency stack:
|
||||
|
||||
> **Design intent, not the shipped assignment.** The per-driver tier tables below record the proposed
|
||||
> assignment (Galaxy/FOCAS in Tier C, the Ab/S7/TwinCAT drivers in Tier B). As shipped every driver is Tier A —
|
||||
> see the banner above and `docs/drivers/README.md`, which is the accurate reference.
|
||||
|
||||
### Tier A — Pure Managed
|
||||
|
||||
Drivers whose entire dependency chain is verifiable .NET. Standard exception handling and Polly are sufficient. Run in-process in the main server.
|
||||
|
||||
Reference in New Issue
Block a user