docs: rewrite IncrementalSync + AddressSpace for v3, fix driver-specs/driver-stability tier claims, update Secrets runbook for pair replication + KEK rotation (#623)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
+58
-77
@@ -1,90 +1,71 @@
|
||||
# Incremental Sync
|
||||
|
||||
> ⚠️ **Accuracy warning (audited 2026-07-27) — most of this page describes machinery that is retired
|
||||
> or was never wired.** Read it as v2-era design history, not as current behaviour.
|
||||
>
|
||||
> - **"Rebuild flow" (below) is not a production path.** It is built on `GenericDriverNodeManager`,
|
||||
> which is Core test scaffolding with **zero production references**
|
||||
> (`GenericDriverNodeManager.cs:71`). Its `_variablesByFullRef` / `_sourceByFullRef` fields do not
|
||||
> exist in `src/`.
|
||||
> - **Driver-backend rediscovery has no consumer.** Nothing in `src/Server/` or `src/Core/`
|
||||
> subscribes to `IRediscoverable.OnRediscoveryNeeded`; `DriverHostActor.HandleDiscoveredNodes`
|
||||
> (`DriverHostActor.cs:986-1002`) short-circuits unconditionally. The `ScopeHint` is read by
|
||||
> nothing. Drivers that raise it today: Galaxy, TwinCAT, MQTT/Sparkplug, MTConnect
|
||||
> ([#518](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/518),
|
||||
> [#507](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/507)).
|
||||
> - **The generation-publish path is retired.** `sp_PublishGeneration` and
|
||||
> `sp_ComputeGenerationDiff` are stubs that `RAISERROR`
|
||||
> (`20260715230637_V3Initial.StoredProcedures.cs:194`); `DraftRevisionToken` does not exist.
|
||||
>
|
||||
> **The one live rebuild path in v3** is the deploy artifact: `ConfigComposer` →
|
||||
> `Deployment.ArtifactBlob` → `DriverHostActor` apply → `AddressSpaceComposer` /
|
||||
> `AddressSpaceApplier` diff-apply.
|
||||
How a change reaches the running server's address space in v3. There is exactly one path that
|
||||
changes the served address space — the **deploy artifact** — and one advisory signal
|
||||
(`IRediscoverable`) that never changes it.
|
||||
|
||||
Two distinct change-detection paths feed the running server: driver-backend rediscovery (Galaxy's `time_of_last_deploy`, TwinCAT's symbol-version-changed) and generation-level config publishes from the Admin UI. Both flow into re-runs of `ITagDiscovery.DiscoverAsync`, but they originate differently.
|
||||
## The deploy-artifact path
|
||||
|
||||
## Driver-backend rediscovery — IRediscoverable
|
||||
1. **Compose.** `ConfigComposer` (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs`)
|
||||
snapshots the whole Config DB into one gzip-compressed JSON blob (`Deployment.ArtifactBlob`) whose
|
||||
`RevisionHash` is the SHA-256 of those exact bytes. A deployment is fleet-wide; each node
|
||||
self-scopes the artifact (`DeploymentArtifact.ResolveClusterScope`).
|
||||
2. **Dispatch.** `ConfigPublishCoordinator` (`.../ControlPlane/Coordinators/`) sends each enabled
|
||||
`ClusterNode` a payload-free dispatch and waits for acks until the apply deadline. A node obtains the
|
||||
bytes either straight from central SQL (`ConfigSource:Mode=Direct`) or over gRPC with a SHA-256
|
||||
check (`FetchAndCache`) — see `docs/Configuration.md` §`ConfigSource`.
|
||||
3. **Apply.** `DriverHostActor` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/`) opens the artifact and
|
||||
reconciles in two halves:
|
||||
- **Drivers** — `DriverSpawnPlanner.Compute` yields `ToStop` / `ToApplyDelta` / `ToSpawn`. A changed
|
||||
driver config reaches the live driver as `IDriver.ReinitializeAsync` on its `DriverInstanceActor`;
|
||||
a removed driver is stopped. An unreadable or empty blob is "no answer": the apply fails and the
|
||||
running children are left alone (never an empty configuration).
|
||||
- **Address space** — `AddressSpaceComposer` builds an `AddressSpaceComposition` from the artifact;
|
||||
`AddressSpacePlan` diffs it against the last applied composition; `AddressSpaceChangeClassifier`
|
||||
picks the delta class; `AddressSpaceApplier`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/`) mutates `OtOpcUaNodeManager` through the sink.
|
||||
4. **Classes** (`AddressSpaceChangeKind`): `Empty`, `AttributeOnly`, `PureAdd`, `PureRemove`,
|
||||
`AddRemoveMix` — all applied surgically, so unrelated MonitoredItems survive — and `Rebuild`, the
|
||||
default-closed fallback for anything the classifier does not whitelist. Any surgical `false`/throw
|
||||
also ratchets to a safe rebuild, and a degraded non-rebuild apply does not advance the last-applied
|
||||
composition (the next deploy re-diffs and re-ensures). The whitelist, the UNS-mirror repair rule and
|
||||
the add/remove announcements are described in the "surgical deploys" paragraph of `CLAUDE.md`
|
||||
and `docs/plans/2026-08-15-surgical-deploys.md`.
|
||||
|
||||
Drivers whose backend has a native change signal implement `IRediscoverable` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs`):
|
||||
Virtual tags and scripted alarms ride the same artifact and the same plan; a script edit is just another
|
||||
changed record in the composition (see `docs/VirtualTags.md`, `docs/ScriptedAlarms.md`).
|
||||
|
||||
```csharp
|
||||
public interface IRediscoverable
|
||||
{
|
||||
event EventHandler<RediscoveryEventArgs>? OnRediscoveryNeeded;
|
||||
}
|
||||
public sealed record RediscoveryEventArgs(string Reason, string? ScopeHint);
|
||||
```
|
||||
## Driver-backend rediscovery is advisory
|
||||
|
||||
The driver fires the event with a reason string (for the diagnostic log) and an optional scope hint — a non-null hint lets Core scope the rebuild surgically to that subtree; null means "the whole address space may have changed".
|
||||
Drivers whose backend signals a tag-set change implement `IRediscoverable`
|
||||
(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs`; `OnRediscoveryNeeded` carries a
|
||||
`Reason` and an optional `ScopeHint`). Raisers today: **Galaxy** (`DeployWatcher`, new
|
||||
`time_of_last_deploy`), **TwinCAT** (ADS symbol-version-changed), **MQTT/Sparkplug** and **MTConnect**.
|
||||
AbCip, AbLegacy, FOCAS, Modbus, S7 and the rest raise nothing.
|
||||
|
||||
Drivers that implement the capability today:
|
||||
`DriverInstanceActor` is the only production consumer: it subscribes in `PreStart`, detaches in
|
||||
`PostStop`, and folds the raise into the driver-health snapshot
|
||||
(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`), which the AdminUI `/hosts` page shows
|
||||
as a **re-browse** chip. **It never rebuilds or edits the served address space** — v3 raw tags are authored
|
||||
explicitly in `/raw`, so an operator re-browses the device and commits, and the result reaches the
|
||||
server as an ordinary deployment. `ScopeHint` is not read by the actor. (The older runtime
|
||||
injection path was removed, #507 closed as superseded.)
|
||||
|
||||
- **Galaxy** — `DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) subscribes to the mxaccessgw gRPC stream (`IGalaxyDeployWatchSource.WatchAsync`) and fires on a new `time_of_last_deploy` value. The gateway polls the Galaxy repository DB internally; the driver side is event-driven.
|
||||
- **TwinCAT** — observes ADS symbol-version-changed notifications (ADS error `DeviceSymbolVersionInvalid`, decimal 1809 / `0x0711`). Note: legacy Beckhoff documentation sometimes cites `0x0702` (`DeviceInvalidGroup`) — that is a transcription error; the correct code is `0x0711` per `TwinCATStatusMapper.AdsSymbolVersionChanged` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATStatusMapper.cs:35`).
|
||||
## Not part of the v3 path
|
||||
|
||||
Static drivers (Modbus, S7, AB CIP, AB Legacy, FOCAS) do not implement `IRediscoverable` — their tags only change when a new generation is published from the Config DB. Core sees absence of the interface and skips change-detection wiring for those drivers (decision #54).
|
||||
|
||||
## Config-DB generation publishes
|
||||
|
||||
Tag-set changes authored in the Admin UI (UNS edits, CSV imports, driver-config edits) accumulate in a draft generation and commit via `sp_PublishGeneration`. The delta between the currently-published generation and the proposed next one is computed by `sp_ComputeGenerationDiff`, which drives:
|
||||
|
||||
- The publish-preview surface in the Admin UI (`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Deployments.razor`, backed by `AdminOperationsClient`) so operators can preview what will change before clicking Publish.
|
||||
- The 409-on-stale-draft flow (decision #161) — a UNS drag-reorder preview carries a `DraftRevisionToken` so Confirm returns `409 Conflict / refresh-required` if the draft advanced between preview and commit.
|
||||
|
||||
After publish, the server's generation applier invokes `IDriver.ReinitializeAsync(driverConfigJson, ct)` on every driver whose `DriverInstance.DriverConfig` row changed in the new generation. Reinitialize is the in-process recovery path for Tier A/B drivers; if it fails the driver is marked `DriverState.Faulted` and its nodes go Bad quality — but the server process stays running. See `docs/v2/driver-stability.md`.
|
||||
|
||||
Drivers whose discovery depends on Config DB state (Modbus register maps, S7 DBs, AB CIP tag lists) re-run their discovery inside `ReinitializeAsync`; Core then diffs the new node set against the current address space.
|
||||
|
||||
## Rebuild flow
|
||||
|
||||
When a rediscovery is triggered (by either source), `GenericDriverNodeManager` re-runs `ITagDiscovery.DiscoverAsync` into the same `CapturingBuilder` it used at first build. The new node set is diffed against the current:
|
||||
|
||||
1. **Diff** — full-name comparison of the new `DriverAttributeInfo` set against the existing `_variablesByFullRef` map. Added / removed / modified references are partitioned.
|
||||
2. **Snapshot subscriptions** — before teardown, Core captures the current monitored-item ref-counts for every affected reference so subscriptions can be replayed after rebuild.
|
||||
3. **Teardown** — removed / modified variable nodes are deleted via `CustomNodeManager2.DeleteNode`. Driver-side subscriptions for those references are unwound via `ISubscribable.UnsubscribeAsync`.
|
||||
4. **Rebuild** — added / modified references get fresh `BaseDataVariableState` nodes via the standard `IAddressSpaceBuilder.Variable(...)` path. Alarm-flagged references re-register their `IAlarmConditionSink` through `CapturingBuilder`.
|
||||
5. **Restore subscriptions** — for every captured reference that still exists after rebuild, Core re-opens the driver subscription and restores the original ref-count.
|
||||
|
||||
Exceptions during teardown are swallowed per decision #12 — a driver throw must not leave the node tree half-deleted.
|
||||
|
||||
## Scope hint
|
||||
|
||||
When `RediscoveryEventArgs.ScopeHint` is non-null (e.g. a folder path), Core restricts the diff to that subtree. This matters for Galaxy Platform-scoped deployments where a `time_of_last_deploy` advance may only affect one platform's subtree. Null scope falls back to a full-tree diff.
|
||||
|
||||
## Virtual tags in the rebuild
|
||||
|
||||
Per [ADR-002](v2/implementation/adr-002-driver-vs-virtual-dispatch.md), virtual (scripted) tags live in the same address space as driver tags and flow through the same rebuild. `EquipmentNodeWalker` (`src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/EquipmentNodeWalker.cs`) emits virtual-tag children alongside driver-tag children with `DriverAttributeInfo.Source = NodeSourceKind.Virtual`, and `DriverNodeManager` registers each variable's source in `_sourceByFullRef` so the dispatch branches correctly after rebuild. Virtual-tag script changes published from the Admin UI land through the same generation-publish path — the `VirtualTagEngine` recompiles its script bundle when its config row changes and `DriverNodeManager` re-registers any added/removed virtual variables through the standard diff path. Subscription restoration after rebuild runs through each source's `ISubscribable` — either the driver's or `VirtualTagSource` — without special-casing.
|
||||
|
||||
## Active subscriptions survive rebuild
|
||||
|
||||
Subscriptions for unchanged references stay live across rebuilds — their ref-count map is not disturbed. Clients monitoring a stable tag never see a data-change gap during a deploy, only clients monitoring a tag that was genuinely removed see the subscription drop.
|
||||
- `sp_PublishGeneration` / `sp_ComputeGenerationDiff` are stubs that `RAISERROR` in the v3 baseline
|
||||
migration; there is no generation-publish flow and no `DraftRevisionToken`.
|
||||
- `GenericDriverNodeManager` (`Core/OpcUa/`) is retained scaffolding with no production caller; the
|
||||
diff / teardown / subscription-replay "rebuild flow" described by earlier versions of this page does not
|
||||
exist in `src/`.
|
||||
|
||||
## Key source files
|
||||
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IRediscoverable.cs` — backend-change capability
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core/OpcUa/GenericDriverNodeManager.cs` — discovery orchestration
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriver.cs` — `ReinitializeAsync` contract
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` — publish-flow driver
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs` — cluster singleton invoked by the Admin UI's `AdminOperationsClient`
|
||||
- `docs/v2/config-db-schema.md` — `sp_PublishGeneration` + `sp_ComputeGenerationDiff`
|
||||
- `docs/v2/admin-ui.md` — DiffViewer + draft-revision-token flow
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs` — artifact composition + hash
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` — dispatch + ack set
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — apply + driver reconcile
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs` — node-side artifact parsing
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceComposer.cs`, `AddressSpacePlan.cs`,
|
||||
`AddressSpaceChangeClassifier.cs`, `AddressSpaceApplier.cs` — compose, diff, classify, apply
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs` — `IRediscoverable` consumer
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/ISurgicalAddressSpaceSink.cs` — in-place update seam
|
||||
|
||||
Reference in New Issue
Block a user