From c06769991b28b4af882acbac23875c28b78c8f7d Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 11 Nov 2026 18:19:14 -0500 Subject: [PATCH] docs(plans): #717 Warm redundancy design spec + implementation plan (draft for owner approval) Spec: per-shape contract (Unset/None/Warm), RedundancySupport + ServerUriArray graph, ServiceLevel bands, per-service standby behaviour, alarm suppression + ConditionRefresh (D5), strict first-delivery (D4), transition sequencing, ordered snapshot revisions, ScadaBridge impact, SDK 2.0.1 re-proof list, open questions. Plan: T0-T13 with wire tests, docker-dev live gate, rollout/rollback. Docs only. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk --- .../2026-11-11-warm-redundancy-717-design.md | 315 ++++++++++++++++++ .../2026-11-11-warm-redundancy-717-plan.md | 277 +++++++++++++++ 2 files changed, 592 insertions(+) create mode 100644 docs/plans/2026-11-11-warm-redundancy-717-design.md create mode 100644 docs/plans/2026-11-11-warm-redundancy-717-plan.md diff --git a/docs/plans/2026-11-11-warm-redundancy-717-design.md b/docs/plans/2026-11-11-warm-redundancy-717-design.md new file mode 100644 index 000000000..86112ca14 --- /dev/null +++ b/docs/plans/2026-11-11-warm-redundancy-717-design.md @@ -0,0 +1,315 @@ +# Warm non-transparent redundancy with a degraded standby (Gitea #717): design spec + +Status: **DRAFT for owner approval.** No product code changes until this spec and the companion plan +(`2026-11-11-warm-redundancy-717-plan.md`) are approved. Parent gate: #615 (G8 CTT, G9 production-client +cutover). These stay open and separate from everything here. + +## 1. Owner decisions this spec builds on (not re-opened) + +| # | Decision | Source | +|---|---|---| +| D1 | Advertise **Warm** for a 2-node pair, with an explicit degraded-standby contract; **None** for standalone. | #717 c7796 | +| D2 | Standby ServiceLevel in the **degraded band 2–199**; ordinary tag Reads/subscriptions on the standby **degrade**, while internal acquisition continues. Today's Secondary serves live Good data at 240, so it cannot just be relabelled. | #717 c7796, c7801, c9329 | +| D3 | Event subscriptions stay established on the standby; **new** alarm events are suppressed there; current retained conditions are refreshed on promotion. | #717 c7804 | +| D4 | **Strict first delivery.** After demotion, any Publish response delivered for the first time must not contain Good ordinary-tag values. That includes samples queued before demotion and batches built before it. Republish history and responses already in flight are preserved. | #717 c7849 | +| D5 | A client-initiated ConditionRefresh on the Warm standby **returns its current retained conditions**. Only new alarm events are suppressed. | #717 c11362 (2026-11-11) | +| D6 | The queue-only spike gate must not be adopted unchanged, because of late-overflow and filtered-retain effects. Product alarm suppression happens **before** SDK event-field projection. Per-tag read projection is the recommended data seam. Every ACL, direct-item and Server-object refresh path needs coverage. Role snapshots need an ordered revision. | #717 c7812, c7853, issue body | +| D7 | Existing mutation and election policies are preserved: the write/native-ack/HistoryUpdate Primary gates, the oldest-Up-member election, and `auto-down`. | #717 c7794 | + +## 2. Where the code is today + +- **Metadata.** No code sets `ServerRedundancy.RedundancySupport` (`i=3709`), so it reads the SDK default. `Server.ServerRedundancy` is the base `ServerRedundancyType` and has no `ServerUriArray` (`i=11314`) (`docs/Redundancy.md:9`). Peers are published only in `Server.ServerArray` (`i=2254`), by appending to `IServerInternal.ServerUris` after start (`OpcUaApplicationHost.cs:378-414`, called at `:241`). +- **Self URI.** It comes from `OpcUa:ApplicationUri`, which defaults to `urn:OtOpcUa` (`OpcUaApplicationHost.cs:35`). If `ApplicationConfigPath` is set, the loaded XML wins (`:602-608`). docker-dev sets neither `OpcUa__ApplicationUri` nor `PeerApplicationUris` on central-1/2 (`docker-dev/docker-compose.yml:289-481`), so both centrals advertise `urn:OtOpcUa`. The seeded `ClusterNode.ApplicationUri` uses `urn:OtOpcUa:` (`docker-dev/seed/seed-clusters.sql:13`). +- **ServiceLevel.** `ServiceLevelCalculator.Compute` returns 240 + 10 for the Primary (`ServiceLevelCalculator.cs:44-64`). The standby therefore sits in the *healthy* band. `OpcUaPublishActor.RecomputeServiceLevel` feeds it (`OpcUaPublishActor.cs:1576-1636`), and the result is published through the #719-fixed retain/replay path (`:1468-1488`, `SdkServiceLevelPublisher.cs:36-60`). +- **Acquisition is role-independent.** Every driver node spawns its drivers and subscribes whatever its role (#717 c7794). Today Primary gating covers only writes, native acks, the alerts emit, alarm-history drain, historization capture and HistoryUpdate (`PrimaryGatePolicy.cs:21,56,97,155`; `docs/Redundancy.md:554-600`). +- **Data variables.** Every ordinary-tag data variable (raw tag, UNS reference, virtual tag, Calculation tag) is an `EquipmentVariableState` created at exactly one site (`OtOpcUaNodeManager.cs:4060`). Values are pushed into it with `PublishValue` (`EquipmentVariableState.cs:28-35`). +- **Alarm events.** Every condition event goes through `ReportConditionEvent` (`OtOpcUaNodeManager.cs:1376`) and then `ReportEventWithoutBlocking` (`:1507`). The callers are engine/driver projection (`:1254`, `:1341`), method transitions (`:2842`) and restore re-report (`:4659`). Server-object items get per-item, ACL-filtered dispatch (#586). Folder and direct items get per-item delivery (#826/#855). Refresh paths: `ConditionRefresh` (`:6586`), `RefreshDirectItems` (`:6974`) and the #854 cut. Model-change and audit events use `ReportToServerObject` (`:6060`), not this path. +- **Role snapshots are unordered.** `RedundancyStateActor` is a cluster singleton. It publishes `RedundancyStateChanged(Nodes, CorrelationId)` on DPS, debounced at 250 ms and heartbeated every 10 s (`RedundancyStateActor.cs:25,33,105-113`). `AsOfUtc` is the *publisher's* wall clock (`:184`) and `CorrelationId` is random (`:110`). Subscribers replace their cached snapshot unconditionally (`OpcUaPublishActor.cs:1494-1498`, `DriverHostActor.cs:2555-2562`). After a singleton handover, a late heartbeat from the old incarnation can overwrite the new incarnation's snapshot. Nothing establishes the order, and there is no lease. +- **Config.** `ServerCluster.RedundancyMode` (None/Warm/Hot) exists in ConfigDb. It is CHECK-coupled to NodeCount (`OtOpcUaConfigDbContext.cs:109-111`; `DraftValidator.cs:958-962`) and travels in the artifact's `Clusters` array (`ConfigComposer.cs:67`), but no runtime code reads it. + +## 3. Contract per deployment shape + +| Shape | `OpcUa:Redundancy:Mode` | `RedundancySupport` (i=3709) | `ServerRedundancy` type | `ServerUriArray` (i=11314) | `ServerArray` (i=2254) | Standby behaviour | +|---|---|---|---|---|---|---| +| Today / not opted in | `Unset` (default) | SDK default (unchanged) | `ServerRedundancyType` (2034) | absent | self + peers (unchanged) | none; Secondary serves Good at 240 (unchanged) | +| Standalone | `None` | `None` (0) | `ServerRedundancyType` (2034) | absent | self only | never; always serves | +| Pair | `Warm` | `Warm` (2) | `NonTransparentRedundancyType` (2039) | `[effectiveSelfUri, peer…]`, read-only String[], self first (decision #155, `docs/v2/plan.md:930`) | unchanged | §4 | + +- The metadata is **startup-static**. The graph is built before node-manager registration through + `StandardServer.CreateMainNodeManagerFactory` + `ConfigurationNodeManager.AddBehaviourToPredefinedNode`. That + route is spike-proven on SDK 1.5.378.106 (#717 c7791) and becomes `AddBehaviourToPredefinedNodeAsync` on 2.0.1. + The passive predefined `i=11314` the SDK visits later must resolve to the same constructed instance, or the wire + value is null (c7791). Changing mode or peers requires a restart. +- `effectiveSelfUri` is the **effective** `ApplicationConfiguration.ApplicationUri` after any XML load, never the + options default (c7787). +- `Hot` is **not advertisable**: continuous acquisition alone does not meet Part 4 §6.6.2.4 Hot (c7794). + `Mode=Hot` fails host start. +- Startup validation (`ValidateOnStart`) runs for `Warm` and fails host start when any of these holds: there is no + peer; a peer URI equals the self URI; a URI is duplicated; the `driver` role is missing. +- Deploy-time cross-check: if the artifact's `ServerCluster.RedundancyMode` for this node's cluster disagrees + with the local mode (`NodeCount=1/None` against local `Warm`, or the reverse), the node logs an Error and records + the metric `otopcua.redundancy.mode_mismatch`. The node keeps running in its local mode. Local config is the + authority, because it is the only value available before SDK start. + +## 4. Warm role contract + +### 4.1 Role verdict (who is Active) + +A Warm node is **Active** iff all of the following hold. Otherwise it is **Standby**. + +1. The local `SelfMember` is Up and `EndpointListening` is true (#762). +2. The node's **own** `Cluster.State` names it the oldest Up `driver` member. This is the same function the + singleton uses (`RedundancyStateActor.SelectOldestUpMemberOfRole`, `:172`), so the verdict cannot drift from + singleton placement (D7). +3. The latest **accepted** snapshot (§4.2) names it `Primary`. + +Any disagreement between (2) and (3) resolves to Standby. A Warning is logged if the disagreement lasts more than +two heartbeats (20 s). When the role is unknown — before the first accepted snapshot, or when the snapshot omits +this node — the node is **Standby**, whatever the member count. This is stricter than `PrimaryGatePolicy`'s +lone-node allow (`PrimaryGatePolicy.cs:21`): a lone node goes Active within one debounce (~250 ms) of its own +singleton's first publish. That gate is **not** changed for writes/acks (D7). The Warm verdict is a separate output. + +Accepted trade: an `auto-down` partition can give two Active nodes, as it already does for the data-plane gate +(`docs/Redundancy.md` §Split-brain). For reads this is benign, because both nodes serve the same Good data. + +### 4.2 Ordered snapshots (the revision problem) + +`RedundancyStateChanged` gains an additive `Revision(PublisherUpNumber, IncarnationTick, Sequence)`: + +- `PublisherUpNumber` is the `Member.UpNumber` of the node that hosts the singleton. The singleton always lives on + the oldest member, and every surviving member has a larger up-number than a departed oldest one. The value is + therefore monotonic across handovers within one cluster incarnation. +- `IncarnationTick` is `Environment.TickCount64` at singleton `PreStart`. It is monotonic for a singleton that + restarts in the same process. +- `Sequence` increments on every publish, including heartbeats. + +Subscribers (`OpcUaPublishActor`, `DriverHostActor`) accept a snapshot only if its revision is lexicographically +greater than the last accepted one. Anything else is dropped and counted in +`otopcua.redundancy.snapshot_rejected{reason}`. A subscriber cannot outlive its cluster incarnation: a removed node +must restart its `ActorSystem` to rejoin. The plan pins this assumption with a test. A snapshot **without** a +revision (from a pre-#717 publisher during a rolling upgrade) is accepted for the existing gates but can never make +a Warm node Active. Warm therefore requires both pair nodes to be upgraded before it is turned on. + +There is no time lease beyond this. Liveness comes from (2), which reads live gossip, and the existing 30 s +staleness input to ServiceLevel. Demoting an Active node on snapshot staleness alone was rejected: the singleton +lives on the Active node itself, so staleness there means a local fault, and demoting would turn it into a pair-wide +data outage. + +### 4.3 ServiceLevel bands (Part 4 §6.6.2.4.2) + +| Node state (Warm) | Byte | Band | +|---|---|---| +| Down / Detached / endpoint failed | 0 | Maintenance (unchanged) | +| Boot, before first computed level | **1** (set in the startup graph; replaces SDK default 255) | NoData | +| Standby, degraded (stale or DB unreachable) | **50** | Degraded | +| Standby, healthy | **90** | Degraded | +| Active, critically degraded | 100 | Degraded (unchanged) | +| Active, stale | 200 | Healthy (unchanged) | +| Active, healthy | 250 | Healthy (unchanged: 240 + 10) | + +Invariant: **every Standby level is below every level an Active node serving data can publish (≥ 100)**. A +"highest ServiceLevel" client therefore never prefers a standby over a data-serving Active node. `Unset` and `None` +keep `ServiceLevelCalculator.Compute` byte-for-byte. + +### 4.4 Behaviour on the Standby, per service + +"Ordinary tag" means the `Value` attribute of an `EquipmentVariableState`. The category is captured at item +creation from the node instance and the `AttributeId`, never from namespace or ClientHandle (c7853). + +| Service | Standby behaviour | Status the client sees | +|---|---|---| +| Read (ordinary tag `Value`) | Projected. The stored value is kept internally and continues to update from acquisition. | `BadNoCommunication`, `Value = null`, `ServerTimestamp = now`. Operation result Good. | +| Read (other attributes, condition children, ns=0 incl. ServiceLevel/ServerStatus/ServerRedundancy) | Unchanged | Good / as today | +| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification `BadNoCommunication`/null | +| Existing data items at demotion | One `BadNoCommunication`/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. **D4:** any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to `BadNoCommunication`/null at Publish/Republish (§4.6). | `BadNoCommunication` | +| Write | **Unchanged (D7):** the existing gate rejects off-Primary through the #5 self-correction (synchronous Good, then revert + audit). Under Warm the revert is invisible because the value is projected Bad anyway. See open question Q1. | as today | +| HistoryRead | Unchanged. Served from the shared backend, which is not live acquisition. | as today | +| HistoryUpdate | Unchanged | `BadInvalidState` off-Primary (#674) | +| Browse / TranslateBrowsePaths / RegisterNodes | Unchanged | as today | +| Call: Part 9 methods | Unchanged (D7): native ack `BadInvalidState` off-Primary; scripted ack/shelve accepted. The resulting condition event counts as a new alarm event and is suppressed (§4.5). See Q4. | as today | +| Event subscriptions (Server object, folders, direct) | Stay established, in Reporting mode, with the ACL unchanged | — | +| Model-change / audit events | **Not suppressed.** They are not alarm events, and browse consistency needs them. | — | +| Session / secure channel / ServerStatus.State | Unchanged (`Running`) | — | + +### 4.5 Alarms and events on the Standby + +- **Suppression point.** `ReportConditionEvent` is split into *stamp* and *deliver*. Stamp sets EventId, Time and + ReceiveTime, and runs `ClearChangeMasks`. Deliver builds the snapshot and calls `ReportEventWithoutBlocking`. On + the Standby only *stamp* runs. Condition node state therefore stays current, and a later refresh carries a valid, + ackable EventId. Because suppression sits before snapshot construction and before every SDK field projection, it + covers the Server-object, folder and direct paths, ACL on or off, by construction. Overflow, filtered retain and + the #838 deferral queue never see a suppressed event. The `RingEventQueueFactory` queue is **not** used as the + gate (D6). +- **Boundary.** The decision is made at report time. An event whose report ran while Active is delivered even if + delivery finishes after demotion: deferred (#838) or already in an item queue. This matches the spike's + observed admission boundary (c7805). Republish of already-built event messages is unchanged. +- **Client ConditionRefresh / ConditionRefresh2 on the Standby (D5)** run the existing paths unchanged: Server + object (#854 cut), `RefreshDirectItems` (#855), with ACL filtering as today. The replay is RefreshStart, the + current retained conditions, then RefreshEnd. +- **Promotion refresh.** On Standby → Active, the server runs a refresh on **every** live event item: Server-object + items, direct items and folder items. The two refresh entry points are reused, so ACL decisions and the #854/#855 + cut apply. Clients see RefreshStart, retained conditions, RefreshEnd, then new events. This is spike-proven on + 1.5 (c7805). A condition that went active **and** stopped being retained entirely while the node was Standby is + not replayed. That is standard refresh semantics, and the transition is not historized twice, because alarm + history is Primary-gated (`docs/Redundancy.md:572-573`). + +### 4.6 Strict first delivery (D4) + +The item-level projection (§4.4) stops new Good samples. It cannot reach samples already queued in an item, nor +`NotificationMessage`s the subscription has prebuilt or cached (c7846, c7850). For those, the design uses the +mechanism that passed the provenance spike (c7851, 66 assertions) and the delivery-history spike (c7852, +53 assertions): + +1. **Provenance.** Ordinary-tag data items are created through a custom sampled item. Its per-value `Publish` + stamps each newly created `MonitoredItemNotification` with an immutable `(NodeId, AttributeId, Category)`. The + stamp survives handle modification, item deletion and reuse. +2. **Delivery history.** `OtOpcUaSdkServer` overrides `PublishAsync` and `RepublishAsync`, calls base first, and + records **one first-delivery choice per cached message object**. The association is weak and keyed by object; + cached SDK objects are never mutated. + - If the node is Active at first delivery, the original message is the choice. + - If the node is Standby, a detached clone is the choice, with every `OrdinaryTag` notification replaced by + `BadNoCommunication`/null. + - A Republish returns the recorded choice. First delivery *through* Republish is decided the same way. + - A short lock orders the choice against the role flip. No lock spans an await. +3. **Out of scope of the rewrite**, by D4: messages already in flight before the demotion step, and Republish of + messages first delivered while Active. Unknown-origin notifications are not rewritten; there is no implicit + policy (c7853). + +### 4.7 Transition sequencing + +One actor, `OpcUaPublishActor`, owns the verdict, the ServiceLevel publish and a new `IStandbyProjection` sink +(in `Commons.OpcUa`, bound late like `IServiceLevelPublisher`). Each transition runs as one serialized handler, in +this order. + +**Promotion (Standby → Active)** + +1. Open the read projection. +2. For each live ordinary-tag item, queue the current stored value, ignoring filters. The precedent is the queue + pattern at `OtOpcUaNodeManager.cs:7303`. +3. Switch the delivery-history choice to Active. +4. Un-suppress alarm delivery. +5. Run the promotion refresh (§4.5). +6. Publish the Active ServiceLevel. + +**Demotion (Active → Standby)** + +1. Publish the Standby ServiceLevel. +2. Switch the delivery-history choice to Standby. This is the D4 boundary. +3. Suppress alarm delivery. +4. Close the read projection. +5. Queue `BadNoCommunication` to each live ordinary-tag item. + +Invariant: **an Active-band ServiceLevel is never visible while the projection is closed.** If the ServiceLevel +publish fails (#719 retry path), a promotion stops before step 6 and retries. A demotion continues anyway: failing +safe means degrading the data. + +### 4.8 Standalone (`None`) and `Unset` + +`None` never projects, never suppresses and never refreshes. It advertises `RedundancySupport=None` with no +`ServerUriArray`, and uses today's ServiceLevel. `Unset` is byte-for-byte today's server, including the documented +unmet A4/A5b gates. + +## 5. ScadaBridge impact (cross-repo, blocking for any Warm flip) + +ScadaBridge consumes this server as an OPC UA client, through `RealOpcUaClient` and `DataConnectionActor`. + +- **Failover is connection-driven, not ServiceLevel-driven.** ScadaBridge swaps primary to backup only after + `FailoverRetryCount` consecutive *unstable disconnects* + (`../ScadaBridge/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs:655-690`; + `RealOpcUaClient.cs` sits in that project's `Adapters/`). + `BadNoCommunication` is not a connection-level status (`:1974-1983`), so it maps to plain Bad quality, and since + ScadaBridge#112 a Bad-with-null update keeps the last-known value. Nothing in ScadaBridge reads `ServiceLevel`, + `RedundancySupport` or `ServerUriArray`. +- **Consequence.** A ScadaBridge connection that lands on the Standby will sit on Bad quality indefinitely on a + healthy session, and never fail over. Today the same connection gets Good data at 240. Warm is therefore an + **outage for ScadaBridge** unless ScadaBridge becomes ServiceLevel-aware: monitor `i=2267` and treat a Degraded + band (< 200) sustained for N s, while the other endpoint reports a higher level, as a failover trigger. That + change must ship and be deployed first. The plan files it as a ScadaBridge issue (`cross-repo`, blocking). +- **Alarms are compatible.** ScadaBridge already handles a RefreshStart/RefreshEnd replay as a snapshot + (`RealOpcUaClient.cs:72`) and triggers its own ConditionRefresh on (re)subscribe (`:1179-1193`). Under D5 that + refresh returns retained conditions on the Standby too. The promotion refresh arrives as an unsolicited + Start/End pair on existing items; this needs a ScadaBridge-side test. +- **Status mapping is compatible.** ScadaBridge#117 maps by severity bits, so `BadNoCommunication` is Bad. + +## 6. Configuration surface + +| Key | Default | Notes | +|---|---|---| +| `OpcUa:Redundancy:Mode` | `Unset` | `Unset` / `None` / `Warm`. `Hot` and unknown values fail host start. Startup-static. Flip **both** pair nodes in one window, after both run a revision-capable build (§4.2). | +| `OpcUa:PeerApplicationUris` | `[]` | Existing key, now also the `ServerUriArray` peer source. Required (≥ 1) for `Warm`, must be empty for `None`. | +| `OpcUa:ApplicationUri` | `urn:OtOpcUa` | Must be unique within the pair under `Warm` (validated against the effective config). | + +There are no ConfigDb schema changes. `ServerCluster.RedundancyMode` is only cross-checked (§3). + +## 7. Observability + +- `otopcua.redundancy.warm_role`: gauge. 1 = Active, 0 = Standby, −1 = not Warm. +- `otopcua.redundancy.warm_transition{direction,reason}`: counter. Reasons: `membership`, `snapshot`, + `endpoint`, `boot`. +- `otopcua.redundancy.snapshot_rejected{reason=older_revision|unrevisioned}`: counter. +- `otopcua.redundancy.mode_mismatch`: counter (§3). +- `otopcua.opcua.standby.projected_notifications`: counter. Counts D4 rewrites at Publish/Republish. +- `otopcua.opcua.standby.suppressed_alarm_events`: counter. +- Logs: + - Information per transition, naming the revision and the verdict inputs. + - Warning when membership and snapshot disagree for more than 20 s. + - A boot banner line with mode, effective self URI and peers. +- `otopcua-cli redundancy` already reads `i=3709`, `i=11314` and `i=2267`. + +## 8. Explicitly out of scope + +- Hot (and Transparent) redundancy. +- Dynamic peer membership. Peers are static config. +- Changing the election (oldest Up driver member), `auto-down`, or the write / native-ack / HistoryUpdate / + alarm-drain / historization gates. +- Stopping acquisition on the Standby. +- TransferSubscriptions across the pair. Non-transparent means the client re-creates its subscriptions. +- CTT (G8) and production-client cutover (G9). Hermetic and live-rig passes are not acceptance. +- Durable subscription restore. +- Sequence-number wrap. +- Encoded-byte fidelity of cloned payloads beyond the plan's bounded matrix (c7890). + +## 9. Open questions for the owner (short) + +1. **Standby writes.** Keep today's optimistic Good + revert (D7), or refuse synchronously with `BadInvalidState` + under Warm, as HistoryUpdate and native ack already do? *Recommend synchronous refusal under Warm only.* +2. **Crash-failover gap.** A SIGKILLed Primary stays an unreachable member until auto-down, ~25–30 s. Under (2)/(3) + the survivor stays Standby for that whole window, with no Good data anywhere, where today the Secondary keeps + serving. Should the survivor be allowed to promote when the Primary is *unreachable* (the #672 pattern, default + on), accepting dual-Active during a real partition? *Recommend yes, behind + `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable`.* +3. **Standby status code.** `BadNoCommunication` is spike-proven and is plain Bad in ScadaBridge, but it is the + same code as real device comms loss; only ServiceLevel tells them apart. Accept it, or use `BadOutOfService`? +4. **Scripted Part 9 methods on the Standby.** They are accepted (D7) but their event is suppressed (D3), so the + caller sees Good and no event until promotion or refresh. Accept, or refuse with `BadInvalidState` under Warm? +5. **ConfigDb `Hot`.** Narrow the CHECK to `None/Warm` with a migration, or leave it and rely on the §3 mismatch + Error? +6. **Bands.** Are Standby 90/50 and boot 1 acceptable (§4.3)? + +## 10. What must be re-proven on SDK 2.0.1 + +The spikes ran on **1.5.378.106**. The repo now pins **2.0.1** (`Directory.Packages.props:129-135`). The 2.0.1 XML +docs confirm that these members still exist: + +- `StandardServer.CreateMainNodeManagerFactory` +- `ConfigurationNodeManager.AddBehaviourToPredefinedNodeAsync`, now **async** +- `NonTransparentRedundancyState.ServerUriArray`, now in `Opc.Ua.Core.Types` +- `StandardServer.PublishAsync` / `RepublishAsync`, whose signatures now carry `RequestLifetime` +- `CreateSubscriptionManager` +- `BaseVariableState.OnSimpleReadValue`, plus new `OnSimpleReadValueAsync` / `OnReadValueAsync` +- `ReadValueAttribute` +- `IServerInternal.ConditionRefresh2` +- a new `MonitoredItemFactory` / `MonitoredItemCreateDecision` creation hook and an + `IMonitoredItemTransferCoordinator` + +Existence is not behaviour. Plan T0 re-proves, on 2.0.1 over the wire: + +- (a) the metadata graph, including the passive-`i=11314` resolution; +- (b) that **every** value read path reaches the per-tag projection: Read, initial sample, change fan-out, + ResendData, size-1 queue and AggregateFilter. In particular, the new async read callbacks must not bypass a sync + `OnSimpleReadValue`; if they do, the hook moves to a `ReadValueAttribute` override on `EquipmentVariableState`; +- (c) provenance through the 2.0.1 creation hook, which replaces the 1.5 manager-subclass route, including + transfer/restore; +- (d) the Publish/Republish choice with the new signatures; +- (e) suppression and promotion refresh through the product's #854/#855 per-item paths, which did not exist when + the 1.5 event spike ran. diff --git a/docs/plans/2026-11-11-warm-redundancy-717-plan.md b/docs/plans/2026-11-11-warm-redundancy-717-plan.md new file mode 100644 index 000000000..e7c5d510c --- /dev/null +++ b/docs/plans/2026-11-11-warm-redundancy-717-plan.md @@ -0,0 +1,277 @@ +# Warm non-transparent redundancy (Gitea #717): implementation plan + +Status: **DRAFT for owner approval**, together with `2026-11-11-warm-redundancy-717-design.md` (referred to below +as "the spec", with § references). No task starts until both documents and the §9 open questions are settled. +Every task ships **dark**: `OpcUa:Redundancy:Mode` defaults to `Unset`, so merging any task changes no deployed +behaviour. + +Conventions: + +- **Wire tests** use a real OPC Foundation client session against a loopback `OtOpcUaSdkServer`, in + `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests` (the home of `DualEndpointTests`, + `ConditionRefreshDeferralWireTests` and `FolderConditionRefreshWireTests`). +- **Actor tests** go in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` and + `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests`. +- Every gate gets an **allow** leg as well as a deny leg (the vacuous-test rule), and a mutation check: revert the + production line and confirm the test fails. + +## T0: SDK 2.0.1 re-proof spikes (throwaway, separately approved) + +The 1.5.378.106 spikes do not carry over (spec §10). Write throwaway loopback fixtures against **2.0.1**, outside +`src/`. No product edits. Each fixture is rerun independently and its evidence is recorded in +`docs/plans/evidence/2026-11-xx-717-sdk2/`. + +- **a. Metadata graph.** Build it via `CreateMainNodeManagerFactory` + `AddBehaviourToPredefinedNodeAsync`. Check + Read/Browse/Translate of `i=2296` (HasTypeDefinition 2039), `i=3709`, and `i=11314` for both Warm and None. The + passive `i=11314` visit must resolve to the constructed instance. +- **b. Read projection coverage.** Cover Read, the initial sample on create, the change fan-out after + `ClearChangeMasks`, ResendData, queue size 1 vs. N, an AggregateFilter item, and Sampling → Reporting. Decide + between `OnSimpleReadValue` and a `ReadValueAttribute` override, and record which async callback paths 2.0.1 + takes. +- **c. Provenance.** Stamp items through the 2.0.1 `MonitoredItemFactory` / `MonitoredItemCreateDecision`. Exercise + ModifyMonitoredItems handle change, delete/reuse, TransferSubscriptions (`IMonitoredItemTransferCoordinator`), + and a restore. +- **d. Publish/Republish.** Override `PublishAsync`/`RepublishAsync` with the 2.0.1 signatures (`RequestLifetime`). + Cover the first-delivery choice, first delivery through Republish, an ack followed by `BadMessageNotAvailable`, + and a cancelled or failed response. +- **e. Events.** Exercise the stamp/deliver split under the #855 per-item folder path and the #854 Server-object + cut. Run the promotion refresh on all three item kinds, and a client ConditionRefresh/2 while Standby. + +**Exit:** a short results note per fixture, and an owner go/no-go. A "no" on (c) or (d) re-opens D4's mechanism. +D4 itself stays an owner decision. + +## T1: Configuration, validation, effective self URI + +**Files:** +- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`: options at `:29-121`. Add + `OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable` if Q2 = yes). +- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`. +- A post-load check in `StartAsync` (`:218-250`) against the effective `ApplicationConfiguration.ApplicationUri`. + +**Rules (spec §3):** +- `Hot` and unknown values fail host start. +- `Warm` requires at least one peer, all URIs distinct, self not listed as a peer, and the `driver` role. +- `None` requires no peers. + +**Tests:** +- A validator matrix. +- An XML `ApplicationConfigPath` whose URI differs from the options value, proving the effective URI is used. +- A boot-banner log assertion. + +## T2: Standard redundancy metadata graph + +**Files:** +- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs`: next to `CreateMasterNodeManager` (`:341`), add a + `CreateMainNodeManagerFactory` override that installs the T0a graph per mode. +- Under `Warm`, the initial `Server.ServiceLevel` is set to **1** in the same pass (spec §4.3). `Unset` must not + touch the graph. + +**Tests (wire):** +- Per mode, check the `i=3709` value, the `i=2296` type definition, the presence or absence of `i=11314`, its + value order `[effectiveSelf, peers…]`, and that it is read-only (a Write returns `BadNotWritable`). +- `ServerArray` (`i=2254`) is unchanged; extend `DualEndpointTests`. +- A Warm server's ServiceLevel reads 1 before the first publish. +- Security profiles and startup are unaffected under `Unset` (golden read of today's values). + +## T3: Ordered redundancy snapshots (spec §4.2) + +**Files:** +- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/RedundancyStateChanged.cs` (`:9-11`): add an optional + `SnapshotRevision? Revision` (additive). +- `RedundancyStateActor.cs`: + - stamp the revision in `PublishIfDirty` (`:105-113`); + - set `IncarnationTick` in `PreStart`; + - read the host's `UpNumber` from `Cluster.SelfMember`. +- Subscribers: accept only a newer revision in `OpcUaPublishActor.HandleRedundancyStateChanged` (`:1494-1498`) and + `DriverHostActor.OnRedundancyStateChanged` (`:2555`). +- An unrevisioned snapshot keeps today's handling for the existing gates. + +**Tests:** +- Ordering unit tests: an older singleton incarnation's heartbeat after a newer snapshot is dropped; a same-process + singleton restart advances. +- A **real 2-node cluster** handover test (pattern: `RedundancyPrimaryElectionTests`). It holds back the old + incarnation's last publish through the `_broadcastOverride` seam (`:111`) and asserts the subscriber keeps the + new Primary. +- Serialization round-trip of the extended message, plus a mixed-version decode with the field absent. +- The pinned assumption: a removed node cannot rejoin without a new `ActorSystem`. + +## T4: Warm role verdict and ServiceLevel bands (spec §4.1, §4.3) + +**Files:** +- New `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/WarmRoleAuthority.cs`: a pure function of membership view, + accepted snapshot, `EndpointListening` and `SelfMember` status. +- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs`: add `ComputeWarm(inputs, isActive)`. + `Compute` itself is unchanged. +- `OpcUaPublishActor.RecomputeServiceLevel` (`:1576-1636`) branches on mode. It subscribes to `MemberUp/Removed` + (and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots. + +**Tests:** +- A truth table covering every row of §4.3 and the invariant *max(Standby) < min(Active serving)*. +- Membership/snapshot disagreement gives Standby, plus the 20 s Warning. +- Boot with an unknown role gives Standby; a lone node goes Active after its own snapshot. +- `None`/`Unset` produce byte-identical output to today's `ServiceLevelCalculatorTests`. +- DB-less mode (#721 timer) is covered. + +## T5: Standby projection seam and transition sequencer (spec §4.7) + +**Files:** +- New `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IStandbyProjection.cs`, with a deferred/late-bound adapter that + mirrors `DeferredServiceLevelPublisher` and retains the latest state across binding (the #719 lesson). +- Bound in `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs` next to the ServiceLevel binding. +- `OpcUaPublishActor` runs promotion and demotion as one serialized handler, in the §4.7 step order. + +**Tests (actor, with a recording fake):** +- The exact step order for both directions. +- A ServiceLevel publish failure halts a promotion before the Active level and retries; a demotion continues. +- A burst of alternating verdicts converges on the last one. +- Binding after the first transition replays the current state. + +## T6: Per-tag read projection (spec §4.4) + +**Files:** +- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs`: the hook decided in T0b, reading a + node-manager-wide volatile projection flag. Stored `Value` is never modified. +- `OtOpcUaNodeManager` implements `IStandbyProjection.SetDataProjection`. + +**Tests (wire):** +- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadNoCommunication`/null. +- A Standby Read of a condition child, `i=2267` or `i=3709` stays Good. +- A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected. +- Acquisition continues: flip to Active and the latest value is read without any new driver publish. +- ACL on: a denied node is still `BadUserAccessDenied` and the projection never leaks existence. + +## T7: Provenance and first-delivery choice (spec §4.6, D4) + +**Files:** +- `OtOpcUaSdkServer.cs`: install the custom sampled item (T0c route) for ordinary-tag `Value` items only. +- `OtOpcUaSdkServer.cs`: add the `PublishAsync`/`RepublishAsync` overrides and a new `FirstDeliveryLedger`, a + `ConditionalWeakTable` keyed by the SDK message object. +- Wire into `OtOpcUaNodeManager.CreateMonitoredItems` (`:5295`) / `OnMonitoredItemCreated` (`:5442`) without + disturbing the ACL filtering or `_liveItems`. + +**Tests (wire):** port the spikes' assertion matrix to product code. +- Queued Good samples, then demotion, then the first Publish carries `BadNoCommunication`. +- Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same + projected payload. +- Good delivered while Active, then demotion, then Republish returns the original Good. +- A first delivery through Republish. +- A mixed subscription (ordinary tag + `i=2267` + condition-child item): only ordinary-tag items are rewritten. +- Handle modified, deleted or reused between queue and publish. +- The cached SDK object is unchanged (sacrificial-clone check). +- A bounded clone matrix (arrays, ExtensionObject, DiagnosticInfo, per c7890). +- Concurrent Publish and role flip, by a stress loop with a deterministic barrier, not sleeps. + +## T8: Promotion and demotion of live data items + +**Files:** +- `OtOpcUaNodeManager`: `IStandbyProjection.QueueCurrentToLiveItems(bool projected)` iterates `_liveItems`. It + reuses the QueueValue-with-`ignoreFilters` pattern (`:7303`), **not** detach or `MonitoringMode.Disabled`; the + disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped. + +**Tests (wire):** +- Demotion delivers exactly one `BadNoCommunication` per item, then silence while values keep changing internally. +- Promotion delivers the current Good value without a driver publish. +- An ACL-stood-down item receives nothing either way. +- A DeadbandFilter item still receives the promotion value. + +## T9: Alarm event suppression and promotion refresh (spec §4.5, D3, D5) + +**Files (`OtOpcUaNodeManager`):** +- Split `ReportConditionEvent` (`:1376`) into stamp and deliver; deliver is gated by the projection flag. +- Audit all callers (`:1254`, `:1341`, `:2842`, `:4659`) and `WireMethodTransitionReporting`. +- `IStandbyProjection.RefreshAllEventItems()` runs the existing `ConditionRefresh` (`:6586`) Server-object path and + `RefreshDirectItems` (`:6974`) over every live event item, from `_allEventsItems` and `_eventItems`. +- `ReportToServerObject` (`:6060`) is untouched. + +**Tests (wire, ACL off and on):** +- A new alarm on the Standby reaches no Server-object, folder or direct item. +- Model-change and audit events are still delivered on the Standby. +- A client ConditionRefresh **and** ConditionRefresh2 on the Standby return Start, retained, End (D5). +- Promotion gives Start, retained, End on all three item kinds, followed by new events. +- An ACL-denied condition is absent from the promotion refresh. +- A condition raised and cleared (Retain=false) while Standby is not replayed. +- An event reported while Active but deferred (#838, via `NotifierStall`) is still delivered after demotion. +- A Standby-stamped EventId is accepted by Acknowledge after promotion. +- The `RingEventQueueFactory` path is untouched (`RingEventMonitoredItemQueueParityTests` stay green). +- Re-run `EventQueueOverflowAclWireTests` and `ConditionMethodEventTests`. + +## T10: Deploy-time mode cross-check and observability (spec §3, §7) + +**Files:** +- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`: read this node's cluster + `RedundancyMode` from the `Clusters` array. +- `OpcUaPublishActor` compares it with the local mode at apply. +- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`: add the six instruments. + +**Tests:** +- A mismatch logs an Error and records the metric. +- Each counter fires on its path. Use a `MeterListener` filtered by **instrument instance**, never by meter name + alone (the #503 lesson). + +## T11: ScadaBridge prerequisite (cross-repo) + +- File a ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes + ServiceLevel-aware: monitor `i=2267`, and fail over when it is below 200 for a configurable window while the + other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items + (spec §5). +- File the lmxopcua companion as `blocked` + `cross-repo`. +- **The Warm flip in T13/rollout is gated on that issue being deployed.** + +## T12: Documentation and index + +- `docs/Redundancy.md`: rewrite the `:9` block, add a Warm contract section, update the tiers table. Remove the + generic "standby / no device contact / exactly-once" wording that contradicts acquisition and at-least-once + (c7794). +- `docs/OpcUaServer.md:70`, `docs/Configuration.md` (the new keys), and `docs/security.md` (ns=0 stays ungated; + projection order vs. ACL). +- `.claude/rules/cluster-topology.md` §Redundancy. +- `docs/plans/phase-6-3-redundancy-interop-plan.md` A2/A4/A5b: under Warm, A2 expects 90 and A4/A5b become + testable. Add `docs/v2/redundancy-interop-playbook.md`. +- Umbrella: update the OtOpcUa row and `sister-project-history/otopcua.md` in `../scadaproj` on **its `main`**, + pushed (root `CLAUDE.md` procedure). + +## T13: Live gate on docker-dev (central pair, `:4840` / `:4841`) + +**Overlay:** new `docker-dev/docker-compose.warm.yml`, central-1/2 only. It sets: + +- `OpcUa__ApplicationUri=urn:OtOpcUa:central-1|2` (matching `seed-clusters.sql`); +- `OpcUa__PeerApplicationUris__0=`; +- `OpcUa__Redundancy__Mode=Warm`. + +Today both centrals advertise the default `urn:OtOpcUa` (spec §2). Login is disabled on the rig, so drive +everything yourself. Use `otopcua-cli` from `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` against both ports. + +| Leg | Procedure | Pass | +|---|---|---| +| L1 metadata | `redundancy` on :4840 and :4841 | `RedundancySupport=Warm`. ServerUriArray is self first and differs per node. Same URIs in both. | +| L2 bands | `read -n i=2267` on both | Primary 250, Standby 90 | +| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadNoCommunication`/null; the Primary shows Good at the same moment | +| L4 standby alarms | `alarms` on both. Trip a scripted alarm via its source tag. | Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5). | +| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadNoCommunication`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). | +| L6 crash | `docker kill central-` | The survivor goes Active within the gap Q2 selects. Record the measured gap. | +| L7 rejoin | Restart the killed node | It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90. | +| L8 standalone | site-b-2 (or a one-node overlay) with `Mode=None` | `RedundancySupport=None`, no `i=11314`, data Good | +| L9 Unset control | Another node left on `Unset` | Byte-identical to pre-change: SL 240/250, SDK-default `i=3709` | +| L10 writes/history | Write and HistoryRead on the Standby | Behaviour per Q1. HistoryRead is Good. | + +Record evidence in `docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md`. Do **not** mark G8/G9 (#615); they +need CTT and a production client (Ignition 8.3 / ScadaBridge after T11). + +## Rollout + +1. Merge T1–T12 dark (`Unset` default). The full CI floor applies. +2. Deploy the ScadaBridge T11 change everywhere ScadaBridge reads OtOpcUa. +3. Upgrade **both** nodes of a pair to a revision-capable build (T3) before flipping either one. +4. Flip both nodes of one docker-dev pair, then run T13. After that, flip one non-production site pair. +5. Production is a separate, owner-approved change per pair. Flip the **Secondary first**, then the Primary, in + one window. + +A mixed window (one `Warm`, one `Unset`) is tolerable: the Unset node serves Good at 240/250. But do not leave it +that way, because `ServerUriArray` then exists on only one node. + +## Rollback + +Set `OpcUa:Redundancy:Mode=Unset` on both nodes and restart them. The setting is startup-static, so there is +nothing persistent to undo: no schema changes and no artifact changes. The `Revision` field stays and is harmless, +because unrevisioned handling is preserved. Rolling back ScadaBridge is independent; its ServiceLevel-aware +failover also works against `Unset` servers.