Merge #717 Warm redundancy design spec + plan (DRAFT, awaiting owner approval)

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk

docs/plans/2026-11-11-warm-redundancy-717-design.md

@@ -0,0 +1,315 @@
+# Warm non-transparent redundancy with a degraded standby (Gitea #717): design spec
+
+Status: **DRAFT for owner approval.** No product code changes until this spec and the companion plan
+(`2026-11-11-warm-redundancy-717-plan.md`) are approved. Parent gate: #615 (G8 CTT, G9 production-client
+cutover). These stay open and separate from everything here.
+
+## 1. Owner decisions this spec builds on (not re-opened)
+
+| # | Decision | Source |
+|---|---|---|
+| D1 | Advertise **Warm** for a 2-node pair, with an explicit degraded-standby contract; **None** for standalone. | #717 c7796 |
+| D2 | Standby ServiceLevel in the **degraded band 2–199**; ordinary tag Reads/subscriptions on the standby **degrade**, while internal acquisition continues. Today's Secondary serves live Good data at 240, so it cannot just be relabelled. | #717 c7796, c7801, c9329 |
+| D3 | Event subscriptions stay established on the standby; **new** alarm events are suppressed there; current retained conditions are refreshed on promotion. | #717 c7804 |
+| D4 | **Strict first delivery.** After demotion, any Publish response delivered for the first time must not contain Good ordinary-tag values. That includes samples queued before demotion and batches built before it. Republish history and responses already in flight are preserved. | #717 c7849 |
+| D5 | A client-initiated ConditionRefresh on the Warm standby **returns its current retained conditions**. Only new alarm events are suppressed. | #717 c11362 (2026-11-11) |
+| D6 | The queue-only spike gate must not be adopted unchanged, because of late-overflow and filtered-retain effects. Product alarm suppression happens **before** SDK event-field projection. Per-tag read projection is the recommended data seam. Every ACL, direct-item and Server-object refresh path needs coverage. Role snapshots need an ordered revision. | #717 c7812, c7853, issue body |
+| D7 | Existing mutation and election policies are preserved: the write/native-ack/HistoryUpdate Primary gates, the oldest-Up-member election, and `auto-down`. | #717 c7794 |
+
+## 2. Where the code is today
+
+- **Metadata.** No code sets `ServerRedundancy.RedundancySupport` (`i=3709`), so it reads the SDK default. `Server.ServerRedundancy` is the base `ServerRedundancyType` and has no `ServerUriArray` (`i=11314`) (`docs/Redundancy.md:9`). Peers are published only in `Server.ServerArray` (`i=2254`), by appending to `IServerInternal.ServerUris` after start (`OpcUaApplicationHost.cs:378-414`, called at `:241`).
+- **Self URI.** It comes from `OpcUa:ApplicationUri`, which defaults to `urn:OtOpcUa` (`OpcUaApplicationHost.cs:35`). If `ApplicationConfigPath` is set, the loaded XML wins (`:602-608`). docker-dev sets neither `OpcUa__ApplicationUri` nor `PeerApplicationUris` on central-1/2 (`docker-dev/docker-compose.yml:289-481`), so both centrals advertise `urn:OtOpcUa`. The seeded `ClusterNode.ApplicationUri` uses `urn:OtOpcUa:<NodeId>` (`docker-dev/seed/seed-clusters.sql:13`).
+- **ServiceLevel.** `ServiceLevelCalculator.Compute` returns 240 + 10 for the Primary (`ServiceLevelCalculator.cs:44-64`). The standby therefore sits in the *healthy* band. `OpcUaPublishActor.RecomputeServiceLevel` feeds it (`OpcUaPublishActor.cs:1576-1636`), and the result is published through the #719-fixed retain/replay path (`:1468-1488`, `SdkServiceLevelPublisher.cs:36-60`).
+- **Acquisition is role-independent.** Every driver node spawns its drivers and subscribes whatever its role (#717 c7794). Today Primary gating covers only writes, native acks, the alerts emit, alarm-history drain, historization capture and HistoryUpdate (`PrimaryGatePolicy.cs:21,56,97,155`; `docs/Redundancy.md:554-600`).
+- **Data variables.** Every ordinary-tag data variable (raw tag, UNS reference, virtual tag, Calculation tag) is an `EquipmentVariableState` created at exactly one site (`OtOpcUaNodeManager.cs:4060`). Values are pushed into it with `PublishValue` (`EquipmentVariableState.cs:28-35`).
+- **Alarm events.** Every condition event goes through `ReportConditionEvent` (`OtOpcUaNodeManager.cs:1376`) and then `ReportEventWithoutBlocking` (`:1507`). The callers are engine/driver projection (`:1254`, `:1341`), method transitions (`:2842`) and restore re-report (`:4659`). Server-object items get per-item, ACL-filtered dispatch (#586). Folder and direct items get per-item delivery (#826/#855). Refresh paths: `ConditionRefresh` (`:6586`), `RefreshDirectItems` (`:6974`) and the #854 cut. Model-change and audit events use `ReportToServerObject` (`:6060`), not this path.
+- **Role snapshots are unordered.** `RedundancyStateActor` is a cluster singleton. It publishes `RedundancyStateChanged(Nodes, CorrelationId)` on DPS, debounced at 250 ms and heartbeated every 10 s (`RedundancyStateActor.cs:25,33,105-113`). `AsOfUtc` is the *publisher's* wall clock (`:184`) and `CorrelationId` is random (`:110`). Subscribers replace their cached snapshot unconditionally (`OpcUaPublishActor.cs:1494-1498`, `DriverHostActor.cs:2555-2562`). After a singleton handover, a late heartbeat from the old incarnation can overwrite the new incarnation's snapshot. Nothing establishes the order, and there is no lease.
+- **Config.** `ServerCluster.RedundancyMode` (None/Warm/Hot) exists in ConfigDb. It is CHECK-coupled to NodeCount (`OtOpcUaConfigDbContext.cs:109-111`; `DraftValidator.cs:958-962`) and travels in the artifact's `Clusters` array (`ConfigComposer.cs:67`), but no runtime code reads it.
+
+## 3. Contract per deployment shape
+
+| Shape | `OpcUa:Redundancy:Mode` | `RedundancySupport` (i=3709) | `ServerRedundancy` type | `ServerUriArray` (i=11314) | `ServerArray` (i=2254) | Standby behaviour |
+|---|---|---|---|---|---|---|
+| Today / not opted in | `Unset` (default) | SDK default (unchanged) | `ServerRedundancyType` (2034) | absent | self + peers (unchanged) | none; Secondary serves Good at 240 (unchanged) |
+| Standalone | `None` | `None` (0) | `ServerRedundancyType` (2034) | absent | self only | never; always serves |
+| Pair | `Warm` | `Warm` (2) | `NonTransparentRedundancyType` (2039) | `[effectiveSelfUri, peer…]`, read-only String[], self first (decision #155, `docs/v2/plan.md:930`) | unchanged | §4 |
+
+- The metadata is **startup-static**. The graph is built before node-manager registration through
+  `StandardServer.CreateMainNodeManagerFactory` + `ConfigurationNodeManager.AddBehaviourToPredefinedNode`. That
+  route is spike-proven on SDK 1.5.378.106 (#717 c7791) and becomes `AddBehaviourToPredefinedNodeAsync` on 2.0.1.
+  The passive predefined `i=11314` the SDK visits later must resolve to the same constructed instance, or the wire
+  value is null (c7791). Changing mode or peers requires a restart.
+- `effectiveSelfUri` is the **effective** `ApplicationConfiguration.ApplicationUri` after any XML load, never the
+  options default (c7787).
+- `Hot` is **not advertisable**: continuous acquisition alone does not meet Part 4 §6.6.2.4 Hot (c7794).
+  `Mode=Hot` fails host start.
+- Startup validation (`ValidateOnStart`) runs for `Warm` and fails host start when any of these holds: there is no
+  peer; a peer URI equals the self URI; a URI is duplicated; the `driver` role is missing.
+- Deploy-time cross-check: if the artifact's `ServerCluster.RedundancyMode` for this node's cluster disagrees
+  with the local mode (`NodeCount=1/None` against local `Warm`, or the reverse), the node logs an Error and records
+  the metric `otopcua.redundancy.mode_mismatch`. The node keeps running in its local mode. Local config is the
+  authority, because it is the only value available before SDK start.
+
+## 4. Warm role contract
+
+### 4.1 Role verdict (who is Active)
+
+A Warm node is **Active** iff all of the following hold. Otherwise it is **Standby**.
+
+1. The local `SelfMember` is Up and `EndpointListening` is true (#762).
+2. The node's **own** `Cluster.State` names it the oldest Up `driver` member. This is the same function the
+   singleton uses (`RedundancyStateActor.SelectOldestUpMemberOfRole`, `:172`), so the verdict cannot drift from
+   singleton placement (D7).
+3. The latest **accepted** snapshot (§4.2) names it `Primary`.
+
+Any disagreement between (2) and (3) resolves to Standby. A Warning is logged if the disagreement lasts more than
+two heartbeats (20 s). When the role is unknown — before the first accepted snapshot, or when the snapshot omits
+this node — the node is **Standby**, whatever the member count. This is stricter than `PrimaryGatePolicy`'s
+lone-node allow (`PrimaryGatePolicy.cs:21`): a lone node goes Active within one debounce (~250 ms) of its own
+singleton's first publish. That gate is **not** changed for writes/acks (D7). The Warm verdict is a separate output.
+
+Accepted trade: an `auto-down` partition can give two Active nodes, as it already does for the data-plane gate
+(`docs/Redundancy.md` §Split-brain). For reads this is benign, because both nodes serve the same Good data.
+
+### 4.2 Ordered snapshots (the revision problem)
+
+`RedundancyStateChanged` gains an additive `Revision(PublisherUpNumber, IncarnationTick, Sequence)`:
+
+- `PublisherUpNumber` is the `Member.UpNumber` of the node that hosts the singleton. The singleton always lives on
+  the oldest member, and every surviving member has a larger up-number than a departed oldest one. The value is
+  therefore monotonic across handovers within one cluster incarnation.
+- `IncarnationTick` is `Environment.TickCount64` at singleton `PreStart`. It is monotonic for a singleton that
+  restarts in the same process.
+- `Sequence` increments on every publish, including heartbeats.
+
+Subscribers (`OpcUaPublishActor`, `DriverHostActor`) accept a snapshot only if its revision is lexicographically
+greater than the last accepted one. Anything else is dropped and counted in
+`otopcua.redundancy.snapshot_rejected{reason}`. A subscriber cannot outlive its cluster incarnation: a removed node
+must restart its `ActorSystem` to rejoin. The plan pins this assumption with a test. A snapshot **without** a
+revision (from a pre-#717 publisher during a rolling upgrade) is accepted for the existing gates but can never make
+a Warm node Active. Warm therefore requires both pair nodes to be upgraded before it is turned on.
+
+There is no time lease beyond this. Liveness comes from (2), which reads live gossip, and the existing 30 s
+staleness input to ServiceLevel. Demoting an Active node on snapshot staleness alone was rejected: the singleton
+lives on the Active node itself, so staleness there means a local fault, and demoting would turn it into a pair-wide
+data outage.
+
+### 4.3 ServiceLevel bands (Part 4 §6.6.2.4.2)
+
+| Node state (Warm) | Byte | Band |
+|---|---|---|
+| Down / Detached / endpoint failed | 0 | Maintenance (unchanged) |
+| Boot, before first computed level | **1** (set in the startup graph; replaces SDK default 255) | NoData |
+| Standby, degraded (stale or DB unreachable) | **50** | Degraded |
+| Standby, healthy | **90** | Degraded |
+| Active, critically degraded | 100 | Degraded (unchanged) |
+| Active, stale | 200 | Healthy (unchanged) |
+| Active, healthy | 250 | Healthy (unchanged: 240 + 10) |
+
+Invariant: **every Standby level is below every level an Active node serving data can publish (≥ 100)**. A
+"highest ServiceLevel" client therefore never prefers a standby over a data-serving Active node. `Unset` and `None`
+keep `ServiceLevelCalculator.Compute` byte-for-byte.
+
+### 4.4 Behaviour on the Standby, per service
+
+"Ordinary tag" means the `Value` attribute of an `EquipmentVariableState`. The category is captured at item
+creation from the node instance and the `AttributeId`, never from namespace or ClientHandle (c7853).
+
+| Service | Standby behaviour | Status the client sees |
+|---|---|---|
+| Read (ordinary tag `Value`) | Projected. The stored value is kept internally and continues to update from acquisition. | `BadNoCommunication`, `Value = null`, `ServerTimestamp = now`. Operation result Good. |
+| Read (other attributes, condition children, ns=0 incl. ServiceLevel/ServerStatus/ServerRedundancy) | Unchanged | Good / as today |
+| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification `BadNoCommunication`/null |
+| Existing data items at demotion | One `BadNoCommunication`/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. **D4:** any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to `BadNoCommunication`/null at Publish/Republish (§4.6). | `BadNoCommunication` |
+| Write | **Unchanged (D7):** the existing gate rejects off-Primary through the #5 self-correction (synchronous Good, then revert + audit). Under Warm the revert is invisible because the value is projected Bad anyway. See open question Q1. | as today |
+| HistoryRead | Unchanged. Served from the shared backend, which is not live acquisition. | as today |
+| HistoryUpdate | Unchanged | `BadInvalidState` off-Primary (#674) |
+| Browse / TranslateBrowsePaths / RegisterNodes | Unchanged | as today |
+| Call: Part 9 methods | Unchanged (D7): native ack `BadInvalidState` off-Primary; scripted ack/shelve accepted. The resulting condition event counts as a new alarm event and is suppressed (§4.5). See Q4. | as today |
+| Event subscriptions (Server object, folders, direct) | Stay established, in Reporting mode, with the ACL unchanged | — |
+| Model-change / audit events | **Not suppressed.** They are not alarm events, and browse consistency needs them. | — |
+| Session / secure channel / ServerStatus.State | Unchanged (`Running`) | — |
+
+### 4.5 Alarms and events on the Standby
+
+- **Suppression point.** `ReportConditionEvent` is split into *stamp* and *deliver*. Stamp sets EventId, Time and
+  ReceiveTime, and runs `ClearChangeMasks`. Deliver builds the snapshot and calls `ReportEventWithoutBlocking`. On
+  the Standby only *stamp* runs. Condition node state therefore stays current, and a later refresh carries a valid,
+  ackable EventId. Because suppression sits before snapshot construction and before every SDK field projection, it
+  covers the Server-object, folder and direct paths, ACL on or off, by construction. Overflow, filtered retain and
+  the #838 deferral queue never see a suppressed event. The `RingEventQueueFactory` queue is **not** used as the
+  gate (D6).
+- **Boundary.** The decision is made at report time. An event whose report ran while Active is delivered even if
+  delivery finishes after demotion: deferred (#838) or already in an item queue. This matches the spike's
+  observed admission boundary (c7805). Republish of already-built event messages is unchanged.
+- **Client ConditionRefresh / ConditionRefresh2 on the Standby (D5)** run the existing paths unchanged: Server
+  object (#854 cut), `RefreshDirectItems` (#855), with ACL filtering as today. The replay is RefreshStart, the
+  current retained conditions, then RefreshEnd.
+- **Promotion refresh.** On Standby → Active, the server runs a refresh on **every** live event item: Server-object
+  items, direct items and folder items. The two refresh entry points are reused, so ACL decisions and the #854/#855
+  cut apply. Clients see RefreshStart, retained conditions, RefreshEnd, then new events. This is spike-proven on
+  1.5 (c7805). A condition that went active **and** stopped being retained entirely while the node was Standby is
+  not replayed. That is standard refresh semantics, and the transition is not historized twice, because alarm
+  history is Primary-gated (`docs/Redundancy.md:572-573`).
+
+### 4.6 Strict first delivery (D4)
+
+The item-level projection (§4.4) stops new Good samples. It cannot reach samples already queued in an item, nor
+`NotificationMessage`s the subscription has prebuilt or cached (c7846, c7850). For those, the design uses the
+mechanism that passed the provenance spike (c7851, 66 assertions) and the delivery-history spike (c7852,
+53 assertions):
+
+1. **Provenance.** Ordinary-tag data items are created through a custom sampled item. Its per-value `Publish`
+   stamps each newly created `MonitoredItemNotification` with an immutable `(NodeId, AttributeId, Category)`. The
+   stamp survives handle modification, item deletion and reuse.
+2. **Delivery history.** `OtOpcUaSdkServer` overrides `PublishAsync` and `RepublishAsync`, calls base first, and
+   records **one first-delivery choice per cached message object**. The association is weak and keyed by object;
+   cached SDK objects are never mutated.
+   - If the node is Active at first delivery, the original message is the choice.
+   - If the node is Standby, a detached clone is the choice, with every `OrdinaryTag` notification replaced by
+     `BadNoCommunication`/null.
+   - A Republish returns the recorded choice. First delivery *through* Republish is decided the same way.
+   - A short lock orders the choice against the role flip. No lock spans an await.
+3. **Out of scope of the rewrite**, by D4: messages already in flight before the demotion step, and Republish of
+   messages first delivered while Active. Unknown-origin notifications are not rewritten; there is no implicit
+   policy (c7853).
+
+### 4.7 Transition sequencing
+
+One actor, `OpcUaPublishActor`, owns the verdict, the ServiceLevel publish and a new `IStandbyProjection` sink
+(in `Commons.OpcUa`, bound late like `IServiceLevelPublisher`). Each transition runs as one serialized handler, in
+this order.
+
+**Promotion (Standby → Active)**
+
+1. Open the read projection.
+2. For each live ordinary-tag item, queue the current stored value, ignoring filters. The precedent is the queue
+   pattern at `OtOpcUaNodeManager.cs:7303`.
+3. Switch the delivery-history choice to Active.
+4. Un-suppress alarm delivery.
+5. Run the promotion refresh (§4.5).
+6. Publish the Active ServiceLevel.
+
+**Demotion (Active → Standby)**
+
+1. Publish the Standby ServiceLevel.
+2. Switch the delivery-history choice to Standby. This is the D4 boundary.
+3. Suppress alarm delivery.
+4. Close the read projection.
+5. Queue `BadNoCommunication` to each live ordinary-tag item.
+
+Invariant: **an Active-band ServiceLevel is never visible while the projection is closed.** If the ServiceLevel
+publish fails (#719 retry path), a promotion stops before step 6 and retries. A demotion continues anyway: failing
+safe means degrading the data.
+
+### 4.8 Standalone (`None`) and `Unset`
+
+`None` never projects, never suppresses and never refreshes. It advertises `RedundancySupport=None` with no
+`ServerUriArray`, and uses today's ServiceLevel. `Unset` is byte-for-byte today's server, including the documented
+unmet A4/A5b gates.
+
+## 5. ScadaBridge impact (cross-repo, blocking for any Warm flip)
+
+ScadaBridge consumes this server as an OPC UA client, through `RealOpcUaClient` and `DataConnectionActor`.
+
+- **Failover is connection-driven, not ServiceLevel-driven.** ScadaBridge swaps primary to backup only after
+  `FailoverRetryCount` consecutive *unstable disconnects*
+  (`../ScadaBridge/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs:655-690`;
+  `RealOpcUaClient.cs` sits in that project's `Adapters/`).
+  `BadNoCommunication` is not a connection-level status (`:1974-1983`), so it maps to plain Bad quality, and since
+  ScadaBridge#112 a Bad-with-null update keeps the last-known value. Nothing in ScadaBridge reads `ServiceLevel`,
+  `RedundancySupport` or `ServerUriArray`.
+- **Consequence.** A ScadaBridge connection that lands on the Standby will sit on Bad quality indefinitely on a
+  healthy session, and never fail over. Today the same connection gets Good data at 240. Warm is therefore an
+  **outage for ScadaBridge** unless ScadaBridge becomes ServiceLevel-aware: monitor `i=2267` and treat a Degraded
+  band (< 200) sustained for N s, while the other endpoint reports a higher level, as a failover trigger. That
+  change must ship and be deployed first. The plan files it as a ScadaBridge issue (`cross-repo`, blocking).
+- **Alarms are compatible.** ScadaBridge already handles a RefreshStart/RefreshEnd replay as a snapshot
+  (`RealOpcUaClient.cs:72`) and triggers its own ConditionRefresh on (re)subscribe (`:1179-1193`). Under D5 that
+  refresh returns retained conditions on the Standby too. The promotion refresh arrives as an unsolicited
+  Start/End pair on existing items; this needs a ScadaBridge-side test.
+- **Status mapping is compatible.** ScadaBridge#117 maps by severity bits, so `BadNoCommunication` is Bad.
+
+## 6. Configuration surface
+
+| Key | Default | Notes |
+|---|---|---|
+| `OpcUa:Redundancy:Mode` | `Unset` | `Unset` / `None` / `Warm`. `Hot` and unknown values fail host start. Startup-static. Flip **both** pair nodes in one window, after both run a revision-capable build (§4.2). |
+| `OpcUa:PeerApplicationUris` | `[]` | Existing key, now also the `ServerUriArray` peer source. Required (≥ 1) for `Warm`, must be empty for `None`. |
+| `OpcUa:ApplicationUri` | `urn:OtOpcUa` | Must be unique within the pair under `Warm` (validated against the effective config). |
+
+There are no ConfigDb schema changes. `ServerCluster.RedundancyMode` is only cross-checked (§3).
+
+## 7. Observability
+
+- `otopcua.redundancy.warm_role`: gauge. 1 = Active, 0 = Standby, −1 = not Warm.
+- `otopcua.redundancy.warm_transition{direction,reason}`: counter. Reasons: `membership`, `snapshot`,
+  `endpoint`, `boot`.
+- `otopcua.redundancy.snapshot_rejected{reason=older_revision|unrevisioned}`: counter.
+- `otopcua.redundancy.mode_mismatch`: counter (§3).
+- `otopcua.opcua.standby.projected_notifications`: counter. Counts D4 rewrites at Publish/Republish.
+- `otopcua.opcua.standby.suppressed_alarm_events`: counter.
+- Logs:
+  - Information per transition, naming the revision and the verdict inputs.
+  - Warning when membership and snapshot disagree for more than 20 s.
+  - A boot banner line with mode, effective self URI and peers.
+- `otopcua-cli redundancy` already reads `i=3709`, `i=11314` and `i=2267`.
+
+## 8. Explicitly out of scope
+
+- Hot (and Transparent) redundancy.
+- Dynamic peer membership. Peers are static config.
+- Changing the election (oldest Up driver member), `auto-down`, or the write / native-ack / HistoryUpdate /
+  alarm-drain / historization gates.
+- Stopping acquisition on the Standby.
+- TransferSubscriptions across the pair. Non-transparent means the client re-creates its subscriptions.
+- CTT (G8) and production-client cutover (G9). Hermetic and live-rig passes are not acceptance.
+- Durable subscription restore.
+- Sequence-number wrap.
+- Encoded-byte fidelity of cloned payloads beyond the plan's bounded matrix (c7890).
+
+## 9. Open questions for the owner (short)
+
+1. **Standby writes.** Keep today's optimistic Good + revert (D7), or refuse synchronously with `BadInvalidState`
+   under Warm, as HistoryUpdate and native ack already do? *Recommend synchronous refusal under Warm only.*
+2. **Crash-failover gap.** A SIGKILLed Primary stays an unreachable member until auto-down, ~25–30 s. Under (2)/(3)
+   the survivor stays Standby for that whole window, with no Good data anywhere, where today the Secondary keeps
+   serving. Should the survivor be allowed to promote when the Primary is *unreachable* (the #672 pattern, default
+   on), accepting dual-Active during a real partition? *Recommend yes, behind
+   `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable`.*
+3. **Standby status code.** `BadNoCommunication` is spike-proven and is plain Bad in ScadaBridge, but it is the
+   same code as real device comms loss; only ServiceLevel tells them apart. Accept it, or use `BadOutOfService`?
+4. **Scripted Part 9 methods on the Standby.** They are accepted (D7) but their event is suppressed (D3), so the
+   caller sees Good and no event until promotion or refresh. Accept, or refuse with `BadInvalidState` under Warm?
+5. **ConfigDb `Hot`.** Narrow the CHECK to `None/Warm` with a migration, or leave it and rely on the §3 mismatch
+   Error?
+6. **Bands.** Are Standby 90/50 and boot 1 acceptable (§4.3)?
+
+## 10. What must be re-proven on SDK 2.0.1
+
+The spikes ran on **1.5.378.106**. The repo now pins **2.0.1** (`Directory.Packages.props:129-135`). The 2.0.1 XML
+docs confirm that these members still exist:
+
+- `StandardServer.CreateMainNodeManagerFactory`
+- `ConfigurationNodeManager.AddBehaviourToPredefinedNodeAsync`, now **async**
+- `NonTransparentRedundancyState.ServerUriArray`, now in `Opc.Ua.Core.Types`
+- `StandardServer.PublishAsync` / `RepublishAsync`, whose signatures now carry `RequestLifetime`
+- `CreateSubscriptionManager`
+- `BaseVariableState.OnSimpleReadValue`, plus new `OnSimpleReadValueAsync` / `OnReadValueAsync`
+- `ReadValueAttribute`
+- `IServerInternal.ConditionRefresh2`
+- a new `MonitoredItemFactory` / `MonitoredItemCreateDecision` creation hook and an
+  `IMonitoredItemTransferCoordinator`
+
+Existence is not behaviour. Plan T0 re-proves, on 2.0.1 over the wire:
+
+- (a) the metadata graph, including the passive-`i=11314` resolution;
+- (b) that **every** value read path reaches the per-tag projection: Read, initial sample, change fan-out,
+  ResendData, size-1 queue and AggregateFilter. In particular, the new async read callbacks must not bypass a sync
+  `OnSimpleReadValue`; if they do, the hook moves to a `ReadValueAttribute` override on `EquipmentVariableState`;
+- (c) provenance through the 2.0.1 creation hook, which replaces the 1.5 manager-subclass route, including
+  transfer/restore;
+- (d) the Publish/Republish choice with the new signatures;
+- (e) suppression and promotion refresh through the product's #854/#855 per-item paths, which did not exist when
+  the 1.5 event spike ran.

docs/plans/2026-11-11-warm-redundancy-717-plan.md

@@ -0,0 +1,277 @@
+# Warm non-transparent redundancy (Gitea #717): implementation plan
+
+Status: **DRAFT for owner approval**, together with `2026-11-11-warm-redundancy-717-design.md` (referred to below
+as "the spec", with § references). No task starts until both documents and the §9 open questions are settled.
+Every task ships **dark**: `OpcUa:Redundancy:Mode` defaults to `Unset`, so merging any task changes no deployed
+behaviour.
+
+Conventions:
+
+- **Wire tests** use a real OPC Foundation client session against a loopback `OtOpcUaSdkServer`, in
+  `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests` (the home of `DualEndpointTests`,
+  `ConditionRefreshDeferralWireTests` and `FolderConditionRefreshWireTests`).
+- **Actor tests** go in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` and
+  `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests`.
+- Every gate gets an **allow** leg as well as a deny leg (the vacuous-test rule), and a mutation check: revert the
+  production line and confirm the test fails.
+
+## T0: SDK 2.0.1 re-proof spikes (throwaway, separately approved)
+
+The 1.5.378.106 spikes do not carry over (spec §10). Write throwaway loopback fixtures against **2.0.1**, outside
+`src/`. No product edits. Each fixture is rerun independently and its evidence is recorded in
+`docs/plans/evidence/2026-11-xx-717-sdk2/`.
+
+- **a. Metadata graph.** Build it via `CreateMainNodeManagerFactory` + `AddBehaviourToPredefinedNodeAsync`. Check
+  Read/Browse/Translate of `i=2296` (HasTypeDefinition 2039), `i=3709`, and `i=11314` for both Warm and None. The
+  passive `i=11314` visit must resolve to the constructed instance.
+- **b. Read projection coverage.** Cover Read, the initial sample on create, the change fan-out after
+  `ClearChangeMasks`, ResendData, queue size 1 vs. N, an AggregateFilter item, and Sampling → Reporting. Decide
+  between `OnSimpleReadValue` and a `ReadValueAttribute` override, and record which async callback paths 2.0.1
+  takes.
+- **c. Provenance.** Stamp items through the 2.0.1 `MonitoredItemFactory` / `MonitoredItemCreateDecision`. Exercise
+  ModifyMonitoredItems handle change, delete/reuse, TransferSubscriptions (`IMonitoredItemTransferCoordinator`),
+  and a restore.
+- **d. Publish/Republish.** Override `PublishAsync`/`RepublishAsync` with the 2.0.1 signatures (`RequestLifetime`).
+  Cover the first-delivery choice, first delivery through Republish, an ack followed by `BadMessageNotAvailable`,
+  and a cancelled or failed response.
+- **e. Events.** Exercise the stamp/deliver split under the #855 per-item folder path and the #854 Server-object
+  cut. Run the promotion refresh on all three item kinds, and a client ConditionRefresh/2 while Standby.
+
+**Exit:** a short results note per fixture, and an owner go/no-go. A "no" on (c) or (d) re-opens D4's mechanism.
+D4 itself stays an owner decision.
+
+## T1: Configuration, validation, effective self URI
+
+**Files:**
+- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`: options at `:29-121`. Add
+  `OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable` if Q2 = yes).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`.
+- A post-load check in `StartAsync` (`:218-250`) against the effective `ApplicationConfiguration.ApplicationUri`.
+
+**Rules (spec §3):**
+- `Hot` and unknown values fail host start.
+- `Warm` requires at least one peer, all URIs distinct, self not listed as a peer, and the `driver` role.
+- `None` requires no peers.
+
+**Tests:**
+- A validator matrix.
+- An XML `ApplicationConfigPath` whose URI differs from the options value, proving the effective URI is used.
+- A boot-banner log assertion.
+
+## T2: Standard redundancy metadata graph
+
+**Files:**
+- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs`: next to `CreateMasterNodeManager` (`:341`), add a
+  `CreateMainNodeManagerFactory` override that installs the T0a graph per mode.
+- Under `Warm`, the initial `Server.ServiceLevel` is set to **1** in the same pass (spec §4.3). `Unset` must not
+  touch the graph.
+
+**Tests (wire):**
+- Per mode, check the `i=3709` value, the `i=2296` type definition, the presence or absence of `i=11314`, its
+  value order `[effectiveSelf, peers…]`, and that it is read-only (a Write returns `BadNotWritable`).
+- `ServerArray` (`i=2254`) is unchanged; extend `DualEndpointTests`.
+- A Warm server's ServiceLevel reads 1 before the first publish.
+- Security profiles and startup are unaffected under `Unset` (golden read of today's values).
+
+## T3: Ordered redundancy snapshots (spec §4.2)
+
+**Files:**
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/RedundancyStateChanged.cs` (`:9-11`): add an optional
+  `SnapshotRevision? Revision` (additive).
+- `RedundancyStateActor.cs`:
+  - stamp the revision in `PublishIfDirty` (`:105-113`);
+  - set `IncarnationTick` in `PreStart`;
+  - read the host's `UpNumber` from `Cluster.SelfMember`.
+- Subscribers: accept only a newer revision in `OpcUaPublishActor.HandleRedundancyStateChanged` (`:1494-1498`) and
+  `DriverHostActor.OnRedundancyStateChanged` (`:2555`).
+- An unrevisioned snapshot keeps today's handling for the existing gates.
+
+**Tests:**
+- Ordering unit tests: an older singleton incarnation's heartbeat after a newer snapshot is dropped; a same-process
+  singleton restart advances.
+- A **real 2-node cluster** handover test (pattern: `RedundancyPrimaryElectionTests`). It holds back the old
+  incarnation's last publish through the `_broadcastOverride` seam (`:111`) and asserts the subscriber keeps the
+  new Primary.
+- Serialization round-trip of the extended message, plus a mixed-version decode with the field absent.
+- The pinned assumption: a removed node cannot rejoin without a new `ActorSystem`.
+
+## T4: Warm role verdict and ServiceLevel bands (spec §4.1, §4.3)
+
+**Files:**
+- New `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/WarmRoleAuthority.cs`: a pure function of membership view,
+  accepted snapshot, `EndpointListening` and `SelfMember` status.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs`: add `ComputeWarm(inputs, isActive)`.
+  `Compute` itself is unchanged.
+- `OpcUaPublishActor.RecomputeServiceLevel` (`:1576-1636`) branches on mode. It subscribes to `MemberUp/Removed`
+  (and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots.
+
+**Tests:**
+- A truth table covering every row of §4.3 and the invariant *max(Standby) < min(Active serving)*.
+- Membership/snapshot disagreement gives Standby, plus the 20 s Warning.
+- Boot with an unknown role gives Standby; a lone node goes Active after its own snapshot.
+- `None`/`Unset` produce byte-identical output to today's `ServiceLevelCalculatorTests`.
+- DB-less mode (#721 timer) is covered.
+
+## T5: Standby projection seam and transition sequencer (spec §4.7)
+
+**Files:**
+- New `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IStandbyProjection.cs`, with a deferred/late-bound adapter that
+  mirrors `DeferredServiceLevelPublisher` and retains the latest state across binding (the #719 lesson).
+- Bound in `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs` next to the ServiceLevel binding.
+- `OpcUaPublishActor` runs promotion and demotion as one serialized handler, in the §4.7 step order.
+
+**Tests (actor, with a recording fake):**
+- The exact step order for both directions.
+- A ServiceLevel publish failure halts a promotion before the Active level and retries; a demotion continues.
+- A burst of alternating verdicts converges on the last one.
+- Binding after the first transition replays the current state.
+
+## T6: Per-tag read projection (spec §4.4)
+
+**Files:**
+- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs`: the hook decided in T0b, reading a
+  node-manager-wide volatile projection flag. Stored `Value` is never modified.
+- `OtOpcUaNodeManager` implements `IStandbyProjection.SetDataProjection`.
+
+**Tests (wire):**
+- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadNoCommunication`/null.
+- A Standby Read of a condition child, `i=2267` or `i=3709` stays Good.
+- A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected.
+- Acquisition continues: flip to Active and the latest value is read without any new driver publish.
+- ACL on: a denied node is still `BadUserAccessDenied` and the projection never leaks existence.
+
+## T7: Provenance and first-delivery choice (spec §4.6, D4)
+
+**Files:**
+- `OtOpcUaSdkServer.cs`: install the custom sampled item (T0c route) for ordinary-tag `Value` items only.
+- `OtOpcUaSdkServer.cs`: add the `PublishAsync`/`RepublishAsync` overrides and a new `FirstDeliveryLedger`, a
+  `ConditionalWeakTable` keyed by the SDK message object.
+- Wire into `OtOpcUaNodeManager.CreateMonitoredItems` (`:5295`) / `OnMonitoredItemCreated` (`:5442`) without
+  disturbing the ACL filtering or `_liveItems`.
+
+**Tests (wire):** port the spikes' assertion matrix to product code.
+- Queued Good samples, then demotion, then the first Publish carries `BadNoCommunication`.
+- Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same
+  projected payload.
+- Good delivered while Active, then demotion, then Republish returns the original Good.
+- A first delivery through Republish.
+- A mixed subscription (ordinary tag + `i=2267` + condition-child item): only ordinary-tag items are rewritten.
+- Handle modified, deleted or reused between queue and publish.
+- The cached SDK object is unchanged (sacrificial-clone check).
+- A bounded clone matrix (arrays, ExtensionObject, DiagnosticInfo, per c7890).
+- Concurrent Publish and role flip, by a stress loop with a deterministic barrier, not sleeps.
+
+## T8: Promotion and demotion of live data items
+
+**Files:**
+- `OtOpcUaNodeManager`: `IStandbyProjection.QueueCurrentToLiveItems(bool projected)` iterates `_liveItems`. It
+  reuses the QueueValue-with-`ignoreFilters` pattern (`:7303`), **not** detach or `MonitoringMode.Disabled`; the
+  disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped.
+
+**Tests (wire):**
+- Demotion delivers exactly one `BadNoCommunication` per item, then silence while values keep changing internally.
+- Promotion delivers the current Good value without a driver publish.
+- An ACL-stood-down item receives nothing either way.
+- A DeadbandFilter item still receives the promotion value.
+
+## T9: Alarm event suppression and promotion refresh (spec §4.5, D3, D5)
+
+**Files (`OtOpcUaNodeManager`):**
+- Split `ReportConditionEvent` (`:1376`) into stamp and deliver; deliver is gated by the projection flag.
+- Audit all callers (`:1254`, `:1341`, `:2842`, `:4659`) and `WireMethodTransitionReporting`.
+- `IStandbyProjection.RefreshAllEventItems()` runs the existing `ConditionRefresh` (`:6586`) Server-object path and
+  `RefreshDirectItems` (`:6974`) over every live event item, from `_allEventsItems` and `_eventItems`.
+- `ReportToServerObject` (`:6060`) is untouched.
+
+**Tests (wire, ACL off and on):**
+- A new alarm on the Standby reaches no Server-object, folder or direct item.
+- Model-change and audit events are still delivered on the Standby.
+- A client ConditionRefresh **and** ConditionRefresh2 on the Standby return Start, retained, End (D5).
+- Promotion gives Start, retained, End on all three item kinds, followed by new events.
+- An ACL-denied condition is absent from the promotion refresh.
+- A condition raised and cleared (Retain=false) while Standby is not replayed.
+- An event reported while Active but deferred (#838, via `NotifierStall`) is still delivered after demotion.
+- A Standby-stamped EventId is accepted by Acknowledge after promotion.
+- The `RingEventQueueFactory` path is untouched (`RingEventMonitoredItemQueueParityTests` stay green).
+- Re-run `EventQueueOverflowAclWireTests` and `ConditionMethodEventTests`.
+
+## T10: Deploy-time mode cross-check and observability (spec §3, §7)
+
+**Files:**
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`: read this node's cluster
+  `RedundancyMode` from the `Clusters` array.
+- `OpcUaPublishActor` compares it with the local mode at apply.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`: add the six instruments.
+
+**Tests:**
+- A mismatch logs an Error and records the metric.
+- Each counter fires on its path. Use a `MeterListener` filtered by **instrument instance**, never by meter name
+  alone (the #503 lesson).
+
+## T11: ScadaBridge prerequisite (cross-repo)
+
+- File a ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes
+  ServiceLevel-aware: monitor `i=2267`, and fail over when it is below 200 for a configurable window while the
+  other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items
+  (spec §5).
+- File the lmxopcua companion as `blocked` + `cross-repo`.
+- **The Warm flip in T13/rollout is gated on that issue being deployed.**
+
+## T12: Documentation and index
+
+- `docs/Redundancy.md`: rewrite the `:9` block, add a Warm contract section, update the tiers table. Remove the
+  generic "standby / no device contact / exactly-once" wording that contradicts acquisition and at-least-once
+  (c7794).
+- `docs/OpcUaServer.md:70`, `docs/Configuration.md` (the new keys), and `docs/security.md` (ns=0 stays ungated;
+  projection order vs. ACL).
+- `.claude/rules/cluster-topology.md` §Redundancy.
+- `docs/plans/phase-6-3-redundancy-interop-plan.md` A2/A4/A5b: under Warm, A2 expects 90 and A4/A5b become
+  testable. Add `docs/v2/redundancy-interop-playbook.md`.
+- Umbrella: update the OtOpcUa row and `sister-project-history/otopcua.md` in `../scadaproj` on **its `main`**,
+  pushed (root `CLAUDE.md` procedure).
+
+## T13: Live gate on docker-dev (central pair, `:4840` / `:4841`)
+
+**Overlay:** new `docker-dev/docker-compose.warm.yml`, central-1/2 only. It sets:
+
+- `OpcUa__ApplicationUri=urn:OtOpcUa:central-1|2` (matching `seed-clusters.sql`);
+- `OpcUa__PeerApplicationUris__0=<the other>`;
+- `OpcUa__Redundancy__Mode=Warm`.
+
+Today both centrals advertise the default `urn:OtOpcUa` (spec §2). Login is disabled on the rig, so drive
+everything yourself. Use `otopcua-cli` from `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` against both ports.
+
+| Leg | Procedure | Pass |
+|---|---|---|
+| L1 metadata | `redundancy` on :4840 and :4841 | `RedundancySupport=Warm`. ServerUriArray is self first and differs per node. Same URIs in both. |
+| L2 bands | `read -n i=2267` on both | Primary 250, Standby 90 |
+| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadNoCommunication`/null; the Primary shows Good at the same moment |
+| L4 standby alarms | `alarms` on both. Trip a scripted alarm via its source tag. | Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5). |
+| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadNoCommunication`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). |
+| L6 crash | `docker kill central-<primary>` | The survivor goes Active within the gap Q2 selects. Record the measured gap. |
+| L7 rejoin | Restart the killed node | It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90. |
+| L8 standalone | site-b-2 (or a one-node overlay) with `Mode=None` | `RedundancySupport=None`, no `i=11314`, data Good |
+| L9 Unset control | Another node left on `Unset` | Byte-identical to pre-change: SL 240/250, SDK-default `i=3709` |
+| L10 writes/history | Write and HistoryRead on the Standby | Behaviour per Q1. HistoryRead is Good. |
+
+Record evidence in `docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md`. Do **not** mark G8/G9 (#615); they
+need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).
+
+## Rollout
+
+1. Merge T1–T12 dark (`Unset` default). The full CI floor applies.
+2. Deploy the ScadaBridge T11 change everywhere ScadaBridge reads OtOpcUa.
+3. Upgrade **both** nodes of a pair to a revision-capable build (T3) before flipping either one.
+4. Flip both nodes of one docker-dev pair, then run T13. After that, flip one non-production site pair.
+5. Production is a separate, owner-approved change per pair. Flip the **Secondary first**, then the Primary, in
+   one window.
+
+A mixed window (one `Warm`, one `Unset`) is tolerable: the Unset node serves Good at 240/250. But do not leave it
+that way, because `ServerUriArray` then exists on only one node.
+
+## Rollback
+
+Set `OpcUa:Redundancy:Mode=Unset` on both nodes and restart them. The setting is startup-static, so there is
+nothing persistent to undo: no schema changes and no artifact changes. The `Revision` field stays and is harmless,
+because unrevisioned handling is preserved. Rolling back ScadaBridge is independent; its ServiceLevel-aware
+failover also works against `Unset` servers.