From a42c9d7f7c5773000d0ec6fb9796ed6a2f11380b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 13 Nov 2026 02:43:00 -0500 Subject: [PATCH] docs(717): record the owner's 2026-11-13 answers in the Warm redundancy spec and plan Standby writes and Part 9 methods refused BadInvalidState under Warm, promote while the Primary is unreachable (OpcUa:Redundancy:PromoteWhilePrimaryUnreachable, default on), standby status code BadOutOfService, no ConfigDb migration, bands as drafted. ScadaBridge#138 is the filed prerequisite. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- .../2026-11-11-warm-redundancy-717-design.md | 66 +++++++++++-------- .../2026-11-11-warm-redundancy-717-plan.md | 34 ++++++---- 2 files changed, 60 insertions(+), 40 deletions(-) diff --git a/docs/plans/2026-11-11-warm-redundancy-717-design.md b/docs/plans/2026-11-11-warm-redundancy-717-design.md index 86112ca14..25f3baf71 100644 --- a/docs/plans/2026-11-11-warm-redundancy-717-design.md +++ b/docs/plans/2026-11-11-warm-redundancy-717-design.md @@ -1,8 +1,9 @@ # Warm non-transparent redundancy with a degraded standby (Gitea #717): design spec -Status: **DRAFT for owner approval.** No product code changes until this spec and the companion plan -(`2026-11-11-warm-redundancy-717-plan.md`) are approved. Parent gate: #615 (G8 CTT, G9 production-client -cutover). These stay open and separate from everything here. +Status: **APPROVED by the owner 2026-11-13** (#717), together with the companion plan +(`2026-11-11-warm-redundancy-717-plan.md`). The §9 questions are answered; the answers are recorded in §9 and +applied throughout. Parent gate: #615 (G8 CTT, G9 production-client cutover). These stay open and separate from +everything here. ## 1. Owner decisions this spec builds on (not re-opened) @@ -69,6 +70,15 @@ this node — the node is **Standby**, whatever the member count. This is strict lone-node allow (`PrimaryGatePolicy.cs:21`): a lone node goes Active within one debounce (~250 ms) of its own singleton's first publish. That gate is **not** changed for writes/acks (D7). The Warm verdict is a separate output. +**Promote while the Primary is unreachable (owner answer Q2, `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable`, +default `true`).** A SIGKILLed Primary stays an Up-but-unreachable member until `auto-down` removes it (~25–30 s), +and the singleton usually died with it, so (2) and (3) would hold the survivor Standby for that whole window. With +the option on, (2) and (3) are evaluated with **unreachable** members excluded: (2′) self is the oldest Up `driver` +member among the members this node can reach, and (3′) the latest accepted snapshot names either self or an +**unreachable** member as `Primary`. This is the #672 pattern (`ContinuousHistorization:CaptureWhilePrimaryUnreachable`). +A node that has never accepted a snapshot is still Standby. The cost is dual-Active while a live Primary is merely +unreachable, and for the whole of a real partition. + Accepted trade: an `auto-down` partition can give two Active nodes, as it already does for the data-plane gate (`docs/Redundancy.md` §Split-brain). For reads this is benign, because both nodes serve the same Good data. @@ -118,15 +128,15 @@ creation from the node instance and the `AttributeId`, never from namespace or C | Service | Standby behaviour | Status the client sees | |---|---|---| -| Read (ordinary tag `Value`) | Projected. The stored value is kept internally and continues to update from acquisition. | `BadNoCommunication`, `Value = null`, `ServerTimestamp = now`. Operation result Good. | +| Read (ordinary tag `Value`) | Projected. The stored value is kept internally and continues to update from acquisition. | `BadOutOfService`, `Value = null`, `ServerTimestamp = now`. Operation result Good. | | Read (other attributes, condition children, ns=0 incl. ServiceLevel/ServerStatus/ServerRedundancy) | Unchanged | Good / as today | -| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification `BadNoCommunication`/null | -| Existing data items at demotion | One `BadNoCommunication`/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. **D4:** any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to `BadNoCommunication`/null at Publish/Republish (§4.6). | `BadNoCommunication` | -| Write | **Unchanged (D7):** the existing gate rejects off-Primary through the #5 self-correction (synchronous Good, then revert + audit). Under Warm the revert is invisible because the value is projected Bad anyway. See open question Q1. | as today | +| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification `BadOutOfService`/null | +| Existing data items at demotion | One `BadOutOfService`/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. **D4:** any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to `BadOutOfService`/null at Publish/Republish (§4.6). | `BadOutOfService` | +| Write (ordinary tag `Value`) | **Refused synchronously under Warm only (owner answer Q1).** The node manager refuses the write before it reaches the driver; nothing is reverted and no device is touched. `Unset`/`None` keep today's #5 self-correction (synchronous Good, then revert + audit) off-Primary (D7). | `BadInvalidState` | | HistoryRead | Unchanged. Served from the shared backend, which is not live acquisition. | as today | | HistoryUpdate | Unchanged | `BadInvalidState` off-Primary (#674) | | Browse / TranslateBrowsePaths / RegisterNodes | Unchanged | as today | -| Call: Part 9 methods | Unchanged (D7): native ack `BadInvalidState` off-Primary; scripted ack/shelve accepted. The resulting condition event counts as a new alarm event and is suppressed (§4.5). See Q4. | as today | +| Call: Part 9 methods | **Refused under Warm (owner answer Q4).** Every Part 9 condition method (Acknowledge, Confirm, AddComment, Enable/Disable, the shelving methods) on a native or scripted condition is refused before it changes any state. `Unset`/`None` keep today's behaviour: native ack `BadInvalidState` off-Primary, scripted ack/shelve accepted (D7). | `BadInvalidState` | | Event subscriptions (Server object, folders, direct) | Stay established, in Reporting mode, with the ACL unchanged | — | | Model-change / audit events | **Not suppressed.** They are not alarm events, and browse consistency needs them. | — | | Session / secure channel / ServerStatus.State | Unchanged (`Running`) | — | @@ -168,7 +178,7 @@ mechanism that passed the provenance spike (c7851, 66 assertions) and the delive cached SDK objects are never mutated. - If the node is Active at first delivery, the original message is the choice. - If the node is Standby, a detached clone is the choice, with every `OrdinaryTag` notification replaced by - `BadNoCommunication`/null. + `BadOutOfService`/null. - A Republish returns the recorded choice. First delivery *through* Republish is decided the same way. - A short lock orders the choice against the role flip. No lock spans an await. 3. **Out of scope of the rewrite**, by D4: messages already in flight before the demotion step, and Republish of @@ -197,7 +207,7 @@ this order. 2. Switch the delivery-history choice to Standby. This is the D4 boundary. 3. Suppress alarm delivery. 4. Close the read projection. -5. Queue `BadNoCommunication` to each live ordinary-tag item. +5. Queue `BadOutOfService` to each live ordinary-tag item. Invariant: **an Active-band ServiceLevel is never visible while the projection is closed.** If the ServiceLevel publish fails (#719 retry path), a promotion stops before step 6 and retries. A demotion continues anyway: failing @@ -217,25 +227,27 @@ ScadaBridge consumes this server as an OPC UA client, through `RealOpcUaClient` `FailoverRetryCount` consecutive *unstable disconnects* (`../ScadaBridge/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs:655-690`; `RealOpcUaClient.cs` sits in that project's `Adapters/`). - `BadNoCommunication` is not a connection-level status (`:1974-1983`), so it maps to plain Bad quality, and since + `BadOutOfService` is not a connection-level status (`:1974-1983`), so it maps to plain Bad quality, and since ScadaBridge#112 a Bad-with-null update keeps the last-known value. Nothing in ScadaBridge reads `ServiceLevel`, `RedundancySupport` or `ServerUriArray`. - **Consequence.** A ScadaBridge connection that lands on the Standby will sit on Bad quality indefinitely on a healthy session, and never fail over. Today the same connection gets Good data at 240. Warm is therefore an **outage for ScadaBridge** unless ScadaBridge becomes ServiceLevel-aware: monitor `i=2267` and treat a Degraded band (< 200) sustained for N s, while the other endpoint reports a higher level, as a failover trigger. That - change must ship and be deployed first. The plan files it as a ScadaBridge issue (`cross-repo`, blocking). + change must ship and be deployed first. It is filed as **ScadaBridge#138** (`cross-repo`, blocking); no tier sets + `Mode=Warm` until it is deployed there. - **Alarms are compatible.** ScadaBridge already handles a RefreshStart/RefreshEnd replay as a snapshot (`RealOpcUaClient.cs:72`) and triggers its own ConditionRefresh on (re)subscribe (`:1179-1193`). Under D5 that refresh returns retained conditions on the Standby too. The promotion refresh arrives as an unsolicited Start/End pair on existing items; this needs a ScadaBridge-side test. -- **Status mapping is compatible.** ScadaBridge#117 maps by severity bits, so `BadNoCommunication` is Bad. +- **Status mapping is compatible.** ScadaBridge#117 maps by severity bits, so `BadOutOfService` is Bad. ## 6. Configuration surface | Key | Default | Notes | |---|---|---| | `OpcUa:Redundancy:Mode` | `Unset` | `Unset` / `None` / `Warm`. `Hot` and unknown values fail host start. Startup-static. Flip **both** pair nodes in one window, after both run a revision-capable build (§4.2). | +| `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable` | `true` | Warm only. Lets the survivor go Active while the Primary is unreachable but not yet downed (§4.1, owner answer Q2). | | `OpcUa:PeerApplicationUris` | `[]` | Existing key, now also the `ServerUriArray` peer source. Required (≥ 1) for `Warm`, must be empty for `None`. | | `OpcUa:ApplicationUri` | `urn:OtOpcUa` | Must be unique within the pair under `Warm` (validated against the effective config). | @@ -269,22 +281,20 @@ There are no ConfigDb schema changes. `ServerCluster.RedundancyMode` is only cro - Sequence-number wrap. - Encoded-byte fidelity of cloned payloads beyond the plan's bounded matrix (c7890). -## 9. Open questions for the owner (short) +## 9. Owner answers (2026-11-13, #717) -1. **Standby writes.** Keep today's optimistic Good + revert (D7), or refuse synchronously with `BadInvalidState` - under Warm, as HistoryUpdate and native ack already do? *Recommend synchronous refusal under Warm only.* -2. **Crash-failover gap.** A SIGKILLed Primary stays an unreachable member until auto-down, ~25–30 s. Under (2)/(3) - the survivor stays Standby for that whole window, with no Good data anywhere, where today the Secondary keeps - serving. Should the survivor be allowed to promote when the Primary is *unreachable* (the #672 pattern, default - on), accepting dual-Active during a real partition? *Recommend yes, behind - `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable`.* -3. **Standby status code.** `BadNoCommunication` is spike-proven and is plain Bad in ScadaBridge, but it is the - same code as real device comms loss; only ServiceLevel tells them apart. Accept it, or use `BadOutOfService`? -4. **Scripted Part 9 methods on the Standby.** They are accepted (D7) but their event is suppressed (D3), so the - caller sees Good and no event until promotion or refresh. Accept, or refuse with `BadInvalidState` under Warm? -5. **ConfigDb `Hot`.** Narrow the CHECK to `None/Warm` with a migration, or leave it and rely on the §3 mismatch - Error? -6. **Bands.** Are Standby 90/50 and boot 1 acceptable (§4.3)? +The draft carried six open questions. The owner answered them on 2026-11-13 and approved the spec and plan +(`2be43aa9`). The answers are applied in the sections above. + +1. **Standby writes.** Refuse synchronously with `BadInvalidState` under `Warm` only (§4.4). `Unset`/`None` keep the + #5 optimistic Good + revert. +2. **Crash-failover gap.** Yes: the survivor may promote while the Primary is unreachable, behind + `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable` (default on) (§4.1). Dual-Active during a real partition is accepted. +3. **Standby status code.** `BadOutOfService`, not the spike's `BadNoCommunication`, so a standby is distinguishable + from real device comms loss without reading ServiceLevel. Re-proven on 2.0.1 in T0. +4. **Scripted Part 9 methods on the Standby.** Refuse with `BadInvalidState` under `Warm` (§4.4). +5. **ConfigDb `Hot`.** Leave the CHECK; rely on the §3 mismatch Error. No migration. +6. **Bands.** Accepted as drafted: Standby 90 (healthy) / 50 (degraded), boot 1 (§4.3). ## 10. What must be re-proven on SDK 2.0.1 diff --git a/docs/plans/2026-11-11-warm-redundancy-717-plan.md b/docs/plans/2026-11-11-warm-redundancy-717-plan.md index e7c5d510c..9fc3f6c03 100644 --- a/docs/plans/2026-11-11-warm-redundancy-717-plan.md +++ b/docs/plans/2026-11-11-warm-redundancy-717-plan.md @@ -1,7 +1,9 @@ # Warm non-transparent redundancy (Gitea #717): implementation plan -Status: **DRAFT for owner approval**, together with `2026-11-11-warm-redundancy-717-design.md` (referred to below -as "the spec", with § references). No task starts until both documents and the §9 open questions are settled. +Status: **APPROVED by the owner 2026-11-13** (#717), together with `2026-11-11-warm-redundancy-717-design.md` +(referred to below as "the spec", with § references). The spec's §9 records the owner's answers: writes and Part 9 +methods are refused `BadInvalidState` under Warm, promote-while-unreachable is on by default, the standby status is +`BadOutOfService`, there is no ConfigDb migration, and the bands are as drafted. Every task ships **dark**: `OpcUa:Redundancy:Mode` defaults to `Unset`, so merging any task changes no deployed behaviour. @@ -44,7 +46,7 @@ D4 itself stays an owner decision. **Files:** - `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`: options at `:29-121`. Add - `OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable` if Q2 = yes). + `OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable`, default `true`, owner answer Q2). - `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`. - A post-load check in `StartAsync` (`:218-250`) against the effective `ApplicationConfiguration.ApplicationUri`. @@ -103,7 +105,7 @@ D4 itself stays an owner decision. - `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs`: add `ComputeWarm(inputs, isActive)`. `Compute` itself is unchanged. - `OpcUaPublishActor.RecomputeServiceLevel` (`:1576-1636`) branches on mode. It subscribes to `MemberUp/Removed` - (and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots. + and to reachability (owner answer Q2) so the verdict re-derives on membership changes, not only on snapshots. **Tests:** - A truth table covering every row of §4.3 and the invariant *max(Standby) < min(Active serving)*. @@ -132,9 +134,13 @@ D4 itself stays an owner decision. - `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs`: the hook decided in T0b, reading a node-manager-wide volatile projection flag. Stored `Value` is never modified. - `OtOpcUaNodeManager` implements `IStandbyProjection.SetDataProjection`. +- Owner answer Q1: under Warm, the ordinary-tag write handler refuses a write on the Standby with `BadInvalidState` + before it reaches the node-write gateway. `Unset`/`None` keep the #5 optimistic path. **Tests (wire):** -- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadNoCommunication`/null. +- Warm Standby: a Write returns `BadInvalidState` and the gateway is never called; on the Active node the same Write + reaches the gateway; under `Unset` a node that is not Primary still takes today's optimistic path. +- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadOutOfService`/null. - A Standby Read of a condition child, `i=2267` or `i=3709` stays Good. - A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected. - Acquisition continues: flip to Active and the latest value is read without any new driver publish. @@ -150,7 +156,7 @@ D4 itself stays an owner decision. disturbing the ACL filtering or `_liveItems`. **Tests (wire):** port the spikes' assertion matrix to product code. -- Queued Good samples, then demotion, then the first Publish carries `BadNoCommunication`. +- Queued Good samples, then demotion, then the first Publish carries `BadOutOfService`. - Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same projected payload. - Good delivered while Active, then demotion, then Republish returns the original Good. @@ -169,7 +175,7 @@ D4 itself stays an owner decision. disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped. **Tests (wire):** -- Demotion delivers exactly one `BadNoCommunication` per item, then silence while values keep changing internally. +- Demotion delivers exactly one `BadOutOfService` per item, then silence while values keep changing internally. - Promotion delivers the current Good value without a driver publish. - An ACL-stood-down item receives nothing either way. - A DeadbandFilter item still receives the promotion value. @@ -182,8 +188,12 @@ D4 itself stays an owner decision. - `IStandbyProjection.RefreshAllEventItems()` runs the existing `ConditionRefresh` (`:6586`) Server-object path and `RefreshDirectItems` (`:6974`) over every live event item, from `_allEventsItems` and `_eventItems`. - `ReportToServerObject` (`:6060`) is untouched. +- Owner answer Q4: under Warm, every Part 9 condition method (native or scripted) is refused `BadInvalidState` on + the Standby before it changes condition state. **Tests (wire, ACL off and on):** +- Warm Standby: Acknowledge and a shelving method on a scripted condition return `BadInvalidState` and route no + command; the Active node accepts them; `Unset` keeps today's scripted-method acceptance. - A new alarm on the Standby reaches no Server-object, folder or direct item. - Model-change and audit events are still delivered on the Standby. - A client ConditionRefresh **and** ConditionRefresh2 on the Standby return Start, retained, End (D5). @@ -210,7 +220,7 @@ D4 itself stays an owner decision. ## T11: ScadaBridge prerequisite (cross-repo) -- File a ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes +- **Filed as ScadaBridge#138** (2026-11-13). A ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes ServiceLevel-aware: monitor `i=2267`, and fail over when it is below 200 for a configurable window while the other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items (spec §5). @@ -245,14 +255,14 @@ everything yourself. Use `otopcua-cli` from `src/Client/ZB.MOM.WW.OtOpcUa.Client |---|---|---| | L1 metadata | `redundancy` on :4840 and :4841 | `RedundancySupport=Warm`. ServerUriArray is self first and differs per node. Same URIs in both. | | L2 bands | `read -n i=2267` on both | Primary 250, Standby 90 | -| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadNoCommunication`/null; the Primary shows Good at the same moment | +| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadOutOfService`/null; the Primary shows Good at the same moment | | L4 standby alarms | `alarms` on both. Trip a scripted alarm via its source tag. | Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5). | -| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadNoCommunication`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). | -| L6 crash | `docker kill central-` | The survivor goes Active within the gap Q2 selects. Record the measured gap. | +| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadOutOfService`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). | +| L6 crash | `docker kill central-` | The survivor goes Active once the Primary is unreachable (`PromoteWhilePrimaryUnreachable`, owner answer Q2). Record the measured gap. | | L7 rejoin | Restart the killed node | It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90. | | L8 standalone | site-b-2 (or a one-node overlay) with `Mode=None` | `RedundancySupport=None`, no `i=11314`, data Good | | L9 Unset control | Another node left on `Unset` | Byte-identical to pre-change: SL 240/250, SDK-default `i=3709` | -| L10 writes/history | Write and HistoryRead on the Standby | Behaviour per Q1. HistoryRead is Good. | +| L10 writes/history | Write and HistoryRead on the Standby | The write is refused `BadInvalidState` (owner answer Q1); HistoryRead is Good. | Record evidence in `docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md`. Do **not** mark G8/G9 (#615); they need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).