dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(717): record the owner's 2026-11-13 answers in the Warm redundancy spec and plan

Browse Source 
Standby writes and Part 9 methods refused BadInvalidState under Warm, promote while the Primary is
unreachable (OpcUa:Redundancy:PromoteWhilePrimaryUnreachable, default on), standby status code
BadOutOfService, no ConfigDb migration, bands as drafted. ScadaBridge#138 is the filed prerequisite.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 02:43:00 -05:00
parent c0ba450d46
commit a42c9d7f7c
2 changed files with 60 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-11-11-warm-redundancy-717-design.md
+38 -28
View File
@@ -1,8 +1,9 @@
# Warm non-transparent redundancy with a degraded standby (Gitea #717): design spec
Status: **DRAFT for owner approval.** No product code changes until this spec and the companion plan
(`2026-11-11-warm-redundancy-717-plan.md`) are approved. Parent gate: #615 (G8 CTT, G9 production-client
cutover). These stay open and separate from everything here.
Status: **APPROVED by the owner 2026-11-13** (#717), together with the companion plan
(`2026-11-11-warm-redundancy-717-plan.md`). The §9 questions are answered; the answers are recorded in §9 and
applied throughout. Parent gate: #615 (G8 CTT, G9 production-client cutover). These stay open and separate from
everything here.
## 1. Owner decisions this spec builds on (not re-opened)
@@ -69,6 +70,15 @@ this node — the node is **Standby**, whatever the member count. This is strict
lone-node allow (`PrimaryGatePolicy.cs:21`): a lone node goes Active within one debounce (~250 ms) of its own
singleton's first publish. That gate is **not** changed for writes/acks (D7). The Warm verdict is a separate output.
**Promote while the Primary is unreachable (owner answer Q2, `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable`,
default `true`).** A SIGKILLed Primary stays an Up-but-unreachable member until `auto-down` removes it (~25–30 s),
and the singleton usually died with it, so (2) and (3) would hold the survivor Standby for that whole window. With
the option on, (2) and (3) are evaluated with **unreachable** members excluded: (2′) self is the oldest Up `driver`
member among the members this node can reach, and (3′) the latest accepted snapshot names either self or an
**unreachable** member as `Primary`. This is the #672 pattern (`ContinuousHistorization:CaptureWhilePrimaryUnreachable`).
A node that has never accepted a snapshot is still Standby. The cost is dual-Active while a live Primary is merely
unreachable, and for the whole of a real partition.
Accepted trade: an `auto-down` partition can give two Active nodes, as it already does for the data-plane gate
(`docs/Redundancy.md` §Split-brain). For reads this is benign, because both nodes serve the same Good data.
@@ -118,15 +128,15 @@ creation from the node instance and the `AttributeId`, never from namespace or C
| Service | Standby behaviour | Status the client sees |
|---|---|---|
| Read (ordinary tag `Value`) | Projected. The stored value is kept internally and continues to update from acquisition. | `BadNoCommunication`, `Value = null`, `ServerTimestamp = now`. Operation result Good. |
| Read (ordinary tag `Value`) | Projected. The stored value is kept internally and continues to update from acquisition. | `BadOutOfService`, `Value = null`, `ServerTimestamp = now`. Operation result Good. |
| Read (other attributes, condition children, ns=0 incl. ServiceLevel/ServerStatus/ServerRedundancy) | Unchanged | Good / as today |
| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification `BadNoCommunication`/null |
| Existing data items at demotion | One `BadNoCommunication`/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. **D4:** any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to `BadNoCommunication`/null at Publish/Republish (§4.6). | `BadNoCommunication` |
| Write | **Unchanged (D7):** the existing gate rejects off-Primary through the #5 self-correction (synchronous Good, then revert + audit). Under Warm the revert is invisible because the value is projected Bad anyway. See open question Q1. | as today |
| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification `BadOutOfService`/null |
| Existing data items at demotion | One `BadOutOfService`/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. **D4:** any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to `BadOutOfService`/null at Publish/Republish (§4.6). | `BadOutOfService` |
| Write (ordinary tag `Value`) | **Refused synchronously under Warm only (owner answer Q1).** The node manager refuses the write before it reaches the driver; nothing is reverted and no device is touched. `Unset`/`None` keep today's #5 self-correction (synchronous Good, then revert + audit) off-Primary (D7). | `BadInvalidState` |
| HistoryRead | Unchanged. Served from the shared backend, which is not live acquisition. | as today |
| HistoryUpdate | Unchanged | `BadInvalidState` off-Primary (#674) |
| Browse / TranslateBrowsePaths / RegisterNodes | Unchanged | as today |
| Call: Part 9 methods | Unchanged (D7): native ack `BadInvalidState` off-Primary; scripted ack/shelve accepted. The resulting condition event counts as a new alarm event and is suppressed (§4.5). See Q4. | as today |
| Call: Part 9 methods | **Refused under Warm (owner answer Q4).** Every Part 9 condition method (Acknowledge, Confirm, AddComment, Enable/Disable, the shelving methods) on a native or scripted condition is refused before it changes any state. `Unset`/`None` keep today's behaviour: native ack `BadInvalidState` off-Primary, scripted ack/shelve accepted (D7). | `BadInvalidState` |
| Event subscriptions (Server object, folders, direct) | Stay established, in Reporting mode, with the ACL unchanged | — |
| Model-change / audit events | **Not suppressed.** They are not alarm events, and browse consistency needs them. | — |
| Session / secure channel / ServerStatus.State | Unchanged (`Running`) | — |
@@ -168,7 +178,7 @@ mechanism that passed the provenance spike (c7851, 66 assertions) and the delive
cached SDK objects are never mutated.
- If the node is Active at first delivery, the original message is the choice.
- If the node is Standby, a detached clone is the choice, with every `OrdinaryTag` notification replaced by
`BadNoCommunication`/null.
`BadOutOfService`/null.
- A Republish returns the recorded choice. First delivery *through* Republish is decided the same way.
- A short lock orders the choice against the role flip. No lock spans an await.
3. **Out of scope of the rewrite**, by D4: messages already in flight before the demotion step, and Republish of
@@ -197,7 +207,7 @@ this order.
2. Switch the delivery-history choice to Standby. This is the D4 boundary.
3. Suppress alarm delivery.
4. Close the read projection.
5. Queue `BadNoCommunication` to each live ordinary-tag item.
5. Queue `BadOutOfService` to each live ordinary-tag item.
Invariant: **an Active-band ServiceLevel is never visible while the projection is closed.** If the ServiceLevel
publish fails (#719 retry path), a promotion stops before step 6 and retries. A demotion continues anyway: failing
@@ -217,25 +227,27 @@ ScadaBridge consumes this server as an OPC UA client, through `RealOpcUaClient`
`FailoverRetryCount` consecutive *unstable disconnects*
(`../ScadaBridge/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs:655-690`;
`RealOpcUaClient.cs` sits in that project's `Adapters/`).
`BadNoCommunication` is not a connection-level status (`:1974-1983`), so it maps to plain Bad quality, and since
`BadOutOfService` is not a connection-level status (`:1974-1983`), so it maps to plain Bad quality, and since
ScadaBridge#112 a Bad-with-null update keeps the last-known value. Nothing in ScadaBridge reads `ServiceLevel`,
`RedundancySupport` or `ServerUriArray`.
- **Consequence.** A ScadaBridge connection that lands on the Standby will sit on Bad quality indefinitely on a
healthy session, and never fail over. Today the same connection gets Good data at 240. Warm is therefore an
**outage for ScadaBridge** unless ScadaBridge becomes ServiceLevel-aware: monitor `i=2267` and treat a Degraded
band (< 200) sustained for N s, while the other endpoint reports a higher level, as a failover trigger. That
change must ship and be deployed first. The plan files it as a ScadaBridge issue (`cross-repo`, blocking).
change must ship and be deployed first. It is filed as **ScadaBridge#138** (`cross-repo`, blocking); no tier sets
`Mode=Warm` until it is deployed there.
- **Alarms are compatible.** ScadaBridge already handles a RefreshStart/RefreshEnd replay as a snapshot
(`RealOpcUaClient.cs:72`) and triggers its own ConditionRefresh on (re)subscribe (`:1179-1193`). Under D5 that
refresh returns retained conditions on the Standby too. The promotion refresh arrives as an unsolicited
Start/End pair on existing items; this needs a ScadaBridge-side test.
- **Status mapping is compatible.** ScadaBridge#117 maps by severity bits, so `BadNoCommunication` is Bad.
- **Status mapping is compatible.** ScadaBridge#117 maps by severity bits, so `BadOutOfService` is Bad.
## 6. Configuration surface
| Key | Default | Notes |
|---|---|---|
| `OpcUa:Redundancy:Mode` | `Unset` | `Unset` / `None` / `Warm`. `Hot` and unknown values fail host start. Startup-static. Flip **both** pair nodes in one window, after both run a revision-capable build (§4.2). |
| `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable` | `true` | Warm only. Lets the survivor go Active while the Primary is unreachable but not yet downed (§4.1, owner answer Q2). |
| `OpcUa:PeerApplicationUris` | `[]` | Existing key, now also the `ServerUriArray` peer source. Required (≥ 1) for `Warm`, must be empty for `None`. |
| `OpcUa:ApplicationUri` | `urn:OtOpcUa` | Must be unique within the pair under `Warm` (validated against the effective config). |
@@ -269,22 +281,20 @@ There are no ConfigDb schema changes. `ServerCluster.RedundancyMode` is only cro
- Sequence-number wrap.
- Encoded-byte fidelity of cloned payloads beyond the plan's bounded matrix (c7890).
## 9. Open questions for the owner (short)
## 9. Owner answers (2026-11-13, #717)
1. **Standby writes.** Keep today's optimistic Good + revert (D7), or refuse synchronously with `BadInvalidState`
under Warm, as HistoryUpdate and native ack already do? *Recommend synchronous refusal under Warm only.*
2. **Crash-failover gap.** A SIGKILLed Primary stays an unreachable member until auto-down, ~25–30 s. Under (2)/(3)
the survivor stays Standby for that whole window, with no Good data anywhere, where today the Secondary keeps
serving. Should the survivor be allowed to promote when the Primary is *unreachable* (the #672 pattern, default
on), accepting dual-Active during a real partition? *Recommend yes, behind
`OpcUa:Redundancy:PromoteWhilePrimaryUnreachable`.*
3. **Standby status code.** `BadNoCommunication` is spike-proven and is plain Bad in ScadaBridge, but it is the
same code as real device comms loss; only ServiceLevel tells them apart. Accept it, or use `BadOutOfService`?
4. **Scripted Part 9 methods on the Standby.** They are accepted (D7) but their event is suppressed (D3), so the
caller sees Good and no event until promotion or refresh. Accept, or refuse with `BadInvalidState` under Warm?
5. **ConfigDb `Hot`.** Narrow the CHECK to `None/Warm` with a migration, or leave it and rely on the §3 mismatch
Error?
6. **Bands.** Are Standby 90/50 and boot 1 acceptable (§4.3)?
The draft carried six open questions. The owner answered them on 2026-11-13 and approved the spec and plan
(`2be43aa9`). The answers are applied in the sections above.
1. **Standby writes.** Refuse synchronously with `BadInvalidState` under `Warm` only (§4.4). `Unset`/`None` keep the
#5 optimistic Good + revert.
2. **Crash-failover gap.** Yes: the survivor may promote while the Primary is unreachable, behind
`OpcUa:Redundancy:PromoteWhilePrimaryUnreachable` (default on) (§4.1). Dual-Active during a real partition is accepted.
3. **Standby status code.** `BadOutOfService`, not the spike's `BadNoCommunication`, so a standby is distinguishable
from real device comms loss without reading ServiceLevel. Re-proven on 2.0.1 in T0.
4. **Scripted Part 9 methods on the Standby.** Refuse with `BadInvalidState` under `Warm` (§4.4).
5. **ConfigDb `Hot`.** Leave the CHECK; rely on the §3 mismatch Error. No migration.
6. **Bands.** Accepted as drafted: Standby 90 (healthy) / 50 (degraded), boot 1 (§4.3).
## 10. What must be re-proven on SDK 2.0.1
docs/plans/2026-11-11-warm-redundancy-717-plan.md
+22 -12
View File
@@ -1,7 +1,9 @@
# Warm non-transparent redundancy (Gitea #717): implementation plan
Status: **DRAFT for owner approval**, together with `2026-11-11-warm-redundancy-717-design.md` (referred to below
as "the spec", with § references). No task starts until both documents and the §9 open questions are settled.
Status: **APPROVED by the owner 2026-11-13** (#717), together with `2026-11-11-warm-redundancy-717-design.md`
(referred to below as "the spec", with § references). The spec's §9 records the owner's answers: writes and Part 9
methods are refused `BadInvalidState` under Warm, promote-while-unreachable is on by default, the standby status is
`BadOutOfService`, there is no ConfigDb migration, and the bands are as drafted.
Every task ships **dark**: `OpcUa:Redundancy:Mode` defaults to `Unset`, so merging any task changes no deployed
behaviour.
@@ -44,7 +46,7 @@ D4 itself stays an owner decision.
**Files:**
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`: options at `:29-121`. Add
`OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable` if Q2 = yes).
`OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable`, default `true`, owner answer Q2).
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`.
- A post-load check in `StartAsync` (`:218-250`) against the effective `ApplicationConfiguration.ApplicationUri`.
@@ -103,7 +105,7 @@ D4 itself stays an owner decision.
- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs`: add `ComputeWarm(inputs, isActive)`.
`Compute` itself is unchanged.
- `OpcUaPublishActor.RecomputeServiceLevel` (`:1576-1636`) branches on mode. It subscribes to `MemberUp/Removed`
(and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots.
and to reachability (owner answer Q2) so the verdict re-derives on membership changes, not only on snapshots.
**Tests:**
- A truth table covering every row of §4.3 and the invariant *max(Standby) < min(Active serving)*.
@@ -132,9 +134,13 @@ D4 itself stays an owner decision.
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs`: the hook decided in T0b, reading a
node-manager-wide volatile projection flag. Stored `Value` is never modified.
- `OtOpcUaNodeManager` implements `IStandbyProjection.SetDataProjection`.
- Owner answer Q1: under Warm, the ordinary-tag write handler refuses a write on the Standby with `BadInvalidState`
before it reaches the node-write gateway. `Unset`/`None` keep the #5 optimistic path.
**Tests (wire):**
- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadNoCommunication`/null.
- Warm Standby: a Write returns `BadInvalidState` and the gateway is never called; on the Active node the same Write
reaches the gateway; under `Unset` a node that is not Primary still takes today's optimistic path.
- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadOutOfService`/null.
- A Standby Read of a condition child, `i=2267` or `i=3709` stays Good.
- A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected.
- Acquisition continues: flip to Active and the latest value is read without any new driver publish.
@@ -150,7 +156,7 @@ D4 itself stays an owner decision.
disturbing the ACL filtering or `_liveItems`.
**Tests (wire):** port the spikes' assertion matrix to product code.
- Queued Good samples, then demotion, then the first Publish carries `BadNoCommunication`.
- Queued Good samples, then demotion, then the first Publish carries `BadOutOfService`.
- Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same
projected payload.
- Good delivered while Active, then demotion, then Republish returns the original Good.
@@ -169,7 +175,7 @@ D4 itself stays an owner decision.
disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped.
**Tests (wire):**
- Demotion delivers exactly one `BadNoCommunication` per item, then silence while values keep changing internally.
- Demotion delivers exactly one `BadOutOfService` per item, then silence while values keep changing internally.
- Promotion delivers the current Good value without a driver publish.
- An ACL-stood-down item receives nothing either way.
- A DeadbandFilter item still receives the promotion value.
@@ -182,8 +188,12 @@ D4 itself stays an owner decision.
- `IStandbyProjection.RefreshAllEventItems()` runs the existing `ConditionRefresh` (`:6586`) Server-object path and
`RefreshDirectItems` (`:6974`) over every live event item, from `_allEventsItems` and `_eventItems`.
- `ReportToServerObject` (`:6060`) is untouched.
- Owner answer Q4: under Warm, every Part 9 condition method (native or scripted) is refused `BadInvalidState` on
the Standby before it changes condition state.
**Tests (wire, ACL off and on):**
- Warm Standby: Acknowledge and a shelving method on a scripted condition return `BadInvalidState` and route no
command; the Active node accepts them; `Unset` keeps today's scripted-method acceptance.
- A new alarm on the Standby reaches no Server-object, folder or direct item.
- Model-change and audit events are still delivered on the Standby.
- A client ConditionRefresh **and** ConditionRefresh2 on the Standby return Start, retained, End (D5).
@@ -210,7 +220,7 @@ D4 itself stays an owner decision.
## T11: ScadaBridge prerequisite (cross-repo)
- File a ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes
- **Filed as ScadaBridge#138** (2026-11-13). A ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes
ServiceLevel-aware: monitor `i=2267`, and fail over when it is below 200 for a configurable window while the
other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items
(spec §5).
@@ -245,14 +255,14 @@ everything yourself. Use `otopcua-cli` from `src/Client/ZB.MOM.WW.OtOpcUa.Client
|---|---|---|
| L1 metadata | `redundancy` on :4840 and :4841 | `RedundancySupport=Warm`. ServerUriArray is self first and differs per node. Same URIs in both. |
| L2 bands | `read -n i=2267` on both | Primary 250, Standby 90 |
| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadNoCommunication`/null; the Primary shows Good at the same moment |
| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadOutOfService`/null; the Primary shows Good at the same moment |
| L4 standby alarms | `alarms` on both. Trip a scripted alarm via its source tag. | Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5). |
| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadNoCommunication`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). |
| L6 crash | `docker kill central-<primary>` | The survivor goes Active within the gap Q2 selects. Record the measured gap. |
| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadOutOfService`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). |
| L6 crash | `docker kill central-<primary>` | The survivor goes Active once the Primary is unreachable (`PromoteWhilePrimaryUnreachable`, owner answer Q2). Record the measured gap. |
| L7 rejoin | Restart the killed node | It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90. |
| L8 standalone | site-b-2 (or a one-node overlay) with `Mode=None` | `RedundancySupport=None`, no `i=11314`, data Good |
| L9 Unset control | Another node left on `Unset` | Byte-identical to pre-change: SL 240/250, SDK-default `i=3709` |
| L10 writes/history | Write and HistoryRead on the Standby | Behaviour per Q1. HistoryRead is Good. |
| L10 writes/history | Write and HistoryRead on the Standby | The write is refused `BadInvalidState` (owner answer Q1); HistoryRead is Good. |
Record evidence in `docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md`. Do **not** mark G8/G9 (#615); they
need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).