Spec: per-shape contract (Unset/None/Warm), RedundancySupport + ServerUriArray graph, ServiceLevel bands, per-service standby behaviour, alarm suppression + ConditionRefresh (D5), strict first-delivery (D4), transition sequencing, ordered snapshot revisions, ScadaBridge impact, SDK 2.0.1 re-proof list, open questions. Plan: T0-T13 with wire tests, docker-dev live gate, rollout/rollback. Docs only. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
25 KiB
Warm non-transparent redundancy with a degraded standby (Gitea #717): design spec
Status: DRAFT for owner approval. No product code changes until this spec and the companion plan
(2026-11-11-warm-redundancy-717-plan.md) are approved. Parent gate: #615 (G8 CTT, G9 production-client
cutover). These stay open and separate from everything here.
1. Owner decisions this spec builds on (not re-opened)
| # | Decision | Source |
|---|---|---|
| D1 | Advertise Warm for a 2-node pair, with an explicit degraded-standby contract; None for standalone. | #717 c7796 |
| D2 | Standby ServiceLevel in the degraded band 2–199; ordinary tag Reads/subscriptions on the standby degrade, while internal acquisition continues. Today's Secondary serves live Good data at 240, so it cannot just be relabelled. | #717 c7796, c7801, c9329 |
| D3 | Event subscriptions stay established on the standby; new alarm events are suppressed there; current retained conditions are refreshed on promotion. | #717 c7804 |
| D4 | Strict first delivery. After demotion, any Publish response delivered for the first time must not contain Good ordinary-tag values. That includes samples queued before demotion and batches built before it. Republish history and responses already in flight are preserved. | #717 c7849 |
| D5 | A client-initiated ConditionRefresh on the Warm standby returns its current retained conditions. Only new alarm events are suppressed. | #717 c11362 (2026-11-11) |
| D6 | The queue-only spike gate must not be adopted unchanged, because of late-overflow and filtered-retain effects. Product alarm suppression happens before SDK event-field projection. Per-tag read projection is the recommended data seam. Every ACL, direct-item and Server-object refresh path needs coverage. Role snapshots need an ordered revision. | #717 c7812, c7853, issue body |
| D7 | Existing mutation and election policies are preserved: the write/native-ack/HistoryUpdate Primary gates, the oldest-Up-member election, and auto-down. |
#717 c7794 |
2. Where the code is today
- Metadata. No code sets
ServerRedundancy.RedundancySupport(i=3709), so it reads the SDK default.Server.ServerRedundancyis the baseServerRedundancyTypeand has noServerUriArray(i=11314) (docs/Redundancy.md:9). Peers are published only inServer.ServerArray(i=2254), by appending toIServerInternal.ServerUrisafter start (OpcUaApplicationHost.cs:378-414, called at:241). - Self URI. It comes from
OpcUa:ApplicationUri, which defaults tourn:OtOpcUa(OpcUaApplicationHost.cs:35). IfApplicationConfigPathis set, the loaded XML wins (:602-608). docker-dev sets neitherOpcUa__ApplicationUrinorPeerApplicationUrison central-1/2 (docker-dev/docker-compose.yml:289-481), so both centrals advertiseurn:OtOpcUa. The seededClusterNode.ApplicationUriusesurn:OtOpcUa:<NodeId>(docker-dev/seed/seed-clusters.sql:13). - ServiceLevel.
ServiceLevelCalculator.Computereturns 240 + 10 for the Primary (ServiceLevelCalculator.cs:44-64). The standby therefore sits in the healthy band.OpcUaPublishActor.RecomputeServiceLevelfeeds it (OpcUaPublishActor.cs:1576-1636), and the result is published through the #719-fixed retain/replay path (:1468-1488,SdkServiceLevelPublisher.cs:36-60). - Acquisition is role-independent. Every driver node spawns its drivers and subscribes whatever its role (#717 c7794). Today Primary gating covers only writes, native acks, the alerts emit, alarm-history drain, historization capture and HistoryUpdate (
PrimaryGatePolicy.cs:21,56,97,155;docs/Redundancy.md:554-600). - Data variables. Every ordinary-tag data variable (raw tag, UNS reference, virtual tag, Calculation tag) is an
EquipmentVariableStatecreated at exactly one site (OtOpcUaNodeManager.cs:4060). Values are pushed into it withPublishValue(EquipmentVariableState.cs:28-35). - Alarm events. Every condition event goes through
ReportConditionEvent(OtOpcUaNodeManager.cs:1376) and thenReportEventWithoutBlocking(:1507). The callers are engine/driver projection (:1254,:1341), method transitions (:2842) and restore re-report (:4659). Server-object items get per-item, ACL-filtered dispatch (#586). Folder and direct items get per-item delivery (#826/#855). Refresh paths:ConditionRefresh(:6586),RefreshDirectItems(:6974) and the #854 cut. Model-change and audit events useReportToServerObject(:6060), not this path. - Role snapshots are unordered.
RedundancyStateActoris a cluster singleton. It publishesRedundancyStateChanged(Nodes, CorrelationId)on DPS, debounced at 250 ms and heartbeated every 10 s (RedundancyStateActor.cs:25,33,105-113).AsOfUtcis the publisher's wall clock (:184) andCorrelationIdis random (:110). Subscribers replace their cached snapshot unconditionally (OpcUaPublishActor.cs:1494-1498,DriverHostActor.cs:2555-2562). After a singleton handover, a late heartbeat from the old incarnation can overwrite the new incarnation's snapshot. Nothing establishes the order, and there is no lease. - Config.
ServerCluster.RedundancyMode(None/Warm/Hot) exists in ConfigDb. It is CHECK-coupled to NodeCount (OtOpcUaConfigDbContext.cs:109-111;DraftValidator.cs:958-962) and travels in the artifact'sClustersarray (ConfigComposer.cs:67), but no runtime code reads it.
3. Contract per deployment shape
| Shape | OpcUa:Redundancy:Mode |
RedundancySupport (i=3709) |
ServerRedundancy type |
ServerUriArray (i=11314) |
ServerArray (i=2254) |
Standby behaviour |
|---|---|---|---|---|---|---|
| Today / not opted in | Unset (default) |
SDK default (unchanged) | ServerRedundancyType (2034) |
absent | self + peers (unchanged) | none; Secondary serves Good at 240 (unchanged) |
| Standalone | None |
None (0) |
ServerRedundancyType (2034) |
absent | self only | never; always serves |
| Pair | Warm |
Warm (2) |
NonTransparentRedundancyType (2039) |
[effectiveSelfUri, peer…], read-only String[], self first (decision #155, docs/v2/plan.md:930) |
unchanged | §4 |
- The metadata is startup-static. The graph is built before node-manager registration through
StandardServer.CreateMainNodeManagerFactory+ConfigurationNodeManager.AddBehaviourToPredefinedNode. That route is spike-proven on SDK 1.5.378.106 (#717 c7791) and becomesAddBehaviourToPredefinedNodeAsyncon 2.0.1. The passive predefinedi=11314the SDK visits later must resolve to the same constructed instance, or the wire value is null (c7791). Changing mode or peers requires a restart. effectiveSelfUriis the effectiveApplicationConfiguration.ApplicationUriafter any XML load, never the options default (c7787).Hotis not advertisable: continuous acquisition alone does not meet Part 4 §6.6.2.4 Hot (c7794).Mode=Hotfails host start.- Startup validation (
ValidateOnStart) runs forWarmand fails host start when any of these holds: there is no peer; a peer URI equals the self URI; a URI is duplicated; thedriverrole is missing. - Deploy-time cross-check: if the artifact's
ServerCluster.RedundancyModefor this node's cluster disagrees with the local mode (NodeCount=1/Noneagainst localWarm, or the reverse), the node logs an Error and records the metricotopcua.redundancy.mode_mismatch. The node keeps running in its local mode. Local config is the authority, because it is the only value available before SDK start.
4. Warm role contract
4.1 Role verdict (who is Active)
A Warm node is Active iff all of the following hold. Otherwise it is Standby.
- The local
SelfMemberis Up andEndpointListeningis true (#762). - The node's own
Cluster.Statenames it the oldest Updrivermember. This is the same function the singleton uses (RedundancyStateActor.SelectOldestUpMemberOfRole,:172), so the verdict cannot drift from singleton placement (D7). - The latest accepted snapshot (§4.2) names it
Primary.
Any disagreement between (2) and (3) resolves to Standby. A Warning is logged if the disagreement lasts more than
two heartbeats (20 s). When the role is unknown — before the first accepted snapshot, or when the snapshot omits
this node — the node is Standby, whatever the member count. This is stricter than PrimaryGatePolicy's
lone-node allow (PrimaryGatePolicy.cs:21): a lone node goes Active within one debounce (~250 ms) of its own
singleton's first publish. That gate is not changed for writes/acks (D7). The Warm verdict is a separate output.
Accepted trade: an auto-down partition can give two Active nodes, as it already does for the data-plane gate
(docs/Redundancy.md §Split-brain). For reads this is benign, because both nodes serve the same Good data.
4.2 Ordered snapshots (the revision problem)
RedundancyStateChanged gains an additive Revision(PublisherUpNumber, IncarnationTick, Sequence):
PublisherUpNumberis theMember.UpNumberof the node that hosts the singleton. The singleton always lives on the oldest member, and every surviving member has a larger up-number than a departed oldest one. The value is therefore monotonic across handovers within one cluster incarnation.IncarnationTickisEnvironment.TickCount64at singletonPreStart. It is monotonic for a singleton that restarts in the same process.Sequenceincrements on every publish, including heartbeats.
Subscribers (OpcUaPublishActor, DriverHostActor) accept a snapshot only if its revision is lexicographically
greater than the last accepted one. Anything else is dropped and counted in
otopcua.redundancy.snapshot_rejected{reason}. A subscriber cannot outlive its cluster incarnation: a removed node
must restart its ActorSystem to rejoin. The plan pins this assumption with a test. A snapshot without a
revision (from a pre-#717 publisher during a rolling upgrade) is accepted for the existing gates but can never make
a Warm node Active. Warm therefore requires both pair nodes to be upgraded before it is turned on.
There is no time lease beyond this. Liveness comes from (2), which reads live gossip, and the existing 30 s staleness input to ServiceLevel. Demoting an Active node on snapshot staleness alone was rejected: the singleton lives on the Active node itself, so staleness there means a local fault, and demoting would turn it into a pair-wide data outage.
4.3 ServiceLevel bands (Part 4 §6.6.2.4.2)
| Node state (Warm) | Byte | Band |
|---|---|---|
| Down / Detached / endpoint failed | 0 | Maintenance (unchanged) |
| Boot, before first computed level | 1 (set in the startup graph; replaces SDK default 255) | NoData |
| Standby, degraded (stale or DB unreachable) | 50 | Degraded |
| Standby, healthy | 90 | Degraded |
| Active, critically degraded | 100 | Degraded (unchanged) |
| Active, stale | 200 | Healthy (unchanged) |
| Active, healthy | 250 | Healthy (unchanged: 240 + 10) |
Invariant: every Standby level is below every level an Active node serving data can publish (≥ 100). A
"highest ServiceLevel" client therefore never prefers a standby over a data-serving Active node. Unset and None
keep ServiceLevelCalculator.Compute byte-for-byte.
4.4 Behaviour on the Standby, per service
"Ordinary tag" means the Value attribute of an EquipmentVariableState. The category is captured at item
creation from the node instance and the AttributeId, never from namespace or ClientHandle (c7853).
| Service | Standby behaviour | Status the client sees |
|---|---|---|
Read (ordinary tag Value) |
Projected. The stored value is kept internally and continues to update from acquisition. | BadNoCommunication, Value = null, ServerTimestamp = now. Operation result Good. |
| Read (other attributes, condition children, ns=0 incl. ServiceLevel/ServerStatus/ServerRedundancy) | Unchanged | Good / as today |
| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification BadNoCommunication/null |
| Existing data items at demotion | One BadNoCommunication/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. D4: any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to BadNoCommunication/null at Publish/Republish (§4.6). |
BadNoCommunication |
| Write | Unchanged (D7): the existing gate rejects off-Primary through the #5 self-correction (synchronous Good, then revert + audit). Under Warm the revert is invisible because the value is projected Bad anyway. See open question Q1. | as today |
| HistoryRead | Unchanged. Served from the shared backend, which is not live acquisition. | as today |
| HistoryUpdate | Unchanged | BadInvalidState off-Primary (#674) |
| Browse / TranslateBrowsePaths / RegisterNodes | Unchanged | as today |
| Call: Part 9 methods | Unchanged (D7): native ack BadInvalidState off-Primary; scripted ack/shelve accepted. The resulting condition event counts as a new alarm event and is suppressed (§4.5). See Q4. |
as today |
| Event subscriptions (Server object, folders, direct) | Stay established, in Reporting mode, with the ACL unchanged | — |
| Model-change / audit events | Not suppressed. They are not alarm events, and browse consistency needs them. | — |
| Session / secure channel / ServerStatus.State | Unchanged (Running) |
— |
4.5 Alarms and events on the Standby
- Suppression point.
ReportConditionEventis split into stamp and deliver. Stamp sets EventId, Time and ReceiveTime, and runsClearChangeMasks. Deliver builds the snapshot and callsReportEventWithoutBlocking. On the Standby only stamp runs. Condition node state therefore stays current, and a later refresh carries a valid, ackable EventId. Because suppression sits before snapshot construction and before every SDK field projection, it covers the Server-object, folder and direct paths, ACL on or off, by construction. Overflow, filtered retain and the #838 deferral queue never see a suppressed event. TheRingEventQueueFactoryqueue is not used as the gate (D6). - Boundary. The decision is made at report time. An event whose report ran while Active is delivered even if delivery finishes after demotion: deferred (#838) or already in an item queue. This matches the spike's observed admission boundary (c7805). Republish of already-built event messages is unchanged.
- Client ConditionRefresh / ConditionRefresh2 on the Standby (D5) run the existing paths unchanged: Server
object (#854 cut),
RefreshDirectItems(#855), with ACL filtering as today. The replay is RefreshStart, the current retained conditions, then RefreshEnd. - Promotion refresh. On Standby → Active, the server runs a refresh on every live event item: Server-object
items, direct items and folder items. The two refresh entry points are reused, so ACL decisions and the #854/#855
cut apply. Clients see RefreshStart, retained conditions, RefreshEnd, then new events. This is spike-proven on
1.5 (c7805). A condition that went active and stopped being retained entirely while the node was Standby is
not replayed. That is standard refresh semantics, and the transition is not historized twice, because alarm
history is Primary-gated (
docs/Redundancy.md:572-573).
4.6 Strict first delivery (D4)
The item-level projection (§4.4) stops new Good samples. It cannot reach samples already queued in an item, nor
NotificationMessages the subscription has prebuilt or cached (c7846, c7850). For those, the design uses the
mechanism that passed the provenance spike (c7851, 66 assertions) and the delivery-history spike (c7852,
53 assertions):
- Provenance. Ordinary-tag data items are created through a custom sampled item. Its per-value
Publishstamps each newly createdMonitoredItemNotificationwith an immutable(NodeId, AttributeId, Category). The stamp survives handle modification, item deletion and reuse. - Delivery history.
OtOpcUaSdkServeroverridesPublishAsyncandRepublishAsync, calls base first, and records one first-delivery choice per cached message object. The association is weak and keyed by object; cached SDK objects are never mutated.- If the node is Active at first delivery, the original message is the choice.
- If the node is Standby, a detached clone is the choice, with every
OrdinaryTagnotification replaced byBadNoCommunication/null. - A Republish returns the recorded choice. First delivery through Republish is decided the same way.
- A short lock orders the choice against the role flip. No lock spans an await.
- Out of scope of the rewrite, by D4: messages already in flight before the demotion step, and Republish of messages first delivered while Active. Unknown-origin notifications are not rewritten; there is no implicit policy (c7853).
4.7 Transition sequencing
One actor, OpcUaPublishActor, owns the verdict, the ServiceLevel publish and a new IStandbyProjection sink
(in Commons.OpcUa, bound late like IServiceLevelPublisher). Each transition runs as one serialized handler, in
this order.
Promotion (Standby → Active)
- Open the read projection.
- For each live ordinary-tag item, queue the current stored value, ignoring filters. The precedent is the queue
pattern at
OtOpcUaNodeManager.cs:7303. - Switch the delivery-history choice to Active.
- Un-suppress alarm delivery.
- Run the promotion refresh (§4.5).
- Publish the Active ServiceLevel.
Demotion (Active → Standby)
- Publish the Standby ServiceLevel.
- Switch the delivery-history choice to Standby. This is the D4 boundary.
- Suppress alarm delivery.
- Close the read projection.
- Queue
BadNoCommunicationto each live ordinary-tag item.
Invariant: an Active-band ServiceLevel is never visible while the projection is closed. If the ServiceLevel publish fails (#719 retry path), a promotion stops before step 6 and retries. A demotion continues anyway: failing safe means degrading the data.
4.8 Standalone (None) and Unset
None never projects, never suppresses and never refreshes. It advertises RedundancySupport=None with no
ServerUriArray, and uses today's ServiceLevel. Unset is byte-for-byte today's server, including the documented
unmet A4/A5b gates.
5. ScadaBridge impact (cross-repo, blocking for any Warm flip)
ScadaBridge consumes this server as an OPC UA client, through RealOpcUaClient and DataConnectionActor.
- Failover is connection-driven, not ServiceLevel-driven. ScadaBridge swaps primary to backup only after
FailoverRetryCountconsecutive unstable disconnects (../ScadaBridge/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs:655-690;RealOpcUaClient.cssits in that project'sAdapters/).BadNoCommunicationis not a connection-level status (:1974-1983), so it maps to plain Bad quality, and since ScadaBridge#112 a Bad-with-null update keeps the last-known value. Nothing in ScadaBridge readsServiceLevel,RedundancySupportorServerUriArray. - Consequence. A ScadaBridge connection that lands on the Standby will sit on Bad quality indefinitely on a
healthy session, and never fail over. Today the same connection gets Good data at 240. Warm is therefore an
outage for ScadaBridge unless ScadaBridge becomes ServiceLevel-aware: monitor
i=2267and treat a Degraded band (< 200) sustained for N s, while the other endpoint reports a higher level, as a failover trigger. That change must ship and be deployed first. The plan files it as a ScadaBridge issue (cross-repo, blocking). - Alarms are compatible. ScadaBridge already handles a RefreshStart/RefreshEnd replay as a snapshot
(
RealOpcUaClient.cs:72) and triggers its own ConditionRefresh on (re)subscribe (:1179-1193). Under D5 that refresh returns retained conditions on the Standby too. The promotion refresh arrives as an unsolicited Start/End pair on existing items; this needs a ScadaBridge-side test. - Status mapping is compatible. ScadaBridge#117 maps by severity bits, so
BadNoCommunicationis Bad.
6. Configuration surface
| Key | Default | Notes |
|---|---|---|
OpcUa:Redundancy:Mode |
Unset |
Unset / None / Warm. Hot and unknown values fail host start. Startup-static. Flip both pair nodes in one window, after both run a revision-capable build (§4.2). |
OpcUa:PeerApplicationUris |
[] |
Existing key, now also the ServerUriArray peer source. Required (≥ 1) for Warm, must be empty for None. |
OpcUa:ApplicationUri |
urn:OtOpcUa |
Must be unique within the pair under Warm (validated against the effective config). |
There are no ConfigDb schema changes. ServerCluster.RedundancyMode is only cross-checked (§3).
7. Observability
otopcua.redundancy.warm_role: gauge. 1 = Active, 0 = Standby, −1 = not Warm.otopcua.redundancy.warm_transition{direction,reason}: counter. Reasons:membership,snapshot,endpoint,boot.otopcua.redundancy.snapshot_rejected{reason=older_revision|unrevisioned}: counter.otopcua.redundancy.mode_mismatch: counter (§3).otopcua.opcua.standby.projected_notifications: counter. Counts D4 rewrites at Publish/Republish.otopcua.opcua.standby.suppressed_alarm_events: counter.- Logs:
- Information per transition, naming the revision and the verdict inputs.
- Warning when membership and snapshot disagree for more than 20 s.
- A boot banner line with mode, effective self URI and peers.
otopcua-cli redundancyalready readsi=3709,i=11314andi=2267.
8. Explicitly out of scope
- Hot (and Transparent) redundancy.
- Dynamic peer membership. Peers are static config.
- Changing the election (oldest Up driver member),
auto-down, or the write / native-ack / HistoryUpdate / alarm-drain / historization gates. - Stopping acquisition on the Standby.
- TransferSubscriptions across the pair. Non-transparent means the client re-creates its subscriptions.
- CTT (G8) and production-client cutover (G9). Hermetic and live-rig passes are not acceptance.
- Durable subscription restore.
- Sequence-number wrap.
- Encoded-byte fidelity of cloned payloads beyond the plan's bounded matrix (c7890).
9. Open questions for the owner (short)
- Standby writes. Keep today's optimistic Good + revert (D7), or refuse synchronously with
BadInvalidStateunder Warm, as HistoryUpdate and native ack already do? Recommend synchronous refusal under Warm only. - Crash-failover gap. A SIGKILLed Primary stays an unreachable member until auto-down, ~25–30 s. Under (2)/(3)
the survivor stays Standby for that whole window, with no Good data anywhere, where today the Secondary keeps
serving. Should the survivor be allowed to promote when the Primary is unreachable (the #672 pattern, default
on), accepting dual-Active during a real partition? Recommend yes, behind
OpcUa:Redundancy:PromoteWhilePrimaryUnreachable. - Standby status code.
BadNoCommunicationis spike-proven and is plain Bad in ScadaBridge, but it is the same code as real device comms loss; only ServiceLevel tells them apart. Accept it, or useBadOutOfService? - Scripted Part 9 methods on the Standby. They are accepted (D7) but their event is suppressed (D3), so the
caller sees Good and no event until promotion or refresh. Accept, or refuse with
BadInvalidStateunder Warm? - ConfigDb
Hot. Narrow the CHECK toNone/Warmwith a migration, or leave it and rely on the §3 mismatch Error? - Bands. Are Standby 90/50 and boot 1 acceptable (§4.3)?
10. What must be re-proven on SDK 2.0.1
The spikes ran on 1.5.378.106. The repo now pins 2.0.1 (Directory.Packages.props:129-135). The 2.0.1 XML
docs confirm that these members still exist:
StandardServer.CreateMainNodeManagerFactoryConfigurationNodeManager.AddBehaviourToPredefinedNodeAsync, now asyncNonTransparentRedundancyState.ServerUriArray, now inOpc.Ua.Core.TypesStandardServer.PublishAsync/RepublishAsync, whose signatures now carryRequestLifetimeCreateSubscriptionManagerBaseVariableState.OnSimpleReadValue, plus newOnSimpleReadValueAsync/OnReadValueAsyncReadValueAttributeIServerInternal.ConditionRefresh2- a new
MonitoredItemFactory/MonitoredItemCreateDecisioncreation hook and anIMonitoredItemTransferCoordinator
Existence is not behaviour. Plan T0 re-proves, on 2.0.1 over the wire:
- (a) the metadata graph, including the passive-
i=11314resolution; - (b) that every value read path reaches the per-tag projection: Read, initial sample, change fan-out,
ResendData, size-1 queue and AggregateFilter. In particular, the new async read callbacks must not bypass a sync
OnSimpleReadValue; if they do, the hook moves to aReadValueAttributeoverride onEquipmentVariableState; - (c) provenance through the 2.0.1 creation hook, which replaces the 1.5 manager-subclass route, including transfer/restore;
- (d) the Publish/Republish choice with the new signatures;
- (e) suppression and promotion refresh through the product's #854/#855 per-item paths, which did not exist when the 1.5 event spike ran.