T13b: the isolated otopcua-t13b pair with OpcUa:Redundancy:Mode absent, on origin/masterf826af8e(it containsc569f806) and A/B on c569f806^ (687256d6). Every check passes on both builds: - levels 250/240; - Good data on both nodes; - writes, Acknowledge, AddComment and alarm delivery on the Secondary; - graceful stop: the survivor reaches 250 259 ms after Leaving (pre-fix 270 ms); - crash: 250 at +43.6 s (pre-fix +42.8 s), with no demotion at auto-down; - rejoin at 240 with no failback. The role-decision changes are gated on IsWarm and do not run under Unset. Plan L5, the design's section 4.3 and the Warm table in Redundancy.md now say a Leaving/Exiting node reports ServiceLevel 0, not 90 (ComputeWarm returns 0 for any member that is not Up/Joining; T13 note A). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
30 KiB
Warm non-transparent redundancy with a degraded standby (Gitea #717): design spec
Status: APPROVED by the owner 2026-11-13 (#717), together with the companion plan
(2026-11-11-warm-redundancy-717-plan.md). The §9 questions are answered; the answers are recorded in §9 and
applied throughout. §4.6 was amended the same day after T0: SDK 2.0.1 broke the object-keyed hooks, and the owner
adopted the T0-proven alternative (§10). Parent gate: #615 (G8 CTT, G9 production-client cutover). These stay open and separate from
everything here.
1. Owner decisions this spec builds on (not re-opened)
| # | Decision | Source |
|---|---|---|
| D1 | Advertise Warm for a 2-node pair, with an explicit degraded-standby contract; None for standalone. | #717 c7796 |
| D2 | Standby ServiceLevel in the degraded band 2–199; ordinary tag Reads/subscriptions on the standby degrade, while internal acquisition continues. Today's Secondary serves live Good data at 240, so it cannot just be relabelled. | #717 c7796, c7801, c9329 |
| D3 | Event subscriptions stay established on the standby; new alarm events are suppressed there; current retained conditions are refreshed on promotion. | #717 c7804 |
| D4 | Strict first delivery. After demotion, any Publish response delivered for the first time must not contain Good ordinary-tag values. That includes samples queued before demotion and batches built before it. Republish history and responses already in flight are preserved. | #717 c7849 |
| D5 | A client-initiated ConditionRefresh on the Warm standby returns its current retained conditions. Only new alarm events are suppressed. | #717 c11362 (2026-11-11) |
| D6 | The queue-only spike gate must not be adopted unchanged, because of late-overflow and filtered-retain effects. Product alarm suppression happens before SDK event-field projection. Per-tag read projection is the recommended data seam. Every ACL, direct-item and Server-object refresh path needs coverage. Role snapshots need an ordered revision. | #717 c7812, c7853, issue body |
| D7 | Existing mutation and election policies are preserved: the write/native-ack/HistoryUpdate Primary gates, the oldest-Up-member election, and auto-down. |
#717 c7794 |
2. Where the code is today
- Metadata. No code sets
ServerRedundancy.RedundancySupport(i=3709), so it reads the SDK default.Server.ServerRedundancyis the baseServerRedundancyTypeand has noServerUriArray(i=11314) (docs/Redundancy.md:9). Peers are published only inServer.ServerArray(i=2254), by appending toIServerInternal.ServerUrisafter start (OpcUaApplicationHost.cs:378-414, called at:241). - Self URI. It comes from
OpcUa:ApplicationUri, which defaults tourn:OtOpcUa(OpcUaApplicationHost.cs:35). IfApplicationConfigPathis set, the loaded XML wins (:602-608). docker-dev sets neitherOpcUa__ApplicationUrinorPeerApplicationUrison central-1/2 (docker-dev/docker-compose.yml:289-481), so both centrals advertiseurn:OtOpcUa. The seededClusterNode.ApplicationUriusesurn:OtOpcUa:<NodeId>(docker-dev/seed/seed-clusters.sql:13). - ServiceLevel.
ServiceLevelCalculator.Computereturns 240 + 10 for the Primary (ServiceLevelCalculator.cs:44-64). The standby therefore sits in the healthy band.OpcUaPublishActor.RecomputeServiceLevelfeeds it (OpcUaPublishActor.cs:1576-1636), and the result is published through the #719-fixed retain/replay path (:1468-1488,SdkServiceLevelPublisher.cs:36-60). - Acquisition is role-independent. Every driver node spawns its drivers and subscribes whatever its role (#717 c7794). Today Primary gating covers only writes, native acks, the alerts emit, alarm-history drain, historization capture and HistoryUpdate (
PrimaryGatePolicy.cs:21,56,97,155;docs/Redundancy.md:554-600). - Data variables. Every ordinary-tag data variable (raw tag, UNS reference, virtual tag, Calculation tag) is an
EquipmentVariableStatecreated at exactly one site (OtOpcUaNodeManager.cs:4060). Values are pushed into it withPublishValue(EquipmentVariableState.cs:28-35). - Alarm events. Every condition event goes through
ReportConditionEvent(OtOpcUaNodeManager.cs:1376) and thenReportEventWithoutBlocking(:1507). The callers are engine/driver projection (:1254,:1341), method transitions (:2842) and restore re-report (:4659). Server-object items get per-item, ACL-filtered dispatch (#586). Folder and direct items get per-item delivery (#826/#855). Refresh paths:ConditionRefresh(:6586),RefreshDirectItems(:6974) and the #854 cut. Model-change and audit events useReportToServerObject(:6060), not this path. - Role snapshots are unordered.
RedundancyStateActoris a cluster singleton. It publishesRedundancyStateChanged(Nodes, CorrelationId)on DPS, debounced at 250 ms and heartbeated every 10 s (RedundancyStateActor.cs:25,33,105-113).AsOfUtcis the publisher's wall clock (:184) andCorrelationIdis random (:110). Subscribers replace their cached snapshot unconditionally (OpcUaPublishActor.cs:1494-1498,DriverHostActor.cs:2555-2562). After a singleton handover, a late heartbeat from the old incarnation can overwrite the new incarnation's snapshot. Nothing establishes the order, and there is no lease. - Config.
ServerCluster.RedundancyMode(None/Warm/Hot) exists in ConfigDb. It is CHECK-coupled to NodeCount (OtOpcUaConfigDbContext.cs:109-111;DraftValidator.cs:958-962) and travels in the artifact'sClustersarray (ConfigComposer.cs:67), but no runtime code reads it.
3. Contract per deployment shape
| Shape | OpcUa:Redundancy:Mode |
RedundancySupport (i=3709) |
ServerRedundancy type |
ServerUriArray (i=11314) |
ServerArray (i=2254) |
Standby behaviour |
|---|---|---|---|---|---|---|
| Today / not opted in | Unset (default) |
SDK default (unchanged) | ServerRedundancyType (2034) |
not browsable; the SDK's orphan reads Good [] by NodeId (unchanged) |
self + peers (unchanged) | none; Secondary serves Good at 240 (unchanged) |
| Standalone | None |
None (0) |
ServerRedundancyType (2034) |
absent (the SDK's orphan i=11314 is deleted, T0a) |
self only | never; always serves |
| Pair | Warm |
Warm (2) |
NonTransparentRedundancyType (2039) |
[effectiveSelfUri, peer…], read-only String[], self first (decision #155, docs/v2/plan.md:930) |
unchanged | §4 |
- The metadata is startup-static. On 2.0.1 it is built in
OnNodeManagerStartedAsyncthroughIPredefinedNodeSubtypeReplacer.ReplacePredefinedInstanceSubtypeAsyncon the diagnostics node manager, which keeps the well-known child NodeIdi=11314(T0a; the 1.5 factory route of c7791 is not needed). Changing mode or peers requires a restart. effectiveSelfUriis the effectiveApplicationConfiguration.ApplicationUriafter any XML load, never the options default (c7787).Hotis not advertisable: continuous acquisition alone does not meet Part 4 §6.6.2.4 Hot (c7794).Mode=Hotfails host start.- Startup validation (
ValidateOnStart) runs forWarmand fails host start when any of these holds: there is no peer; a peer URI equals the self URI; a URI is duplicated; thedriverrole is missing. - Deploy-time cross-check: if the artifact's
ServerCluster.RedundancyModefor this node's cluster disagrees with the local mode (NodeCount=1/Noneagainst localWarm, or the reverse), the node logs an Error and records the metricotopcua.redundancy.mode_mismatch. The node keeps running in its local mode. Local config is the authority, because it is the only value available before SDK start.
4. Warm role contract
4.1 Role verdict (who is Active)
A Warm node is Active iff all of the following hold. Otherwise it is Standby.
- The local
SelfMemberis Up andEndpointListeningis true (#762). - The node's own
Cluster.Statenames it the oldest Updrivermember. This is the same function the singleton uses (RedundancyStateActor.SelectOldestUpMemberOfRole,:172), so the verdict cannot drift from singleton placement (D7). - The latest accepted snapshot (§4.2) names it
Primary.
Any disagreement between (2) and (3) resolves to Standby. A Warning is logged if the disagreement lasts more than
two heartbeats (20 s). When the role is unknown — before the first accepted snapshot, or when the snapshot omits
this node — the node is Standby, whatever the member count. This is stricter than PrimaryGatePolicy's
lone-node allow (PrimaryGatePolicy.cs:21): a lone node goes Active within one debounce (~250 ms) of its own
singleton's first publish. That gate is not changed for writes/acks (D7). The Warm verdict is a separate output.
Promote while the Primary is unreachable (owner answer Q2, OpcUa:Redundancy:PromoteWhilePrimaryUnreachable,
default true). A SIGKILLed Primary stays an Up-but-unreachable member until auto-down removes it (~25–30 s),
and the singleton usually died with it, so (2) and (3) would hold the survivor Standby for that whole window. With
the option on, (2) and (3) are evaluated with unreachable members excluded: (2′) self is the oldest Up driver
member among the members this node can reach, and (3′) the latest accepted snapshot names either self or an
unreachable member as Primary. This is the #672 pattern (ContinuousHistorization:CaptureWhilePrimaryUnreachable).
A node that has never accepted a snapshot is still Standby. The cost is dual-Active while a live Primary is merely
unreachable, and for the whole of a real partition.
Accepted trade: an auto-down partition can give two Active nodes, as it already does for the data-plane gate
(docs/Redundancy.md §Split-brain). For reads this is benign, because both nodes serve the same Good data.
4.2 Ordered snapshots (the revision problem)
RedundancyStateChanged gains an additive Revision(PublisherUpNumber, IncarnationTick, Sequence):
PublisherUpNumberis theMember.UpNumberof the node that hosts the singleton. The singleton always lives on the oldest member, and every surviving member has a larger up-number than a departed oldest one. The value is therefore monotonic across handovers within one cluster incarnation.IncarnationTickisEnvironment.TickCount64at singletonPreStart. It is monotonic for a singleton that restarts in the same process.Sequenceincrements on every publish, including heartbeats.
Subscribers (OpcUaPublishActor, DriverHostActor) accept a snapshot only if its revision is lexicographically
greater than the last accepted one. Anything else is dropped and counted in
otopcua.redundancy.snapshot_rejected{reason}. A subscriber cannot outlive its cluster incarnation: a removed node
must restart its ActorSystem to rejoin. The plan pins this assumption with a test. A snapshot without a
revision (from a pre-#717 publisher during a rolling upgrade) is accepted for the existing gates but can never make
a Warm node Active. Warm therefore requires both pair nodes to be upgraded before it is turned on.
There is no time lease beyond this. Liveness comes from (2), which reads live gossip, and the existing 30 s staleness input to ServiceLevel. Demoting an Active node on snapshot staleness alone was rejected: the singleton lives on the Active node itself, so staleness there means a local fault, and demoting would turn it into a pair-wide data outage.
4.3 ServiceLevel bands (Part 4 §6.6.2.4.2)
| Node state (Warm) | Byte | Band |
|---|---|---|
| Leaving / Exiting / Down / Detached / endpoint failed (any member not Up/Joining) | 0 | Maintenance (unchanged) |
| Boot, before first computed level | 1 (set in the startup graph; replaces SDK default 255) | NoData |
| Standby, degraded (stale or DB unreachable) | 50 | Degraded |
| Standby, healthy | 90 | Degraded |
| Active, critically degraded | 100 | Degraded (unchanged) |
| Active, stale | 200 | Healthy (unchanged) |
| Active, healthy | 250 | Healthy (unchanged: 240 + 10) |
Invariant: every Standby level is below every level an Active node serving data can publish (≥ 100). A
"highest ServiceLevel" client therefore never prefers a standby over a data-serving Active node. Unset and None
keep ServiceLevelCalculator.Compute byte-for-byte.
4.4 Behaviour on the Standby, per service
"Ordinary tag" means the Value attribute of an EquipmentVariableState. The category is captured at item
creation from the node instance and the AttributeId, never from namespace or ClientHandle (c7853).
| Service | Standby behaviour | Status the client sees |
|---|---|---|
Read (ordinary tag Value) |
Projected. The stored value is kept internally and continues to update from acquisition. | BadOutOfService, Value = null, ServerTimestamp = now. Operation result Good. |
| Read (other attributes, condition children, ns=0 incl. ServiceLevel/ServerStatus/ServerRedundancy) | Unchanged | Good / as today |
| CreateMonitoredItems (ordinary tag) | Created normally (ACL unchanged). Initial value is projected. | Item result Good; first notification BadOutOfService/null |
| Existing data items at demotion | One BadOutOfService/null notification is queued, ignoring filters. Later samples project to the same value, so the SDK's change filter emits nothing further. D4: any Good ordinary-tag notification that would be first-delivered after the demotion step is rewritten to BadOutOfService/null at Publish/Republish (§4.6). |
BadOutOfService |
Write (ordinary tag Value) |
Refused synchronously under Warm only (owner answer Q1). The node manager refuses the write before it reaches the driver; nothing is reverted and no device is touched. Unset/None keep today's #5 self-correction (synchronous Good, then revert + audit) off-Primary (D7). |
BadInvalidState |
| HistoryRead | Unchanged. Served from the shared backend, which is not live acquisition. | as today |
| HistoryUpdate | Unchanged | BadInvalidState off-Primary (#674) |
| Browse / TranslateBrowsePaths / RegisterNodes | Unchanged | as today |
| Call: Part 9 methods | Refused under Warm (owner answer Q4). Every Part 9 condition method (Acknowledge, Confirm, AddComment, Enable/Disable, the shelving methods) on a native or scripted condition is refused before it changes any state. Unset/None keep today's behaviour: native ack BadInvalidState off-Primary, scripted ack/shelve accepted (D7). |
BadInvalidState |
| Event subscriptions (Server object, folders, direct) | Stay established, in Reporting mode, with the ACL unchanged | — |
| Model-change / audit events | Not suppressed. They are not alarm events, and browse consistency needs them. | — |
| Session / secure channel / ServerStatus.State | Unchanged (Running) |
— |
4.5 Alarms and events on the Standby
- Suppression point.
ReportConditionEventis split into stamp and deliver. Stamp sets EventId, Time and ReceiveTime, and runsClearChangeMasks. Deliver builds the snapshot and callsReportEventWithoutBlocking. On the Standby only stamp runs. Condition node state therefore stays current, and a later refresh carries a valid, ackable EventId. Because suppression sits before snapshot construction and before every SDK field projection, it covers the Server-object, folder and direct paths, ACL on or off, by construction. Overflow, filtered retain and the #838 deferral queue never see a suppressed event. TheRingEventQueueFactoryqueue is not used as the gate (D6). - Not affected by T0. The stamp/deliver split is product code before any SDK object exists, so the 2.0.1 clone/pool behaviour (§4.6) does not touch it.
- Boundary. The decision is made at report time. An event whose report ran while Active is delivered even if delivery finishes after demotion: deferred (#838) or already in an item queue. This matches the spike's observed admission boundary (c7805). Republish of already-built event messages is unchanged.
- Client ConditionRefresh / ConditionRefresh2 on the Standby (D5) run the existing paths unchanged: Server
object (#854 cut),
RefreshDirectItems(#855), with ACL filtering as today. The replay is RefreshStart, the current retained conditions, then RefreshEnd. - Promotion refresh. On Standby → Active, the server runs a refresh on every live event item: Server-object
items, direct items and folder items. The two refresh entry points are reused, so ACL decisions and the #854/#855
cut apply. Clients see RefreshStart, retained conditions, RefreshEnd, then new events. This is spike-proven on
1.5 (c7805). A condition that went active and stopped being retained entirely while the node was Standby is
not replayed. That is standard refresh semantics, and the transition is not historized twice, because alarm
history is Primary-gated (
docs/Redundancy.md:572-573).
4.6 Strict first delivery (D4)
The item-level projection (§4.4) stops new Good samples. It cannot reach samples already queued in an item, nor
NotificationMessages the subscription has prebuilt or cached (c7846, c7850). The 1.5 spikes (c7851, c7852) proved
an object-keyed mechanism; SDK 2.0.1 breaks it (T0, docs/plans/evidence/2026-11-13-717-sdk2/RESULTS.md): the
SentMessageQueue hands every Publish and Republish a fresh CoreUtils.Clone of the cached message, and notification
payloads are pooled and reused across items. The owner adopted the T0-proven alternative on 2026-11-13 (#717). It keeps
the same structure (a custom item marks at build time; an override decides at first delivery) and changes both keys:
- Ordinary-tag marker on an SDK-copied, never-encoded field. Ordinary-tag
Valueitems are created through a custom sampled item, installed by re-implementingIMonitoredItemManager.CreateMonitoredItemon aMonitoredNodeMonitoredItemManagersubclass (the 2.0.1MonitoredItemFactoryhook exists only onAsyncCustomNodeManager). Its per-valuePublishruns the base first (so the timestamp filter has been applied) and then setsMonitoredItemNotification.Message(a public helper reference the binary encoder never writes and the SDK'sMemberwiseClonecopies) to an immutable marker holding the notification'sClientHandleand finalDataValue. At delivery a notification counts as ordinary-tag only if its marker matches its ownClientHandleandDataValue; a stale marker left on a pooled object (Initialize()does not clear the helper) fails the match and the notification is treated as unknown-origin. A false match needs the same handle and an identical value, status and both timestamps on a recycled object. That is improbable, not impossible; it is accepted, and the residual case can only make a Good value read projected, never the reverse. - Decision record keyed by
(SubscriptionId, SequenceNumber), never by object.OtOpcUaSdkServeroverridesPublishAsyncandRepublishAsync, calls base first, and on the first delivery of a sequence number records Active or Standby. Every later delivery of that number (Republish included) applies the recorded decision to the fresh clone the SDK returned.- Active: the clone is returned as is.
- Standby: the clone is projected: every marked ordinary-tag notification whose status is not Bad is replaced by
BadOutOfService/null (server timestamp kept). The rewrite allocates newDataChangeNotification/MonitoredItemNotificationobjects; the SDK's cached copy is never mutated. - A short lock orders the record against the role flip. No lock spans an await.
- The record is pruned against the
AvailableSequenceNumberseach Publish returns and bounded per subscription and in total, so dead subscriptions cannot grow it.
- Out of scope of the rewrite, by D4: responses already in flight before the demotion step, and Republish of messages first delivered while Active. Unknown-origin notifications are not rewritten; there is no implicit policy (c7853).
- 2.0.1 facts the matrix follows. A Republish of a prebuilt but never-sent sequence number is refused
BadMessageNotAvailableby the SDK, so "first delivery through Republish" cannot occur for queued messages. TransferSubscriptions moves items (and their markers) with the subscription; a decision record recorded under the old session stays keyed by the same subscription id. Durable subscription restore is exercised in T13.
A regression test pins the SDK behaviour this relies on: the helper field survives the per-delivery clone, and it is never encoded onto the wire. An SDK upgrade that changes either fails it loudly.
4.7 Transition sequencing
One actor, OpcUaPublishActor, owns the verdict, the ServiceLevel publish and a new IStandbyProjection sink
(in Commons.OpcUa, bound late like IServiceLevelPublisher). Each transition runs as one serialized handler, in
this order.
Promotion (Standby → Active)
- Open the read projection.
- For each live ordinary-tag item, queue the current stored value, ignoring filters. The precedent is the queue
pattern at
OtOpcUaNodeManager.cs:7303. - Switch the delivery-history choice to Active.
- Un-suppress alarm delivery.
- Run the promotion refresh (§4.5).
- Publish the Active ServiceLevel.
Demotion (Active → Standby)
- Publish the Standby ServiceLevel.
- Switch the delivery-history choice to Standby. This is the D4 boundary.
- Suppress alarm delivery.
- Close the read projection.
- Queue
BadOutOfServiceto each live ordinary-tag item.
Invariant: an Active-band ServiceLevel is never visible while the projection is closed. If the ServiceLevel publish fails (#719 retry path), a promotion stops before step 6 and retries. A demotion continues anyway: failing safe means degrading the data.
4.8 Standalone (None) and Unset
None never projects, never suppresses and never refreshes. It advertises RedundancySupport=None with no
ServerUriArray, and uses today's ServiceLevel. Unset is byte-for-byte today's server, including the documented
unmet A4/A5b gates.
5. ScadaBridge impact (cross-repo, blocking for any Warm flip)
ScadaBridge consumes this server as an OPC UA client, through RealOpcUaClient and DataConnectionActor.
- Failover is connection-driven, not ServiceLevel-driven. ScadaBridge swaps primary to backup only after
FailoverRetryCountconsecutive unstable disconnects (../ScadaBridge/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs:655-690;RealOpcUaClient.cssits in that project'sAdapters/).BadOutOfServiceis not a connection-level status (:1974-1983), so it maps to plain Bad quality, and since ScadaBridge#112 a Bad-with-null update keeps the last-known value. Nothing in ScadaBridge readsServiceLevel,RedundancySupportorServerUriArray. - Consequence. A ScadaBridge connection that lands on the Standby will sit on Bad quality indefinitely on a
healthy session, and never fail over. Today the same connection gets Good data at 240. Warm is therefore an
outage for ScadaBridge unless ScadaBridge becomes ServiceLevel-aware: monitor
i=2267and treat a Degraded band (< 200) sustained for N s, while the other endpoint reports a higher level, as a failover trigger. That change must ship and be deployed first. It is filed as ScadaBridge#138 (cross-repo, blocking); no tier setsMode=Warmuntil it is deployed there. - Alarms are compatible. ScadaBridge already handles a RefreshStart/RefreshEnd replay as a snapshot
(
RealOpcUaClient.cs:72) and triggers its own ConditionRefresh on (re)subscribe (:1179-1193). Under D5 that refresh returns retained conditions on the Standby too. The promotion refresh arrives as an unsolicited Start/End pair on existing items; this needs a ScadaBridge-side test. - Status mapping is compatible. ScadaBridge#117 maps by severity bits, so
BadOutOfServiceis Bad.
6. Configuration surface
| Key | Default | Notes |
|---|---|---|
OpcUa:Redundancy:Mode |
Unset |
Unset / None / Warm. Hot and unknown values fail host start. Startup-static. Flip both pair nodes in one window, after both run a revision-capable build (§4.2). |
OpcUa:Redundancy:PromoteWhilePrimaryUnreachable |
true |
Warm only. Lets the survivor go Active while the Primary is unreachable but not yet downed (§4.1, owner answer Q2). |
OpcUa:PeerApplicationUris |
[] |
Existing key, now also the ServerUriArray peer source. Required (≥ 1) for Warm, must be empty for None. |
OpcUa:ApplicationUri |
urn:OtOpcUa |
Must be unique within the pair under Warm (validated against the effective config). |
There are no ConfigDb schema changes. ServerCluster.RedundancyMode is only cross-checked (§3).
7. Observability
otopcua.redundancy.warm_role: gauge. 1 = Active, 0 = Standby, −1 = not Warm.otopcua.redundancy.warm_transition{direction,reason}: counter. Reasons:membership,snapshot,endpoint,boot.otopcua.redundancy.snapshot_rejected{reason=older_revision|unrevisioned}: counter.otopcua.redundancy.mode_mismatch: counter (§3).otopcua.opcua.standby.projected_notifications: counter. Counts D4 rewrites at Publish/Republish.otopcua.opcua.standby.suppressed_alarm_events: counter.- Logs:
- Information per transition, naming the revision and the verdict inputs.
- Warning when membership and snapshot disagree for more than 20 s.
- A boot banner line with mode, effective self URI and peers.
otopcua-cli redundancyalready readsi=3709,i=11314andi=2267.
8. Explicitly out of scope
- Hot (and Transparent) redundancy.
- Dynamic peer membership. Peers are static config.
- Changing the election (oldest Up driver member),
auto-down, or the write / native-ack / HistoryUpdate / alarm-drain / historization gates. - Stopping acquisition on the Standby.
- TransferSubscriptions across the pair. Non-transparent means the client re-creates its subscriptions.
- CTT (G8) and production-client cutover (G9). Hermetic and live-rig passes are not acceptance.
- Durable subscription restore.
- Sequence-number wrap.
- Encoded-byte fidelity of cloned payloads beyond the plan's bounded matrix (c7890).
9. Owner answers (2026-11-13, #717)
The draft carried six open questions. The owner answered them on 2026-11-13 and approved the spec and plan
(2be43aa9). The answers are applied in the sections above.
- Standby writes. Refuse synchronously with
BadInvalidStateunderWarmonly (§4.4).Unset/Nonekeep the #5 optimistic Good + revert. - Crash-failover gap. Yes: the survivor may promote while the Primary is unreachable, behind
OpcUa:Redundancy:PromoteWhilePrimaryUnreachable(default on) (§4.1). Dual-Active during a real partition is accepted. - Standby status code.
BadOutOfService, not the spike'sBadNoCommunication, so a standby is distinguishable from real device comms loss without reading ServiceLevel. Re-proven on 2.0.1 in T0. - Scripted Part 9 methods on the Standby. Refuse with
BadInvalidStateunderWarm(§4.4). - ConfigDb
Hot. Leave the CHECK; rely on the §3 mismatch Error. No migration. - Bands. Accepted as drafted: Standby 90 (healthy) / 50 (degraded), boot 1 (§4.3).
10. What must be re-proven on SDK 2.0.1
The spikes ran on 1.5.378.106. The repo now pins 2.0.1 (Directory.Packages.props:129-135). The 2.0.1 XML
docs confirm that these members still exist:
StandardServer.CreateMainNodeManagerFactoryConfigurationNodeManager.AddBehaviourToPredefinedNodeAsync, now asyncNonTransparentRedundancyState.ServerUriArray, now inOpc.Ua.Core.TypesStandardServer.PublishAsync/RepublishAsync, whose signatures now carryRequestLifetimeCreateSubscriptionManagerBaseVariableState.OnSimpleReadValue, plus newOnSimpleReadValueAsync/OnReadValueAsyncReadValueAttributeIServerInternal.ConditionRefresh2- a new
MonitoredItemFactory/MonitoredItemCreateDecisioncreation hook and anIMonitoredItemTransferCoordinator
Existence is not behaviour. Plan T0 re-proves, on 2.0.1 over the wire:
- (a) the metadata graph, including the passive-
i=11314resolution; - (b) that every value read path reaches the per-tag projection: Read, initial sample, change fan-out,
ResendData, size-1 queue and AggregateFilter. In particular, the new async read callbacks must not bypass a sync
OnSimpleReadValue; if they do, the hook moves to aReadValueAttributeoverride onEquipmentVariableState; - (c) provenance through the 2.0.1 creation hook, including transfer/restore;
- (d) the Publish/Republish choice with the new signatures;
- (e) suppression and promotion refresh through the product's #854/#855 per-item paths, which did not exist when the 1.5 event spike ran.
Result (2026-11-13): (a) and (b) are re-proven on 2.0.1. The metadata graph uses IPredefinedNodeSubtypeReplacer
in OnNodeManagerStartedAsync, which is simpler than the factory route. (c) and (d) fail as written in §4.6:
2.0.1 clones the cached message on every delivery and pools notification payloads, so neither object-keyed
provenance nor an object-keyed choice can work. An alternative mechanism passed the spike; the owner adopted it on
2026-11-13 and §4.6 now describes it: a decision record keyed by (SubscriptionId, SequenceNumber) and a
(ClientHandle, DataValue)-validated marker on MonitoredItemNotification.Message. The custom item is installed by
re-implementing IMonitoredItemManager on a MonitoredNodeMonitoredItemManager subclass. The T0 record is kept as
evidence: docs/plans/evidence/2026-11-13-717-sdk2/RESULTS.md.
The T0 finding on the metadata graph also applies: in None/Unset the predefined i=11314 is an orphan, readable by
NodeId though not browsable. §3's "absent" is implemented for None by deleting that orphan at startup; Unset keeps
the SDK's graph untouched (today's behaviour) and docs/Redundancy.md says so.