Spec: per-shape contract (Unset/None/Warm), RedundancySupport + ServerUriArray graph, ServiceLevel bands, per-service standby behaviour, alarm suppression + ConditionRefresh (D5), strict first-delivery (D4), transition sequencing, ordered snapshot revisions, ScadaBridge impact, SDK 2.0.1 re-proof list, open questions. Plan: T0-T13 with wire tests, docker-dev live gate, rollout/rollback. Docs only. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
17 KiB
Warm non-transparent redundancy (Gitea #717): implementation plan
Status: DRAFT for owner approval, together with 2026-11-11-warm-redundancy-717-design.md (referred to below
as "the spec", with § references). No task starts until both documents and the §9 open questions are settled.
Every task ships dark: OpcUa:Redundancy:Mode defaults to Unset, so merging any task changes no deployed
behaviour.
Conventions:
- Wire tests use a real OPC Foundation client session against a loopback
OtOpcUaSdkServer, intests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests(the home ofDualEndpointTests,ConditionRefreshDeferralWireTestsandFolderConditionRefreshWireTests). - Actor tests go in
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Testsandtests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests. - Every gate gets an allow leg as well as a deny leg (the vacuous-test rule), and a mutation check: revert the production line and confirm the test fails.
T0: SDK 2.0.1 re-proof spikes (throwaway, separately approved)
The 1.5.378.106 spikes do not carry over (spec §10). Write throwaway loopback fixtures against 2.0.1, outside
src/. No product edits. Each fixture is rerun independently and its evidence is recorded in
docs/plans/evidence/2026-11-xx-717-sdk2/.
- a. Metadata graph. Build it via
CreateMainNodeManagerFactory+AddBehaviourToPredefinedNodeAsync. Check Read/Browse/Translate ofi=2296(HasTypeDefinition 2039),i=3709, andi=11314for both Warm and None. The passivei=11314visit must resolve to the constructed instance. - b. Read projection coverage. Cover Read, the initial sample on create, the change fan-out after
ClearChangeMasks, ResendData, queue size 1 vs. N, an AggregateFilter item, and Sampling → Reporting. Decide betweenOnSimpleReadValueand aReadValueAttributeoverride, and record which async callback paths 2.0.1 takes. - c. Provenance. Stamp items through the 2.0.1
MonitoredItemFactory/MonitoredItemCreateDecision. Exercise ModifyMonitoredItems handle change, delete/reuse, TransferSubscriptions (IMonitoredItemTransferCoordinator), and a restore. - d. Publish/Republish. Override
PublishAsync/RepublishAsyncwith the 2.0.1 signatures (RequestLifetime). Cover the first-delivery choice, first delivery through Republish, an ack followed byBadMessageNotAvailable, and a cancelled or failed response. - e. Events. Exercise the stamp/deliver split under the #855 per-item folder path and the #854 Server-object cut. Run the promotion refresh on all three item kinds, and a client ConditionRefresh/2 while Standby.
Exit: a short results note per fixture, and an owner go/no-go. A "no" on (c) or (d) re-opens D4's mechanism. D4 itself stays an owner decision.
T1: Configuration, validation, effective self URI
Files:
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs: options at:29-121. AddOpcUaRedundancyOptions Redundancy(Mode;PromoteWhilePrimaryUnreachableif Q2 = yes).src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs.- A post-load check in
StartAsync(:218-250) against the effectiveApplicationConfiguration.ApplicationUri.
Rules (spec §3):
Hotand unknown values fail host start.Warmrequires at least one peer, all URIs distinct, self not listed as a peer, and thedriverrole.Nonerequires no peers.
Tests:
- A validator matrix.
- An XML
ApplicationConfigPathwhose URI differs from the options value, proving the effective URI is used. - A boot-banner log assertion.
T2: Standard redundancy metadata graph
Files:
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs: next toCreateMasterNodeManager(:341), add aCreateMainNodeManagerFactoryoverride that installs the T0a graph per mode.- Under
Warm, the initialServer.ServiceLevelis set to 1 in the same pass (spec §4.3).Unsetmust not touch the graph.
Tests (wire):
- Per mode, check the
i=3709value, thei=2296type definition, the presence or absence ofi=11314, its value order[effectiveSelf, peers…], and that it is read-only (a Write returnsBadNotWritable). ServerArray(i=2254) is unchanged; extendDualEndpointTests.- A Warm server's ServiceLevel reads 1 before the first publish.
- Security profiles and startup are unaffected under
Unset(golden read of today's values).
T3: Ordered redundancy snapshots (spec §4.2)
Files:
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/RedundancyStateChanged.cs(:9-11): add an optionalSnapshotRevision? Revision(additive).RedundancyStateActor.cs:- stamp the revision in
PublishIfDirty(:105-113); - set
IncarnationTickinPreStart; - read the host's
UpNumberfromCluster.SelfMember.
- stamp the revision in
- Subscribers: accept only a newer revision in
OpcUaPublishActor.HandleRedundancyStateChanged(:1494-1498) andDriverHostActor.OnRedundancyStateChanged(:2555). - An unrevisioned snapshot keeps today's handling for the existing gates.
Tests:
- Ordering unit tests: an older singleton incarnation's heartbeat after a newer snapshot is dropped; a same-process singleton restart advances.
- A real 2-node cluster handover test (pattern:
RedundancyPrimaryElectionTests). It holds back the old incarnation's last publish through the_broadcastOverrideseam (:111) and asserts the subscriber keeps the new Primary. - Serialization round-trip of the extended message, plus a mixed-version decode with the field absent.
- The pinned assumption: a removed node cannot rejoin without a new
ActorSystem.
T4: Warm role verdict and ServiceLevel bands (spec §4.1, §4.3)
Files:
- New
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/WarmRoleAuthority.cs: a pure function of membership view, accepted snapshot,EndpointListeningandSelfMemberstatus. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs: addComputeWarm(inputs, isActive).Computeitself is unchanged.OpcUaPublishActor.RecomputeServiceLevel(:1576-1636) branches on mode. It subscribes toMemberUp/Removed(and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots.
Tests:
- A truth table covering every row of §4.3 and the invariant max(Standby) < min(Active serving).
- Membership/snapshot disagreement gives Standby, plus the 20 s Warning.
- Boot with an unknown role gives Standby; a lone node goes Active after its own snapshot.
None/Unsetproduce byte-identical output to today'sServiceLevelCalculatorTests.- DB-less mode (#721 timer) is covered.
T5: Standby projection seam and transition sequencer (spec §4.7)
Files:
- New
src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IStandbyProjection.cs, with a deferred/late-bound adapter that mirrorsDeferredServiceLevelPublisherand retains the latest state across binding (the #719 lesson). - Bound in
src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.csnext to the ServiceLevel binding. OpcUaPublishActorruns promotion and demotion as one serialized handler, in the §4.7 step order.
Tests (actor, with a recording fake):
- The exact step order for both directions.
- A ServiceLevel publish failure halts a promotion before the Active level and retries; a demotion continues.
- A burst of alternating verdicts converges on the last one.
- Binding after the first transition replays the current state.
T6: Per-tag read projection (spec §4.4)
Files:
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs: the hook decided in T0b, reading a node-manager-wide volatile projection flag. StoredValueis never modified.OtOpcUaNodeManagerimplementsIStandbyProjection.SetDataProjection.
Tests (wire):
- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns
BadNoCommunication/null. - A Standby Read of a condition child,
i=2267ori=3709stays Good. - A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected.
- Acquisition continues: flip to Active and the latest value is read without any new driver publish.
- ACL on: a denied node is still
BadUserAccessDeniedand the projection never leaks existence.
T7: Provenance and first-delivery choice (spec §4.6, D4)
Files:
OtOpcUaSdkServer.cs: install the custom sampled item (T0c route) for ordinary-tagValueitems only.OtOpcUaSdkServer.cs: add thePublishAsync/RepublishAsyncoverrides and a newFirstDeliveryLedger, aConditionalWeakTablekeyed by the SDK message object.- Wire into
OtOpcUaNodeManager.CreateMonitoredItems(:5295) /OnMonitoredItemCreated(:5442) without disturbing the ACL filtering or_liveItems.
Tests (wire): port the spikes' assertion matrix to product code.
- Queued Good samples, then demotion, then the first Publish carries
BadNoCommunication. - Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same projected payload.
- Good delivered while Active, then demotion, then Republish returns the original Good.
- A first delivery through Republish.
- A mixed subscription (ordinary tag +
i=2267+ condition-child item): only ordinary-tag items are rewritten. - Handle modified, deleted or reused between queue and publish.
- The cached SDK object is unchanged (sacrificial-clone check).
- A bounded clone matrix (arrays, ExtensionObject, DiagnosticInfo, per c7890).
- Concurrent Publish and role flip, by a stress loop with a deterministic barrier, not sleeps.
T8: Promotion and demotion of live data items
Files:
OtOpcUaNodeManager:IStandbyProjection.QueueCurrentToLiveItems(bool projected)iterates_liveItems. It reuses the QueueValue-with-ignoreFilterspattern (:7303), not detach orMonitoringMode.Disabled; the disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped.
Tests (wire):
- Demotion delivers exactly one
BadNoCommunicationper item, then silence while values keep changing internally. - Promotion delivers the current Good value without a driver publish.
- An ACL-stood-down item receives nothing either way.
- A DeadbandFilter item still receives the promotion value.
T9: Alarm event suppression and promotion refresh (spec §4.5, D3, D5)
Files (OtOpcUaNodeManager):
- Split
ReportConditionEvent(:1376) into stamp and deliver; deliver is gated by the projection flag. - Audit all callers (
:1254,:1341,:2842,:4659) andWireMethodTransitionReporting. IStandbyProjection.RefreshAllEventItems()runs the existingConditionRefresh(:6586) Server-object path andRefreshDirectItems(:6974) over every live event item, from_allEventsItemsand_eventItems.ReportToServerObject(:6060) is untouched.
Tests (wire, ACL off and on):
- A new alarm on the Standby reaches no Server-object, folder or direct item.
- Model-change and audit events are still delivered on the Standby.
- A client ConditionRefresh and ConditionRefresh2 on the Standby return Start, retained, End (D5).
- Promotion gives Start, retained, End on all three item kinds, followed by new events.
- An ACL-denied condition is absent from the promotion refresh.
- A condition raised and cleared (Retain=false) while Standby is not replayed.
- An event reported while Active but deferred (#838, via
NotifierStall) is still delivered after demotion. - A Standby-stamped EventId is accepted by Acknowledge after promotion.
- The
RingEventQueueFactorypath is untouched (RingEventMonitoredItemQueueParityTestsstay green). - Re-run
EventQueueOverflowAclWireTestsandConditionMethodEventTests.
T10: Deploy-time mode cross-check and observability (spec §3, §7)
Files:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs: read this node's clusterRedundancyModefrom theClustersarray.OpcUaPublishActorcompares it with the local mode at apply.src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs: add the six instruments.
Tests:
- A mismatch logs an Error and records the metric.
- Each counter fires on its path. Use a
MeterListenerfiltered by instrument instance, never by meter name alone (the #503 lesson).
T11: ScadaBridge prerequisite (cross-repo)
- File a ScadaBridge issue (
type:feature,cross-repo,priority:p1, linked to #717). The DCL becomes ServiceLevel-aware: monitori=2267, and fail over when it is below 200 for a configurable window while the other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items (spec §5). - File the lmxopcua companion as
blocked+cross-repo. - The Warm flip in T13/rollout is gated on that issue being deployed.
T12: Documentation and index
docs/Redundancy.md: rewrite the:9block, add a Warm contract section, update the tiers table. Remove the generic "standby / no device contact / exactly-once" wording that contradicts acquisition and at-least-once (c7794).docs/OpcUaServer.md:70,docs/Configuration.md(the new keys), anddocs/security.md(ns=0 stays ungated; projection order vs. ACL)..claude/rules/cluster-topology.md§Redundancy.docs/plans/phase-6-3-redundancy-interop-plan.mdA2/A4/A5b: under Warm, A2 expects 90 and A4/A5b become testable. Adddocs/v2/redundancy-interop-playbook.md.- Umbrella: update the OtOpcUa row and
sister-project-history/otopcua.mdin../scadaprojon itsmain, pushed (rootCLAUDE.mdprocedure).
T13: Live gate on docker-dev (central pair, :4840 / :4841)
Overlay: new docker-dev/docker-compose.warm.yml, central-1/2 only. It sets:
OpcUa__ApplicationUri=urn:OtOpcUa:central-1|2(matchingseed-clusters.sql);OpcUa__PeerApplicationUris__0=<the other>;OpcUa__Redundancy__Mode=Warm.
Today both centrals advertise the default urn:OtOpcUa (spec §2). Login is disabled on the rig, so drive
everything yourself. Use otopcua-cli from src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI against both ports.
| Leg | Procedure | Pass |
|---|---|---|
| L1 metadata | redundancy on :4840 and :4841 |
RedundancySupport=Warm. ServerUriArray is self first and differs per node. Same URIs in both. |
| L2 bands | read -n i=2267 on both |
Primary 250, Standby 90 |
| L3 standby data | read and subscribe of a raw tag and its UNS reference on the Standby |
BadNoCommunication/null; the Primary shows Good at the same moment |
| L4 standby alarms | alarms on both. Trip a scripted alarm via its source tag. |
Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5). |
| L5 manual failover | AdminUI manual failover (docs/Redundancy.md §Manual failover). Keep the CLI subscriptions open on both. |
Old Primary: SL drops to 90, then BadNoCommunication, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). |
| L6 crash | docker kill central-<primary> |
The survivor goes Active within the gap Q2 selects. Record the measured gap. |
| L7 rejoin | Restart the killed node | It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90. |
| L8 standalone | site-b-2 (or a one-node overlay) with Mode=None |
RedundancySupport=None, no i=11314, data Good |
| L9 Unset control | Another node left on Unset |
Byte-identical to pre-change: SL 240/250, SDK-default i=3709 |
| L10 writes/history | Write and HistoryRead on the Standby | Behaviour per Q1. HistoryRead is Good. |
Record evidence in docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md. Do not mark G8/G9 (#615); they
need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).
Rollout
- Merge T1–T12 dark (
Unsetdefault). The full CI floor applies. - Deploy the ScadaBridge T11 change everywhere ScadaBridge reads OtOpcUa.
- Upgrade both nodes of a pair to a revision-capable build (T3) before flipping either one.
- Flip both nodes of one docker-dev pair, then run T13. After that, flip one non-production site pair.
- Production is a separate, owner-approved change per pair. Flip the Secondary first, then the Primary, in one window.
A mixed window (one Warm, one Unset) is tolerable: the Unset node serves Good at 240/250. But do not leave it
that way, because ServerUriArray then exists on only one node.
Rollback
Set OpcUa:Redundancy:Mode=Unset on both nodes and restart them. The setting is startup-static, so there is
nothing persistent to undo: no schema changes and no artifact changes. The Revision field stays and is harmless,
because unrevisioned handling is preserved. Rolling back ScadaBridge is independent; its ServiceLevel-aware
failover also works against Unset servers.