Files
lmxopcua/docs/plans/2026-11-11-warm-redundancy-717-plan.md
T
Joseph Doherty c06769991b docs(plans): #717 Warm redundancy design spec + implementation plan (draft for owner approval)
Spec: per-shape contract (Unset/None/Warm), RedundancySupport + ServerUriArray graph,
ServiceLevel bands, per-service standby behaviour, alarm suppression + ConditionRefresh
(D5), strict first-delivery (D4), transition sequencing, ordered snapshot revisions,
ScadaBridge impact, SDK 2.0.1 re-proof list, open questions. Plan: T0-T13 with wire
tests, docker-dev live gate, rollout/rollback. Docs only.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 18:19:14 -05:00

17 KiB
Raw Blame History

Warm non-transparent redundancy (Gitea #717): implementation plan

Status: DRAFT for owner approval, together with 2026-11-11-warm-redundancy-717-design.md (referred to below as "the spec", with § references). No task starts until both documents and the §9 open questions are settled. Every task ships dark: OpcUa:Redundancy:Mode defaults to Unset, so merging any task changes no deployed behaviour.

Conventions:

  • Wire tests use a real OPC Foundation client session against a loopback OtOpcUaSdkServer, in tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests (the home of DualEndpointTests, ConditionRefreshDeferralWireTests and FolderConditionRefreshWireTests).
  • Actor tests go in tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests and tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.
  • Every gate gets an allow leg as well as a deny leg (the vacuous-test rule), and a mutation check: revert the production line and confirm the test fails.

T0: SDK 2.0.1 re-proof spikes (throwaway, separately approved)

The 1.5.378.106 spikes do not carry over (spec §10). Write throwaway loopback fixtures against 2.0.1, outside src/. No product edits. Each fixture is rerun independently and its evidence is recorded in docs/plans/evidence/2026-11-xx-717-sdk2/.

  • a. Metadata graph. Build it via CreateMainNodeManagerFactory + AddBehaviourToPredefinedNodeAsync. Check Read/Browse/Translate of i=2296 (HasTypeDefinition 2039), i=3709, and i=11314 for both Warm and None. The passive i=11314 visit must resolve to the constructed instance.
  • b. Read projection coverage. Cover Read, the initial sample on create, the change fan-out after ClearChangeMasks, ResendData, queue size 1 vs. N, an AggregateFilter item, and Sampling → Reporting. Decide between OnSimpleReadValue and a ReadValueAttribute override, and record which async callback paths 2.0.1 takes.
  • c. Provenance. Stamp items through the 2.0.1 MonitoredItemFactory / MonitoredItemCreateDecision. Exercise ModifyMonitoredItems handle change, delete/reuse, TransferSubscriptions (IMonitoredItemTransferCoordinator), and a restore.
  • d. Publish/Republish. Override PublishAsync/RepublishAsync with the 2.0.1 signatures (RequestLifetime). Cover the first-delivery choice, first delivery through Republish, an ack followed by BadMessageNotAvailable, and a cancelled or failed response.
  • e. Events. Exercise the stamp/deliver split under the #855 per-item folder path and the #854 Server-object cut. Run the promotion refresh on all three item kinds, and a client ConditionRefresh/2 while Standby.

Exit: a short results note per fixture, and an owner go/no-go. A "no" on (c) or (d) re-opens D4's mechanism. D4 itself stays an owner decision.

T1: Configuration, validation, effective self URI

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs: options at :29-121. Add OpcUaRedundancyOptions Redundancy (Mode; PromoteWhilePrimaryUnreachable if Q2 = yes).
  • src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs.
  • A post-load check in StartAsync (:218-250) against the effective ApplicationConfiguration.ApplicationUri.

Rules (spec §3):

  • Hot and unknown values fail host start.
  • Warm requires at least one peer, all URIs distinct, self not listed as a peer, and the driver role.
  • None requires no peers.

Tests:

  • A validator matrix.
  • An XML ApplicationConfigPath whose URI differs from the options value, proving the effective URI is used.
  • A boot-banner log assertion.

T2: Standard redundancy metadata graph

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs: next to CreateMasterNodeManager (:341), add a CreateMainNodeManagerFactory override that installs the T0a graph per mode.
  • Under Warm, the initial Server.ServiceLevel is set to 1 in the same pass (spec §4.3). Unset must not touch the graph.

Tests (wire):

  • Per mode, check the i=3709 value, the i=2296 type definition, the presence or absence of i=11314, its value order [effectiveSelf, peers…], and that it is read-only (a Write returns BadNotWritable).
  • ServerArray (i=2254) is unchanged; extend DualEndpointTests.
  • A Warm server's ServiceLevel reads 1 before the first publish.
  • Security profiles and startup are unaffected under Unset (golden read of today's values).

T3: Ordered redundancy snapshots (spec §4.2)

Files:

  • src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/RedundancyStateChanged.cs (:9-11): add an optional SnapshotRevision? Revision (additive).
  • RedundancyStateActor.cs:
    • stamp the revision in PublishIfDirty (:105-113);
    • set IncarnationTick in PreStart;
    • read the host's UpNumber from Cluster.SelfMember.
  • Subscribers: accept only a newer revision in OpcUaPublishActor.HandleRedundancyStateChanged (:1494-1498) and DriverHostActor.OnRedundancyStateChanged (:2555).
  • An unrevisioned snapshot keeps today's handling for the existing gates.

Tests:

  • Ordering unit tests: an older singleton incarnation's heartbeat after a newer snapshot is dropped; a same-process singleton restart advances.
  • A real 2-node cluster handover test (pattern: RedundancyPrimaryElectionTests). It holds back the old incarnation's last publish through the _broadcastOverride seam (:111) and asserts the subscriber keeps the new Primary.
  • Serialization round-trip of the extended message, plus a mixed-version decode with the field absent.
  • The pinned assumption: a removed node cannot rejoin without a new ActorSystem.

T4: Warm role verdict and ServiceLevel bands (spec §4.1, §4.3)

Files:

  • New src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/WarmRoleAuthority.cs: a pure function of membership view, accepted snapshot, EndpointListening and SelfMember status.
  • src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs: add ComputeWarm(inputs, isActive). Compute itself is unchanged.
  • OpcUaPublishActor.RecomputeServiceLevel (:1576-1636) branches on mode. It subscribes to MemberUp/Removed (and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots.

Tests:

  • A truth table covering every row of §4.3 and the invariant max(Standby) < min(Active serving).
  • Membership/snapshot disagreement gives Standby, plus the 20 s Warning.
  • Boot with an unknown role gives Standby; a lone node goes Active after its own snapshot.
  • None/Unset produce byte-identical output to today's ServiceLevelCalculatorTests.
  • DB-less mode (#721 timer) is covered.

T5: Standby projection seam and transition sequencer (spec §4.7)

Files:

  • New src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IStandbyProjection.cs, with a deferred/late-bound adapter that mirrors DeferredServiceLevelPublisher and retains the latest state across binding (the #719 lesson).
  • Bound in src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs next to the ServiceLevel binding.
  • OpcUaPublishActor runs promotion and demotion as one serialized handler, in the §4.7 step order.

Tests (actor, with a recording fake):

  • The exact step order for both directions.
  • A ServiceLevel publish failure halts a promotion before the Active level and retries; a demotion continues.
  • A burst of alternating verdicts converges on the last one.
  • Binding after the first transition replays the current state.

T6: Per-tag read projection (spec §4.4)

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs: the hook decided in T0b, reading a node-manager-wide volatile projection flag. Stored Value is never modified.
  • OtOpcUaNodeManager implements IStandbyProjection.SetDataProjection.

Tests (wire):

  • Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns BadNoCommunication/null.
  • A Standby Read of a condition child, i=2267 or i=3709 stays Good.
  • A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected.
  • Acquisition continues: flip to Active and the latest value is read without any new driver publish.
  • ACL on: a denied node is still BadUserAccessDenied and the projection never leaks existence.

T7: Provenance and first-delivery choice (spec §4.6, D4)

Files:

  • OtOpcUaSdkServer.cs: install the custom sampled item (T0c route) for ordinary-tag Value items only.
  • OtOpcUaSdkServer.cs: add the PublishAsync/RepublishAsync overrides and a new FirstDeliveryLedger, a ConditionalWeakTable keyed by the SDK message object.
  • Wire into OtOpcUaNodeManager.CreateMonitoredItems (:5295) / OnMonitoredItemCreated (:5442) without disturbing the ACL filtering or _liveItems.

Tests (wire): port the spikes' assertion matrix to product code.

  • Queued Good samples, then demotion, then the first Publish carries BadNoCommunication.
  • Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same projected payload.
  • Good delivered while Active, then demotion, then Republish returns the original Good.
  • A first delivery through Republish.
  • A mixed subscription (ordinary tag + i=2267 + condition-child item): only ordinary-tag items are rewritten.
  • Handle modified, deleted or reused between queue and publish.
  • The cached SDK object is unchanged (sacrificial-clone check).
  • A bounded clone matrix (arrays, ExtensionObject, DiagnosticInfo, per c7890).
  • Concurrent Publish and role flip, by a stress loop with a deterministic barrier, not sleeps.

T8: Promotion and demotion of live data items

Files:

  • OtOpcUaNodeManager: IStandbyProjection.QueueCurrentToLiveItems(bool projected) iterates _liveItems. It reuses the QueueValue-with-ignoreFilters pattern (:7303), not detach or MonitoringMode.Disabled; the disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped.

Tests (wire):

  • Demotion delivers exactly one BadNoCommunication per item, then silence while values keep changing internally.
  • Promotion delivers the current Good value without a driver publish.
  • An ACL-stood-down item receives nothing either way.
  • A DeadbandFilter item still receives the promotion value.

T9: Alarm event suppression and promotion refresh (spec §4.5, D3, D5)

Files (OtOpcUaNodeManager):

  • Split ReportConditionEvent (:1376) into stamp and deliver; deliver is gated by the projection flag.
  • Audit all callers (:1254, :1341, :2842, :4659) and WireMethodTransitionReporting.
  • IStandbyProjection.RefreshAllEventItems() runs the existing ConditionRefresh (:6586) Server-object path and RefreshDirectItems (:6974) over every live event item, from _allEventsItems and _eventItems.
  • ReportToServerObject (:6060) is untouched.

Tests (wire, ACL off and on):

  • A new alarm on the Standby reaches no Server-object, folder or direct item.
  • Model-change and audit events are still delivered on the Standby.
  • A client ConditionRefresh and ConditionRefresh2 on the Standby return Start, retained, End (D5).
  • Promotion gives Start, retained, End on all three item kinds, followed by new events.
  • An ACL-denied condition is absent from the promotion refresh.
  • A condition raised and cleared (Retain=false) while Standby is not replayed.
  • An event reported while Active but deferred (#838, via NotifierStall) is still delivered after demotion.
  • A Standby-stamped EventId is accepted by Acknowledge after promotion.
  • The RingEventQueueFactory path is untouched (RingEventMonitoredItemQueueParityTests stay green).
  • Re-run EventQueueOverflowAclWireTests and ConditionMethodEventTests.

T10: Deploy-time mode cross-check and observability (spec §3, §7)

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs: read this node's cluster RedundancyMode from the Clusters array.
  • OpcUaPublishActor compares it with the local mode at apply.
  • src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs: add the six instruments.

Tests:

  • A mismatch logs an Error and records the metric.
  • Each counter fires on its path. Use a MeterListener filtered by instrument instance, never by meter name alone (the #503 lesson).

T11: ScadaBridge prerequisite (cross-repo)

  • File a ScadaBridge issue (type:feature, cross-repo, priority:p1, linked to #717). The DCL becomes ServiceLevel-aware: monitor i=2267, and fail over when it is below 200 for a configurable window while the other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items (spec §5).
  • File the lmxopcua companion as blocked + cross-repo.
  • The Warm flip in T13/rollout is gated on that issue being deployed.

T12: Documentation and index

  • docs/Redundancy.md: rewrite the :9 block, add a Warm contract section, update the tiers table. Remove the generic "standby / no device contact / exactly-once" wording that contradicts acquisition and at-least-once (c7794).
  • docs/OpcUaServer.md:70, docs/Configuration.md (the new keys), and docs/security.md (ns=0 stays ungated; projection order vs. ACL).
  • .claude/rules/cluster-topology.md §Redundancy.
  • docs/plans/phase-6-3-redundancy-interop-plan.md A2/A4/A5b: under Warm, A2 expects 90 and A4/A5b become testable. Add docs/v2/redundancy-interop-playbook.md.
  • Umbrella: update the OtOpcUa row and sister-project-history/otopcua.md in ../scadaproj on its main, pushed (root CLAUDE.md procedure).

T13: Live gate on docker-dev (central pair, :4840 / :4841)

Overlay: new docker-dev/docker-compose.warm.yml, central-1/2 only. It sets:

  • OpcUa__ApplicationUri=urn:OtOpcUa:central-1|2 (matching seed-clusters.sql);
  • OpcUa__PeerApplicationUris__0=<the other>;
  • OpcUa__Redundancy__Mode=Warm.

Today both centrals advertise the default urn:OtOpcUa (spec §2). Login is disabled on the rig, so drive everything yourself. Use otopcua-cli from src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI against both ports.

Leg Procedure Pass
L1 metadata redundancy on :4840 and :4841 RedundancySupport=Warm. ServerUriArray is self first and differs per node. Same URIs in both.
L2 bands read -n i=2267 on both Primary 250, Standby 90
L3 standby data read and subscribe of a raw tag and its UNS reference on the Standby BadNoCommunication/null; the Primary shows Good at the same moment
L4 standby alarms alarms on both. Trip a scripted alarm via its source tag. Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5).
L5 manual failover AdminUI manual failover (docs/Redundancy.md §Manual failover). Keep the CLI subscriptions open on both. Old Primary: SL drops to 90, then BadNoCommunication, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps).
L6 crash docker kill central-<primary> The survivor goes Active within the gap Q2 selects. Record the measured gap.
L7 rejoin Restart the killed node It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90.
L8 standalone site-b-2 (or a one-node overlay) with Mode=None RedundancySupport=None, no i=11314, data Good
L9 Unset control Another node left on Unset Byte-identical to pre-change: SL 240/250, SDK-default i=3709
L10 writes/history Write and HistoryRead on the Standby Behaviour per Q1. HistoryRead is Good.

Record evidence in docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md. Do not mark G8/G9 (#615); they need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).

Rollout

  1. Merge T1–T12 dark (Unset default). The full CI floor applies.
  2. Deploy the ScadaBridge T11 change everywhere ScadaBridge reads OtOpcUa.
  3. Upgrade both nodes of a pair to a revision-capable build (T3) before flipping either one.
  4. Flip both nodes of one docker-dev pair, then run T13. After that, flip one non-production site pair.
  5. Production is a separate, owner-approved change per pair. Flip the Secondary first, then the Primary, in one window.

A mixed window (one Warm, one Unset) is tolerable: the Unset node serves Good at 240/250. But do not leave it that way, because ServerUriArray then exists on only one node.

Rollback

Set OpcUa:Redundancy:Mode=Unset on both nodes and restart them. The setting is startup-static, so there is nothing persistent to undo: no schema changes and no artifact changes. The Revision field stays and is harmless, because unrevisioned handling is preserved. Rolling back ScadaBridge is independent; its ServiceLevel-aware failover also works against Unset servers.