Files
lmxopcua/docs/plans/2026-11-11-warm-redundancy-717-plan.md
T
Joseph Doherty c06769991b docs(plans): #717 Warm redundancy design spec + implementation plan (draft for owner approval)
Spec: per-shape contract (Unset/None/Warm), RedundancySupport + ServerUriArray graph,
ServiceLevel bands, per-service standby behaviour, alarm suppression + ConditionRefresh
(D5), strict first-delivery (D4), transition sequencing, ordered snapshot revisions,
ScadaBridge impact, SDK 2.0.1 re-proof list, open questions. Plan: T0-T13 with wire
tests, docker-dev live gate, rollout/rollback. Docs only.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 18:19:14 -05:00

278 lines
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Warm non-transparent redundancy (Gitea #717): implementation plan
Status: **DRAFT for owner approval**, together with `2026-11-11-warm-redundancy-717-design.md` (referred to below
as "the spec", with § references). No task starts until both documents and the §9 open questions are settled.
Every task ships **dark**: `OpcUa:Redundancy:Mode` defaults to `Unset`, so merging any task changes no deployed
behaviour.
Conventions:
- **Wire tests** use a real OPC Foundation client session against a loopback `OtOpcUaSdkServer`, in
`tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests` (the home of `DualEndpointTests`,
`ConditionRefreshDeferralWireTests` and `FolderConditionRefreshWireTests`).
- **Actor tests** go in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` and
`tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests`.
- Every gate gets an **allow** leg as well as a deny leg (the vacuous-test rule), and a mutation check: revert the
production line and confirm the test fails.
## T0: SDK 2.0.1 re-proof spikes (throwaway, separately approved)
The 1.5.378.106 spikes do not carry over (spec §10). Write throwaway loopback fixtures against **2.0.1**, outside
`src/`. No product edits. Each fixture is rerun independently and its evidence is recorded in
`docs/plans/evidence/2026-11-xx-717-sdk2/`.
- **a. Metadata graph.** Build it via `CreateMainNodeManagerFactory` + `AddBehaviourToPredefinedNodeAsync`. Check
Read/Browse/Translate of `i=2296` (HasTypeDefinition 2039), `i=3709`, and `i=11314` for both Warm and None. The
passive `i=11314` visit must resolve to the constructed instance.
- **b. Read projection coverage.** Cover Read, the initial sample on create, the change fan-out after
`ClearChangeMasks`, ResendData, queue size 1 vs. N, an AggregateFilter item, and Sampling → Reporting. Decide
between `OnSimpleReadValue` and a `ReadValueAttribute` override, and record which async callback paths 2.0.1
takes.
- **c. Provenance.** Stamp items through the 2.0.1 `MonitoredItemFactory` / `MonitoredItemCreateDecision`. Exercise
ModifyMonitoredItems handle change, delete/reuse, TransferSubscriptions (`IMonitoredItemTransferCoordinator`),
and a restore.
- **d. Publish/Republish.** Override `PublishAsync`/`RepublishAsync` with the 2.0.1 signatures (`RequestLifetime`).
Cover the first-delivery choice, first delivery through Republish, an ack followed by `BadMessageNotAvailable`,
and a cancelled or failed response.
- **e. Events.** Exercise the stamp/deliver split under the #855 per-item folder path and the #854 Server-object
cut. Run the promotion refresh on all three item kinds, and a client ConditionRefresh/2 while Standby.
**Exit:** a short results note per fixture, and an owner go/no-go. A "no" on (c) or (d) re-opens D4's mechanism.
D4 itself stays an owner decision.
## T1: Configuration, validation, effective self URI
**Files:**
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`: options at `:29-121`. Add
`OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable` if Q2 = yes).
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`.
- A post-load check in `StartAsync` (`:218-250`) against the effective `ApplicationConfiguration.ApplicationUri`.
**Rules (spec §3):**
- `Hot` and unknown values fail host start.
- `Warm` requires at least one peer, all URIs distinct, self not listed as a peer, and the `driver` role.
- `None` requires no peers.
**Tests:**
- A validator matrix.
- An XML `ApplicationConfigPath` whose URI differs from the options value, proving the effective URI is used.
- A boot-banner log assertion.
## T2: Standard redundancy metadata graph
**Files:**
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs`: next to `CreateMasterNodeManager` (`:341`), add a
`CreateMainNodeManagerFactory` override that installs the T0a graph per mode.
- Under `Warm`, the initial `Server.ServiceLevel` is set to **1** in the same pass (spec §4.3). `Unset` must not
touch the graph.
**Tests (wire):**
- Per mode, check the `i=3709` value, the `i=2296` type definition, the presence or absence of `i=11314`, its
value order `[effectiveSelf, peers…]`, and that it is read-only (a Write returns `BadNotWritable`).
- `ServerArray` (`i=2254`) is unchanged; extend `DualEndpointTests`.
- A Warm server's ServiceLevel reads 1 before the first publish.
- Security profiles and startup are unaffected under `Unset` (golden read of today's values).
## T3: Ordered redundancy snapshots (spec §4.2)
**Files:**
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/RedundancyStateChanged.cs` (`:9-11`): add an optional
`SnapshotRevision? Revision` (additive).
- `RedundancyStateActor.cs`:
- stamp the revision in `PublishIfDirty` (`:105-113`);
- set `IncarnationTick` in `PreStart`;
- read the host's `UpNumber` from `Cluster.SelfMember`.
- Subscribers: accept only a newer revision in `OpcUaPublishActor.HandleRedundancyStateChanged` (`:1494-1498`) and
`DriverHostActor.OnRedundancyStateChanged` (`:2555`).
- An unrevisioned snapshot keeps today's handling for the existing gates.
**Tests:**
- Ordering unit tests: an older singleton incarnation's heartbeat after a newer snapshot is dropped; a same-process
singleton restart advances.
- A **real 2-node cluster** handover test (pattern: `RedundancyPrimaryElectionTests`). It holds back the old
incarnation's last publish through the `_broadcastOverride` seam (`:111`) and asserts the subscriber keeps the
new Primary.
- Serialization round-trip of the extended message, plus a mixed-version decode with the field absent.
- The pinned assumption: a removed node cannot rejoin without a new `ActorSystem`.
## T4: Warm role verdict and ServiceLevel bands (spec §4.1, §4.3)
**Files:**
- New `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/WarmRoleAuthority.cs`: a pure function of membership view,
accepted snapshot, `EndpointListening` and `SelfMember` status.
- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs`: add `ComputeWarm(inputs, isActive)`.
`Compute` itself is unchanged.
- `OpcUaPublishActor.RecomputeServiceLevel` (`:1576-1636`) branches on mode. It subscribes to `MemberUp/Removed`
(and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots.
**Tests:**
- A truth table covering every row of §4.3 and the invariant *max(Standby) < min(Active serving)*.
- Membership/snapshot disagreement gives Standby, plus the 20 s Warning.
- Boot with an unknown role gives Standby; a lone node goes Active after its own snapshot.
- `None`/`Unset` produce byte-identical output to today's `ServiceLevelCalculatorTests`.
- DB-less mode (#721 timer) is covered.
## T5: Standby projection seam and transition sequencer (spec §4.7)
**Files:**
- New `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IStandbyProjection.cs`, with a deferred/late-bound adapter that
mirrors `DeferredServiceLevelPublisher` and retains the latest state across binding (the #719 lesson).
- Bound in `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs` next to the ServiceLevel binding.
- `OpcUaPublishActor` runs promotion and demotion as one serialized handler, in the §4.7 step order.
**Tests (actor, with a recording fake):**
- The exact step order for both directions.
- A ServiceLevel publish failure halts a promotion before the Active level and retries; a demotion continues.
- A burst of alternating verdicts converges on the last one.
- Binding after the first transition replays the current state.
## T6: Per-tag read projection (spec §4.4)
**Files:**
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs`: the hook decided in T0b, reading a
node-manager-wide volatile projection flag. Stored `Value` is never modified.
- `OtOpcUaNodeManager` implements `IStandbyProjection.SetDataProjection`.
**Tests (wire):**
- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadNoCommunication`/null.
- A Standby Read of a condition child, `i=2267` or `i=3709` stays Good.
- A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected.
- Acquisition continues: flip to Active and the latest value is read without any new driver publish.
- ACL on: a denied node is still `BadUserAccessDenied` and the projection never leaks existence.
## T7: Provenance and first-delivery choice (spec §4.6, D4)
**Files:**
- `OtOpcUaSdkServer.cs`: install the custom sampled item (T0c route) for ordinary-tag `Value` items only.
- `OtOpcUaSdkServer.cs`: add the `PublishAsync`/`RepublishAsync` overrides and a new `FirstDeliveryLedger`, a
`ConditionalWeakTable` keyed by the SDK message object.
- Wire into `OtOpcUaNodeManager.CreateMonitoredItems` (`:5295`) / `OnMonitoredItemCreated` (`:5442`) without
disturbing the ACL filtering or `_liveItems`.
**Tests (wire):** port the spikes' assertion matrix to product code.
- Queued Good samples, then demotion, then the first Publish carries `BadNoCommunication`.
- Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same
projected payload.
- Good delivered while Active, then demotion, then Republish returns the original Good.
- A first delivery through Republish.
- A mixed subscription (ordinary tag + `i=2267` + condition-child item): only ordinary-tag items are rewritten.
- Handle modified, deleted or reused between queue and publish.
- The cached SDK object is unchanged (sacrificial-clone check).
- A bounded clone matrix (arrays, ExtensionObject, DiagnosticInfo, per c7890).
- Concurrent Publish and role flip, by a stress loop with a deterministic barrier, not sleeps.
## T8: Promotion and demotion of live data items
**Files:**
- `OtOpcUaNodeManager`: `IStandbyProjection.QueueCurrentToLiveItems(bool projected)` iterates `_liveItems`. It
reuses the QueueValue-with-`ignoreFilters` pattern (`:7303`), **not** detach or `MonitoringMode.Disabled`; the
disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped.
**Tests (wire):**
- Demotion delivers exactly one `BadNoCommunication` per item, then silence while values keep changing internally.
- Promotion delivers the current Good value without a driver publish.
- An ACL-stood-down item receives nothing either way.
- A DeadbandFilter item still receives the promotion value.
## T9: Alarm event suppression and promotion refresh (spec §4.5, D3, D5)
**Files (`OtOpcUaNodeManager`):**
- Split `ReportConditionEvent` (`:1376`) into stamp and deliver; deliver is gated by the projection flag.
- Audit all callers (`:1254`, `:1341`, `:2842`, `:4659`) and `WireMethodTransitionReporting`.
- `IStandbyProjection.RefreshAllEventItems()` runs the existing `ConditionRefresh` (`:6586`) Server-object path and
`RefreshDirectItems` (`:6974`) over every live event item, from `_allEventsItems` and `_eventItems`.
- `ReportToServerObject` (`:6060`) is untouched.
**Tests (wire, ACL off and on):**
- A new alarm on the Standby reaches no Server-object, folder or direct item.
- Model-change and audit events are still delivered on the Standby.
- A client ConditionRefresh **and** ConditionRefresh2 on the Standby return Start, retained, End (D5).
- Promotion gives Start, retained, End on all three item kinds, followed by new events.
- An ACL-denied condition is absent from the promotion refresh.
- A condition raised and cleared (Retain=false) while Standby is not replayed.
- An event reported while Active but deferred (#838, via `NotifierStall`) is still delivered after demotion.
- A Standby-stamped EventId is accepted by Acknowledge after promotion.
- The `RingEventQueueFactory` path is untouched (`RingEventMonitoredItemQueueParityTests` stay green).
- Re-run `EventQueueOverflowAclWireTests` and `ConditionMethodEventTests`.
## T10: Deploy-time mode cross-check and observability (spec §3, §7)
**Files:**
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`: read this node's cluster
`RedundancyMode` from the `Clusters` array.
- `OpcUaPublishActor` compares it with the local mode at apply.
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`: add the six instruments.
**Tests:**
- A mismatch logs an Error and records the metric.
- Each counter fires on its path. Use a `MeterListener` filtered by **instrument instance**, never by meter name
alone (the #503 lesson).
## T11: ScadaBridge prerequisite (cross-repo)
- File a ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes
ServiceLevel-aware: monitor `i=2267`, and fail over when it is below 200 for a configurable window while the
other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items
(spec §5).
- File the lmxopcua companion as `blocked` + `cross-repo`.
- **The Warm flip in T13/rollout is gated on that issue being deployed.**
## T12: Documentation and index
- `docs/Redundancy.md`: rewrite the `:9` block, add a Warm contract section, update the tiers table. Remove the
generic "standby / no device contact / exactly-once" wording that contradicts acquisition and at-least-once
(c7794).
- `docs/OpcUaServer.md:70`, `docs/Configuration.md` (the new keys), and `docs/security.md` (ns=0 stays ungated;
projection order vs. ACL).
- `.claude/rules/cluster-topology.md` §Redundancy.
- `docs/plans/phase-6-3-redundancy-interop-plan.md` A2/A4/A5b: under Warm, A2 expects 90 and A4/A5b become
testable. Add `docs/v2/redundancy-interop-playbook.md`.
- Umbrella: update the OtOpcUa row and `sister-project-history/otopcua.md` in `../scadaproj` on **its `main`**,
pushed (root `CLAUDE.md` procedure).
## T13: Live gate on docker-dev (central pair, `:4840` / `:4841`)
**Overlay:** new `docker-dev/docker-compose.warm.yml`, central-1/2 only. It sets:
- `OpcUa__ApplicationUri=urn:OtOpcUa:central-1|2` (matching `seed-clusters.sql`);
- `OpcUa__PeerApplicationUris__0=<the other>`;
- `OpcUa__Redundancy__Mode=Warm`.
Today both centrals advertise the default `urn:OtOpcUa` (spec §2). Login is disabled on the rig, so drive
everything yourself. Use `otopcua-cli` from `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` against both ports.
| Leg | Procedure | Pass |
|---|---|---|
| L1 metadata | `redundancy` on :4840 and :4841 | `RedundancySupport=Warm`. ServerUriArray is self first and differs per node. Same URIs in both. |
| L2 bands | `read -n i=2267` on both | Primary 250, Standby 90 |
| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadNoCommunication`/null; the Primary shows Good at the same moment |
| L4 standby alarms | `alarms` on both. Trip a scripted alarm via its source tag. | Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5). |
| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadNoCommunication`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). |
| L6 crash | `docker kill central-<primary>` | The survivor goes Active within the gap Q2 selects. Record the measured gap. |
| L7 rejoin | Restart the killed node | It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90. |
| L8 standalone | site-b-2 (or a one-node overlay) with `Mode=None` | `RedundancySupport=None`, no `i=11314`, data Good |
| L9 Unset control | Another node left on `Unset` | Byte-identical to pre-change: SL 240/250, SDK-default `i=3709` |
| L10 writes/history | Write and HistoryRead on the Standby | Behaviour per Q1. HistoryRead is Good. |
Record evidence in `docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md`. Do **not** mark G8/G9 (#615); they
need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).
## Rollout
1. Merge T1–T12 dark (`Unset` default). The full CI floor applies.
2. Deploy the ScadaBridge T11 change everywhere ScadaBridge reads OtOpcUa.
3. Upgrade **both** nodes of a pair to a revision-capable build (T3) before flipping either one.
4. Flip both nodes of one docker-dev pair, then run T13. After that, flip one non-production site pair.
5. Production is a separate, owner-approved change per pair. Flip the **Secondary first**, then the Primary, in
one window.
A mixed window (one `Warm`, one `Unset`) is tolerable: the Unset node serves Good at 240/250. But do not leave it
that way, because `ServerUriArray` then exists on only one node.
## Rollback
Set `OpcUa:Redundancy:Mode=Unset` on both nodes and restart them. The setting is startup-static, so there is
nothing persistent to undo: no schema changes and no artifact changes. The `Revision` field stays and is harmless,
because unrevisioned handling is preserved. Rolling back ScadaBridge is independent; its ServiceLevel-aware
failover also works against `Unset` servers.