c06769991b
Spec: per-shape contract (Unset/None/Warm), RedundancySupport + ServerUriArray graph, ServiceLevel bands, per-service standby behaviour, alarm suppression + ConditionRefresh (D5), strict first-delivery (D4), transition sequencing, ordered snapshot revisions, ScadaBridge impact, SDK 2.0.1 re-proof list, open questions. Plan: T0-T13 with wire tests, docker-dev live gate, rollout/rollback. Docs only. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
278 lines
17 KiB
Markdown
278 lines
17 KiB
Markdown
# Warm non-transparent redundancy (Gitea #717): implementation plan
|
||
|
||
Status: **DRAFT for owner approval**, together with `2026-11-11-warm-redundancy-717-design.md` (referred to below
|
||
as "the spec", with § references). No task starts until both documents and the §9 open questions are settled.
|
||
Every task ships **dark**: `OpcUa:Redundancy:Mode` defaults to `Unset`, so merging any task changes no deployed
|
||
behaviour.
|
||
|
||
Conventions:
|
||
|
||
- **Wire tests** use a real OPC Foundation client session against a loopback `OtOpcUaSdkServer`, in
|
||
`tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests` (the home of `DualEndpointTests`,
|
||
`ConditionRefreshDeferralWireTests` and `FolderConditionRefreshWireTests`).
|
||
- **Actor tests** go in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` and
|
||
`tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests`.
|
||
- Every gate gets an **allow** leg as well as a deny leg (the vacuous-test rule), and a mutation check: revert the
|
||
production line and confirm the test fails.
|
||
|
||
## T0: SDK 2.0.1 re-proof spikes (throwaway, separately approved)
|
||
|
||
The 1.5.378.106 spikes do not carry over (spec §10). Write throwaway loopback fixtures against **2.0.1**, outside
|
||
`src/`. No product edits. Each fixture is rerun independently and its evidence is recorded in
|
||
`docs/plans/evidence/2026-11-xx-717-sdk2/`.
|
||
|
||
- **a. Metadata graph.** Build it via `CreateMainNodeManagerFactory` + `AddBehaviourToPredefinedNodeAsync`. Check
|
||
Read/Browse/Translate of `i=2296` (HasTypeDefinition 2039), `i=3709`, and `i=11314` for both Warm and None. The
|
||
passive `i=11314` visit must resolve to the constructed instance.
|
||
- **b. Read projection coverage.** Cover Read, the initial sample on create, the change fan-out after
|
||
`ClearChangeMasks`, ResendData, queue size 1 vs. N, an AggregateFilter item, and Sampling → Reporting. Decide
|
||
between `OnSimpleReadValue` and a `ReadValueAttribute` override, and record which async callback paths 2.0.1
|
||
takes.
|
||
- **c. Provenance.** Stamp items through the 2.0.1 `MonitoredItemFactory` / `MonitoredItemCreateDecision`. Exercise
|
||
ModifyMonitoredItems handle change, delete/reuse, TransferSubscriptions (`IMonitoredItemTransferCoordinator`),
|
||
and a restore.
|
||
- **d. Publish/Republish.** Override `PublishAsync`/`RepublishAsync` with the 2.0.1 signatures (`RequestLifetime`).
|
||
Cover the first-delivery choice, first delivery through Republish, an ack followed by `BadMessageNotAvailable`,
|
||
and a cancelled or failed response.
|
||
- **e. Events.** Exercise the stamp/deliver split under the #855 per-item folder path and the #854 Server-object
|
||
cut. Run the promotion refresh on all three item kinds, and a client ConditionRefresh/2 while Standby.
|
||
|
||
**Exit:** a short results note per fixture, and an owner go/no-go. A "no" on (c) or (d) re-opens D4's mechanism.
|
||
D4 itself stays an owner decision.
|
||
|
||
## T1: Configuration, validation, effective self URI
|
||
|
||
**Files:**
|
||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`: options at `:29-121`. Add
|
||
`OpcUaRedundancyOptions Redundancy` (`Mode`; `PromoteWhilePrimaryUnreachable` if Q2 = yes).
|
||
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`.
|
||
- A post-load check in `StartAsync` (`:218-250`) against the effective `ApplicationConfiguration.ApplicationUri`.
|
||
|
||
**Rules (spec §3):**
|
||
- `Hot` and unknown values fail host start.
|
||
- `Warm` requires at least one peer, all URIs distinct, self not listed as a peer, and the `driver` role.
|
||
- `None` requires no peers.
|
||
|
||
**Tests:**
|
||
- A validator matrix.
|
||
- An XML `ApplicationConfigPath` whose URI differs from the options value, proving the effective URI is used.
|
||
- A boot-banner log assertion.
|
||
|
||
## T2: Standard redundancy metadata graph
|
||
|
||
**Files:**
|
||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs`: next to `CreateMasterNodeManager` (`:341`), add a
|
||
`CreateMainNodeManagerFactory` override that installs the T0a graph per mode.
|
||
- Under `Warm`, the initial `Server.ServiceLevel` is set to **1** in the same pass (spec §4.3). `Unset` must not
|
||
touch the graph.
|
||
|
||
**Tests (wire):**
|
||
- Per mode, check the `i=3709` value, the `i=2296` type definition, the presence or absence of `i=11314`, its
|
||
value order `[effectiveSelf, peers…]`, and that it is read-only (a Write returns `BadNotWritable`).
|
||
- `ServerArray` (`i=2254`) is unchanged; extend `DualEndpointTests`.
|
||
- A Warm server's ServiceLevel reads 1 before the first publish.
|
||
- Security profiles and startup are unaffected under `Unset` (golden read of today's values).
|
||
|
||
## T3: Ordered redundancy snapshots (spec §4.2)
|
||
|
||
**Files:**
|
||
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/RedundancyStateChanged.cs` (`:9-11`): add an optional
|
||
`SnapshotRevision? Revision` (additive).
|
||
- `RedundancyStateActor.cs`:
|
||
- stamp the revision in `PublishIfDirty` (`:105-113`);
|
||
- set `IncarnationTick` in `PreStart`;
|
||
- read the host's `UpNumber` from `Cluster.SelfMember`.
|
||
- Subscribers: accept only a newer revision in `OpcUaPublishActor.HandleRedundancyStateChanged` (`:1494-1498`) and
|
||
`DriverHostActor.OnRedundancyStateChanged` (`:2555`).
|
||
- An unrevisioned snapshot keeps today's handling for the existing gates.
|
||
|
||
**Tests:**
|
||
- Ordering unit tests: an older singleton incarnation's heartbeat after a newer snapshot is dropped; a same-process
|
||
singleton restart advances.
|
||
- A **real 2-node cluster** handover test (pattern: `RedundancyPrimaryElectionTests`). It holds back the old
|
||
incarnation's last publish through the `_broadcastOverride` seam (`:111`) and asserts the subscriber keeps the
|
||
new Primary.
|
||
- Serialization round-trip of the extended message, plus a mixed-version decode with the field absent.
|
||
- The pinned assumption: a removed node cannot rejoin without a new `ActorSystem`.
|
||
|
||
## T4: Warm role verdict and ServiceLevel bands (spec §4.1, §4.3)
|
||
|
||
**Files:**
|
||
- New `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/WarmRoleAuthority.cs`: a pure function of membership view,
|
||
accepted snapshot, `EndpointListening` and `SelfMember` status.
|
||
- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs`: add `ComputeWarm(inputs, isActive)`.
|
||
`Compute` itself is unchanged.
|
||
- `OpcUaPublishActor.RecomputeServiceLevel` (`:1576-1636`) branches on mode. It subscribes to `MemberUp/Removed`
|
||
(and to reachability if Q2 = yes) so the verdict re-derives on membership changes, not only on snapshots.
|
||
|
||
**Tests:**
|
||
- A truth table covering every row of §4.3 and the invariant *max(Standby) < min(Active serving)*.
|
||
- Membership/snapshot disagreement gives Standby, plus the 20 s Warning.
|
||
- Boot with an unknown role gives Standby; a lone node goes Active after its own snapshot.
|
||
- `None`/`Unset` produce byte-identical output to today's `ServiceLevelCalculatorTests`.
|
||
- DB-less mode (#721 timer) is covered.
|
||
|
||
## T5: Standby projection seam and transition sequencer (spec §4.7)
|
||
|
||
**Files:**
|
||
- New `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IStandbyProjection.cs`, with a deferred/late-bound adapter that
|
||
mirrors `DeferredServiceLevelPublisher` and retains the latest state across binding (the #719 lesson).
|
||
- Bound in `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs` next to the ServiceLevel binding.
|
||
- `OpcUaPublishActor` runs promotion and demotion as one serialized handler, in the §4.7 step order.
|
||
|
||
**Tests (actor, with a recording fake):**
|
||
- The exact step order for both directions.
|
||
- A ServiceLevel publish failure halts a promotion before the Active level and retries; a demotion continues.
|
||
- A burst of alternating verdicts converges on the last one.
|
||
- Binding after the first transition replays the current state.
|
||
|
||
## T6: Per-tag read projection (spec §4.4)
|
||
|
||
**Files:**
|
||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs`: the hook decided in T0b, reading a
|
||
node-manager-wide volatile projection flag. Stored `Value` is never modified.
|
||
- `OtOpcUaNodeManager` implements `IStandbyProjection.SetDataProjection`.
|
||
|
||
**Tests (wire):**
|
||
- Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns `BadNoCommunication`/null.
|
||
- A Standby Read of a condition child, `i=2267` or `i=3709` stays Good.
|
||
- A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected.
|
||
- Acquisition continues: flip to Active and the latest value is read without any new driver publish.
|
||
- ACL on: a denied node is still `BadUserAccessDenied` and the projection never leaks existence.
|
||
|
||
## T7: Provenance and first-delivery choice (spec §4.6, D4)
|
||
|
||
**Files:**
|
||
- `OtOpcUaSdkServer.cs`: install the custom sampled item (T0c route) for ordinary-tag `Value` items only.
|
||
- `OtOpcUaSdkServer.cs`: add the `PublishAsync`/`RepublishAsync` overrides and a new `FirstDeliveryLedger`, a
|
||
`ConditionalWeakTable` keyed by the SDK message object.
|
||
- Wire into `OtOpcUaNodeManager.CreateMonitoredItems` (`:5295`) / `OnMonitoredItemCreated` (`:5442`) without
|
||
disturbing the ACL filtering or `_liveItems`.
|
||
|
||
**Tests (wire):** port the spikes' assertion matrix to product code.
|
||
- Queued Good samples, then demotion, then the first Publish carries `BadNoCommunication`.
|
||
- Prebuilt sequences, then demotion: the first delivery is projected and a later Republish returns the same
|
||
projected payload.
|
||
- Good delivered while Active, then demotion, then Republish returns the original Good.
|
||
- A first delivery through Republish.
|
||
- A mixed subscription (ordinary tag + `i=2267` + condition-child item): only ordinary-tag items are rewritten.
|
||
- Handle modified, deleted or reused between queue and publish.
|
||
- The cached SDK object is unchanged (sacrificial-clone check).
|
||
- A bounded clone matrix (arrays, ExtensionObject, DiagnosticInfo, per c7890).
|
||
- Concurrent Publish and role flip, by a stress loop with a deterministic barrier, not sleeps.
|
||
|
||
## T8: Promotion and demotion of live data items
|
||
|
||
**Files:**
|
||
- `OtOpcUaNodeManager`: `IStandbyProjection.QueueCurrentToLiveItems(bool projected)` iterates `_liveItems`. It
|
||
reuses the QueueValue-with-`ignoreFilters` pattern (`:7303`), **not** detach or `MonitoringMode.Disabled`; the
|
||
disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped.
|
||
|
||
**Tests (wire):**
|
||
- Demotion delivers exactly one `BadNoCommunication` per item, then silence while values keep changing internally.
|
||
- Promotion delivers the current Good value without a driver publish.
|
||
- An ACL-stood-down item receives nothing either way.
|
||
- A DeadbandFilter item still receives the promotion value.
|
||
|
||
## T9: Alarm event suppression and promotion refresh (spec §4.5, D3, D5)
|
||
|
||
**Files (`OtOpcUaNodeManager`):**
|
||
- Split `ReportConditionEvent` (`:1376`) into stamp and deliver; deliver is gated by the projection flag.
|
||
- Audit all callers (`:1254`, `:1341`, `:2842`, `:4659`) and `WireMethodTransitionReporting`.
|
||
- `IStandbyProjection.RefreshAllEventItems()` runs the existing `ConditionRefresh` (`:6586`) Server-object path and
|
||
`RefreshDirectItems` (`:6974`) over every live event item, from `_allEventsItems` and `_eventItems`.
|
||
- `ReportToServerObject` (`:6060`) is untouched.
|
||
|
||
**Tests (wire, ACL off and on):**
|
||
- A new alarm on the Standby reaches no Server-object, folder or direct item.
|
||
- Model-change and audit events are still delivered on the Standby.
|
||
- A client ConditionRefresh **and** ConditionRefresh2 on the Standby return Start, retained, End (D5).
|
||
- Promotion gives Start, retained, End on all three item kinds, followed by new events.
|
||
- An ACL-denied condition is absent from the promotion refresh.
|
||
- A condition raised and cleared (Retain=false) while Standby is not replayed.
|
||
- An event reported while Active but deferred (#838, via `NotifierStall`) is still delivered after demotion.
|
||
- A Standby-stamped EventId is accepted by Acknowledge after promotion.
|
||
- The `RingEventQueueFactory` path is untouched (`RingEventMonitoredItemQueueParityTests` stay green).
|
||
- Re-run `EventQueueOverflowAclWireTests` and `ConditionMethodEventTests`.
|
||
|
||
## T10: Deploy-time mode cross-check and observability (spec §3, §7)
|
||
|
||
**Files:**
|
||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`: read this node's cluster
|
||
`RedundancyMode` from the `Clusters` array.
|
||
- `OpcUaPublishActor` compares it with the local mode at apply.
|
||
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`: add the six instruments.
|
||
|
||
**Tests:**
|
||
- A mismatch logs an Error and records the metric.
|
||
- Each counter fires on its path. Use a `MeterListener` filtered by **instrument instance**, never by meter name
|
||
alone (the #503 lesson).
|
||
|
||
## T11: ScadaBridge prerequisite (cross-repo)
|
||
|
||
- File a ScadaBridge issue (`type:feature`, `cross-repo`, `priority:p1`, linked to #717). The DCL becomes
|
||
ServiceLevel-aware: monitor `i=2267`, and fail over when it is below 200 for a configurable window while the
|
||
other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items
|
||
(spec §5).
|
||
- File the lmxopcua companion as `blocked` + `cross-repo`.
|
||
- **The Warm flip in T13/rollout is gated on that issue being deployed.**
|
||
|
||
## T12: Documentation and index
|
||
|
||
- `docs/Redundancy.md`: rewrite the `:9` block, add a Warm contract section, update the tiers table. Remove the
|
||
generic "standby / no device contact / exactly-once" wording that contradicts acquisition and at-least-once
|
||
(c7794).
|
||
- `docs/OpcUaServer.md:70`, `docs/Configuration.md` (the new keys), and `docs/security.md` (ns=0 stays ungated;
|
||
projection order vs. ACL).
|
||
- `.claude/rules/cluster-topology.md` §Redundancy.
|
||
- `docs/plans/phase-6-3-redundancy-interop-plan.md` A2/A4/A5b: under Warm, A2 expects 90 and A4/A5b become
|
||
testable. Add `docs/v2/redundancy-interop-playbook.md`.
|
||
- Umbrella: update the OtOpcUa row and `sister-project-history/otopcua.md` in `../scadaproj` on **its `main`**,
|
||
pushed (root `CLAUDE.md` procedure).
|
||
|
||
## T13: Live gate on docker-dev (central pair, `:4840` / `:4841`)
|
||
|
||
**Overlay:** new `docker-dev/docker-compose.warm.yml`, central-1/2 only. It sets:
|
||
|
||
- `OpcUa__ApplicationUri=urn:OtOpcUa:central-1|2` (matching `seed-clusters.sql`);
|
||
- `OpcUa__PeerApplicationUris__0=<the other>`;
|
||
- `OpcUa__Redundancy__Mode=Warm`.
|
||
|
||
Today both centrals advertise the default `urn:OtOpcUa` (spec §2). Login is disabled on the rig, so drive
|
||
everything yourself. Use `otopcua-cli` from `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` against both ports.
|
||
|
||
| Leg | Procedure | Pass |
|
||
|---|---|---|
|
||
| L1 metadata | `redundancy` on :4840 and :4841 | `RedundancySupport=Warm`. ServerUriArray is self first and differs per node. Same URIs in both. |
|
||
| L2 bands | `read -n i=2267` on both | Primary 250, Standby 90 |
|
||
| L3 standby data | `read` and `subscribe` of a raw tag and its UNS reference on the Standby | `BadNoCommunication`/null; the Primary shows Good at the same moment |
|
||
| L4 standby alarms | `alarms` on both. Trip a scripted alarm via its source tag. | Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5). |
|
||
| L5 manual failover | AdminUI manual failover (`docs/Redundancy.md` §Manual failover). Keep the CLI subscriptions open on both. | Old Primary: SL drops to 90, then `BadNoCommunication`, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps). |
|
||
| L6 crash | `docker kill central-<primary>` | The survivor goes Active within the gap Q2 selects. Record the measured gap. |
|
||
| L7 rejoin | Restart the killed node | It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90. |
|
||
| L8 standalone | site-b-2 (or a one-node overlay) with `Mode=None` | `RedundancySupport=None`, no `i=11314`, data Good |
|
||
| L9 Unset control | Another node left on `Unset` | Byte-identical to pre-change: SL 240/250, SDK-default `i=3709` |
|
||
| L10 writes/history | Write and HistoryRead on the Standby | Behaviour per Q1. HistoryRead is Good. |
|
||
|
||
Record evidence in `docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md`. Do **not** mark G8/G9 (#615); they
|
||
need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).
|
||
|
||
## Rollout
|
||
|
||
1. Merge T1–T12 dark (`Unset` default). The full CI floor applies.
|
||
2. Deploy the ScadaBridge T11 change everywhere ScadaBridge reads OtOpcUa.
|
||
3. Upgrade **both** nodes of a pair to a revision-capable build (T3) before flipping either one.
|
||
4. Flip both nodes of one docker-dev pair, then run T13. After that, flip one non-production site pair.
|
||
5. Production is a separate, owner-approved change per pair. Flip the **Secondary first**, then the Primary, in
|
||
one window.
|
||
|
||
A mixed window (one `Warm`, one `Unset`) is tolerable: the Unset node serves Good at 240/250. But do not leave it
|
||
that way, because `ServerUriArray` then exists on only one node.
|
||
|
||
## Rollback
|
||
|
||
Set `OpcUa:Redundancy:Mode=Unset` on both nodes and restart them. The setting is startup-static, so there is
|
||
nothing persistent to undo: no schema changes and no artifact changes. The `Revision` field stays and is harmless,
|
||
because unrevisioned handling is preserved. Rolling back ScadaBridge is independent; its ServiceLevel-aware
|
||
failover also works against `Unset` servers.
|