test(717): deferred-event boundary, ACL-filtered promotion refresh, native-ack refusal, effective-URI XML load; docs + T13 overlay
v2-ci / unit-tests (push) Has been cancelled

- Wire: an event reported while Active but deferred (#838) is delivered after demotion; the promotion refresh
  withholds an ACL-denied condition (positive control alongside); ConditionRefresh2 and model-change events on the
  Standby; a native Acknowledge is refused and never routed; Warm validates and advertises the EFFECTIVE
  ApplicationUri from an ApplicationConfigPath XML load.
- docs/Redundancy.md: Warm section, the Unset/None i=11314 behaviour (orphan readable by NodeId under Unset,
  deleted under None), instruments; OpcUaServer.md, Configuration.md, security.md, cluster-topology rule.
- docker-dev/docker-compose.warm.yml for the T13 gate (central pair, Mode=Warm).
- Plan: implementation status and the test gaps left for T13/follow-up.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 03:39:49 -05:00
parent 7829b937d1
commit f817c77e02
9 changed files with 357 additions and 14 deletions
+144 -2
View File
@@ -6,7 +6,17 @@ OtOpcUa runs as a 2-node **non-transparent** redundant pair (client-side failove
> **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. Clients should read `Server.ServerArray` for peer discovery today.
>
> **Standard redundancy metadata — not yet advertised (#717).** Nothing in `src/` sets `Server.ServerRedundancy.RedundancySupport` (`i=3709`), so it reads the SDK default regardless of the cluster's `RedundancyMode`, and `Server.ServerRedundancy` is the base `ServerRedundancyType` with **no** `ServerUriArray` (`i=11314`, mandatory only on `NonTransparentRedundancyType`). `ServerArray` is **not** a substitute for it. This is **not** blocked on an SDK upgrade: hermetic spikes on the pinned SDK 1.5.378.106 built the correct startup-static graph (object `i=2296` typed `NonTransparentRedundancyType` `i=2039`, `RedundancySupport`, `ServerUriArray` = effective self URI then peers) through `StandardServer.CreateMainNodeManagerFactory` + `ConfigurationNodeManager.AddBehaviourToPredefinedNode`. What blocks it is the contract: the owner-selected target is **Warm with an explicit degraded standby** (standby ServiceLevel in the degraded 2–199 band, ordinary standby tag Reads/subscriptions degraded while internal acquisition continues, standby alarm events suppressed and refreshed on promotion) and **None** for standalone deployments. Today's Secondary serves live Good data at ServiceLevel 240, so advertising `Warm` without that standby behaviour would misdescribe the server; the design, spec and plan for it are pending under #717. Interop gates A4/A5b in `docs/plans/phase-6-3-redundancy-interop-plan.md` stay unmet until then.
> **Standard redundancy metadata (#717) — opt-in, default off.** `OpcUa:Redundancy:Mode` selects the contract. The
> default `Unset` is byte-for-byte the pre-#717 server: nothing writes `Server.ServerRedundancy.RedundancySupport`
> (`i=3709`, so it reads the SDK default `None`), `Server.ServerRedundancy` stays the base `ServerRedundancyType`, the
> Secondary serves live Good data at ServiceLevel 240, and interop gates A4/A5b stay unmet. Under `Unset` the SDK's own
> predefined `ServerUriArray` node (`i=11314`) is an **orphan**: it is not referenced from `ServerRedundancy`, so Browse
> and TranslateBrowsePaths do not reach it, but a Read **by NodeId** returns Good with an empty `String[]` (T0a,
> `docs/plans/evidence/2026-11-13-717-sdk2/RESULTS.md`). That is left as the SDK builds it, because `Unset` changes
> nothing. `None` (standalone) advertises `RedundancySupport=None` and deletes the orphan, so `i=11314` is absent by
> NodeId as well (`BadNodeIdUnknown`). `Warm` (a pair) advertises the full non-transparent graph and the degraded-standby
> contract; see [Warm redundancy](#warm-non-transparent-redundancy-717). No tier sets `Warm` until ScadaBridge#138
> (ServiceLevel-aware failover) is deployed there.
> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster membership** — the oldest Up member carrying the `driver` role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.
@@ -280,6 +290,132 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
## Warm non-transparent redundancy (#717)
Spec: `docs/plans/2026-11-11-warm-redundancy-717-design.md`; plan: `docs/plans/2026-11-11-warm-redundancy-717-plan.md`.
Everything below applies only under `OpcUa:Redundancy:Mode=Warm`. `Unset` (the default) and `None` keep every behaviour
described elsewhere on this page.
### Modes
| Mode | `RedundancySupport` (`i=3709`) | `ServerRedundancy` type | `ServerUriArray` (`i=11314`) | Standby behaviour |
|---|---|---|---|---|
| `Unset` (default) | SDK default (`None`), never written | `ServerRedundancyType` | orphan, Good `[]` by NodeId, not browsable | none; Secondary serves Good at 240 |
| `None` | `None` | `ServerRedundancyType` | absent (orphan deleted) | never |
| `Warm` | `Warm` | `NonTransparentRedundancyType` (`i=2039`) | read-only, `[effectiveSelfUri, peers…]` | degraded standby (below) |
`Hot` and any other value fail host start (continuous acquisition alone is not Part 4 Hot). The mode is
**startup-static**. `Warm` needs at least one `OpcUa:PeerApplicationUris` entry, distinct URIs, the node's own
**effective** `ApplicationUri` (the one an `ApplicationConfigPath` XML load produces, not the options default) not
listed as a peer, and the `driver` role. `None` needs no peers. The metadata is installed in
`OtOpcUaSdkServer.OnNodeManagerStartedAsync` through the SDK 2.0.1 `IPredefinedNodeSubtypeReplacer`, which keeps the
well-known `i=11314`. Under `Warm` the server boots at ServiceLevel **1** (NoData) until the first computed level.
At each deploy the node compares the artifact's `ServerCluster.RedundancyMode` for its cluster with its local mode
(`None` or `Warm` only). A mismatch, `Hot` included, logs an Error and counts `otopcua.redundancy.mode_mismatch`; the
node keeps its local mode.
### Role verdict and bands
A Warm node is **Active** iff it is Up with a listening endpoint, its own cluster state names it the oldest Up `driver`
member, and the latest **accepted** redundancy snapshot names it Primary. Otherwise it is **Standby**, including before
the first accepted snapshot. With `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable` (default `true`) the oldest
**reachable** driver member also goes Active while the snapshot's Primary is unreachable (a SIGKILLed Primary before
`auto-down`); the cost is dual-Active during a real partition, which serves the same Good data on both nodes. The
verdict is separate from `PrimaryGatePolicy`, which is unchanged for the data-plane gates below.
Snapshots are ordered: `RedundancyStateChanged.Revision` = (publisher up-number, incarnation tick, sequence). Under
`Warm`, `OpcUaPublishActor` and `DriverHostActor` drop a snapshot whose revision is not newer than the last accepted
one (a late heartbeat of an old singleton incarnation) and count `otopcua.redundancy.snapshot_rejected`. A snapshot
without a revision (a pre-#717 publisher) still feeds the health inputs but can never make a Warm node Active, so
upgrade both nodes before flipping either. `Unset`/`None` keep the unconditional replace.
| Warm state | ServiceLevel |
|---|---|
| Down / Detached / endpoint failed | 0 |
| Boot, before the first computed level | 1 |
| Standby, stale or DB unreachable | 50 |
| Standby, healthy | 90 |
| Active, critically degraded | 100 |
| Active, stale | 200 |
| Active, healthy | 250 |
Every Standby level is below every level a data-serving Active node publishes (`ServiceLevelCalculator.ComputeWarm`).
### What the Standby does
- **Ordinary tags** (the `Value` of an `EquipmentVariableState`: raw tag, UNS reference, virtual tag, Calculation tag):
Read, the initial sample, change fan-out, ResendData and AggregateFilter input all answer `BadOutOfService`/null. The
stored value keeps updating from acquisition, which never stops on the Standby. Other attributes, condition children
and ns=0 (ServiceLevel, ServerStatus, ServerRedundancy) are unchanged. ACL decisions run first, so a denied node is
still `BadUserAccessDenied`.
- **Writes** to an ordinary tag are refused `BadInvalidState` synchronously, after authorization and before anything
reaches the driver (nothing to revert). `Unset`/`None` keep the #5 optimistic Good + revert.
- **Part 9 condition methods** (Acknowledge, Confirm, AddComment, Enable/Disable, shelving) are refused
`BadInvalidState` before any state change, native and scripted. `Unset`/`None` keep today's behaviour.
- **New alarm events** are not delivered on any item kind (Server object, folders, direct), ACL on or off. The
condition node is still stamped (EventId, Time), so a ConditionRefresh on the Standby returns RefreshStart, the
current retained conditions, RefreshEnd, and a Standby-stamped EventId acknowledges after promotion. The decision is
made at report time: an event reported while Active is delivered even if delivery finishes after demotion.
Model-change and audit events are not suppressed.
- HistoryRead, HistoryUpdate (Primary-gated, #674), Browse and sessions are unchanged.
### Strict first delivery (D4)
After demotion no Publish response delivered **for the first time** carries a Good ordinary-tag value, including
samples queued in an item and messages the subscription built before the demotion. SDK 2.0.1 clones the cached
message on every delivery and pools notification payloads, so the mechanism is keyed by value, not by object (T0):
- Ordinary-tag `Value` items are created as `OrdinaryTagMonitoredItem`, which sets
`MonitoredItemNotification.Message` (a helper the encoder never writes and the SDK clone copies) to an
`OrdinaryTagMarker(ClientHandle, DataValue)`. A notification counts as ordinary-tag only when the marker matches its
own handle and value, so a stale marker on a recycled payload is ignored. `WarmStandbyUnitTests` pins both SDK
behaviours this relies on.
- `OtOpcUaSdkServer` overrides `PublishAsync`/`RepublishAsync` and records one decision per
`(SubscriptionId, SequenceNumber)` on first delivery. A Standby decision rewrites the SDK's fresh clone; the cached
copy is never mutated. Republish returns the recorded decision, so a message first delivered while Active
republishes Good. The record is pruned by `AvailableSequenceNumbers` and bounded.
- Responses already in flight at the demotion step are out of scope. A Republish of a prebuilt, never-sent message is
`BadMessageNotAvailable` in 2.0.1, so first delivery through Republish cannot happen.
### Transitions
`OpcUaPublishActor` owns the verdict and drives the node manager through `IStandbyProjection` (late-bound by
`DeferredStandbyProjection`, which replays the latest state) in a fixed order:
- **Promotion:** open the read projection → queue the current value to each live ordinary-tag item (ignoring filters)
→ first deliveries Active → un-suppress alarms → ConditionRefresh2 on every live event item (RefreshStart, retained,
RefreshEnd) → publish the Active ServiceLevel **last**. If that publish fails, the data plane is Active and the level is
retried every second; the Standby level stays visible meanwhile.
- **Demotion:** publish the Standby ServiceLevel **first** (continue even if it fails) → first deliveries Standby (the D4
boundary) → suppress alarms → close the read projection → queue `BadOutOfService` to each live ordinary-tag item.
Invariant: an Active-band ServiceLevel is never visible while the projection is closed.
### Configuration
```jsonc
{
"OpcUa": {
"ApplicationUri": "urn:OtOpcUa:node-a", // unique within the pair
"PeerApplicationUris": ["urn:OtOpcUa:node-b"],
"Redundancy": {
"Mode": "Warm", // Unset (default) | None | Warm
"PromoteWhilePrimaryUnreachable": true
}
}
}
```
Flip both pair nodes in one window, after both run a revision-capable build. Rollback: set `Mode=Unset` on both and
restart; nothing persistent changes.
### Not covered
TransferSubscriptions across the pair (non-transparent: clients re-create), durable subscription restore (the product
does not enable durable subscriptions; an SDK-restored item would not carry the marker), Hot/Transparent redundancy,
dynamic peers. CTT (G8) and production-client cutover (G9) stay open under #615.
## Split-brain / downing
**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
@@ -577,7 +713,7 @@ Before archreview 03/S4 the unknown-role case defaulted to *allow* unconditional
Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
- **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
- **`HistorianAdapterActor` historization** — likewise Primary-gated so each alarm transition historizes once in steady state (at-least-once across a failover, see the drain below). The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. The gate applies only to a node whose queue a peer actually holds — `LocalDbReplicationPolicy.SharesRowsWithPeer`: `LocalDb:Replication:ApiKey` AND (`PeerAddress` OR `LocalDb:SyncListenPort` > 0); any other node always drains (review R1 m3). Within a pair BOTH halves stand down for their partner: the dialling half knows it by the `PeerAddress` host, the passive half (no `PeerAddress`) as its per-cluster mesh partner (review R6 MINOR-4 — before that the passive half never stood down, so both halves drained and every event reached the historian twice; a passive half with no `cluster-{ClusterId}` role still cannot identify its partner and keeps draining). Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
- **Continuous-historization value CAPTURE** (Gitea #537) — a third gate on the same role rule, and the
@@ -782,6 +918,12 @@ The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F
| Metric | Type | Notes |
|---|---|---|
| `otopcua.redundancy.service_level_change` | Counter (`{change}`) | OPC UA `Server.ServiceLevel` transitions emitted by the redundancy state. Tagged with `level` = the new byte. |
| `otopcua.redundancy.warm_role` | Gauge | Warm only: 1 Active, 0 Standby, −1 not Warm. |
| `otopcua.redundancy.warm_transition` | Counter (`{transition}`) | Warm role transitions; `direction` = `promote` \| `demote`, `reason` = `boot` \| `membership` \| `snapshot` \| `endpoint` \| `unreachable-primary` \| `agreed`. |
| `otopcua.redundancy.snapshot_rejected` | Counter (`{snapshot}`) | Warm only: snapshots dropped by a subscriber; `reason` = `older_revision` \| `unrevisioned`. |
| `otopcua.redundancy.mode_mismatch` | Counter (`{apply}`) | The deployed cluster `RedundancyMode` disagrees with `OpcUa:Redundancy:Mode` (`local`, `cluster` tags). |
| `otopcua.opcua.standby.projected_notifications` | Counter (`{notification}`) | Ordinary-tag notifications rewritten `BadOutOfService` at a Warm Standby's first delivery (D4). |
| `otopcua.opcua.standby.suppressed_alarm_events` | Counter (`{event}`) | New alarm events not delivered on a Warm Standby. |
| `otopcua.redundancy.primary_gate_denied` | Counter (`{denial}`) | Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged `site` = `write` \| `ack` \| `alarm-emit` and `reason` = `secondary` \| `detached` \| `role-unknown`. A non-zero `role-unknown` rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering). |
| `otopcua.opcua.apply.failed` | Counter (`{apply}`) | Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged `kind` = `rebuild` (the rebuild threw) \| `nodes` (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — `OpcUaPublishActor` also logs the degraded apply at **Error**. |
| `galaxy.writes.advise_failed` | Counter (`{write}`) | Galaxy writes short-circuited to `Bad` because `AdviseSupervisory` failed — the value could not have committed, so the write **fails closed** (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter `ZB.MOM.WW.OtOpcUa.Driver.Galaxy`. |