test(717): deferred-event boundary, ACL-filtered promotion refresh, native-ack refusal, effective-URI XML load; docs + T13 overlay
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
- Wire: an event reported while Active but deferred (#838) is delivered after demotion; the promotion refresh withholds an ACL-denied condition (positive control alongside); ConditionRefresh2 and model-change events on the Standby; a native Acknowledge is refused and never routed; Warm validates and advertises the EFFECTIVE ApplicationUri from an ApplicationConfigPath XML load. - docs/Redundancy.md: Warm section, the Unset/None i=11314 behaviour (orphan readable by NodeId under Unset, deleted under None), instruments; OpcUaServer.md, Configuration.md, security.md, cluster-topology rule. - docker-dev/docker-compose.warm.yml for the T13 gate (central pair, Mode=Warm). - Plan: implementation status and the test gaps left for T13/follow-up. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
+144
-2
@@ -6,7 +6,17 @@ OtOpcUa runs as a 2-node **non-transparent** redundant pair (client-side failove
|
||||
|
||||
> **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. Clients should read `Server.ServerArray` for peer discovery today.
|
||||
>
|
||||
> **Standard redundancy metadata — not yet advertised (#717).** Nothing in `src/` sets `Server.ServerRedundancy.RedundancySupport` (`i=3709`), so it reads the SDK default regardless of the cluster's `RedundancyMode`, and `Server.ServerRedundancy` is the base `ServerRedundancyType` with **no** `ServerUriArray` (`i=11314`, mandatory only on `NonTransparentRedundancyType`). `ServerArray` is **not** a substitute for it. This is **not** blocked on an SDK upgrade: hermetic spikes on the pinned SDK 1.5.378.106 built the correct startup-static graph (object `i=2296` typed `NonTransparentRedundancyType` `i=2039`, `RedundancySupport`, `ServerUriArray` = effective self URI then peers) through `StandardServer.CreateMainNodeManagerFactory` + `ConfigurationNodeManager.AddBehaviourToPredefinedNode`. What blocks it is the contract: the owner-selected target is **Warm with an explicit degraded standby** (standby ServiceLevel in the degraded 2–199 band, ordinary standby tag Reads/subscriptions degraded while internal acquisition continues, standby alarm events suppressed and refreshed on promotion) and **None** for standalone deployments. Today's Secondary serves live Good data at ServiceLevel 240, so advertising `Warm` without that standby behaviour would misdescribe the server; the design, spec and plan for it are pending under #717. Interop gates A4/A5b in `docs/plans/phase-6-3-redundancy-interop-plan.md` stay unmet until then.
|
||||
> **Standard redundancy metadata (#717) — opt-in, default off.** `OpcUa:Redundancy:Mode` selects the contract. The
|
||||
> default `Unset` is byte-for-byte the pre-#717 server: nothing writes `Server.ServerRedundancy.RedundancySupport`
|
||||
> (`i=3709`, so it reads the SDK default `None`), `Server.ServerRedundancy` stays the base `ServerRedundancyType`, the
|
||||
> Secondary serves live Good data at ServiceLevel 240, and interop gates A4/A5b stay unmet. Under `Unset` the SDK's own
|
||||
> predefined `ServerUriArray` node (`i=11314`) is an **orphan**: it is not referenced from `ServerRedundancy`, so Browse
|
||||
> and TranslateBrowsePaths do not reach it, but a Read **by NodeId** returns Good with an empty `String[]` (T0a,
|
||||
> `docs/plans/evidence/2026-11-13-717-sdk2/RESULTS.md`). That is left as the SDK builds it, because `Unset` changes
|
||||
> nothing. `None` (standalone) advertises `RedundancySupport=None` and deletes the orphan, so `i=11314` is absent by
|
||||
> NodeId as well (`BadNodeIdUnknown`). `Warm` (a pair) advertises the full non-transparent graph and the degraded-standby
|
||||
> contract; see [Warm redundancy](#warm-non-transparent-redundancy-717). No tier sets `Warm` until ScadaBridge#138
|
||||
> (ServiceLevel-aware failover) is deployed there.
|
||||
|
||||
> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster membership** — the oldest Up member carrying the `driver` role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.
|
||||
|
||||
@@ -280,6 +290,132 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
|
||||
|
||||
Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
|
||||
|
||||
## Warm non-transparent redundancy (#717)
|
||||
|
||||
Spec: `docs/plans/2026-11-11-warm-redundancy-717-design.md`; plan: `docs/plans/2026-11-11-warm-redundancy-717-plan.md`.
|
||||
Everything below applies only under `OpcUa:Redundancy:Mode=Warm`. `Unset` (the default) and `None` keep every behaviour
|
||||
described elsewhere on this page.
|
||||
|
||||
### Modes
|
||||
|
||||
| Mode | `RedundancySupport` (`i=3709`) | `ServerRedundancy` type | `ServerUriArray` (`i=11314`) | Standby behaviour |
|
||||
|---|---|---|---|---|
|
||||
| `Unset` (default) | SDK default (`None`), never written | `ServerRedundancyType` | orphan, Good `[]` by NodeId, not browsable | none; Secondary serves Good at 240 |
|
||||
| `None` | `None` | `ServerRedundancyType` | absent (orphan deleted) | never |
|
||||
| `Warm` | `Warm` | `NonTransparentRedundancyType` (`i=2039`) | read-only, `[effectiveSelfUri, peers…]` | degraded standby (below) |
|
||||
|
||||
`Hot` and any other value fail host start (continuous acquisition alone is not Part 4 Hot). The mode is
|
||||
**startup-static**. `Warm` needs at least one `OpcUa:PeerApplicationUris` entry, distinct URIs, the node's own
|
||||
**effective** `ApplicationUri` (the one an `ApplicationConfigPath` XML load produces, not the options default) not
|
||||
listed as a peer, and the `driver` role. `None` needs no peers. The metadata is installed in
|
||||
`OtOpcUaSdkServer.OnNodeManagerStartedAsync` through the SDK 2.0.1 `IPredefinedNodeSubtypeReplacer`, which keeps the
|
||||
well-known `i=11314`. Under `Warm` the server boots at ServiceLevel **1** (NoData) until the first computed level.
|
||||
|
||||
At each deploy the node compares the artifact's `ServerCluster.RedundancyMode` for its cluster with its local mode
|
||||
(`None` or `Warm` only). A mismatch, `Hot` included, logs an Error and counts `otopcua.redundancy.mode_mismatch`; the
|
||||
node keeps its local mode.
|
||||
|
||||
### Role verdict and bands
|
||||
|
||||
A Warm node is **Active** iff it is Up with a listening endpoint, its own cluster state names it the oldest Up `driver`
|
||||
member, and the latest **accepted** redundancy snapshot names it Primary. Otherwise it is **Standby**, including before
|
||||
the first accepted snapshot. With `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable` (default `true`) the oldest
|
||||
**reachable** driver member also goes Active while the snapshot's Primary is unreachable (a SIGKILLed Primary before
|
||||
`auto-down`); the cost is dual-Active during a real partition, which serves the same Good data on both nodes. The
|
||||
verdict is separate from `PrimaryGatePolicy`, which is unchanged for the data-plane gates below.
|
||||
|
||||
Snapshots are ordered: `RedundancyStateChanged.Revision` = (publisher up-number, incarnation tick, sequence). Under
|
||||
`Warm`, `OpcUaPublishActor` and `DriverHostActor` drop a snapshot whose revision is not newer than the last accepted
|
||||
one (a late heartbeat of an old singleton incarnation) and count `otopcua.redundancy.snapshot_rejected`. A snapshot
|
||||
without a revision (a pre-#717 publisher) still feeds the health inputs but can never make a Warm node Active, so
|
||||
upgrade both nodes before flipping either. `Unset`/`None` keep the unconditional replace.
|
||||
|
||||
| Warm state | ServiceLevel |
|
||||
|---|---|
|
||||
| Down / Detached / endpoint failed | 0 |
|
||||
| Boot, before the first computed level | 1 |
|
||||
| Standby, stale or DB unreachable | 50 |
|
||||
| Standby, healthy | 90 |
|
||||
| Active, critically degraded | 100 |
|
||||
| Active, stale | 200 |
|
||||
| Active, healthy | 250 |
|
||||
|
||||
Every Standby level is below every level a data-serving Active node publishes (`ServiceLevelCalculator.ComputeWarm`).
|
||||
|
||||
### What the Standby does
|
||||
|
||||
- **Ordinary tags** (the `Value` of an `EquipmentVariableState`: raw tag, UNS reference, virtual tag, Calculation tag):
|
||||
Read, the initial sample, change fan-out, ResendData and AggregateFilter input all answer `BadOutOfService`/null. The
|
||||
stored value keeps updating from acquisition, which never stops on the Standby. Other attributes, condition children
|
||||
and ns=0 (ServiceLevel, ServerStatus, ServerRedundancy) are unchanged. ACL decisions run first, so a denied node is
|
||||
still `BadUserAccessDenied`.
|
||||
- **Writes** to an ordinary tag are refused `BadInvalidState` synchronously, after authorization and before anything
|
||||
reaches the driver (nothing to revert). `Unset`/`None` keep the #5 optimistic Good + revert.
|
||||
- **Part 9 condition methods** (Acknowledge, Confirm, AddComment, Enable/Disable, shelving) are refused
|
||||
`BadInvalidState` before any state change, native and scripted. `Unset`/`None` keep today's behaviour.
|
||||
- **New alarm events** are not delivered on any item kind (Server object, folders, direct), ACL on or off. The
|
||||
condition node is still stamped (EventId, Time), so a ConditionRefresh on the Standby returns RefreshStart, the
|
||||
current retained conditions, RefreshEnd, and a Standby-stamped EventId acknowledges after promotion. The decision is
|
||||
made at report time: an event reported while Active is delivered even if delivery finishes after demotion.
|
||||
Model-change and audit events are not suppressed.
|
||||
- HistoryRead, HistoryUpdate (Primary-gated, #674), Browse and sessions are unchanged.
|
||||
|
||||
### Strict first delivery (D4)
|
||||
|
||||
After demotion no Publish response delivered **for the first time** carries a Good ordinary-tag value, including
|
||||
samples queued in an item and messages the subscription built before the demotion. SDK 2.0.1 clones the cached
|
||||
message on every delivery and pools notification payloads, so the mechanism is keyed by value, not by object (T0):
|
||||
|
||||
- Ordinary-tag `Value` items are created as `OrdinaryTagMonitoredItem`, which sets
|
||||
`MonitoredItemNotification.Message` (a helper the encoder never writes and the SDK clone copies) to an
|
||||
`OrdinaryTagMarker(ClientHandle, DataValue)`. A notification counts as ordinary-tag only when the marker matches its
|
||||
own handle and value, so a stale marker on a recycled payload is ignored. `WarmStandbyUnitTests` pins both SDK
|
||||
behaviours this relies on.
|
||||
- `OtOpcUaSdkServer` overrides `PublishAsync`/`RepublishAsync` and records one decision per
|
||||
`(SubscriptionId, SequenceNumber)` on first delivery. A Standby decision rewrites the SDK's fresh clone; the cached
|
||||
copy is never mutated. Republish returns the recorded decision, so a message first delivered while Active
|
||||
republishes Good. The record is pruned by `AvailableSequenceNumbers` and bounded.
|
||||
- Responses already in flight at the demotion step are out of scope. A Republish of a prebuilt, never-sent message is
|
||||
`BadMessageNotAvailable` in 2.0.1, so first delivery through Republish cannot happen.
|
||||
|
||||
### Transitions
|
||||
|
||||
`OpcUaPublishActor` owns the verdict and drives the node manager through `IStandbyProjection` (late-bound by
|
||||
`DeferredStandbyProjection`, which replays the latest state) in a fixed order:
|
||||
|
||||
- **Promotion:** open the read projection → queue the current value to each live ordinary-tag item (ignoring filters)
|
||||
→ first deliveries Active → un-suppress alarms → ConditionRefresh2 on every live event item (RefreshStart, retained,
|
||||
RefreshEnd) → publish the Active ServiceLevel **last**. If that publish fails, the data plane is Active and the level is
|
||||
retried every second; the Standby level stays visible meanwhile.
|
||||
- **Demotion:** publish the Standby ServiceLevel **first** (continue even if it fails) → first deliveries Standby (the D4
|
||||
boundary) → suppress alarms → close the read projection → queue `BadOutOfService` to each live ordinary-tag item.
|
||||
|
||||
Invariant: an Active-band ServiceLevel is never visible while the projection is closed.
|
||||
|
||||
### Configuration
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"OpcUa": {
|
||||
"ApplicationUri": "urn:OtOpcUa:node-a", // unique within the pair
|
||||
"PeerApplicationUris": ["urn:OtOpcUa:node-b"],
|
||||
"Redundancy": {
|
||||
"Mode": "Warm", // Unset (default) | None | Warm
|
||||
"PromoteWhilePrimaryUnreachable": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Flip both pair nodes in one window, after both run a revision-capable build. Rollback: set `Mode=Unset` on both and
|
||||
restart; nothing persistent changes.
|
||||
|
||||
### Not covered
|
||||
|
||||
TransferSubscriptions across the pair (non-transparent: clients re-create), durable subscription restore (the product
|
||||
does not enable durable subscriptions; an SDK-restored item would not carry the marker), Hot/Transparent redundancy,
|
||||
dynamic peers. CTT (G8) and production-client cutover (G9) stay open under #615.
|
||||
|
||||
## Split-brain / downing
|
||||
|
||||
**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
|
||||
@@ -577,7 +713,7 @@ Before archreview 03/S4 the unknown-role case defaulted to *allow* unconditional
|
||||
Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
|
||||
|
||||
- **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
|
||||
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
|
||||
- **`HistorianAdapterActor` historization** — likewise Primary-gated so each alarm transition historizes once in steady state (at-least-once across a failover, see the drain below). The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
|
||||
- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. The gate applies only to a node whose queue a peer actually holds — `LocalDbReplicationPolicy.SharesRowsWithPeer`: `LocalDb:Replication:ApiKey` AND (`PeerAddress` OR `LocalDb:SyncListenPort` > 0); any other node always drains (review R1 m3). Within a pair BOTH halves stand down for their partner: the dialling half knows it by the `PeerAddress` host, the passive half (no `PeerAddress`) as its per-cluster mesh partner (review R6 MINOR-4 — before that the passive half never stood down, so both halves drained and every event reached the historian twice; a passive half with no `cluster-{ClusterId}` role still cannot identify its partner and keeps draining). Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
|
||||
|
||||
- **Continuous-historization value CAPTURE** (Gitea #537) — a third gate on the same role rule, and the
|
||||
@@ -782,6 +918,12 @@ The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F
|
||||
| Metric | Type | Notes |
|
||||
|---|---|---|
|
||||
| `otopcua.redundancy.service_level_change` | Counter (`{change}`) | OPC UA `Server.ServiceLevel` transitions emitted by the redundancy state. Tagged with `level` = the new byte. |
|
||||
| `otopcua.redundancy.warm_role` | Gauge | Warm only: 1 Active, 0 Standby, −1 not Warm. |
|
||||
| `otopcua.redundancy.warm_transition` | Counter (`{transition}`) | Warm role transitions; `direction` = `promote` \| `demote`, `reason` = `boot` \| `membership` \| `snapshot` \| `endpoint` \| `unreachable-primary` \| `agreed`. |
|
||||
| `otopcua.redundancy.snapshot_rejected` | Counter (`{snapshot}`) | Warm only: snapshots dropped by a subscriber; `reason` = `older_revision` \| `unrevisioned`. |
|
||||
| `otopcua.redundancy.mode_mismatch` | Counter (`{apply}`) | The deployed cluster `RedundancyMode` disagrees with `OpcUa:Redundancy:Mode` (`local`, `cluster` tags). |
|
||||
| `otopcua.opcua.standby.projected_notifications` | Counter (`{notification}`) | Ordinary-tag notifications rewritten `BadOutOfService` at a Warm Standby's first delivery (D4). |
|
||||
| `otopcua.opcua.standby.suppressed_alarm_events` | Counter (`{event}`) | New alarm events not delivered on a Warm Standby. |
|
||||
| `otopcua.redundancy.primary_gate_denied` | Counter (`{denial}`) | Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged `site` = `write` \| `ack` \| `alarm-emit` and `reason` = `secondary` \| `detached` \| `role-unknown`. A non-zero `role-unknown` rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering). |
|
||||
| `otopcua.opcua.apply.failed` | Counter (`{apply}`) | Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged `kind` = `rebuild` (the rebuild threw) \| `nodes` (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — `OpcUaPublishActor` also logs the degraded apply at **Error**. |
|
||||
| `galaxy.writes.advise_failed` | Counter (`{write}`) | Galaxy writes short-circuited to `Bad` because `AdviseSupervisory` failed — the value could not have committed, so the write **fails closed** (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter `ZB.MOM.WW.OtOpcUa.Driver.Galaxy`. |
|
||||
|
||||
Reference in New Issue
Block a user