test(717): deferred Warm coverage - 2-node handover, DataEncoding, deleted/reused items, clone matrix, ACL stand-down, per-counter metrics
v2-ci / unit-tests (push) Has been cancelled

- T3 WarmSingletonHandoverTests: real 2-node cluster, singleton under ClusterSingletonManager on both nodes; the old
  incarnation's held-back publish (older revision, higher sequence) reaches a real Warm OpcUaPublishActor after the
  handover and never demotes the new Primary. Pin: a removed node cannot rejoin from its old ActorSystem; a new one
  rejoins younger (higher up-number).
- T6 DataEncoding: Binary/XML encoded reads of a structured tag projected on the Standby; scalar encoding error kept;
  a DefaultXml item is projected, then promoted through its own encoding (TypeId i=888).
- T7 item deleted between queue and publish; ClientHandle reused by i=2267 on a payload recycled through the SDK 2.0.1
  pool (precondition asserted); WarmStandbyCloneMatrixTests over DataChange+DiagnosticInfos/EventNotificationList/
  StatusChange, clone/cached immutability, encode round-trip, real-pool stale marker.
- T8 ACL-stood-down item receives nothing from demotion or promotion; restored grant reaches it again.
- T10 per-counter MeterListener tests (by instrument instance): warm_transition, warm_role gauge, snapshot_rejected
  (publish actor both reasons + DriverHostActor's ordering gate, first actor test of it), projected_notifications,
  suppressed_alarm_events. Process-wide state tests run in non-parallel collections.
- Every test mutation-checked (26 mutations killed); no product bug found. Plan status updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 04:31:11 -05:00
parent 0c19f26c81
commit 687256d68e
7 changed files with 1152 additions and 16 deletions
@@ -307,9 +307,63 @@ Test homes: `WarmStandbyUnitTests` (OpcUaServer.Tests: marker, rewrite, decision
option rules), `WarmRedundancyWireTests` (OpcUaServer.IntegrationTests: metadata per mode, read projection, write and
Part 9 refusal, the three T0 demotion timings, live-item queue, alarm suppression/refresh, effective-URI XML load),
`WarmRedundancyRevisionTests` + `WarmServiceLevelBandTests` (ControlPlane.Tests), `WarmRoleAuthorityTests` +
`OpcUaPublishActorWarmTests` (Runtime.Tests), `OpcUaRedundancyValidationTests` (Host.IntegrationTests).
`OpcUaPublishActorWarmTests` (Runtime.Tests), `OpcUaRedundancyValidationTests` (Host.IntegrationTests). The deferred
depth below adds `WarmSingletonHandoverTests` (ControlPlane.Tests), `WarmRedundancyMetricsTests` (Runtime.Tests),
`WarmStandbyCloneMatrixTests` (OpcUaServer.Tests), `WarmRedundancyWireTests.Deferred.cs` and `WarmRedundancyPooledWireTests`
(OpcUaServer.IntegrationTests).
Not yet covered by a test (left for T13 or a follow-up): the real 2-node singleton-handover test and the
"removed node cannot rejoin" pin (T3; ordering is covered at gate and actor level), a DataEncoding read, a
deleted/reused monitored item between queue and publish and the ExtensionObject/DiagnosticInfo clone matrix (T7), the
ACL-stood-down item on transition (T8), and per-counter MeterListener tests beyond `mode_mismatch` (T10).
**Deferred coverage added (2026-11-13, in-process, no docker).** Each test was mutation-checked against a deliberately
broken implementation and failed. No product bug was found.
- **T3:** `WarmSingletonHandoverTests` (ControlPlane.Tests). A real 2-node cluster runs the redundancy singleton under
`ClusterSingletonManager` on both nodes. The old incarnation's last publish is held back through the broadcast seam,
its node is downed, and the singleton hands over. The held publish is then delivered to a real Warm
`OpcUaPublishActor` on the survivor. That publish is the older revision but has the higher sequence. The survivor
stays Active. The second test pins that a removed node cannot rejoin from its old ActorSystem. A new ActorSystem on
the same address rejoins as a younger member, with a higher up-number. Mutations killed:
- the gate accepts older revisions;
- the revision drops the up-number and incarnation (sequence only);
- for the pin, skipping the Down (a harness negative control).
- **T6:** `A_DataEncoding_read_and_item_are_projected_…` (wire). On the Standby, a Binary or XML DataEncoding read of
a structured (EUInformation) tag is projected. An encoding a scalar cannot take still returns the base read's own
`BadDataEncodingUnsupported`. An item created with `DefaultXml` gets the projected value on the Standby, and on
promotion it receives the value read through its own encoding (TypeId `i=888`). Mutations killed:
- DataEncoding reads are not projected;
- the promotion queue ignores the item's DataEncoding.
- **T7:**
- `An_item_deleted_between_queue_and_publish_…` (wire). The deleted item's queued samples never surface. Its
sibling is still first-delivered projected, and a Republish gives the same result.
- `WarmRedundancyPooledWireTests.A_client_handle_reused_after_delete_…` (wire, non-parallel collection). A marked
payload goes back through SDK 2.0.1's process-wide payload pool, and its ClientHandle is reused by an `i=2267`
item. That item stays Good. The test first asserts that the pool really hands out a marked payload.
- `WarmStandbyCloneMatrixTests` (unit). One delivered clone carries two DataChangeNotifications (one with
DiagnosticInfos), an EventNotificationList and a StatusChangeNotification. Only marked values are rewritten.
Diagnostics, events and status pass through in order. Neither the clone nor the cached message is mutated, and
the result encodes. Two clones project to identical bytes. A recycled payload's stale marker, rented through the
real pool with the same handle, never validates.
- Mutations killed:
- the marker is not validated (wire and unit);
- the marker is not set;
- DiagnosticInfos are dropped;
- non-data payloads are dropped;
- the rewrite happens in place;
- only the first DataChangeNotification is rewritten.
- **T8:** `An_ACL_stood_down_item_receives_nothing_…` (wire). An item that `RevalidateMonitoredItems` stood down gets
nothing from a demotion or a promotion (`QueueCurrentToLiveItems` returns 1, not 2). Its permitted sibling gets
both. Once the grant is restored, the next demotion reaches the item. Mutation killed: stood-down items are not
skipped.
- **T10:**
- `WarmRedundancyMetricsTests` (Runtime.Tests, non-parallel collection) checks four things:
- `warm_transition`, with direction and reason (`agreed` / `snapshot`), and no count on a same-role snapshot;
- the `warm_role` gauge, -1, then 0, then 1, then 0;
- `snapshot_rejected` at the publish actor for `older_revision` and `unrevisioned`;
- at `DriverHostActor`, a stale snapshot is dropped under Warm and counted, and its role view does not move. The
Unset control does move. This is the first actor test of the driver host's ordering gate.
- `WarmRedundancyPooledWireTests` checks `standby.projected_notifications`: it equals the rewritten count, counts
again on Republish, and stays 0 while Active. It also checks `standby.suppressed_alarm_events`: one per withheld
transition, none once Active.
- Every listener selects by instrument instance.
- Mutations killed: each counter's `Add` was removed, the direction tag inverted, the gauge never set, and the
driver-host gate ignored.
Still open (live-only, T13): the durable restore, TransferSubscriptions under one identity, and the rig legs.