Files
lmxopcua/docs/plans/2026-11-11-warm-redundancy-717-plan.md
T
Joseph Doherty 36eb6d558d
v2-ci / unit-tests (push) Has been cancelled
docs(717): Mode=Unset re-check after c569f806 (no change); a leaving node reports 0
T13b: the isolated otopcua-t13b pair with OpcUa:Redundancy:Mode absent, on origin/master f826af8e
(it contains c569f806) and A/B on c569f806^ (687256d6). Every check passes on both builds:
- levels 250/240;
- Good data on both nodes;
- writes, Acknowledge, AddComment and alarm delivery on the Secondary;
- graceful stop: the survivor reaches 250 259 ms after Leaving (pre-fix 270 ms);
- crash: 250 at +43.6 s (pre-fix +42.8 s), with no demotion at auto-down;
- rejoin at 240 with no failback.
The role-decision changes are gated on IsWarm and do not run under Unset.

Plan L5, the design's section 4.3 and the Warm table in Redundancy.md now say a Leaving/Exiting node
reports ServiceLevel 0, not 90 (ComputeWarm returns 0 for any member that is not Up/Joining; T13 note A).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 06:44:46 -05:00

26 KiB
Raw Blame History

Warm non-transparent redundancy (Gitea #717): implementation plan

Status: APPROVED by the owner 2026-11-13 (#717), together with 2026-11-11-warm-redundancy-717-design.md (referred to below as "the spec", with § references). The spec's §9 records the owner's answers: writes and Part 9 methods are refused BadInvalidState under Warm, promote-while-unreachable is on by default, the standby status is BadOutOfService, there is no ConfigDb migration, and the bands are as drafted. Every task ships dark: OpcUa:Redundancy:Mode defaults to Unset, so merging any task changes no deployed behaviour.

Conventions:

  • Wire tests use a real OPC Foundation client session against a loopback OtOpcUaSdkServer, in tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests (the home of DualEndpointTests, ConditionRefreshDeferralWireTests and FolderConditionRefreshWireTests).
  • Actor tests go in tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests and tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.
  • Every gate gets an allow leg as well as a deny leg (the vacuous-test rule), and a mutation check: revert the production line and confirm the test fails.

T0: SDK 2.0.1 re-proof spikes (throwaway, separately approved)

The 1.5.378.106 spikes do not carry over (spec §10). Write throwaway loopback fixtures against 2.0.1, outside src/. No product edits. Each fixture is rerun independently and its evidence is recorded in docs/plans/evidence/2026-11-xx-717-sdk2/.

  • a. Metadata graph. Build it via CreateMainNodeManagerFactory + AddBehaviourToPredefinedNodeAsync. Check Read/Browse/Translate of i=2296 (HasTypeDefinition 2039), i=3709, and i=11314 for both Warm and None. The passive i=11314 visit must resolve to the constructed instance.
  • b. Read projection coverage. Cover Read, the initial sample on create, the change fan-out after ClearChangeMasks, ResendData, queue size 1 vs. N, an AggregateFilter item, and Sampling → Reporting. Decide between OnSimpleReadValue and a ReadValueAttribute override, and record which async callback paths 2.0.1 takes.
  • c. Provenance. Stamp items through the 2.0.1 MonitoredItemFactory / MonitoredItemCreateDecision. Exercise ModifyMonitoredItems handle change, delete/reuse, TransferSubscriptions (IMonitoredItemTransferCoordinator), and a restore.
  • d. Publish/Republish. Override PublishAsync/RepublishAsync with the 2.0.1 signatures (RequestLifetime). Cover the first-delivery choice, first delivery through Republish, an ack followed by BadMessageNotAvailable, and a cancelled or failed response.
  • e. Events. Exercise the stamp/deliver split under the #855 per-item folder path and the #854 Server-object cut. Run the promotion refresh on all three item kinds, and a client ConditionRefresh/2 while Standby.

Exit: a short results note per fixture, and an owner go/no-go. A "no" on (c) or (d) re-opens D4's mechanism. D4 itself stays an owner decision.

T0 run 2026-11-13: (a) and (b) pass; (c) and (d) are no-go as written. SDK 2.0.1 hands every Publish and Republish a fresh clone of the cached message and pools the notification payloads, so provenance and the first-delivery choice cannot be keyed by object. An alternative, keyed by (SubscriptionId, SequenceNumber) with a clone-surviving provenance token, passed the same spike. The owner adopted it on 2026-11-13; spec §4.6 and T2/T7 below describe it. Evidence: docs/plans/evidence/2026-11-13-717-sdk2/RESULTS.md.

T1: Configuration, validation, effective self URI

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs: options at :29-121. Add OpcUaRedundancyOptions Redundancy (Mode; PromoteWhilePrimaryUnreachable, default true, owner answer Q2).
  • src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs.
  • A post-load check in StartAsync (:218-250) against the effective ApplicationConfiguration.ApplicationUri.

Rules (spec §3):

  • Hot and unknown values fail host start.
  • Warm requires at least one peer, all URIs distinct, self not listed as a peer, and the driver role.
  • None requires no peers.

Tests:

  • A validator matrix.
  • An XML ApplicationConfigPath whose URI differs from the options value, proving the effective URI is used.
  • A boot-banner log assertion.

T2: Standard redundancy metadata graph

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs: in the existing OnNodeManagerStartedAsync override, install the T0a graph per mode through IPredefinedNodeSubtypeReplacer.ReplacePredefinedInstanceSubtypeAsync (no CreateMainNodeManagerFactory override is needed on 2.0.1).
  • Under Warm, the initial Server.ServiceLevel is set to 1 in the same pass (spec §4.3). Unset must not touch the graph. Under None the SDK's orphan i=11314 (T0a finding) is deleted so it is absent by NodeId too.

Tests (wire):

  • Per mode, check the i=3709 value, the i=2296 type definition, the presence or absence of i=11314, its value order [effectiveSelf, peers…], and that it is read-only (a Write returns BadNotWritable).
  • ServerArray (i=2254) is unchanged; extend DualEndpointTests.
  • A Warm server's ServiceLevel reads 1 before the first publish.
  • Security profiles and startup are unaffected under Unset (golden read of today's values).

T3: Ordered redundancy snapshots (spec §4.2)

Files:

  • src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/RedundancyStateChanged.cs (:9-11): add an optional SnapshotRevision? Revision (additive).
  • RedundancyStateActor.cs:
    • stamp the revision in PublishIfDirty (:105-113);
    • set IncarnationTick in PreStart;
    • read the host's UpNumber from Cluster.SelfMember.
  • Subscribers: accept only a newer revision in OpcUaPublishActor.HandleRedundancyStateChanged (:1494-1498) and DriverHostActor.OnRedundancyStateChanged (:2555).
  • An unrevisioned snapshot keeps today's handling for the existing gates.

Tests:

  • Ordering unit tests: an older singleton incarnation's heartbeat after a newer snapshot is dropped; a same-process singleton restart advances.
  • A real 2-node cluster handover test (pattern: RedundancyPrimaryElectionTests). It holds back the old incarnation's last publish through the _broadcastOverride seam (:111) and asserts the subscriber keeps the new Primary.
  • Serialization round-trip of the extended message, plus a mixed-version decode with the field absent.
  • The pinned assumption: a removed node cannot rejoin without a new ActorSystem.

T4: Warm role verdict and ServiceLevel bands (spec §4.1, §4.3)

Files:

  • New src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/WarmRoleAuthority.cs: a pure function of membership view, accepted snapshot, EndpointListening and SelfMember status.
  • src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs: add ComputeWarm(inputs, isActive). Compute itself is unchanged.
  • OpcUaPublishActor.RecomputeServiceLevel (:1576-1636) branches on mode. It subscribes to MemberUp/Removed and to reachability (owner answer Q2) so the verdict re-derives on membership changes, not only on snapshots.

Tests:

  • A truth table covering every row of §4.3 and the invariant max(Standby) < min(Active serving).
  • Membership/snapshot disagreement gives Standby, plus the 20 s Warning.
  • Boot with an unknown role gives Standby; a lone node goes Active after its own snapshot.
  • None/Unset produce byte-identical output to today's ServiceLevelCalculatorTests.
  • DB-less mode (#721 timer) is covered.

T5: Standby projection seam and transition sequencer (spec §4.7)

Files:

  • New src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IStandbyProjection.cs, with a deferred/late-bound adapter that mirrors DeferredServiceLevelPublisher and retains the latest state across binding (the #719 lesson).
  • Bound in src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs next to the ServiceLevel binding.
  • OpcUaPublishActor runs promotion and demotion as one serialized handler, in the §4.7 step order.

Tests (actor, with a recording fake):

  • The exact step order for both directions.
  • A ServiceLevel publish failure halts a promotion before the Active level and retries; a demotion continues.
  • A burst of alternating verdicts converges on the last one.
  • Binding after the first transition replays the current state.

T6: Per-tag read projection (spec §4.4)

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EquipmentVariableState.cs: the hook decided in T0b, reading a node-manager-wide volatile projection flag. Stored Value is never modified.
  • OtOpcUaNodeManager implements IStandbyProjection.SetDataProjection.
  • Owner answer Q1: under Warm, the ordinary-tag write handler refuses a write on the Standby with BadInvalidState before it reaches the node-write gateway. Unset/None keep the #5 optimistic path.

Tests (wire):

  • Warm Standby: a Write returns BadInvalidState and the gateway is never called; on the Active node the same Write reaches the gateway; under Unset a node that is not Primary still takes today's optimistic path.
  • Standby Read of a raw tag, a UNS reference, a virtual tag and a Calculation tag returns BadOutOfService/null.
  • A Standby Read of a condition child, i=2267 or i=3709 stays Good.
  • A Read with an IndexRange on an array tag is projected; a DataEncoding request is projected.
  • Acquisition continues: flip to Active and the latest value is read without any new driver publish.
  • ACL on: a denied node is still BadUserAccessDenied and the projection never leaks existence.

T7: Ordinary-tag marker and first-delivery decision record (spec §4.6, D4)

The T0-adopted alternative (owner, 2026-11-13). The 1.5 object-keyed provenance and choice table are not used.

Files:

  • OtOpcUaNodeManager: replace m_monitoredItemManager with a MonitoredNodeMonitoredItemManager subclass that re-implements IMonitoredItemManager.CreateMonitoredItem and creates a custom sampled item for ordinary-tag (EquipmentVariableState) Value items only. Its Publish runs base first and then sets MonitoredItemNotification.Message to an OrdinaryTagMarker(ClientHandle, DataValue).
  • OtOpcUaSdkServer.cs: PublishAsync/RepublishAsync overrides and a new FirstDeliveryRecord keyed by (SubscriptionId, SequenceNumber), pruned by AvailableSequenceNumbers and bounded.
  • Wire into OtOpcUaNodeManager.CreateMonitoredItems / OnMonitoredItemCreated without disturbing ACL filtering or _liveItems.

Tests (wire):

  • Queued Good samples, a ClientHandle change, then demotion: the first Publish carries BadOutOfService, and a Republish returns the same projected payload.
  • Prebuilt sequences (maxNotificationsPerPublish=1), then demotion: the next Publish is projected, and Republish returns the recorded projected choice. A Republish of a prebuilt, never-sent number is BadMessageNotAvailable.
  • Good delivered while Active, then demotion: Republish returns the original Good; after an ack it is BadMessageNotAvailable.
  • A mixed subscription (ordinary tag + i=2267 + condition-child item): only ordinary-tag items are rewritten.
  • The marker is validated: a notification whose marker names another handle or value is not rewritten (unit).
  • SDK-field regression pin: the helper survives CoreUtils.Clone and is not encoded by the binary encoder.
  • The record is bounded, and pruned against AvailableSequenceNumbers.
  • Concurrent Publish and role flip, by a stress loop with a deterministic barrier, not sleeps.

T8: Promotion and demotion of live data items

Files:

  • OtOpcUaNodeManager: IStandbyProjection.QueueCurrentToLiveItems(bool projected) iterates _liveItems. It reuses the QueueValue-with-ignoreFilters pattern (:7303), not detach or MonitoringMode.Disabled; the disabled-mode-drops-queue lesson applies. ACL-suppressed items are skipped.

Tests (wire):

  • Demotion delivers exactly one BadOutOfService per item, then silence while values keep changing internally.
  • Promotion delivers the current Good value without a driver publish.
  • An ACL-stood-down item receives nothing either way.
  • A DeadbandFilter item still receives the promotion value.

T9: Alarm event suppression and promotion refresh (spec §4.5, D3, D5)

Files (OtOpcUaNodeManager):

  • Split ReportConditionEvent (:1376) into stamp and deliver; deliver is gated by the projection flag.
  • Audit all callers (:1254, :1341, :2842, :4659) and WireMethodTransitionReporting.
  • IStandbyProjection.RefreshAllEventItems() runs the existing ConditionRefresh (:6586) Server-object path and RefreshDirectItems (:6974) over every live event item, from _allEventsItems and _eventItems.
  • ReportToServerObject (:6060) is untouched.
  • Owner answer Q4: under Warm, every Part 9 condition method (native or scripted) is refused BadInvalidState on the Standby before it changes condition state.

Tests (wire, ACL off and on):

  • Warm Standby: Acknowledge and a shelving method on a scripted condition return BadInvalidState and route no command; the Active node accepts them; Unset keeps today's scripted-method acceptance.
  • A new alarm on the Standby reaches no Server-object, folder or direct item.
  • Model-change and audit events are still delivered on the Standby.
  • A client ConditionRefresh and ConditionRefresh2 on the Standby return Start, retained, End (D5).
  • Promotion gives Start, retained, End on all three item kinds, followed by new events.
  • An ACL-denied condition is absent from the promotion refresh.
  • A condition raised and cleared (Retain=false) while Standby is not replayed.
  • An event reported while Active but deferred (#838, via NotifierStall) is still delivered after demotion.
  • A Standby-stamped EventId is accepted by Acknowledge after promotion.
  • The RingEventQueueFactory path is untouched (RingEventMonitoredItemQueueParityTests stay green).
  • Re-run EventQueueOverflowAclWireTests and ConditionMethodEventTests.

T10: Deploy-time mode cross-check and observability (spec §3, §7)

Files:

  • src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs: read this node's cluster RedundancyMode from the Clusters array.
  • OpcUaPublishActor compares it with the local mode at apply.
  • src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs: add the six instruments.

Tests:

  • A mismatch logs an Error and records the metric.
  • Each counter fires on its path. Use a MeterListener filtered by instrument instance, never by meter name alone (the #503 lesson).

T11: ScadaBridge prerequisite (cross-repo)

  • Filed as ScadaBridge#138 (2026-11-13). A ScadaBridge issue (type:feature, cross-repo, priority:p1, linked to #717). The DCL becomes ServiceLevel-aware: monitor i=2267, and fail over when it is below 200 for a configurable window while the other endpoint reports higher. Also add a test for an unsolicited RefreshStart/End on existing alarm items (spec §5).
  • File the lmxopcua companion as blocked + cross-repo.
  • The Warm flip in T13/rollout is gated on that issue being deployed.

T12: Documentation and index

  • docs/Redundancy.md: rewrite the :9 block, add a Warm contract section, update the tiers table. Remove the generic "standby / no device contact / exactly-once" wording that contradicts acquisition and at-least-once (c7794).
  • docs/OpcUaServer.md:70, docs/Configuration.md (the new keys), and docs/security.md (ns=0 stays ungated; projection order vs. ACL).
  • .claude/rules/cluster-topology.md §Redundancy.
  • docs/plans/phase-6-3-redundancy-interop-plan.md A2/A4/A5b: under Warm, A2 expects 90 and A4/A5b become testable. Add docs/v2/redundancy-interop-playbook.md.
  • Umbrella: update the OtOpcUa row and sister-project-history/otopcua.md in ../scadaproj on its main, pushed (root CLAUDE.md procedure).

T13: Live gate on docker-dev (central pair, :4840 / :4841)

Overlay: docker-dev/docker-compose.warm.yml, central-1/2 only. It sets OpcUa__Redundancy__Mode=Warm. The base compose file already gives each central its own OpcUa__ApplicationUri=urn:OtOpcUa:central-1|2 (matching seed-clusters.sql) and its partner in OpcUa__PeerApplicationUris__0 (lmxopcua#801, after spec §2 was written). Login is disabled on the rig, so drive everything yourself. Use otopcua-cli from src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI against both ports.

Leg Procedure Pass
L1 metadata redundancy on :4840 and :4841 RedundancySupport=Warm. ServerUriArray is self first and differs per node. Same URIs in both.
L2 bands read -n i=2267 on both Primary 250, Standby 90
L3 standby data read and subscribe of a raw tag and its UNS reference on the Standby BadOutOfService/null; the Primary shows Good at the same moment
L4 standby alarms alarms on both. Trip a scripted alarm via its source tag. Event on the Primary only. A ConditionRefresh on the Standby lists it as retained (D5).
L5 manual failover AdminUI manual failover (docs/Redundancy.md §Manual failover). Keep the CLI subscriptions open on both. Old Primary: SL drops to 0 (a Leaving/Exiting member is not Up/Joining, so ServiceLevelCalculator.ComputeWarm returns 0; T13 note A), then BadOutOfService, with no Good delivered after the drop. New Primary: Good values, then RefreshStart / retained / RefreshEnd, then SL 250, in that order (timestamps).
L6 crash docker kill central-<primary> The survivor goes Active once the Primary is unreachable (PromoteWhilePrimaryUnreachable, owner answer Q2). Record the measured gap.
L7 rejoin Restart the killed node It comes back Standby (90) and never shows 250 or 255. The boot read is 1 or 90.
L8 standalone site-b-2 (or a one-node overlay) with Mode=None RedundancySupport=None, no i=11314, data Good
L9 Unset control Another node left on Unset Byte-identical to pre-change: SL 240/250, SDK-default i=3709
L10 writes/history Write and HistoryRead on the Standby The write is refused BadInvalidState (owner answer Q1); HistoryRead is Good.
L11 transfer A UserName session's subscription on the Standby is taken over by a second session of the same identity (TransferSubscriptions), then the node is demoted/promoted The transferred items keep their marker; a demotion still projects; a promotion delivers Good.
L12 durable restore A durable subscription, if the rig's SDK configuration enables them, across a node restart The restored items are created through the custom item (projected on the Standby). If durable subscriptions are not enabled on the rig, record that and leave the leg open.

Record evidence in docs/plans/2026-11-xx-warm-redundancy-717-live-gate.md. Do not mark G8/G9 (#615); they need CTT and a production client (Ignition 8.3 / ScadaBridge after T11).

Rollout

  1. Merge T1–T12 dark (Unset default). The full CI floor applies.
  2. Deploy the ScadaBridge T11 change everywhere ScadaBridge reads OtOpcUa.
  3. Upgrade both nodes of a pair to a revision-capable build (T3) before flipping either one.
  4. Flip both nodes of one docker-dev pair, then run T13. After that, flip one non-production site pair.
  5. Production is a separate, owner-approved change per pair. Flip the Secondary first, then the Primary, in one window.

A mixed window (one Warm, one Unset) is tolerable: the Unset node serves Good at 240/250. But do not leave it that way, because ServerUriArray then exists on only one node.

Rollback

Set OpcUa:Redundancy:Mode=Unset on both nodes and restart them. The setting is startup-static, so there is nothing persistent to undo: no schema changes and no artifact changes. The Revision field stays and is harmless, because unrevisioned handling is preserved. Rolling back ScadaBridge is independent; its ServiceLevel-aware failover also works against Unset servers.

Implementation status (2026-11-13)

T1–T10 and T12 are implemented on master, default-off (OpcUa:Redundancy:Mode defaults to Unset, pinned by WarmRedundancyWireTests.Unset_is_todays_server_exactly and OpcUaPublishActorWarmTests.Unset_never_drives_the_projection_and_keeps_todays_levels). T11 is ScadaBridge#138 (separate). T13 ran 2026-11-13 on an isolated two-node pair (not the shared docker-dev rig): every applicable leg passes after c569f806, which fixes two crash-failover defects the gate found. The survivor no longer demotes when auto-down removes the Primary, and the verdict reads its own cluster events instead of the lagging read view. Evidence: docs/plans/evidence/2026-11-13-717-t13/RESULTS.md. L8 (None) was not run, and L12 is not applicable (durable subscriptions are off). G8/G9 (#615) stay open.

Test homes: WarmStandbyUnitTests (OpcUaServer.Tests: marker, rewrite, decision record, the SDK-field regression pin, option rules), WarmRedundancyWireTests (OpcUaServer.IntegrationTests: metadata per mode, read projection, write and Part 9 refusal, the three T0 demotion timings, live-item queue, alarm suppression/refresh, effective-URI XML load), WarmRedundancyRevisionTests + WarmServiceLevelBandTests (ControlPlane.Tests), WarmRoleAuthorityTests + OpcUaPublishActorWarmTests (Runtime.Tests), OpcUaRedundancyValidationTests (Host.IntegrationTests). The deferred depth below adds WarmSingletonHandoverTests (ControlPlane.Tests), WarmRedundancyMetricsTests (Runtime.Tests), WarmStandbyCloneMatrixTests (OpcUaServer.Tests), WarmRedundancyWireTests.Deferred.cs and WarmRedundancyPooledWireTests (OpcUaServer.IntegrationTests).

Deferred coverage added (2026-11-13, in-process, no docker). Each test was mutation-checked against a deliberately broken implementation and failed. No product bug was found.

  • T3: WarmSingletonHandoverTests (ControlPlane.Tests). A real 2-node cluster runs the redundancy singleton under ClusterSingletonManager on both nodes. The old incarnation's last publish is held back through the broadcast seam, its node is downed, and the singleton hands over. The held publish is then delivered to a real Warm OpcUaPublishActor on the survivor. That publish is the older revision but has the higher sequence. The survivor stays Active. The second test pins that a removed node cannot rejoin from its old ActorSystem. A new ActorSystem on the same address rejoins as a younger member, with a higher up-number. Mutations killed:
    • the gate accepts older revisions;
    • the revision drops the up-number and incarnation (sequence only);
    • for the pin, skipping the Down (a harness negative control).
  • T6: A_DataEncoding_read_and_item_are_projected_… (wire). On the Standby, a Binary or XML DataEncoding read of a structured (EUInformation) tag is projected. An encoding a scalar cannot take still returns the base read's own BadDataEncodingUnsupported. An item created with DefaultXml gets the projected value on the Standby, and on promotion it receives the value read through its own encoding (TypeId i=888). Mutations killed:
    • DataEncoding reads are not projected;
    • the promotion queue ignores the item's DataEncoding.
  • T7:
    • An_item_deleted_between_queue_and_publish_… (wire). The deleted item's queued samples never surface. Its sibling is still first-delivered projected, and a Republish gives the same result.
    • WarmRedundancyPooledWireTests.A_client_handle_reused_after_delete_… (wire, non-parallel collection). A marked payload goes back through SDK 2.0.1's process-wide payload pool, and its ClientHandle is reused by an i=2267 item. That item stays Good. The test first asserts that the pool really hands out a marked payload.
    • WarmStandbyCloneMatrixTests (unit). One delivered clone carries two DataChangeNotifications (one with DiagnosticInfos), an EventNotificationList and a StatusChangeNotification. Only marked values are rewritten. Diagnostics, events and status pass through in order. Neither the clone nor the cached message is mutated, and the result encodes. Two clones project to identical bytes. A recycled payload's stale marker, rented through the real pool with the same handle, never validates.
    • Mutations killed:
      • the marker is not validated (wire and unit);
      • the marker is not set;
      • DiagnosticInfos are dropped;
      • non-data payloads are dropped;
      • the rewrite happens in place;
      • only the first DataChangeNotification is rewritten.
  • T8: An_ACL_stood_down_item_receives_nothing_… (wire). An item that RevalidateMonitoredItems stood down gets nothing from a demotion or a promotion (QueueCurrentToLiveItems returns 1, not 2). Its permitted sibling gets both. Once the grant is restored, the next demotion reaches the item. Mutation killed: stood-down items are not skipped.
  • T10:
    • WarmRedundancyMetricsTests (Runtime.Tests, non-parallel collection) checks four things:
      • warm_transition, with direction and reason (agreed / snapshot), and no count on a same-role snapshot;
      • the warm_role gauge, -1, then 0, then 1, then 0;
      • snapshot_rejected at the publish actor for older_revision and unrevisioned;
      • at DriverHostActor, a stale snapshot is dropped under Warm and counted, and its role view does not move. The Unset control does move. This is the first actor test of the driver host's ordering gate.
    • WarmRedundancyPooledWireTests checks standby.projected_notifications: it equals the rewritten count, counts again on Republish, and stays 0 while Active. It also checks standby.suppressed_alarm_events: one per withheld transition, none once Active.
    • Every listener selects by instrument instance.
    • Mutations killed: each counter's Add was removed, the direction tag inverted, the gauge never set, and the driver-host gate ignored.

Still open (live-only, T13): the durable restore, TransferSubscriptions under one identity, and the rig legs.