Every applicable T13 leg passes on0c19f26c+c569f806. The leg list covers: - metadata and ServiceLevel bands; - standby projection; - alarm suppression with Standby ConditionRefresh and the promotion refresh; - the three D4 demotion timings, run live; - standby write and Part 9 refusal (BadInvalidState); - graceful and crash failover, with no failback; - subscription transfer; - the Mode=Unset regression. The two crash-path defects the gate found are fixed inc569f806. Evidence, rig files and the raw-client harness are in docs/plans/evidence/2026-11-13-717-t13/. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
12 KiB
Gitea #717 T13 live gate: Warm redundancy on an isolated docker pair (2026-11-13)
Result: every leg that applies passes on the fixed build. The gate found two product defects on the crash-failover
path, both fixed in c569f806 with tests (mutation-checked) and re-verified live.
Rig
An isolated two-node pair on the Mac, separate from the shared otopcua-dev rig, which was not touched.
| Item | Value |
|---|---|
| Build | origin/master 0c19f26c (runs 1 and 2), then 0c19f26c + fix c569f806 (runs 2b, 3 and 4). Published on the host and copied into a runtime-only aspnet:10.0 image (rig/Dockerfile). |
| Images | otopcua-host:t13-0c19f26c and otopcua-host:t13-0c19f26c-fix. Both were removed at teardown. |
| Compose | project otopcua-t13 (rig/compose.t13.yml) with its own network, its own SQL Server 2022 and its own volumes (t13-mssql-data, t13-c1, t13-c2). Ports: OPC UA 14840/14841, HTTP 19210/19211, SQL 14339. |
| Topology | central-1 + central-2, roles admin,driver,cluster-MAIN, self-first seeds, per-node ApplicationUri with the partner in PeerApplicationUris (as in docker-dev). OpcUa__Redundancy__Mode=Warm on both nodes, and Unset for the last run. PromoteWhilePrimaryUnreachable is at its default (true). |
| External contact | None. LDAP points at loopback (AdminUI login is disabled), the historian is off, and no driver reaches a device. |
| Schema | dotnet ef database update from the host against :14339. |
| Config | rig/seed.sql plus rig/fixeq.sql, then POST /api/deployments. The deploy was rejected once with EquipmentIdNotDerived; fixeq.sql renames the equipment to the canonical EQ-3326c7a4728e. The config contains:• a Calculation driver t13-calc/Device1, with calc-seconds (Int32, Read, a 1 s timer, value = UTC seconds-of-day) and calc-writable (ReadWrite);• UNS t13/warm/t13-equipment, with references to both tags;• VirtualTag vt-second;• scripted alarm t13-even-minute (OffNormal, Retain), active on even UTC minutes;• one API key t13-operator (principal t13-op) with a Cluster-scope grant (WriteOperate, AlarmAck and the rest).Deployment 5c90c82f, sealed with acks from both nodes. |
| Client | harness/ is a raw OPC UA client on SDK 2.0.1. It uses raw Publish/Republish/TransferSubscriptions so sequence numbers are visible. meta/probe sessions are anonymous or API-key; watch, timing* and transfer sessions use the API-key UserName identity. |
Logs: logs/run1 (master, the first graceful + crash run), logs/run2 (the first fix, crash only), logs/run3
(final fix: graceful, crash, timings and transfer), logs/run4-unset. logs/actions.log holds the operator timeline.
The server-*.log files are the containers' stdout. From run 2 onwards they carry millisecond timestamps.
Legs
| Leg | Result | Evidence |
|---|---|---|
| L1 metadata | PASS | L1-L2-meta-warm.txt. Both nodes: RedundancySupport=2 (Warm), i=2296 type definition NonTransparentRedundancyType (i=2039). ServerUriArray lists self first: [central-1, central-2] on :14840 and [central-2, central-1] on :14841. A Write to it returns BadNotWritable. ServerArray is unchanged and lists the same URIs. |
| L2 bands | PASS | Steady state: Primary 250, Standby 90. Boot reads on a rejoin were 1 (run 3) or 50 (run 1, the Standby-degraded band). |
| L3 standby data | PASS | L3-L10-probe-warm.txt. Standby reads of the raw tag, its UNS reference, the VirtualTag and the writable tag all return BadOutOfService/null, while the Primary reads Good at the same moment. Standby subscriptions deliver one BadOutOfService per item and then silence. Acquisition continues: on promotion the current value arrives Good without a new driver publish (run1/watch-c2.log 09:06:00.371). |
| L4 standby alarms | PASS | run1/watch-c*.log 09:02:40–09:05:54: the Primary delivered 3 transitions (09:03, 09:04, 09:05) on both the Server-object item and the equipment-folder item; the Standby delivered 0. Reconfirmed after the crash (run 3, 09:26:00). A ConditionRefresh on the Standby returns RefreshStart, the retained T13 even minute condition, RefreshEnd. |
| Unsolicited promotion refresh | PASS | On every promotion, each live event item received RefreshStart, the retained condition, RefreshEnd, followed by new events (run 1 09:06:00.371; run 3 09:23:28.650 and 09:10:59/09:24:49 crash). |
L5 graceful failover (docker stop of the Primary, i.e. CoordinatedShutdown → cluster Leave) |
PASS (see note A) | Run 3, final build: the old Primary demoted at 09:23:28.332, the same millisecond it was marked Leaving. Its watcher saw ServiceLevel 0, then BadOutOfService on all three tags, with the last Good at src .213 and no Good after the drop. The survivor promoted at .601–.609 in the spec order: RefreshStart/retained/RefreshEnd, then SL 250, then Good data. There was no dual-Active window (~270 ms with no Active node). |
| Three demotion timings (first delivery after demotion) | PASS | run3/timing1..3.txt, each across a live graceful demotion (detected 96–127 ms after docker stop). T1: 4 Good samples queued with publishing disabled, then the client handle changed 1→99, then demotion. The first Publish carried 5× BadOutOfService/null for h99, while i=2267 (h2) stayed Good, so it was not rewritten. Republish of the same sequence number returned the same projected payload. T2: maxNotificationsPerPublish=1 with prebuilt messages (MoreNotifications=true). Message a (seq 2) was Good while Active. After demotion, Republish of the prebuilt never-sent seq 3 returned BadMessageNotAvailable, and Publish then delivered seq 3 projected; its Republish was also projected. T3: Good was delivered while Active. After demotion, Republish returned the original Good; after the ack it returned BadMessageNotAvailable. |
| L10 writes / Part 9 methods on the Standby | PASS | L3-L10-probe-warm.txt (API-key session with WriteOperate and AlarmAck). Standby: Write on the raw and UNS calc-writable returns BadInvalidState, and so do Acknowledge and AddComment on the condition. Active, same session shape: the Write is Good (the optimistic path; the Calculation driver later reverts it), Acknowledge is Good and AddComment is Good. HistoryRead returns BadHistoryOperationUnsupported on both nodes because the rig has no historian, so it is not refused by role. |
L6 crash failover (docker kill of the Primary, PromoteWhilePrimaryUnreachable on) |
FAIL on 0c19f26c → PASS after c569f806 (findings 1 and 2) |
Run 3, final build: kill at 09:24:37.877; unreachable at 09:24:49.263; the survivor promoted at 09:24:49.275 (unreachable-primary, 12 ms after its own event). The measured gap from kill to Active is 11.4 s, all of it the failure detector. The survivor stayed Active through auto-down's removal (09:25:05) with Good data throughout. ServiceLevel went 250 → 200 (Active-stale, because no singleton heartbeat arrived while the Primary was unreachable) → 250 once its own singleton published (09:25:20). |
| L7 rejoin / no failback | PASS | After the graceful stop and after the crash, the restarted node came back Standby: the boot read was 1 or 50, then 90, and 250/255 never appeared. The Active node stayed at 250 throughout (run1, run3 09:24:00, 09:25:48). |
| L11 transfer | PASS | run3/transfer-c1.log. Session A (API key) created a subscription on the Standby, and its items read BadOutOfService. Session B, with the same identity, called TransferSubscriptions(sendInitialValues=true) and got Good; A received GoodSubscriptionTransferred. B's initial values were projected (BadOutOfService), and B received no Good until promotion. The Primary was then stopped and the node promoted: B got SL 250 and Good at 09:29:16.621. The other node was restarted and this node was stopped: B got SL 0 and BadOutOfService for both tags (09:30:09.514) before the session ended. |
L9 Mode=Unset regression |
PASS | run4-unset/, same pair and same fixed build, recreated with OpcUa__Redundancy__Mode=Unset. The levels are 250/240. RedundancySupport=0 and i=2296 is ServerRedundancyType, as the SDK default. The orphan i=11314 reads Good []. Both nodes serve Good data, the secondary accepts writes and AddComment (today's path), and the secondary delivers alarm events (09:33:00) with 0 BadOutOfService in 190 Good notifications. |
L8 Mode=None standalone |
not run | It was not in this gate's scope. It is covered by WarmRedundancyWireTests (None metadata). |
| L12 durable restore | not applicable | Durable subscriptions are not enabled (SDK default off), as recorded in #717. |
Findings
Finding 1: the crash survivor demoted when auto-down removed the Primary (fixed). On 0c19f26c (run1,
09:10:47 kill), the survivor promoted at 09:10:59 because the Primary was unreachable. At 09:11:15 auto-down removed the
Primary. The snapshot's Primary then left the unreachable set, so rule (3′) no longer held, and the survivor demoted
to Standby (reason=snapshot, SL 50, BadOutOfService to every client). It was Active again only at 09:11:30, when
its own singleton finished the hand-over retries and published. The result was a ~15 s outage immediately after a
successful crash failover. Fix: a Primary that is Down, or that this node saw removed (and not back since), counts as
gone (WarmRoleAuthority.SnapshotPrimaryGone, removal tracked from MemberRemoved in OpcUaPublishActor). A member
that is only Leaving/Exiting, or that the node never saw, does not count. Re-verified in run 2 (09:17:53) and run 3
(09:24:37): no demotion through the removal.
Finding 2: the Warm verdict read Akka's read view, which lags the event being handled (fixed). OpcUaPublishActor
re-evaluated on each member/reachability event, but it read Cluster.State/SelfMember. Those come from
ClusterReadView, which is a separate subscriber to the same events. In run 2 the survivor stayed Standby 4.3 s
after its own UnreachableMember event (09:18:05.722 → 09:18:09.988); it promoted only on the next health tick. The
same lag shows in run 1's graceful handover. The old Primary demoted only at Exiting (SL src 09:06:00.552), while the
survivor was already Active from .316. That left a ~240 ms dual-Active window, in which the old Primary delivered a Good
sample at .400. Fix: the Warm verdict uses a membership view built from the actor's own subscription, falling back to
the read view only before the first event. Re-verified: promotion now comes 12 ms after the unreachable event (run 3),
and the graceful demotion lands at the same millisecond as Leaving, with no overlap (run 3, 09:23:28.332).
Tests (Runtime.Tests, 1058 passed, 3 skipped):
WarmRoleAuthorityTests.A_snapshot_Primary_that_auto_down_removed_is_still_gone_and_the_survivor_stays_Active, a truth table that includes the Leaving/Exiting/never-seen negatives;OpcUaPublishActorWarmTests.The_crash_survivor_promotes_on_its_own_unreachable_event_and_stays_Active_through_the_removal, which uses synthetic events that the read view never sees, with a positive control: the Primary comes back and demotes the node.
Mutations: a read-view-only verdict fails the actor test at promotion; turning off the removed arm fails both tests.
Notes
- A. The old Primary's level on a graceful leave is 0, not 90.
ServiceLevelCalculator.ComputeWarmreturns 0 for a member that is not Up/Joining, so a leaving Primary drops 250 → 0. The plan's L5 text says "drops to 90". 0 is below every band, and it is what the code specifies for a non-Up member. This is a wording difference, not a defect. - B. The Active-stale band after a crash. The survivor sits at 200 (Active, stale snapshot) for ~10–20 s until its
own singleton publishes. Data stays Good. This is expected from
ComputeWarm, and clients still see it as the higher-level node. - C. The manual-failover button (AdminUI) was not used.
docker stopdrives the same mechanism (CoordinatedShutdown → cluster Leave), and it can be scripted with millisecond timing. - Both nodes ran the same build in every run. The pre-fix image was used only for runs 1 and 2.