DCL: graceful stop of the Warm Primary still waits out the connectivity retries (66 s) - ServiceLevel 0 is seen but cleared as unknown #141

New Issue

2026-11-13T09:00:26-05:00

dohertj2 commented

2026-11-13 09:00:26 -05:00

Found in the QA Warm gate for #138 / lmxopcua#907 on 2026-11-13. The record is lmxopcua docs/deploy/2026-11-13-qa-warm-717.md, step 4a.

What happened. The WarmCanary connection (vq03 primary, vq04 backup) was on vq03 when vq03's OtOpcUa was stopped gracefully. vq03 was the Warm Primary.

13:52:57.435Z: the DCL read ServiceLevel 0 and logged "Primary" endpoint reports ServiceLevel 0 (Degraded band, < 200) — will check the other endpoint if it persists for 10s. vq04 had already promoted to 250 (13:52:57.3Z).
13:53:03.432Z: the next poll found the level unreadable because the server was closing. The DCL logged ServiceLevel is unknown — degraded watch cleared.
13:53:06.4Z: AdapterDisconnected. The connectivity trigger then retried the stopped primary FailoverRetryCount (3) times, about 17 s each (connect timeout + 5 s), before switching to the backup at 13:53:57.9Z.
Good again on the backup at 13:54:02.9Z, 66 s after the stop. The connectivity-only baseline under Mode=Unset was the same: 67 s.

A graceful stop reports 0 for only about 6 s before the session dies (#906 makes the leave fast). That is shorter than the 10 s degraded window, so #138 never fires on the most common failover. The 66 s is the old connectivity path.

Proposal (pick one or both):

On AdapterDisconnected, if the last ServiceLevel sample of this session was below 200, probe the other endpoint at once. If it answers Healthy, switch without the retry cycle (expected: about 10 s, all keep-alive detection).
Treat "unknown after a degraded sample" as still degraded rather than clearing the streak. A server that stops answering after reporting 0 is going away, not recovering.

Either option has to keep the anti-flap rules (Healthy and higher, dwell).

Not blocking: Warm is live on QA and every #138 gate passed. The connect-time preference moved the connection off the Standby in 0.1 s.

https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

Found in the QA Warm gate for #138 / lmxopcua#907 on 2026-11-13. The record is lmxopcua `docs/deploy/2026-11-13-qa-warm-717.md`, step 4a. **What happened.** The `WarmCanary` connection (vq03 primary, vq04 backup) was on vq03 when vq03's OtOpcUa was stopped gracefully. vq03 was the Warm Primary. - 13:52:57.435Z: the DCL read ServiceLevel **0** and logged `"Primary" endpoint reports ServiceLevel 0 (Degraded band, < 200) — will check the other endpoint if it persists for 10s`. vq04 had already promoted to 250 (13:52:57.3Z). - 13:53:03.432Z: the next poll found the level unreadable because the server was closing. The DCL logged `ServiceLevel is unknown — degraded watch cleared`. - 13:53:06.4Z: `AdapterDisconnected`. The connectivity trigger then retried the stopped primary `FailoverRetryCount` (3) times, about 17 s each (connect timeout + 5 s), before switching to the backup at 13:53:57.9Z. - Good again on the backup at 13:54:02.9Z, **66 s** after the stop. The connectivity-only baseline under `Mode=Unset` was the same: 67 s. A graceful stop reports 0 for only about 6 s before the session dies (#906 makes the leave fast). That is shorter than the 10 s degraded window, so #138 never fires on the most common failover. The 66 s is the old connectivity path. **Proposal (pick one or both):** - On `AdapterDisconnected`, if the last ServiceLevel sample of this session was below 200, probe the other endpoint at once. If it answers Healthy, switch without the retry cycle (expected: about 10 s, all keep-alive detection). - Treat "unknown after a degraded sample" as *still degraded* rather than clearing the streak. A server that stops answering after reporting 0 is going away, not recovering. Either option has to keep the anti-flap rules (Healthy and higher, dwell). Not blocking: Warm is live on QA and every #138 gate passed. The connect-time preference moved the connection off the Standby in 0.1 s. https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

dohertj2 added the type:followup priority:p3 cross-repo labels 2026-11-13 09:00:26 -05:00

dohertj2 referenced this issue from a commit

2026-11-13 09:01:26 -05:00

docs(deploy): record QA WarmCanary OPC UA connection (vq03/vq04 OtOpcUa) and the #138 live gate

dohertj2 referenced this issue

2026-11-13 09:02:49 -05:00

OPC UA adapter: ServiceLevel-aware failover (prerequisite for OtOpcUa Warm redundancy, lmxopcua#717) #138

dohertj2 referenced this issue from a commit

2026-11-13 09:37:12 -05:00

fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)

dohertj2 commented

2026-11-13 09:37:25 -05:00

Fixed on main in c4ec51e8. Not deployed; QA live re-check pending.

Design: both proposals, plus the ServiceLevel-0 shortcut. All three keep the anti-flap rules: the other endpoint must report Healthy (≥ 200) and higher, and the dwell applies.

ServiceLevel 0 skips the degraded window. In OPC UA Part 4, 0 means Maintenance (the server is not available for use), and OtOpcUa reports it during a graceful stop. On a 0, the other endpoint is probed at once. The shortcut fires once per degraded streak, so a peer that is not Healthy is re-probed once per window, not on every poll. Other Degraded levels (standby 90/50, booting 1) still wait out the window. In the QA timeline this fires at the first 0 sample (13:52:57.4), about 9 s before the session dies.
A lost link after a Degraded level probes the other endpoint first. This is proposal 1. The actor keeps the session's last known level, and an unknown sample does not overwrite it. That covers proposal 2's case without changing the rule that an unknown sample ends the streak. If the link drops while that level is below 200, the reconnect timer is held and the other endpoint is probed. If it is Healthy, the connection switches and reconnects at zero delay, and the stopped node is never retried. Otherwise the ordinary retry cycle runs on the same endpoint, minus the time the probe took.

Rationale: the 0 trigger is the fast path, because it fires before the session closes. The lost-link trigger is the backstop for a 0 that was seen too late, or a Degraded level followed by a drop. The disconnect probe is not applied to every drop: after only Healthy levels (250/240, Mode=Unset without a 0), inside the dwell, or with the feature off, the retry-then-failover behaviour is exactly as before. A network blip on a 250 Primary therefore still does not move the connection to the 240 Secondary. ServerState Shutdown was not added: it needs a second read per poll, and OtOpcUa already signals the stop with 0.

Evidence

DataConnectionGoingAwayFailoverTests: 10 cases. Each trigger and non-trigger has a positive control, and the "no shortcut" cases assert on the order in which sessions were opened. Cases: 0 skips a 30 s window; 90 and 1 still wait (then 0 moves the connection); 0 with a Degraded peer stays put and probes it exactly once; 0 inside the dwell waits, then switches; the QA sequence (Degraded → unknown → link lost) makes no connect to the stopped node; a lost link after Healthy levels gives 3× Primary, then Backup, as before; a lost link with a Degraded peer gives probe → normal retry; a lost link inside the dwell gives retry, no probe; with the feature disabled there is no probe.
Real in-process OPC UA server pair, end to end, on production defaults (5 s reconnect, 2 s poll, 10 s window, 60 s dwell, 3 retries). The Warm Primary goes to 0 while the peer promotes to 250, then the server stops 1 s later. Failover took 1.1 s. With only the lost-link trigger: 12.1 s, dominated by keep-alive detection. With neither trigger: 33.1 s, which fails the test's 10 s bound.
Mutation testing: 13 mutants, all killed. They covered: shortcut off; shortcut on every poll; shortcut for any Degraded level; lost-link probe off; unknown overwrites last-known; probe after Healthy; lost-link ignores the dwell; lost-link switches to a non-Healthy peer; fallback never re-arms; 0 ignores the dwell; and the e2e run with both triggers off and with lost-link only.
ZB.MOM.WW.ScadaBridge.slnx: 0 warnings, 0 errors. DataConnectionLayer.Tests: 608/608 on 3 consecutive full runs.
Docs updated: Component-DataConnectionLayer.md → ServiceLevel-aware failover → A server going away (Gitea #141), and docs/components/DataConnectionLayer.md.

QA re-check to do after deploy: repeat step 4a of lmxopcua docs/deploy/2026-11-13-qa-warm-717.md. The expected result is Good on vq04 within a few seconds of the stop, with the site event Failover from Primary to Backup (ServiceLevel).

https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

Fixed on `main` in c4ec51e8. **Not deployed; QA live re-check pending.** **Design: both proposals, plus the ServiceLevel-0 shortcut.** All three keep the anti-flap rules: the other endpoint must report Healthy (≥ 200) and higher, and the dwell applies. 1. **ServiceLevel 0 skips the degraded window.** In OPC UA Part 4, 0 means Maintenance (the server is not available for use), and OtOpcUa reports it during a graceful stop. On a 0, the other endpoint is probed at once. The shortcut fires once per degraded streak, so a peer that is not Healthy is re-probed once per window, not on every poll. Other Degraded levels (standby 90/50, booting 1) still wait out the window. In the QA timeline this fires at the first 0 sample (13:52:57.4), about 9 s before the session dies. 2. **A lost link after a Degraded level probes the other endpoint first.** This is proposal 1. The actor keeps the session's last *known* level, and an unknown sample does not overwrite it. That covers proposal 2's case without changing the rule that an unknown sample ends the streak. If the link drops while that level is below 200, the reconnect timer is held and the other endpoint is probed. If it is Healthy, the connection switches and reconnects at zero delay, and the stopped node is never retried. Otherwise the ordinary retry cycle runs on the same endpoint, minus the time the probe took. Rationale: the 0 trigger is the fast path, because it fires before the session closes. The lost-link trigger is the backstop for a 0 that was seen too late, or a Degraded level followed by a drop. The disconnect probe is not applied to every drop: after only Healthy levels (250/240, `Mode=Unset` without a 0), inside the dwell, or with the feature off, the retry-then-failover behaviour is exactly as before. A network blip on a 250 Primary therefore still does not move the connection to the 240 Secondary. ServerState Shutdown was not added: it needs a second read per poll, and OtOpcUa already signals the stop with 0. **Evidence** - `DataConnectionGoingAwayFailoverTests`: 10 cases. Each trigger and non-trigger has a positive control, and the "no shortcut" cases assert on the order in which sessions were opened. Cases: 0 skips a 30 s window; 90 and 1 still wait (then 0 moves the connection); 0 with a Degraded peer stays put and probes it exactly once; 0 inside the dwell waits, then switches; the QA sequence (Degraded → unknown → link lost) makes no connect to the stopped node; a lost link after Healthy levels gives 3× Primary, then Backup, as before; a lost link with a Degraded peer gives probe → normal retry; a lost link inside the dwell gives retry, no probe; with the feature disabled there is no probe. - Real in-process OPC UA server pair, end to end, on production defaults (5 s reconnect, 2 s poll, 10 s window, 60 s dwell, 3 retries). The Warm Primary goes to 0 while the peer promotes to 250, then the server stops 1 s later. **Failover took 1.1 s.** With only the lost-link trigger: 12.1 s, dominated by keep-alive detection. With neither trigger: 33.1 s, which fails the test's 10 s bound. - Mutation testing: 13 mutants, all killed. They covered: shortcut off; shortcut on every poll; shortcut for any Degraded level; lost-link probe off; unknown overwrites last-known; probe after Healthy; lost-link ignores the dwell; lost-link switches to a non-Healthy peer; fallback never re-arms; 0 ignores the dwell; and the e2e run with both triggers off and with lost-link only. - `ZB.MOM.WW.ScadaBridge.slnx`: 0 warnings, 0 errors. DataConnectionLayer.Tests: 608/608 on 3 consecutive full runs. - Docs updated: `Component-DataConnectionLayer.md` → *ServiceLevel-aware failover* → *A server going away (Gitea #141)*, and `docs/components/DataConnectionLayer.md`. QA re-check to do after deploy: repeat step 4a of lmxopcua `docs/deploy/2026-11-13-qa-warm-717.md`. The expected result is Good on vq04 within a few seconds of the stop, with the site event `Failover from Primary to Backup (ServiceLevel)`. https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

dohertj2 closed this issue

2026-11-13 09:37:27 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#141