dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

OPC UA adapter: ServiceLevel-aware failover (prerequisite for OtOpcUa Warm redundancy, lmxopcua#717) #138

New Issue
Closed
opened 2026-11-13 02:33:45 -05:00 by dohertj2 · 3 comments
dohertj2 commented 2026-11-13 02:33:45 -05:00
Owner
Copy Link
Copy Source

Prerequisite for lmxopcua#717 (Warm non-transparent redundancy, spec docs/plans/2026-11-11-warm-redundancy-717-design.md §5 in OtOpcUa).

ScadaBridge fails over between OPC UA endpoints only after FailoverRetryCount consecutive unstable disconnects (DataConnectionLayer/Actors/DataConnectionActor.cs:655-690) and never reads ServiceLevel (i=2267), RedundancySupport or ServerUriArray. Under OtOpcUa Warm, the standby keeps a healthy session but serves BadOutOfService data at a standby ServiceLevel band (< 200), so a ScadaBridge connection on the standby would sit on Bad quality indefinitely and never fail over: an outage.

Needed: make the OPC UA adapter ServiceLevel-aware. Monitor i=2267 on the active endpoint, and treat a sustained Degraded band (< 200 for N s, configurable) while the other endpoint reports a higher level as a failover trigger (and prefer the higher-ServiceLevel endpoint on connect). Also add a test for the unsolicited RefreshStart/RefreshEnd pair OtOpcUa sends on promotion.

Blocking: no OtOpcUa tier may set OpcUa:Redundancy:Mode=Warm until this ships and is deployed to that tier.

Prerequisite for lmxopcua#717 (Warm non-transparent redundancy, spec `docs/plans/2026-11-11-warm-redundancy-717-design.md` §5 in OtOpcUa). ScadaBridge fails over between OPC UA endpoints only after `FailoverRetryCount` consecutive unstable disconnects (`DataConnectionLayer/Actors/DataConnectionActor.cs:655-690`) and never reads `ServiceLevel` (`i=2267`), `RedundancySupport` or `ServerUriArray`. Under OtOpcUa Warm, the standby keeps a healthy session but serves `BadOutOfService` data at a standby ServiceLevel band (< 200), so a ScadaBridge connection on the standby would sit on Bad quality indefinitely and never fail over: an outage. Needed: make the OPC UA adapter ServiceLevel-aware. Monitor `i=2267` on the active endpoint, and treat a sustained Degraded band (< 200 for N s, configurable) while the other endpoint reports a higher level as a failover trigger (and prefer the higher-ServiceLevel endpoint on connect). Also add a test for the unsolicited RefreshStart/RefreshEnd pair OtOpcUa sends on promotion. **Blocking:** no OtOpcUa tier may set `OpcUa:Redundancy:Mode=Warm` until this ships and is deployed to that tier.
dohertj2 added the cross-repopriority:p2type:feature labels 2026-11-13 02:34:01 -05:00
dohertj2 commented 2026-11-13 03:07:45 -05:00
Author
Owner
Copy Link
Copy Source

Implemented and pushed to main as 712c5117 (not deployed).

Behaviour

An OPC UA connection with a backup now polls Server_ServiceLevel (i=2267) on the connected endpoint every 2 s. MxGateway connections, single-endpoint connections and disabled connections never read it.

  • Trigger. The connected level stays below 200 for the degraded window (default 10 s). Any sample of 200 or more, or an unreadable sample, breaks the streak. The actor then probes the other endpoint with a short-lived session (connect, read i=2267, dispose; 15 s budget). It fails over only if the other endpoint reports a Healthy level (200 or more) that is higher than the connected one.
  • Stays put when the other endpoint is also below 200 (90/50, 50/90, 90/199), when it is unreachable, or when the connected level is unknown. The window then re-arms, so the other endpoint is probed at most once per window.
  • Connect / reconnect preference. The first sample of every session is evaluated at once, without the window. A connection that comes up on the Warm standby (90) or a booting node (1) moves straight to a Healthy peer.
  • Anti-flap. No second ServiceLevel switch within the minimum dwell (default 60 s), and that includes the connect-time check. With the unbroken-streak rule, a pair that flips roles cannot bounce the connection.
  • One failover path. All three triggers (connect failures, unstable disconnects, ServiceLevel) now share SwitchToOtherEndpoint(). The ServiceLevel switch then runs BecomeReconnecting(ReconnectCause.ServiceLevelFailover): zero delay, no unstable-disconnect accounting, and ReSubscribeAll / ReSubscribeAllAlarms on the new endpoint. Site event: Failover from Primary to Backup (ServiceLevel), with both levels. Poll and probe results are fenced by adapter generation.
  • Today's OtOpcUa (250/240, both Active band) is unchanged. A level of 200 or more never starts a streak, so there is no probe and no switch. A 240 Secondary never triggers (tested for 240/250, 250/240 and 200/250). That is why the feature is on by default. The one behaviour change against non-Warm servers is intended: a node reporting 100/110 or 0/10 while its peer reports 200 or more is left after the window.

Config knobs

  • Per connection, in the primary endpoint's OPC UA JSON. The backup's copy is ignored.
    • Shape: "serviceLevelFailover": {"enabled": true, "degradedWindowSeconds": 10, "minDwellSeconds": 60}.
    • Absent or null means those defaults.
    • Validation: window 1–3600 s, dwell 0–86400 s.
    • It lives inside the protocol JSON, which is already encrypted at rest. No column, no migration, no artifact or wire change, and no hash churn for stored rows.
    • Central UI: a checkbox and two fields in the Backup endpoint card, next to Failover Retry Count. CLI: part of --primary-config (documented in Component-CLI).
  • Global, under ScadaBridge:DataConnection:
    • ServiceLevelPollInterval (2 s)
    • ServiceLevelProbeTimeout (15 s)

Tests

  • DataConnectionServiceLevelFailoverTests: 19 Akka TestKit cases. Each trigger and non-trigger case is covered, and every "no switch" case has a positive control (the reads or the probe did happen).
  • OpcUaServiceLevelTests: 6 cases.
    • Adapter mapping: Bad status or a non-byte value is unknown, and a read failure never raises Disconnected.
    • Real in-process OPC UA servers (SecureOpcUaTestServer.SetServiceLevel): the real i=2267 read tracks 250/240/90/50/1.
    • A real server pair through the production DataConnectionFactory: standby primary 90 + backup 250 ends up on the backup; 240 + 250 stays on the primary.
  • Promotion refresh (design §5):
    • RealOpcUaPromotionRefreshTests: an unsolicited RefreshStart / conditions / RefreshEnd on an existing item is framed as its own snapshot with a fresh epoch, and live derivation resumes after it.
    • DataConnectionPromotionRefreshTests: the existing snapshot handling forwards it in order and reconciles the cache. A condition the promoted node no longer retains is gone from a late subscriber's replay.
  • Commons serializer and validator tests (including the upgrade-safety case: an existing row has serviceLevelFailover null and adds no flat keys) and two bUnit form tests.
  • Mutation-checked: 15 of 15 mutants killed.
    • Threshold 200→241, no window, no connect check, no dwell, other endpoint need not be Healthy, unknown counted as degraded, Enabled ignored, a Healthy sample not breaking the streak, unreachable treated as switchable, policy read from the backup, no-backup check removed.
    • Adapter raising Disconnected on a failed read, adapter ignoring Bad status.
    • RefreshEnd not leaving refresh mode, actor merging instead of replacing on SnapshotComplete.
  • Release build of ZB.MOM.WW.ScadaBridge.slnx: 0 warnings, 0 errors.
  • Green: DataConnectionLayer (596 passed / 1 skipped), Commons (986), CentralUI (1551), SiteRuntime (1062), TemplateEngine (628), Transport (236), DeploymentManager (314), ManagementService (793).
  • ExecutionAwareLoggingTests.DisabledLevel_PublishesNothing failed once under full-suite load. It is unrelated to this change and passes in isolation and on the rerun.

Keeping this open

I'm leaving this issue open as the deploy gate for lmxopcua#717. The blocking condition in the description is "ships and is deployed to that tier", and nothing is deployed: DEV vd03 runs 424db155 and QA runs 0e540291. Close it when a tier that will set OpcUa:Redundancy:Mode=Warm runs 712c5117 or later. On QA, confirm the OtOpcUa pair's connection has the backup configured: the feature is inert without one.

Docs: docs/requirements/Component-DataConnectionLayer.md (new section ServiceLevel-aware failover, settings tables, alarm note), docs/components/DataConnectionLayer.md, docs/requirements/Component-CLI.md.

Implemented and pushed to `main` as **712c5117** (not deployed). ## Behaviour An OPC UA connection with a backup now polls `Server_ServiceLevel` (`i=2267`) on the connected endpoint every 2 s. MxGateway connections, single-endpoint connections and disabled connections never read it. - **Trigger.** The connected level stays **below 200** for the degraded window (default **10 s**). Any sample of 200 or more, or an unreadable sample, breaks the streak. The actor then probes the **other** endpoint with a short-lived session (connect, read `i=2267`, dispose; 15 s budget). It fails over **only if the other endpoint reports a Healthy level (200 or more) that is higher than the connected one.** - **Stays put** when the other endpoint is also below 200 (90/50, 50/90, 90/199), when it is unreachable, or when the connected level is unknown. The window then re-arms, so the other endpoint is probed at most once per window. - **Connect / reconnect preference.** The first sample of every session is evaluated at once, without the window. A connection that comes up on the Warm standby (90) or a booting node (1) moves straight to a Healthy peer. - **Anti-flap.** No second ServiceLevel switch within the minimum dwell (default **60 s**), and that includes the connect-time check. With the unbroken-streak rule, a pair that flips roles cannot bounce the connection. - **One failover path.** All three triggers (connect failures, unstable disconnects, ServiceLevel) now share `SwitchToOtherEndpoint()`. The ServiceLevel switch then runs `BecomeReconnecting(ReconnectCause.ServiceLevelFailover)`: zero delay, no unstable-disconnect accounting, and `ReSubscribeAll` / `ReSubscribeAllAlarms` on the new endpoint. Site event: `Failover from Primary to Backup (ServiceLevel)`, with both levels. Poll and probe results are fenced by adapter generation. - **Today's OtOpcUa (250/240, both Active band) is unchanged.** A level of 200 or more never starts a streak, so there is no probe and no switch. A 240 Secondary never triggers (tested for 240/250, 250/240 and 200/250). That is why the feature is **on by default**. The one behaviour change against non-Warm servers is intended: a node reporting 100/110 or 0/10 while its peer reports 200 or more is left after the window. ## Config knobs - **Per connection**, in the **primary** endpoint's OPC UA JSON. The backup's copy is ignored. - Shape: `"serviceLevelFailover": {"enabled": true, "degradedWindowSeconds": 10, "minDwellSeconds": 60}`. - Absent or `null` means those defaults. - Validation: window 1–3600 s, dwell 0–86400 s. - It lives inside the protocol JSON, which is already encrypted at rest. No column, no migration, no artifact or wire change, and no hash churn for stored rows. - Central UI: a checkbox and two fields in the *Backup endpoint* card, next to *Failover Retry Count*. CLI: part of `--primary-config` (documented in Component-CLI). - **Global**, under `ScadaBridge:DataConnection`: - `ServiceLevelPollInterval` (2 s) - `ServiceLevelProbeTimeout` (15 s) ## Tests - `DataConnectionServiceLevelFailoverTests`: 19 Akka TestKit cases. Each trigger and non-trigger case is covered, and every "no switch" case has a positive control (the reads or the probe did happen). - `OpcUaServiceLevelTests`: 6 cases. - Adapter mapping: Bad status or a non-byte value is unknown, and a read failure never raises `Disconnected`. - Real in-process OPC UA servers (`SecureOpcUaTestServer.SetServiceLevel`): the real `i=2267` read tracks 250/240/90/50/1. - A real **server pair** through the production `DataConnectionFactory`: standby primary 90 + backup 250 ends up on the backup; 240 + 250 stays on the primary. - **Promotion refresh** (design §5): - `RealOpcUaPromotionRefreshTests`: an unsolicited RefreshStart / conditions / RefreshEnd on an existing item is framed as its own snapshot with a fresh epoch, and live derivation resumes after it. - `DataConnectionPromotionRefreshTests`: the existing snapshot handling forwards it in order and **reconciles the cache**. A condition the promoted node no longer retains is gone from a late subscriber's replay. - Commons serializer and validator tests (including the upgrade-safety case: an existing row has `serviceLevelFailover` null and adds no flat keys) and two bUnit form tests. - **Mutation-checked: 15 of 15 mutants killed.** - Threshold 200→241, no window, no connect check, no dwell, other endpoint need not be Healthy, unknown counted as degraded, `Enabled` ignored, a Healthy sample not breaking the streak, unreachable treated as switchable, policy read from the backup, no-backup check removed. - Adapter raising Disconnected on a failed read, adapter ignoring Bad status. - RefreshEnd not leaving refresh mode, actor merging instead of replacing on SnapshotComplete. - Release build of `ZB.MOM.WW.ScadaBridge.slnx`: **0 warnings, 0 errors**. - Green: DataConnectionLayer (596 passed / 1 skipped), Commons (986), CentralUI (1551), SiteRuntime (1062), TemplateEngine (628), Transport (236), DeploymentManager (314), ManagementService (793). - `ExecutionAwareLoggingTests.DisabledLevel_PublishesNothing` failed once under full-suite load. It is unrelated to this change and passes in isolation and on the rerun. ## Keeping this open I'm leaving this issue **open** as the deploy gate for lmxopcua#717. The blocking condition in the description is "ships **and is deployed** to that tier", and nothing is deployed: DEV vd03 runs `424db155` and QA runs `0e540291`. Close it when a tier that will set `OpcUa:Redundancy:Mode=Warm` runs 712c5117 or later. On QA, confirm the OtOpcUa pair's connection has the backup configured: the feature is inert without one. Docs: `docs/requirements/Component-DataConnectionLayer.md` (new section *ServiceLevel-aware failover*, settings tables, alarm note), `docs/components/DataConnectionLayer.md`, `docs/requirements/Component-CLI.md`.
dohertj2 commented 2026-11-13 08:34:27 -05:00
Author
Owner
Copy Link
Copy Source

Deployed to QA vq03/vq04 inside build 8e32f353 (2026-11-13, record ScadaBridge 8ba5d5d7; 712c5117 is an ancestor) and to DEV vd03 in the same build (581e25ca).

It does nothing on either tier yet: QA ScadaBridge has a single data connection (MxGateway, no backup) and no OPC UA connection, so no connection polls ServiceLevel. Stays open as the lmxopcua#717/#907 prerequisite until a QA OPC UA primary/backup connection to the OtOpcUa pair exists and the Warm flip is gated.

Deployed to QA vq03/vq04 inside build `8e32f353` (2026-11-13, record ScadaBridge `8ba5d5d7`; `712c5117` is an ancestor) and to DEV vd03 in the same build (`581e25ca`). It does nothing on either tier yet: QA ScadaBridge has a single data connection (`MxGateway`, no backup) and no OPC UA connection, so no connection polls ServiceLevel. Stays open as the lmxopcua#717/#907 prerequisite until a QA OPC UA primary/backup connection to the OtOpcUa pair exists and the Warm flip is gated.
dohertj2 commented 2026-11-13 09:02:49 -05:00
Author
Owner
Copy Link
Copy Source

Gated live on QA, 2026-11-13, against OtOpcUa running Mode=Warm (lmxopcua#907). Closing.

QA runs 8e32f353, which contains 712c5117. New test connection WarmCanary:

  • id 2, primary OtOpcUa vq03, backup vq04;
  • serviceLevelFailover left at its defaults: enabled, 10 s, 60 s;
  • template 27 / instance 16 bound to OtOpcUa's 1 s script-runner canaries.

Records: ScadaBridge docs/deployment/deployment-records/2026-11-13-qa-warmcanary-opcua-connection.md (933da568) and lmxopcua docs/deploy/2026-11-13-qa-warm-717.md (1862fab6).

Case DCL log (site vq03) Result
Healthy Primary at 250, Mode Unset none, by design Good at 1/s; no streak
Unset: stop the Primary AdapterDisconnected → 3 connect retries → Failing over from "Primary" to "Backup" Good again in 67.1 s (connectivity path, baseline)
Pair restart into Warm; the connection reaches the booting Standby "Backup" endpoint reports ServiceLevel 50 (Degraded band, < 200) — will check the other endpoint now (connect-time preference), then Failing over from "Backup" to "Primary" (ServiceLevel 50 Degraded for 4s; "Primary" reports 250) Good 8.6 s after the Standby connect
Warm: stop the Primary "Primary" endpoint reports ServiceLevel 0 … (+0.77 s), then ServiceLevel is unknown — degraded watch cleared (+6.8 s), then the connectivity failover Good on the promoted node in 66.3 s. See below
Old Primary back as Standby (90) none No failback; stayed on 250 for 110 s with no Bad sample
On the Standby (90) over a healthy session: an in-place reconfigure re-arms on the primary, which is now the Standby Reconnected successfully on "Primary" 13:56:52.401 → reports ServiceLevel 90 … connect-time preference .413 → Failing over from "Primary" to "Backup" (ServiceLevel 90 Degraded for 0s; "Backup" reports 250) .436 → Reconnected … "Backup" .475 Good 0.1 s after landing on the Standby; no disconnect. The case only #138 handles

MxGateway was unaffected throughout: Connected, 2196/2196 Good. The artifact redeploys were hash-guarded no-ops for it.

Not shown live:

  • The 60 s dwell. An in-place reconfigure resets it by design, and a role flip on a healthy session needs the Primary to leave, which drops the session. No flapping was seen.
  • The "poll active" log on a Healthy endpoint: Healthy samples log nothing at Information. The poll is evidenced by the 0/50/90 Warnings, each logged within 15–35 ms.

Follow-up #141. A graceful stop of the Warm Primary shows ServiceLevel 0 for only about 6 s, which is shorter than the 10 s window. The streak is then cleared as unknown when the server closes, so that case still costs the 66 s connectivity retry cycle. This is not a regression; #141 proposes probing the other endpoint at once on a disconnect that follows a degraded sample.

https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

**Gated live on QA, 2026-11-13, against OtOpcUa running `Mode=Warm` (lmxopcua#907). Closing.** QA runs `8e32f353`, which contains `712c5117`. New test connection `WarmCanary`: - id 2, primary OtOpcUa vq03, backup vq04; - `serviceLevelFailover` left at its defaults: enabled, 10 s, 60 s; - template 27 / instance 16 bound to OtOpcUa's 1 s script-runner canaries. Records: ScadaBridge `docs/deployment/deployment-records/2026-11-13-qa-warmcanary-opcua-connection.md` (`933da568`) and lmxopcua `docs/deploy/2026-11-13-qa-warm-717.md` (`1862fab6`). | Case | DCL log (site vq03) | Result | |---|---|---| | Healthy Primary at 250, Mode Unset | none, by design | Good at 1/s; no streak | | Unset: stop the Primary | `AdapterDisconnected` → 3 connect retries → `Failing over from "Primary" to "Backup"` | Good again in **67.1 s** (connectivity path, baseline) | | Pair restart into Warm; the connection reaches the **booting Standby** | `"Backup" endpoint reports ServiceLevel 50 (Degraded band, < 200) — will check the other endpoint now (connect-time preference)`, then `Failing over from "Backup" to "Primary" (ServiceLevel 50 Degraded for 4s; "Primary" reports 250)` | Good 8.6 s after the Standby connect | | Warm: stop the Primary | `"Primary" endpoint reports ServiceLevel 0 …` (+0.77 s), then `ServiceLevel is unknown — degraded watch cleared` (+6.8 s), then the connectivity failover | Good on the promoted node in **66.3 s**. See below | | Old Primary back as Standby (90) | none | No failback; stayed on 250 for 110 s with no Bad sample | | **On the Standby (90) over a healthy session**: an in-place reconfigure re-arms on the primary, which is now the Standby | `Reconnected successfully on "Primary"` 13:56:52.401 → `reports ServiceLevel 90 … connect-time preference` .413 → `Failing over from "Primary" to "Backup" (ServiceLevel 90 Degraded for 0s; "Backup" reports 250)` .436 → `Reconnected … "Backup"` .475 | **Good 0.1 s after landing on the Standby; no disconnect.** The case only #138 handles | `MxGateway` was unaffected throughout: Connected, 2196/2196 Good. The artifact redeploys were hash-guarded no-ops for it. **Not shown live:** - The 60 s dwell. An in-place reconfigure resets it by design, and a role flip on a healthy session needs the Primary to leave, which drops the session. No flapping was seen. - The "poll active" log on a Healthy endpoint: Healthy samples log nothing at Information. The poll is evidenced by the 0/50/90 Warnings, each logged within 15–35 ms. **Follow-up #141.** A graceful stop of the Warm Primary shows ServiceLevel 0 for only about 6 s, which is shorter than the 10 s window. The streak is then cleared as unknown when the server closes, so that case still costs the 66 s connectivity retry cycle. This is not a regression; #141 proposes probing the other endpoint at once on a disconnect that follows a degraded sample. https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Blocked by dependency or external hold
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label cross-repo priority:p2 type:feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#138
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?