SDK 2.0.1: Server.ServiceLevel reads 255 on both pair nodes — SDK session-headroom calculator overwrites 250/240/200 (publisher never claims IServerServiceLevelControl) #884

New Issue

2026-11-12T10:23:11-05:00

dohertj2 commented

2026-11-12 10:23:11 -05:00

Found while running the #801 pair-behaviour gate on the local docker-dev rig (master 438484fd, SDK 2.0.1).

Symptom

On the docker-dev central pair, Server.ServiceLevel (i=2267) reads 255 on both nodes, so a client can't tell the Primary from the Secondary. The nodes' own metric shows the right levels:

node	`otopcua_redundancy_service_level_change_total` (computed and published)	wire `i=2267`
central-2 (oldest Up `driver` member = Primary)	`240:1 250:1`	255
central-1 (Secondary)	`240:1`	255

Polling i=2267 on central-1 just after a restart shows the bootstrap 100 (14:43:40). The value then jumps to 255 with a fresh SourceTimestamp at the moment the next client session opens (14:43:48). It stays at 255 even though the node published 240.

Cause

SDK 2.0.1 adds a session-headroom ServiceLevel calculator. ServerInternalData.OnSessionCountChanged runs on every SessionCreated and SessionClosing, and UpdateServerServiceLevel overwrites ServerObject.ServiceLevel with ServerServiceLevelCalculator.CalculateTarget(...) whenever the current value is ≥ 200 and no explicit owner has claimed it. That target is 255 when the server is lightly loaded. A provider opts out by calling IServerServiceLevelControl.ClaimServiceLevelControl(), which returns the writer to use from then on. SdkServiceLevelPublisher never claims it. It writes ServerObject.ServiceLevel.Value inside UpdateServerDiagnostics, so:

Primary 250, Secondary 240 and the Stale 200 tier all read back as 255 after the next session opens or closes. A stale node therefore advertises full service.
100 and 0 (critically degraded, down, detached, endpoint failed) are below the SDK's 200 floor and survive.
ServiceLevelBindingReplayTests didn't catch it. It publishes 0/100 before the session opens and reads once, so no session-count change happens after a ≥200 publish.

Repro

In-process: Published_level_survives_later_sessions(250|240|200) (on branch fix/801-servicelevel-claim) publishes a level, opens and closes sessions, and reads i=2267. On master's SdkServiceLevelPublisher all three cases fail on the first session (should be 250/240/200, read 255).
Rig: run docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2, then otopcua-cli redundancy -u opc.tcp://localhost:4840 and the same on :4841. Both nodes show Service Level: 255, while curl central-N:9000/metrics | grep service_level_change shows 250 and 240.

Fix (branch, not merged)

fix/801-servicelevel-claim @ cc76249a (rebased on master 0f001bcc) is a small change. SdkServiceLevelPublisher claims IServerServiceLevelControl in its constructor and publishes through the returned writer. A server without the claim (the unit-test DispatchProxy doubles) keeps the old path. If the claim fails, the existing binding catch marks the endpoint failed instead of leaving a silent 255. Tests:

the new test passes with the fix and fails without it (mutation-checked);
OpcUaServer.Tests ServiceLevel 7/7, Runtime.Tests ServiceLevel 4/4 and Host.Tests OtOpcUaServerHostedService* 15/15 still pass.

On the docker-dev pair with the fix image, the wire reads 250 and 240 and follows failover. See the #801 results comment.

Impact

Every SDK 2.0.1 build. DEV vd03 runs 5fb4b4db (SDK 2.0.1) since 2026-11-12. It's a single node, so it advertises 255 instead of 250. That's harmless there, except that a Stale 200 would also read 255.
It blocks the QA SDK 2 rollout under #801. On the vq03/vq04 pair both nodes would advertise 255, so ServiceLevel-driven clients lose Primary preference. ScadaBridge isn't affected, because its failover is endpoint-level and doesn't read ServiceLevel. Other OPC UA clients that pick a server by ServiceLevel are affected.

Found while running the #801 pair-behaviour gate on the local docker-dev rig (master `438484fd`, SDK 2.0.1). ## Symptom On the docker-dev central pair, `Server.ServiceLevel` (`i=2267`) reads **255 on both nodes**, so a client can't tell the Primary from the Secondary. The nodes' own metric shows the right levels: | node | `otopcua_redundancy_service_level_change_total` (computed and published) | wire `i=2267` | |---|---|---| | central-2 (oldest Up `driver` member = Primary) | `240:1 250:1` | **255** | | central-1 (Secondary) | `240:1` | **255** | Polling `i=2267` on central-1 just after a restart shows the bootstrap `100` (14:43:40). The value then jumps to `255` with a fresh SourceTimestamp at the moment the next client session opens (14:43:48). It stays at 255 even though the node published 240. ## Cause SDK 2.0.1 adds a session-headroom ServiceLevel calculator. `ServerInternalData.OnSessionCountChanged` runs on every `SessionCreated` and `SessionClosing`, and `UpdateServerServiceLevel` overwrites `ServerObject.ServiceLevel` with `ServerServiceLevelCalculator.CalculateTarget(...)` whenever the current value is **≥ 200** and no explicit owner has claimed it. That target is 255 when the server is lightly loaded. A provider opts out by calling `IServerServiceLevelControl.ClaimServiceLevelControl()`, which returns the writer to use from then on. `SdkServiceLevelPublisher` never claims it. It writes `ServerObject.ServiceLevel.Value` inside `UpdateServerDiagnostics`, so: - Primary 250, Secondary 240 and the **Stale 200** tier all read back as 255 after the next session opens or closes. A stale node therefore advertises *full* service. - 100 and 0 (critically degraded, down, detached, endpoint failed) are below the SDK's 200 floor and survive. - `ServiceLevelBindingReplayTests` didn't catch it. It publishes 0/100 before the session opens and reads once, so no session-count change happens after a ≥200 publish. ## Repro 1. In-process: `Published_level_survives_later_sessions(250|240|200)` (on branch `fix/801-servicelevel-claim`) publishes a level, opens and closes sessions, and reads `i=2267`. On master's `SdkServiceLevelPublisher` all three cases fail on the first session (`should be 250/240/200`, read 255). 2. Rig: run `docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2`, then `otopcua-cli redundancy -u opc.tcp://localhost:4840` and the same on `:4841`. Both nodes show `Service Level: 255`, while `curl central-N:9000/metrics | grep service_level_change` shows 250 and 240. ## Fix (branch, not merged) `fix/801-servicelevel-claim` @ `cc76249a` (rebased on master `0f001bcc`) is a small change. `SdkServiceLevelPublisher` claims `IServerServiceLevelControl` in its constructor and publishes through the returned writer. A server without the claim (the unit-test DispatchProxy doubles) keeps the old path. If the claim fails, the existing binding catch marks the endpoint failed instead of leaving a silent 255. Tests: - the new test passes with the fix and fails without it (mutation-checked); - `OpcUaServer.Tests` ServiceLevel 7/7, `Runtime.Tests` ServiceLevel 4/4 and `Host.Tests` `OtOpcUaServerHostedService*` 15/15 still pass. On the docker-dev pair with the fix image, the wire reads 250 and 240 and follows failover. See the #801 results comment. ## Impact - Every SDK 2.0.1 build. **DEV vd03 runs `5fb4b4db` (SDK 2.0.1) since 2026-11-12.** It's a single node, so it advertises 255 instead of 250. That's harmless there, except that a Stale 200 would also read 255. - **It blocks the QA SDK 2 rollout under #801.** On the vq03/vq04 pair both nodes would advertise 255, so ServiceLevel-driven clients lose Primary preference. ScadaBridge isn't affected, because its failover is endpoint-level and doesn't read ServiceLevel. Other OPC UA clients that pick a server by ServiceLevel are affected.

dohertj2 added the type:bug priority:p1 labels 2026-11-12 10:23:11 -05:00

dohertj2 referenced this issue

2026-11-12 10:23:34 -05:00

Restarting pair node advertises ServiceLevel 255 for ~10 s before its first redundancy snapshot (outranks the real Primary) #885

dohertj2 referenced this issue

2026-11-12 10:45:14 -05:00

SDK2 live gate: AVEVA history, consumer compatibility and paired-cluster rollout #801

dohertj2 referenced this issue from a commit

2026-11-12 11:03:25 -05:00

test(redundancy): exercise SessionClosing in the #884 ServiceLevel claim test

dohertj2 commented

2026-11-12 11:04:02 -05:00

Merged to master: f66b9dae (the fix, rebased from cc76249a onto da948e5f) and 5e3fb422 (test hardening). Fast-forward.

Review of the claim

I checked each write path against the decompiled SDK 2.0.1 (ServerInternalData.ClaimServiceLevelControl / PublishServiceLevel / UpdateServerServiceLevel):

Role change, Stale 200, degraded 100, 0. In src/ the only writer is OpcUaPublishActor → DeferredServiceLevelPublisher → SdkServiceLevelPublisher. After the bind, every tier goes through the claimed writer. Nothing else writes Server.ServiceLevel.
Startup. The claim happens at bind (OtOpcUaServerHostedService.StartAsync), and the deferred publisher replays the latest request under its lock. There are two gaps before that. From SDK Running to the bind, the SDK default 255 still stands. If no redundancy snapshot has arrived yet, no request is held, so 255 stays after the bind too. Both are the #885 window, fixed separately.
Shutdown. DetachSinks unbinds, and nothing publishes on stop, because connection-refused is the failover signal. The claim lasts for the server's lifetime, so sessions closing while the node stops can no longer lift the last value to 255. Before the fix they could.
Thread safety. The SDK's writer and its headroom calculator take the same m_serviceLevelLock, and the calculator checks the owner flag under that lock. A session event therefore can't overwrite a claimed value, even when the two interleave. On our side, DeferredServiceLevelPublisher._gate serialises publish and bind. The write no longer goes through UpdateServerDiagnostics. That lock never protected ServiceLevel from the calculator, so nothing is lost.
Session churn. The claim covers both SessionCreated and SessionClosing, since they share one handler and one owner check.
One claim per server. A second ClaimServiceLevelControl on the same ServerInternalData throws. There's one construction per StartAsync, and each start builds a fresh OtOpcUaSdkServer, so this can't happen. A failed claim now fails the endpoint at bind. Before, a missing ServiceLevel node failed only on the first publish. That's stricter, and the right way round.
Minor. The claimed writer doesn't set StatusCode = Good the way the old path did. That makes no difference, because nothing ever sets it to anything else.

Test gap, fixed in 5e3fb422. The new test disposed its sessions. That can leave a session to time out instead of raising SessionClosing, so only the SessionCreated half was proven. The 200 case's second level was 190, which is below the SDK's 200 floor, so that step was vacuous. Sessions now CloseAsync explicitly. A second level (230) is published and read back on a surviving session after another session closes on its own.

Mutation checks

Claim disabled (if (false && …)): Published_level_survives_later_sessions fails in 3/3 cases on the first session read (255).
Claim disabled, with a probe variant that publishes after the third session opens and then only closes it: the close-only read fails in 3/3 cases. That confirms the SessionClosing path is exercised and the assertion can fail.

Tests (rebased tree)

OpcUaServer.IntegrationTests: full suite 309 passed / 35 skipped / 0 failed. ServiceLevelBindingReplayTests 5/5.
OpcUaServer.Tests ServiceLevel 7/7. Runtime.Tests ServiceLevel + OpcUaPublishActor* 108/108. Host.Tests OtOpcUaServerHostedService* + FailedServiceLevelBindingCleanup 18/18. Commons.Tests Deferred* 15/15. Cluster.Tests ServiceLevel/Redundancy 16/16. ControlPlane.Tests ServiceLevel/Redundancy 25/25.

Rig verification of the wire pair values follows on #801 once #885 has landed. DEV vd03 (5fb4b4db) still advertises 255 until it's redeployed.

**Merged to `master`: `f66b9dae` (the fix, rebased from `cc76249a` onto `da948e5f`) and `5e3fb422` (test hardening). Fast-forward.** ## Review of the claim I checked each write path against the decompiled SDK 2.0.1 (`ServerInternalData.ClaimServiceLevelControl` / `PublishServiceLevel` / `UpdateServerServiceLevel`): - **Role change, Stale 200, degraded 100, 0.** In `src/` the only writer is `OpcUaPublishActor` → `DeferredServiceLevelPublisher` → `SdkServiceLevelPublisher`. After the bind, every tier goes through the claimed writer. Nothing else writes `Server.ServiceLevel`. - **Startup.** The claim happens at bind (`OtOpcUaServerHostedService.StartAsync`), and the deferred publisher replays the latest request under its lock. There are two gaps before that. From SDK `Running` to the bind, the SDK default 255 still stands. If no redundancy snapshot has arrived yet, no request is held, so 255 stays after the bind too. Both are the **#885** window, fixed separately. - **Shutdown.** `DetachSinks` unbinds, and nothing publishes on stop, because connection-refused is the failover signal. The claim lasts for the server's lifetime, so sessions closing while the node stops can no longer lift the last value to 255. Before the fix they could. - **Thread safety.** The SDK's writer and its headroom calculator take the same `m_serviceLevelLock`, and the calculator checks the owner flag under that lock. A session event therefore can't overwrite a claimed value, even when the two interleave. On our side, `DeferredServiceLevelPublisher._gate` serialises publish and bind. The write no longer goes through `UpdateServerDiagnostics`. That lock never protected ServiceLevel from the calculator, so nothing is lost. - **Session churn.** The claim covers both `SessionCreated` and `SessionClosing`, since they share one handler and one owner check. - **One claim per server.** A second `ClaimServiceLevelControl` on the same `ServerInternalData` throws. There's one construction per `StartAsync`, and each start builds a fresh `OtOpcUaSdkServer`, so this can't happen. A failed claim now fails the endpoint at bind. Before, a missing ServiceLevel node failed only on the first publish. That's stricter, and the right way round. - **Minor.** The claimed writer doesn't set `StatusCode = Good` the way the old path did. That makes no difference, because nothing ever sets it to anything else. **Test gap, fixed in `5e3fb422`.** The new test *disposed* its sessions. That can leave a session to time out instead of raising `SessionClosing`, so only the `SessionCreated` half was proven. The 200 case's second level was 190, which is below the SDK's 200 floor, so that step was vacuous. Sessions now `CloseAsync` explicitly. A second level (230) is published and read back on a surviving session after another session closes **on its own**. ## Mutation checks - Claim disabled (`if (false && …)`): `Published_level_survives_later_sessions` fails in **3/3** cases on the first session read (255). - Claim disabled, with a probe variant that publishes after the third session opens and then only closes it: the close-only read fails in **3/3** cases. That confirms the `SessionClosing` path is exercised and the assertion can fail. ## Tests (rebased tree) - `OpcUaServer.IntegrationTests`: full suite **309 passed / 35 skipped / 0 failed**. `ServiceLevelBindingReplayTests` 5/5. - `OpcUaServer.Tests` ServiceLevel 7/7. `Runtime.Tests` ServiceLevel + `OpcUaPublishActor*` 108/108. `Host.Tests` `OtOpcUaServerHostedService*` + `FailedServiceLevelBindingCleanup` 18/18. `Commons.Tests` `Deferred*` 15/15. `Cluster.Tests` ServiceLevel/Redundancy 16/16. `ControlPlane.Tests` ServiceLevel/Redundancy 25/25. Rig verification of the wire pair values follows on #801 once #885 has landed. DEV vd03 (`5fb4b4db`) still advertises 255 until it's redeployed.

dohertj2 closed this issue

2026-11-12 11:04:10 -05:00

dohertj2 referenced this issue from a commit

2026-11-12 11:11:15 -05:00

fix(redundancy): advertise ServiceLevel 0 until the node has a redundancy level (#885)

dohertj2 referenced this issue

2026-11-12 11:20:44 -05:00

Restarting pair node advertises ServiceLevel 255 for ~10 s before its first redundancy snapshot (outranks the real Primary) #885

dohertj2 referenced this issue

2026-11-12 11:20:59 -05:00

SDK2 live gate: AVEVA history, consumer compatibility and paired-cluster rollout #801

Sign in to join this conversation.