ServiceLevel/election ignores OPC UA endpoint start failure (IOpcUaEndpointState unconsumed, lane C3 follow-up) #762

New Issue

2026-11-09T11:43:14-05:00

dohertj2 commented

2026-11-09 11:43:14 -05:00

Follow-up from review-remediation lane C (#733), item C3.

f5579e6e made an OPC UA SDK start failure visible: it added IOpcUaEndpointState, OpcUaEndpointHealthCheck and a StopAsync fix. Its message says the fact "is exposed for the Wave 3 follow-up to consume". That follow-up was never done. On master (dc0a24f2), IOpcUaEndpointState is referenced only under src/Server/ZB.MOM.WW.OtOpcUa.Host. No ServiceLevel or election code in Cluster or Runtime reads it.

Effect: a node whose OPC UA endpoint failed to start (for example, port 4840 taken) still advertises ServiceLevel 250 or 240 and can stay Primary. Clients and the peer believe it is serving when it is not.

Do: feed IOpcUaEndpointState into the ServiceLevel calculator, and step down from Primary while the endpoint is not listening. Add a test where a failed endpoint start lowers ServiceLevel and moves the role to the peer. The docker-dev gate for C (port 4840 taken → /health/ready 503) is tracked in #761.

Follow-up from review-remediation lane C (#733), item C3. `f5579e6e` made an OPC UA SDK start failure visible: it added `IOpcUaEndpointState`, `OpcUaEndpointHealthCheck` and a StopAsync fix. Its message says the fact "is exposed for the Wave 3 follow-up to consume". That follow-up was never done. On `master` (`dc0a24f2`), `IOpcUaEndpointState` is referenced only under `src/Server/ZB.MOM.WW.OtOpcUa.Host`. No ServiceLevel or election code in Cluster or Runtime reads it. **Effect:** a node whose OPC UA endpoint failed to start (for example, port 4840 taken) still advertises ServiceLevel 250 or 240 and can stay Primary. Clients and the peer believe it is serving when it is not. **Do:** feed `IOpcUaEndpointState` into the ServiceLevel calculator, and step down from Primary while the endpoint is not listening. Add a test where a failed endpoint start lowers ServiceLevel and moves the role to the peer. The docker-dev gate for C (port 4840 taken → `/health/ready` 503) is tracked in #761.

dohertj2 added the priority:p2 label 2026-11-09 11:43:14 -05:00

dohertj2 referenced this issue

2026-11-09 11:43:35 -05:00

Review remediation 2026-10-21: run the Wave-4 integration gates (lanes A–G, H1, drivers) #761

dohertj2 added the type:followup label 2026-11-09 11:44:13 -05:00

dohertj2 referenced this issue

2026-11-09 11:44:21 -05:00

Review remediation: Lane C — Control plane / host / security (fable for C1–C2, opus for C3–C5; review fable) #733

dohertj2 commented

2026-11-09 12:16:39 -05:00

Fixed on master in acfb4fba.

IOpcUaEndpointState moved from Host to Commons/OpcUa so Runtime can consume it.
ServiceLevelCalculator gains EndpointListening (default true): a Failed endpoint forces ServiceLevel 0, overriding every other input incl. the +10 Primary bonus. NotStarted/Stopped (boot/shutdown windows) do not lower it. OpcUaPublishActor feeds it on all three ServiceLevel paths (DB-backed, DB-less, role-only fallback).
New OpcUaEndpointStepDownActor: every driver node publishes its endpoint state on DistributedPubSub topic opcua-endpoint-state every 5 s. A node steps down only when (1) its endpoint is Failed, (2) it is the driver Primary per ClusterActiveNode.SelfIsActive (the election's own primitive), and (3) an Up driver peer reported a listening endpoint within 20 s. Step-down = graceful Cluster.Leave (same path as manual failover): singletons hand over, the peer becomes the oldest Up member → Primary/250, the node exits and is restarted by its supervisor as a Secondary at ServiceLevel 0. The election itself is not bent (Primary must stay with the singletons). Guard (3) prevents ping-pong when both endpoints failed and self-shutdown with no peer; a peer on an older build never reports, so a failed Primary holds during rolling upgrades.
Docs: docs/Redundancy.md (new section + tier-table row), .claude/rules/cluster-topology.md.
15 tests incl. a two-node TestKit cluster test (failed Primary holds while the peer is not listening, leaves once it is, peer becomes Primary; failed Secondary / healthy Primary never leave). Mutation-checked: removing the 0 override, the leave, the peer guard or the Primary check each breaks the matching tests.
Results: Cluster.Tests 255/0, Commons.Tests 363/0, Runtime.Tests 933 passed/3 skipped, Host.Tests 219/0; solution builds clean.

Not verified here: Host.IntegrationTests, docker-dev, and the real process restart after leave. The live gate (port 4840 taken → /health/ready 503 → Primary moves to the peer) is part of #761.

Fixed on master in acfb4fba. - `IOpcUaEndpointState` moved from Host to `Commons/OpcUa` so Runtime can consume it. - `ServiceLevelCalculator` gains `EndpointListening` (default true): a **Failed** endpoint forces ServiceLevel 0, overriding every other input incl. the +10 Primary bonus. NotStarted/Stopped (boot/shutdown windows) do not lower it. `OpcUaPublishActor` feeds it on all three ServiceLevel paths (DB-backed, DB-less, role-only fallback). - New `OpcUaEndpointStepDownActor`: every driver node publishes its endpoint state on DistributedPubSub topic `opcua-endpoint-state` every 5 s. A node steps down only when (1) its endpoint is Failed, (2) it is the driver Primary per `ClusterActiveNode.SelfIsActive` (the election's own primitive), and (3) an Up driver peer reported a listening endpoint within 20 s. Step-down = graceful `Cluster.Leave` (same path as manual failover): singletons hand over, the peer becomes the oldest Up member → Primary/250, the node exits and is restarted by its supervisor as a Secondary at ServiceLevel 0. The election itself is not bent (Primary must stay with the singletons). Guard (3) prevents ping-pong when both endpoints failed and self-shutdown with no peer; a peer on an older build never reports, so a failed Primary holds during rolling upgrades. - Docs: `docs/Redundancy.md` (new section + tier-table row), `.claude/rules/cluster-topology.md`. - 15 tests incl. a two-node TestKit cluster test (failed Primary holds while the peer is not listening, leaves once it is, peer becomes Primary; failed Secondary / healthy Primary never leave). Mutation-checked: removing the 0 override, the leave, the peer guard or the Primary check each breaks the matching tests. - Results: Cluster.Tests 255/0, Commons.Tests 363/0, Runtime.Tests 933 passed/3 skipped, Host.Tests 219/0; solution builds clean. Not verified here: Host.IntegrationTests, docker-dev, and the real process restart after leave. The live gate (port 4840 taken → `/health/ready` 503 → Primary moves to the peer) is part of #761.

dohertj2 closed this issue

2026-11-09 12:16:39 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#762