fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)

IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs

@@ -2,12 +2,24 @@ using Akka.Cluster;
 
 namespace ZB.MOM.WW.OtOpcUa.Cluster.Redundancy;
 
+/// <summary>The health facts <see cref="ServiceLevelCalculator.Compute"/> tiers a node on.</summary>
+/// <param name="MemberState">This node's Akka cluster member status.</param>
+/// <param name="DbReachable">Whether the config DB answered (constant <c>true</c> on a DB-less driver node).</param>
+/// <param name="OpcUaProbeOk">A peer's recent TCP probe verdict about this node's OPC UA port (benefit of the
+/// doubt when there is none). Observes the PORT only: when another process holds it, the probe succeeds against
+/// that process — which is why <paramref name="EndpointListening"/> exists.</param>
+/// <param name="Stale">Whether this node is running behind on config.</param>
+/// <param name="IsDriverPrimary">Whether this node is the driver Primary (oldest Up driver member).</param>
+/// <param name="EndpointListening">This node's OWN view of its OPC UA endpoint: <c>false</c> when the SDK failed
+/// to start (<c>IOpcUaEndpointState</c> reports <c>Failed</c>). Defaults to <c>true</c> so a caller with no
+/// endpoint fact (a harness, an admin-only node) keeps the previous tiering. lmxopcua#762.</param>
 public readonly record struct NodeHealthInputs(
     MemberStatus MemberState,
     bool DbReachable,
     bool OpcUaProbeOk,
     bool Stale,
-    bool IsDriverPrimary);
+    bool IsDriverPrimary,
+    bool EndpointListening = true);
 
 /// <summary>
 /// Pure ServiceLevel computation per design §6. Output range 0–255, where higher = "more
@@ -16,6 +28,8 @@ public readonly record struct NodeHealthInputs(
 ///
 /// Tiering:
 /// - Member not Up/Joining: 0 (cluster cannot trust this node).
+/// - OPC UA endpoint not listening (the SDK failed to start on this node): 0, with no Primary bonus — the node
+///   cannot serve a single client, so it must never out-rank a peer that can (lmxopcua#762).
 /// - DB reachable + OPC UA probe ok + not stale: 240 (full service).
 /// - Stale config (DB reachable or not, OPC UA probe state ignored): 100 or 200 depending on DB.
 /// - +10 bonus when this node is the driver Primary (the oldest Up member with the "driver" role —
@@ -32,6 +46,12 @@ public static class ServiceLevelCalculator
         if (h.MemberState is not (MemberStatus.Up or MemberStatus.Joining))
             return 0;
 
+        // lmxopcua#762: a node whose own SDK failed to start serves nothing, whatever its DB, probe, staleness or
+        // Primary status say. Checked before the tiering so no input can lift it — in particular not the +10
+        // Primary bonus, which is exactly what used to keep a dead endpoint advertising 250.
+        if (!h.EndpointListening)
+            return 0;
+
         var basis = (h.DbReachable, h.OpcUaProbeOk, h.Stale) switch
         {
             (true,  true,  false) => 240,

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaEndpointState.cs

@@ -0,0 +1,82 @@
+namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
+
+/// <summary>Lifecycle of this node's OPC UA endpoint as the Host's <c>OtOpcUaServerHostedService</c> drives it.</summary>
+public enum OpcUaEndpointStatus
+{
+    /// <summary>The hosted service has not run <c>StartAsync</c> yet.</summary>
+    NotStarted,
+
+    /// <summary>The SDK is listening and the address-space / ServiceLevel sinks are bound.</summary>
+    Started,
+
+    /// <summary>The SDK failed to start (port in use, PKI unwritable, bad <c>OpcUa:*</c> override, …).</summary>
+    Failed,
+
+    /// <summary>The hosted service stopped the SDK; the port is released.</summary>
+    Stopped,
+}
+
+/// <summary>
+///     The one place the "did the OPC UA SDK actually start?" fact lives (R03:2.1). Before it, a failed
+///     start was one Error line: <c>/health/ready</c> stayed 200, the node joined its mesh, could be
+///     elected driver Primary and acked deploys as Applied through the Null sink — while clients got
+///     connection-refused from the node the fleet called healthy.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Lives in Commons (not the Host, where it started) so the Runtime can read it. Three consumers:
+///     </para>
+///     <list type="bullet">
+///         <item><c>OpcUaEndpointHealthCheck</c> (Host) — Ready tier everywhere, Active tier on a driver-only node.</item>
+///         <item><c>OpcUaPublishActor</c> (Runtime) — a <see cref="OpcUaEndpointStatus.Failed"/> endpoint feeds
+///         <c>EndpointListening=false</c> into <c>ServiceLevelCalculator</c>, so the node computes ServiceLevel 0
+///         instead of 250/240 (lmxopcua#762).</item>
+///         <item><c>OpcUaEndpointStepDownActor</c> (Runtime) — a driver Primary whose endpoint failed gracefully
+///         leaves its mesh once a peer reports a listening endpoint, so the peer becomes the oldest Up driver
+///         member and therefore Primary (lmxopcua#762).</item>
+///     </list>
+/// </remarks>
+public interface IOpcUaEndpointState
+{
+    /// <summary>The endpoint's current lifecycle status.</summary>
+    OpcUaEndpointStatus Status { get; }
+
+    /// <summary>Why the start failed, when <see cref="Status"/> is <see cref="OpcUaEndpointStatus.Failed"/>.</summary>
+    string? FailureReason { get; }
+}
+
+/// <summary>Mutable <see cref="IOpcUaEndpointState"/>; written by the hosted service, read by the health check, the
+/// ServiceLevel computation and the Primary step-down.</summary>
+public sealed class OpcUaEndpointState : IOpcUaEndpointState
+{
+    private volatile OpcUaEndpointStatus _status = OpcUaEndpointStatus.NotStarted;
+    private volatile string? _failureReason;
+
+    /// <inheritdoc />
+    public OpcUaEndpointStatus Status => _status;
+
+    /// <inheritdoc />
+    public string? FailureReason => _failureReason;
+
+    /// <summary>Records a successful SDK start.</summary>
+    public void MarkStarted()
+    {
+        _failureReason = null;
+        _status = OpcUaEndpointStatus.Started;
+    }
+
+    /// <summary>Records a failed SDK start.</summary>
+    /// <param name="reason">The failure, as the health probe should report it.</param>
+    public void MarkFailed(string reason)
+    {
+        _failureReason = reason;
+        _status = OpcUaEndpointStatus.Failed;
+    }
+
+    /// <summary>Records that the hosted service stopped the SDK.</summary>
+    public void MarkStopped()
+    {
+        _failureReason = null;
+        _status = OpcUaEndpointStatus.Stopped;
+    }
+}