fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)

IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.

docs/Redundancy.md

@@ -30,7 +30,7 @@ The runtime pieces live in:
 `RedundancyStateChanged` snapshot, then forwards the result through
 `IServiceLevelPublisher` to the SDK's `Server.ServiceLevel` Variable.
 
-The four inputs are sourced locally per driver node:
+The inputs are sourced locally per driver node:
 
 | Input | Source |
 |---|---|
@@ -39,12 +39,14 @@ The four inputs are sourced locally per driver node:
 | `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. |
 | `Stale` (derived) | `!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`. |
 | `IsDriverPrimary` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
+| `EndpointListening` | This node's OWN `IOpcUaEndpointState` (`Commons.OpcUa`), written by `OtOpcUaServerHostedService`: `false` only when the SDK start is recorded `Failed` (`NotStarted`/`Stopped` are boot/shutdown windows, not faults). Read on every recompute. A LOCAL fact on purpose — when another process holds port 4840, the peer TCP probe behind `OpcUaProbeOk` connects to *that* process and reports Ok. lmxopcua#762. |
 
 The resulting truth table (all tiers are now reachable at runtime):
 
 | Tier | Byte | Condition |
 |---|---|---|
 | Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. |
+| Endpoint failed | 0 | This node's OPC UA SDK failed to start (`EndpointListening=false`). Wins over every other input, including the +10 Primary bonus; the legacy role-only seam applies it too. See [Endpoint start failure: Primary step-down](#endpoint-start-failure-primary-step-down). |
 | Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
 | Stale | 200 | Data stale but ConfigDb reachable. |
 | Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not the driver Primary. |
@@ -384,6 +386,40 @@ Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.
 > testable — deferred to Phase 7's failover drill. See
 > `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
 
+## Endpoint start failure: Primary step-down
+
+lmxopcua#762 (follow-up F-C1 of review-remediation lane C3). A fused admin+driver node whose OPC UA SDK fails to
+start (port 4840 taken, PKI unwritable, a bad `OpcUa:*` override) keeps running so the operator keeps the AdminUI
+(a driver-only node instead rethrows and exits). Before #762 it also stayed **Primary** — the Primary-gated data
+plane stayed on a node no client could reach while the reachable peer refused writes as Secondary.
+
+Now two things happen:
+
+1. **ServiceLevel 0.** `OpcUaPublishActor` feeds `EndpointListening=false` into `ServiceLevelCalculator`
+   (above). Nothing can actually read it over OPC UA — the SDK is not running — but the metric and every
+   in-process consumer see the truth instead of 250/240.
+2. **The Primary moves.** `OpcUaEndpointStepDownActor` (`Runtime.Redundancy`, one per driver node, spawned when
+   the host registers `IOpcUaEndpointState`) makes a failed Primary **leave its mesh** — exactly the
+   [manual failover](#manual-failover) mechanism, triggered automatically. The election is deliberately *not*
+   bent to skip the node: the Primary must stay where `ClusterSingletonManager` places the singletons. The node
+   hands its singletons to the peer, exits, is restarted by its supervisor, and rejoins as the youngest member —
+   a Secondary at ServiceLevel 0, with its AdminUI back up. The peer advertises 250 and the `/health/active`
+   tier follows it.
+
+The step-down fires only when **all** hold, re-checked every 5 s:
+
+| Guard | Why |
+|---|---|
+| Own endpoint is `Failed` | `NotStarted`/`Stopped` are boot/shutdown windows. |
+| This node is the driver Primary — `ClusterActiveNode.SelfIsActive(cluster, "driver")` | The same primitive the election delegates to, so the two cannot disagree. A failed Secondary has nothing to hand over. |
+| Another Up driver member reported **its own** endpoint listening within the last 20 s | Every driver node publishes an `EndpointReport(node, listening)` on the `opcua-endpoint-state` DistributedPubSub topic each tick. Without this guard a pair whose endpoints **both** failed would bounce the Primary between them forever (each leave restarts a node), and a lone node would shut itself down with nowhere to fail over to. A peer on an older build never reports, so it is never handed the role — the safe direction during a rolling upgrade. |
+
+While a guard holds the role, the node logs one Warning naming the reason (repeated only when the reason changes).
+Pinned by `OpcUaEndpointStepDownActorTests` (real two-node mesh: holds while the peer is not listening, then
+leaves and the peer becomes the oldest Up driver; a failed Secondary and a healthy Primary never leave),
+`OpcUaPublishActorEndpointStateTests` and `ServiceLevelCalculatorTests`. The docker-dev gate (port 4840 taken
+⇒ `/health/ready` 503) is lmxopcua#761.
+
 ## Bootstrap: self-first seed ordering
 
 Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any