lmxopcua/docs/plans/2026-06-15-stillpending-phase-2-servicelevel-design.md

Phase 2 — Health-aware redundancy ServiceLevel (H3) — design

Status: approved 2026-06-15. Parent roadmap: docs/plans/2026-06-15-stillpending-backlog-design.md (Phase 2). Backlog item H3 in stillpending.md §1. Branch: feat/stillpending-phase-2-servicelevel off master 4bd7180e. Classification: high-risk (actor model + redundancy correctness). Live /run on the 2-node rig is the acceptance gate; unit tests cannot prove the cross-node wiring (per project_redundancy_state_delivery).

Problem (H3)

The SDK's Server.ServiceLevel byte is driven solely by the coarse role map today (OpcUaPublishActor.HandleRedundancyStateChanged: Primary-leader→240, Primary→200, Secondary→100, Detached→0). The richer ServiceLevelCalculator.Compute(NodeHealthInputs) — which folds in DB reachability, an OPC-UA liveness probe, and a staleness signal — is never invoked in production. The two health producers are dead or half-wired:

• DbHealthProbeActor — spawned per node, but only feeds /health/ready; its DbReachable never reaches the published byte.
• PeerOpcUaProbeActor — never spawned anywhere; its OpcUaProbeResult has zero consumers.
• NodeHealthInputs.Stale — no producer exists anywhere in the codebase.

Consequence: a DB-unreachable / OPC-UA-down primary still advertises its full role-based ServiceLevel, so redundant clients keep subscribing to a degraded node instead of failing over.

Goal

Make each driver node publish a ServiceLevel computed by ServiceLevelCalculator.Compute from its real local health, so an unhealthy node drops below its role-based level (and below a healthy peer's level, triggering client failover). Honor the already-documented tiers in docs/Redundancy.md (250 / 240 / 200 / 100 / 0) — do not reshape the calculator's truth table. No EF migration; no change to the RedundancyStateChanged / NodeRedundancyState message contract.

Locked decisions (from brainstorming 2026-06-15)

1. Compute site = local per-node (OpcUaPublishActor), not the admin RedundancyStateActor singleton. DB reachability is an inherently local fact; centralizing it would require a new cluster-wide health-broadcast contract for no benefit. The publish actor already runs per node, already subscribes to the redundancy-state topic, already holds the role snapshot, and already writes the byte.
2. Stale is derived from signal freshness, with DB-unreachable ⟹ stale. This makes the documented 200/100 graceful-degradation tiers reachable, so a node that loses its DB settles into 100 ("degraded, still serving cached values") rather than slamming to 0. The 0 tier stays reserved for "not a healthy cluster member / detached." A richer driver-input staleness signal is deferred (flagged, not built).
3. OpcUaProbeOk = peer-probes-me, reusing the existing PeerOpcUaProbeActor (finally spawned), closing the audit's "never spawned / zero consumers" gaps. Absence of a fresh peer result → true (benefit of the doubt: don't penalize a node for its peer being down); only an actively observed recent failure demotes. Single-node / no-peer cluster → true.

Architecture

All changes are per-driver-node, inside Runtime (WithOtOpcUaRuntimeActors). No ControlPlane / admin-singleton change. No new Commons message types.

WithOtOpcUaRuntimeActors (per driver node)
  ├─ DbHealthProbeActor            (exists; now ALSO consulted by the publish actor)
  ├─ PeerProbeSupervisor   (NEW)   watches cluster membership →
  │     └─ PeerOpcUaProbeActor(peer)  one child per OTHER driver member;
  │            publishes OpcUaProbeResult(peer, ok) on the redundancy-state topic
  └─ OpcUaPublishActor      (MODIFIED)
        subscribes redundancy-state topic  → RedundancyStateChanged  (role/leadership)
                                           → OpcUaProbeResult         (peer's verdict on ME)
        holds IActorRef dbHealthProbe      → Ask<DbHealthStatus> on a periodic HealthTick
        Cluster.SelfMember.Status          → MemberState (local)
        on any trigger:  ServiceLevelCalculator.Compute(inputs) → publish-if-changed (existing dedup)

Components

1. PeerProbeSupervisor (NEW, Runtime/Health/PeerProbeSupervisor.cs) — a small per-node actor whose only job is the probe lifecycle (kept separate so the pinned-dispatcher OpcUaPublishActor never parents network-probe children):

• Subscribes to cluster membership (IMemberEvent / ReachabilityEvent) and/or the redundancy snapshot.
• Maintains one PeerOpcUaProbeActor child per other driver-role member, (re)spawning on membership change, stopping children for departed members. Resolves the peer's OPC UA host from its node id (host:port → host) and the configured OPC UA port.
• Children publish OpcUaProbeResult to the redundancy-state topic (existing behavior).
• Spawned on the default dispatcher (not the OPC UA pinned one).

2. OpcUaPublishActor (MODIFIED) — replace the role-only switch in HandleRedundancyStateChanged with a RecomputeServiceLevel() that calls the calculator. New Props params (both optional, defaulted so test Props/harnesses keep working): IActorRef? dbHealthProbe, and the freshness windows / self-OPC-UA-port (injectable for tests).

• New state: _dbReachable + _dbAsOfUtc (from the DB Ask), _lastSnapshotAsOfUtc, a per-peer map of (ok, asOfUtc) for results about my own node, and the current RedundancyStateChanged local entry.
• New Receive<OpcUaProbeResult>: if msg.NodeId == _localNode, record (msg.Ok, now) for the reporting peer; recompute.
• New Receive<HealthTick> (periodic, e.g. 5 s): dbHealthProbe.Ask<DbHealthStatus>(GetStatus, 1s).PipeTo(Self).
• New Receive<DbHealthStatus>: cache _dbReachable + _dbAsOfUtc; recompute.
• RecomputeServiceLevel() builds NodeHealthInputs and calls ServiceLevelCalculator.Compute, then Self.Tell(new ServiceLevelChanged(level)) (the existing handler dedups + publishes + emits the metric).

Input derivation (the load-bearing logic)

For the local node, on each recompute:

NodeHealthInputs field	Source
MemberState	Cluster.Get(system).SelfMember.Status (local; most accurate)
IsDriverRoleLeader	local entry of the latest RedundancyStateChanged snapshot (IsRoleLeaderForDriver)
DbReachable	latest DbHealthStatus.Reachable from the local DbHealthProbeActor
OpcUaProbeOk	true if no fresh peer result about me, else the latest such result's Ok (only an actively-observed recent failure → false); single-node → true
Stale	!DbReachable OR (now − _dbAsOfUtc) > StaleWindow OR (now − _lastSnapshotAsOfUtc) > StaleWindow

Guards (calculator does not model these):

• If there is no local snapshot entry, or the local entry's Role == Detached, publish 0 directly and skip the calculator (a healthy detached node would otherwise compute 240 because the calculator only checks MemberState + the leader bonus). In steady state a node running OpcUaPublishActor always carries the driver role, so this is defensive — it preserves the current Detached→0 behavior during transitions.

Resulting ServiceLevel truth table (now fully reachable)

Node condition	Byte	How reached
Not a healthy member / detached	0	MemberState not Up/Joining, or Detached guard
DB unreachable, sustained	100	DbReachable=false ⇒ Stale=true → (false,_,true)
DB reachable but signals stale	200	(true,_,true)
Healthy follower (DB ok + probe ok + fresh)	240	(true,true,false)
Healthy leader	250	follower 240 + IsDriverRoleLeader +10
DB ok + fresh but peer reports me unreachable	0	(true,false,false) → falls through

Behavior change to be aware of (already documented as the target): a healthy Secondary moves 100 → 240. Both healthy nodes sit at 240/250 with the leader preferred by the +10 bonus; role-leadership is stable so the bytes do not flap. The "DB ok + probe-fail" → 0 edge is intentionally sharp but debounced by the OpcUaProbeOk freshness rule (a single missed probe does not flip it; only a sustained, actively observed failure does).

Error handling / edge cases

• DB Ask timeout → treat as DbReachable=false for that cycle (fail-safe demote); the next tick re-Asks.
• No peer (single node) → OpcUaProbeOk=true; supervisor spawns no children.
• Peer departs → its prior result ages out of the freshness window → OpcUaProbeOk reverts to true (we don't penalize a node for its peer's absence). Supervisor stops the child for the departed member.
• Cluster forming / role-leader not yet resolved → no local snapshot entry yet → Detached guard → 0, same as today, until the first snapshot arrives.
• HealthTick before the first snapshot → no local entry → 0 (no spurious 240 before role is known).

Testing strategy (xUnit + Shouldly, TDD; NO bUnit)

Unit-testable (the wiring that can be proven in a TestKit):

• OpcUaPublishActor input-derivation: feed a RedundancyStateChanged (Primary-leader) + a cached DbHealthStatus(reachable) + an OpcUaProbeResult(me, ok) → assert published byte 250; flip DB unreachable → 100; stale snapshot → 200; healthy secondary → 240; Detached entry → 0; probe Ok=false about me with DB-ok+fresh → 0. Use PropsForTests with a broadcast/serviceLevel capture + injected dbHealthProbe test ref + short windows.
• OpcUaProbeOk freshness/debounce: a single stale/absent result → true; a recent explicit false → false.
• PeerProbeSupervisor: on a membership snapshot with one peer driver member, spawns exactly one child targeting that peer; on the peer leaving, stops it; single-node → no children. (TestKit, with a probe factory injected so no real TCP is attempted.)
• Reuse existing ServiceLevelCalculatorTests unchanged (the pure function is not modified).

Not unit-testable (acceptance gate): the live cross-node ServiceLevel on the 2-node rig — proven by /run (user-driven). A DB-unreachable primary must be observed dropping below the secondary, and the secondary taking over as the higher-ServiceLevel endpoint. See project_redundancy_state_delivery.

Verification

• dotnet build clean (production projects are TreatWarningsAsErrors); full dotnet test green.
• High-risk review chain (serial spec → code → final integration).
• Live /run on the 2-node / docker-dev rig (user-driven; agent does not sign in): confirm steady-state 250/240, then kill a node's DB reachability and confirm its ServiceLevel drops to 100 and the peer is preferred. Update docs/Redundancy.md to mark the calculator path wired (remove the "not yet wired into the live driver publish path" caveats) and document the freshness/peer-probe sourcing.

Alternatives considered

• Central compute in RedundancyStateActor — rejected: the admin singleton can't see each node's local DB health without a new cluster-wide health-broadcast contract; heavier, bigger blast radius, no benefit.
• Local self-probe (TCP to own localhost:4840 / SDK server-state) instead of the peer-probe — simpler (no membership-tracking supervisor) and a defensible "is my OPC UA serving" signal, but leaves PeerOpcUaProbeActor dead and misses the cross-node vantage. Rejected to honor the locked design and close the audit's dead-code gap; recorded here so the trade-off is explicit if the supervisor proves heavier than valued (then it becomes a follow-up swap, not a redesign).
• Reshape the calculator to role-baseline-demoted-by-health (keep the wide 240-vs-100 gap; health only lowers) — rejected: the calculator's 250/240/200/100/0 tiers are already approved, documented, and tested; reshaping re-opens a settled decision. The +10 leader bonus already preserves client preference for the primary.

Hard constraints (carried from the roadmap)

NO Configuration entity / EF migration. Stage by path — never git add .; never stage sql_login.txt, src/Server/.../Host/pki/, pending.md, current.md, docker-dev/docker-compose.yml, stillpending.md. Never echo or commit secrets. No force-push, no --no-verify. Razor/runtime cross-node behavior proven only by live /run, never bUnit.