diff --git a/docs/Redundancy.md b/docs/Redundancy.md index 69ccf39f..06f0f2f4 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -15,75 +15,97 @@ The runtime pieces live in: | `RedundancyStateActor` | `OtOpcUa.ControlPlane.Redundancy` | Admin-role cluster singleton; subscribes to cluster topology events, debounces 250ms, broadcasts `RedundancyStateChanged` on the `redundancy-state` DPS topic. | | `OpcUaPublishActor` | `OtOpcUa.Runtime.OpcUa` | Per-driver-node; subscribes to the `redundancy-state` topic, maps the local node's role to a ServiceLevel byte (see below), and forwards it to `IServiceLevelPublisher`. | | `IServiceLevelPublisher` / `SdkServiceLevelPublisher` | `OtOpcUa.Commons.OpcUa` / `OtOpcUa.OpcUaServer` | Writes the byte into the SDK's `Server.ServiceLevel` Variable. Production binds `DeferredServiceLevelPublisher`, which swaps in the real `SdkServiceLevelPublisher` once the SDK is up (it needs `IServerInternal`, available only after `StandardServer.Start`); until then writes route through `NullServiceLevelPublisher`. | -| `ServiceLevelCalculator` | `OtOpcUa.ControlPlane.Redundancy` | Pure function `(NodeHealthInputs) → byte` — the fuller DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`; **not yet wired into the live driver publish path**, which uses the coarse role mapping in `OpcUaPublishActor`. | +| `ServiceLevelCalculator` | `OtOpcUa.Cluster.Redundancy` (`Core.Cluster`) | Pure function `(NodeHealthInputs) → byte` — the DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`. **Now the live publish path** — `OpcUaPublishActor` calls it on every `HealthTick` and `RedundancyStateChanged` event. Moved to `Core.Cluster` so Runtime can reach it without a Runtime→ControlPlane reference. | | `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by health endpoint. | | `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; pings peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes the result on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. | | `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. | ## ServiceLevel tiers -### Live driver-side mapping (current) +### Health-aware tiering (`ServiceLevelCalculator` — live path) -`OpcUaPublishActor.HandleRedundancyStateChanged` maps the local node's role -(from the `RedundancyStateChanged` snapshot) to a ServiceLevel byte and forwards -it through `IServiceLevelPublisher` to the SDK's `Server.ServiceLevel` Variable: +`ServiceLevelCalculator.Compute(NodeHealthInputs)` is the live publish path. +`OpcUaPublishActor` calls it on every `HealthTick` (~5 s) and on each +`RedundancyStateChanged` snapshot, then forwards the result through +`IServiceLevelPublisher` to the SDK's `Server.ServiceLevel` Variable. -| Local role | Byte | +The four inputs are sourced locally per driver node: + +| Input | Source | |---|---| -| `Primary` and `driver` role-leader | 240 | -| `Primary` (not role-leader) | 200 | -| `Secondary` | 100 | -| `Detached` (no `driver` role) | 0 | +| `MemberState` | Local `SelfMember.Status` from the Akka cluster (Up / Joining / Leaving / …). | +| `DbReachable` | Local `DbHealthProbeActor` — `OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. | +| `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. | +| `Stale` (derived) | `!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`. | +| `IsDriverRoleLeader` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. | + +The resulting truth table (all tiers are now reachable at runtime): + +| Tier | Byte | Condition | +|---|---|---| +| Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. | +| Critically degraded | 100 | ConfigDb unreachable AND data is stale. | +| Stale | 200 | Data stale but ConfigDb reachable. | +| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not role-leader. | +| Healthy leader | 250 | Same as healthy follower + this node is the `driver` role-leader (+10 bonus). | + +> **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary +> published 100 (coarse role-only mapping). It now publishes **240** — both +> nodes sit at 240/250 under healthy conditions, with the leader still preferred +> by the +10 bonus. Clients with the standard "pick highest ServiceLevel" +> heuristic continue to prefer the primary. + +#### Backward-compatible fallback (legacy seam) + +A node with no `DbHealthStatus` wired (e.g. early bootstrap window before the +first `DbHealthProbeActor` reply) falls back to the old role-only mapping: +Primary-leader → 240, Primary → 200, Secondary → 100, Detached → 0. Once the +first `DbHealthStatus` arrives the calculator takes over. The first computed +ServiceLevel (even 0) is always published so no node lingers at the SDK default +255. Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver` role is `Primary` when it holds the `driver` role-leader lease, otherwise `Secondary`; a node without the `driver` role is `Detached`. -### Full health-aware tiering (`ServiceLevelCalculator`) - -`ServiceLevelCalculator.Compute(NodeHealthInputs)` is the fuller, DB/probe-aware -calculation. It is unit-tested but **not yet on the live publish path** — the -driver-side mapping above is what actually drives the SDK today. - -| Tier | Byte | Condition | -|---|---|---| -| Down | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting). | -| Critically degraded | 100 | ConfigDb unreachable AND data is stale. | -| Stale | 200 | Data stale but ConfigDb reachable. | -| Healthy follower | 240 | DB ok + OPC UA probe ok + not stale. | -| Healthy leader | 250 | Healthy follower (240) + a `+10` bonus when this node is the `driver` role-leader. | - -Either way, clients with the standard redundancy heuristic ("pick the highest -ServiceLevel") prefer the `driver` role-leader and fall back to followers on its -degradation. - ## Data flow ``` -Cluster topology event ──┐ - ▼ - RedundancyStateActor (admin singleton) - │ debounce 250ms - ▼ - DPS topic "redundancy-state" - │ - ▼ - Driver nodes' OpcUaPublishActor - │ role → byte (240/200/100/0) - ▼ +Cluster topology event ──────────────────────────────────────────┐ + ▼ + RedundancyStateActor (admin singleton) + │ debounce 250ms + ▼ + DPS topic "redundancy-state" + │ ▲ + ┌───────────────────────┘ │ + │ │ + ▼ │ + Driver node: OpcUaPublishActor │ + ┌─────────────────────────────────────────────────────────┐ │ + │ Inputs collected per ~5s HealthTick: │ │ + │ • MemberState ← Akka SelfMember.Status │ │ + │ • DbReachable ← DbHealthProbeActor (Ask, timeout→F) │ │ + │ • OpcUaProbeOk ← OpcUaProbeResult about THIS node │──────┘ + │ • Stale ← derived from above timestamps │ PeerProbeSupervisor + │ • IsLeader ← RedundancyStateChanged snapshot │ → PeerOpcUaProbeActor(s) + │ │ publish OpcUaProbeResult + │ ServiceLevelCalculator.Compute(NodeHealthInputs) │ on "redundancy-state" + │ → byte (0/100/200/240/250) │ + └───────────────────────────────────────────────────────-─┘ + │ + ▼ IServiceLevelPublisher (SdkServiceLevelPublisher) - │ - ▼ + │ + ▼ OPC UA Server.ServiceLevel Variable ``` -Today only cluster topology drives the published ServiceLevel. -`PeerOpcUaProbeActor` and `DbHealthProbeActor` also run per-node — the peer probe -publishes `OpcUaProbeResult` onto the `redundancy-state` topic and the DB probe -backs the health endpoint — but their outputs are not yet consumed by -`RedundancyStateActor` or folded into the published byte. They are the inputs the -fuller `ServiceLevelCalculator` truth table is designed to use once that path goes -live. +Both `DbHealthProbeActor` and `PeerOpcUaProbeActor` feed the live publish path. +The peer probe publishes `OpcUaProbeResult` on the `redundancy-state` topic; +`OpcUaPublishActor` consumes only results whose target is itself and applies +freshness-debouncing before passing them to the calculator. `DbHealthProbeActor` +is queried directly via Ask on each `HealthTick`. The admin singleton is the cluster's only `RedundancyStateActor`. If the admin leader fails over, the new admin node spins up its replacement, re-subscribes to cluster events, and publishes a fresh snapshot from the current `Cluster.State`. There is no DB-persisted state to recover. @@ -113,7 +135,7 @@ There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` c > **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers). -The `ClusterNode.ServiceLevelBase` column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed from cluster role/health and published per the mapping above, independent of this stored preference. +The `ClusterNode.ServiceLevelBase` column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed by `ServiceLevelCalculator` from cluster role and live health inputs, independent of this stored preference. ### Peer URI advertising