docs(redundancy): ServiceLevelCalculator is wired into the live publish path
This commit is contained in:
Joseph Doherty
+71
-49
@@ -15,75 +15,97 @@ The runtime pieces live in:
|
||||
| `RedundancyStateActor` | `OtOpcUa.ControlPlane.Redundancy` | Admin-role cluster singleton; subscribes to cluster topology events, debounces 250ms, broadcasts `RedundancyStateChanged` on the `redundancy-state` DPS topic. |
|
||||
| `OpcUaPublishActor` | `OtOpcUa.Runtime.OpcUa` | Per-driver-node; subscribes to the `redundancy-state` topic, maps the local node's role to a ServiceLevel byte (see below), and forwards it to `IServiceLevelPublisher`. |
|
||||
| `IServiceLevelPublisher` / `SdkServiceLevelPublisher` | `OtOpcUa.Commons.OpcUa` / `OtOpcUa.OpcUaServer` | Writes the byte into the SDK's `Server.ServiceLevel` Variable. Production binds `DeferredServiceLevelPublisher`, which swaps in the real `SdkServiceLevelPublisher` once the SDK is up (it needs `IServerInternal`, available only after `StandardServer.Start`); until then writes route through `NullServiceLevelPublisher`. |
|
||||
| `ServiceLevelCalculator` | `OtOpcUa.ControlPlane.Redundancy` | Pure function `(NodeHealthInputs) → byte` — the fuller DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`; **not yet wired into the live driver publish path**, which uses the coarse role mapping in `OpcUaPublishActor`. |
|
||||
| `ServiceLevelCalculator` | `OtOpcUa.Cluster.Redundancy` (`Core.Cluster`) | Pure function `(NodeHealthInputs) → byte` — the DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`. **Now the live publish path** — `OpcUaPublishActor` calls it on every `HealthTick` and `RedundancyStateChanged` event. Moved to `Core.Cluster` so Runtime can reach it without a Runtime→ControlPlane reference. |
|
||||
| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by health endpoint. |
|
||||
| `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; pings peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes the result on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. |
|
||||
| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. |
|
||||
|
||||
## ServiceLevel tiers
|
||||
|
||||
### Live driver-side mapping (current)
|
||||
### Health-aware tiering (`ServiceLevelCalculator` — live path)
|
||||
|
||||
`OpcUaPublishActor.HandleRedundancyStateChanged` maps the local node's role
|
||||
(from the `RedundancyStateChanged` snapshot) to a ServiceLevel byte and forwards
|
||||
it through `IServiceLevelPublisher` to the SDK's `Server.ServiceLevel` Variable:
|
||||
`ServiceLevelCalculator.Compute(NodeHealthInputs)` is the live publish path.
|
||||
`OpcUaPublishActor` calls it on every `HealthTick` (~5 s) and on each
|
||||
`RedundancyStateChanged` snapshot, then forwards the result through
|
||||
`IServiceLevelPublisher` to the SDK's `Server.ServiceLevel` Variable.
|
||||
|
||||
| Local role | Byte |
|
||||
The four inputs are sourced locally per driver node:
|
||||
|
||||
| Input | Source |
|
||||
|---|---|
|
||||
| `Primary` and `driver` role-leader | 240 |
|
||||
| `Primary` (not role-leader) | 200 |
|
||||
| `Secondary` | 100 |
|
||||
| `Detached` (no `driver` role) | 0 |
|
||||
| `MemberState` | Local `SelfMember.Status` from the Akka cluster (Up / Joining / Leaving / …). |
|
||||
| `DbReachable` | Local `DbHealthProbeActor` — `OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. |
|
||||
| `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. |
|
||||
| `Stale` (derived) | `!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`. |
|
||||
| `IsDriverRoleLeader` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
|
||||
|
||||
The resulting truth table (all tiers are now reachable at runtime):
|
||||
|
||||
| Tier | Byte | Condition |
|
||||
|---|---|---|
|
||||
| Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. |
|
||||
| Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
|
||||
| Stale | 200 | Data stale but ConfigDb reachable. |
|
||||
| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not role-leader. |
|
||||
| Healthy leader | 250 | Same as healthy follower + this node is the `driver` role-leader (+10 bonus). |
|
||||
|
||||
> **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary
|
||||
> published 100 (coarse role-only mapping). It now publishes **240** — both
|
||||
> nodes sit at 240/250 under healthy conditions, with the leader still preferred
|
||||
> by the +10 bonus. Clients with the standard "pick highest ServiceLevel"
|
||||
> heuristic continue to prefer the primary.
|
||||
|
||||
#### Backward-compatible fallback (legacy seam)
|
||||
|
||||
A node with no `DbHealthStatus` wired (e.g. early bootstrap window before the
|
||||
first `DbHealthProbeActor` reply) falls back to the old role-only mapping:
|
||||
Primary-leader → 240, Primary → 200, Secondary → 100, Detached → 0. Once the
|
||||
first `DbHealthStatus` arrives the calculator takes over. The first computed
|
||||
ServiceLevel (even 0) is always published so no node lingers at the SDK default
|
||||
255.
|
||||
|
||||
Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
|
||||
role is `Primary` when it holds the `driver` role-leader lease, otherwise
|
||||
`Secondary`; a node without the `driver` role is `Detached`.
|
||||
|
||||
### Full health-aware tiering (`ServiceLevelCalculator`)
|
||||
|
||||
`ServiceLevelCalculator.Compute(NodeHealthInputs)` is the fuller, DB/probe-aware
|
||||
calculation. It is unit-tested but **not yet on the live publish path** — the
|
||||
driver-side mapping above is what actually drives the SDK today.
|
||||
|
||||
| Tier | Byte | Condition |
|
||||
|---|---|---|
|
||||
| Down | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting). |
|
||||
| Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
|
||||
| Stale | 200 | Data stale but ConfigDb reachable. |
|
||||
| Healthy follower | 240 | DB ok + OPC UA probe ok + not stale. |
|
||||
| Healthy leader | 250 | Healthy follower (240) + a `+10` bonus when this node is the `driver` role-leader. |
|
||||
|
||||
Either way, clients with the standard redundancy heuristic ("pick the highest
|
||||
ServiceLevel") prefer the `driver` role-leader and fall back to followers on its
|
||||
degradation.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
Cluster topology event ──┐
|
||||
▼
|
||||
RedundancyStateActor (admin singleton)
|
||||
│ debounce 250ms
|
||||
▼
|
||||
DPS topic "redundancy-state"
|
||||
│
|
||||
▼
|
||||
Driver nodes' OpcUaPublishActor
|
||||
│ role → byte (240/200/100/0)
|
||||
▼
|
||||
Cluster topology event ──────────────────────────────────────────┐
|
||||
▼
|
||||
RedundancyStateActor (admin singleton)
|
||||
│ debounce 250ms
|
||||
▼
|
||||
DPS topic "redundancy-state"
|
||||
│ ▲
|
||||
┌───────────────────────┘ │
|
||||
│ │
|
||||
▼ │
|
||||
Driver node: OpcUaPublishActor │
|
||||
┌─────────────────────────────────────────────────────────┐ │
|
||||
│ Inputs collected per ~5s HealthTick: │ │
|
||||
│ • MemberState ← Akka SelfMember.Status │ │
|
||||
│ • DbReachable ← DbHealthProbeActor (Ask, timeout→F) │ │
|
||||
│ • OpcUaProbeOk ← OpcUaProbeResult about THIS node │──────┘
|
||||
│ • Stale ← derived from above timestamps │ PeerProbeSupervisor
|
||||
│ • IsLeader ← RedundancyStateChanged snapshot │ → PeerOpcUaProbeActor(s)
|
||||
│ │ publish OpcUaProbeResult
|
||||
│ ServiceLevelCalculator.Compute(NodeHealthInputs) │ on "redundancy-state"
|
||||
│ → byte (0/100/200/240/250) │
|
||||
└───────────────────────────────────────────────────────-─┘
|
||||
│
|
||||
▼
|
||||
IServiceLevelPublisher (SdkServiceLevelPublisher)
|
||||
│
|
||||
▼
|
||||
│
|
||||
▼
|
||||
OPC UA Server.ServiceLevel Variable
|
||||
```
|
||||
|
||||
Today only cluster topology drives the published ServiceLevel.
|
||||
`PeerOpcUaProbeActor` and `DbHealthProbeActor` also run per-node — the peer probe
|
||||
publishes `OpcUaProbeResult` onto the `redundancy-state` topic and the DB probe
|
||||
backs the health endpoint — but their outputs are not yet consumed by
|
||||
`RedundancyStateActor` or folded into the published byte. They are the inputs the
|
||||
fuller `ServiceLevelCalculator` truth table is designed to use once that path goes
|
||||
live.
|
||||
Both `DbHealthProbeActor` and `PeerOpcUaProbeActor` feed the live publish path.
|
||||
The peer probe publishes `OpcUaProbeResult` on the `redundancy-state` topic;
|
||||
`OpcUaPublishActor` consumes only results whose target is itself and applies
|
||||
freshness-debouncing before passing them to the calculator. `DbHealthProbeActor`
|
||||
is queried directly via Ask on each `HealthTick`.
|
||||
|
||||
The admin singleton is the cluster's only `RedundancyStateActor`. If the admin leader fails over, the new admin node spins up its replacement, re-subscribes to cluster events, and publishes a fresh snapshot from the current `Cluster.State`. There is no DB-persisted state to recover.
|
||||
|
||||
@@ -113,7 +135,7 @@ There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` c
|
||||
|
||||
> **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).
|
||||
|
||||
The `ClusterNode.ServiceLevelBase` column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed from cluster role/health and published per the mapping above, independent of this stored preference.
|
||||
The `ClusterNode.ServiceLevelBase` column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed by `ServiceLevelCalculator` from cluster role and live health inputs, independent of this stored preference.
|
||||
|
||||
### Peer URI advertising
|
||||
|
||||
|
||||
Reference in New Issue
Block a user