fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised 250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could reach while the reachable peer refused writes as Secondary. - Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so the Runtime can consume it. - ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0, ahead of every other input including the +10 Primary bonus. - OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted / Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only seam returns 0 for a failed endpoint too. - New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive, the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is Primary. The election itself is not bent. Guarded on a peer having recently reported its own endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with both endpoints failed does not bounce the role and restart nodes forever, and a lone node never shuts itself down. - Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode). - Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section; cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed). Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less, DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
@@ -11,7 +11,7 @@ using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
|
||||
@@ -45,11 +45,12 @@ public static class HealthEndpoints
|
||||
// The ACTIVE tier depends on the roles (coordinator ruling, fix round 1 / I6):
|
||||
// - driver-only: Active too — a site node's whole purpose is its OPC UA endpoint, so nothing may
|
||||
// route to it;
|
||||
// - fused admin+driver: NOT Active, until the ServiceLevel calculator consumes IOpcUaEndpointState
|
||||
// (follow-up F-C1). Traefik routes the AdminUI on /health/active, so putting the probe there would
|
||||
// take the operator's only console out of rotation precisely when they need it to diagnose the
|
||||
// failed endpoint, while the OPC UA failover (ServiceLevel) still would not move. Once F-C1 lands
|
||||
// the peer takes Primary and the active tier follows it on its own.
|
||||
// - fused admin+driver: NOT Active. Traefik routes the AdminUI on /health/active, so putting the probe
|
||||
// there would take the operator's only console out of rotation precisely when they need it to
|
||||
// diagnose the failed endpoint. The OPC UA failover is handled by the Runtime instead (F-C1,
|
||||
// lmxopcua#762): the failed node computes ServiceLevel 0 and, when it is the Primary and its peer
|
||||
// reports a listening endpoint, OpcUaEndpointStepDownActor makes it leave its mesh — the peer becomes
|
||||
// the oldest Up member, so the active tier follows it on its own.
|
||||
// Factory-registered so the state singleton is resolved per probe.
|
||||
if (hasDriver)
|
||||
{
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
|
||||
@@ -12,8 +12,9 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
/// <remarks>
|
||||
/// Registered on driver-role nodes only (<see cref="HealthEndpoints.AddOtOpcUaHealth"/>); an
|
||||
/// admin-only node serves no OPC UA endpoint and must not be degraded by one it never starts. On a fused
|
||||
/// admin+driver node it is Ready-only until ServiceLevel consumes the fact (F-C1), because the Active tier
|
||||
/// is what Traefik routes the AdminUI on.
|
||||
/// admin+driver node it is Ready-only, because the Active tier is what Traefik routes the AdminUI on; the
|
||||
/// Primary moves through the Runtime instead — ServiceLevel 0 plus <c>OpcUaEndpointStepDownActor</c>
|
||||
/// (lmxopcua#762).
|
||||
/// </remarks>
|
||||
public sealed class OpcUaEndpointHealthCheck : IHealthCheck
|
||||
{
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.OpcUa;
|
||||
|
||||
/// <summary>Lifecycle of this node's OPC UA endpoint as <see cref="OtOpcUaServerHostedService"/> drives it.</summary>
|
||||
public enum OpcUaEndpointStatus
|
||||
{
|
||||
/// <summary>The hosted service has not run <c>StartAsync</c> yet.</summary>
|
||||
NotStarted,
|
||||
|
||||
/// <summary>The SDK is listening and the address-space / ServiceLevel sinks are bound.</summary>
|
||||
Started,
|
||||
|
||||
/// <summary>The SDK failed to start (port in use, PKI unwritable, bad <c>OpcUa:*</c> override, …).</summary>
|
||||
Failed,
|
||||
|
||||
/// <summary>The hosted service stopped the SDK; the port is released.</summary>
|
||||
Stopped,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The one place the "did the OPC UA SDK actually start?" fact lives (R03:2.1). Before it, a failed
|
||||
/// start was one Error line: <c>/health/ready</c> stayed 200, the node joined its mesh, could be
|
||||
/// elected driver Primary and acked deploys as Applied through the Null sink — while clients got
|
||||
/// connection-refused from the node the fleet called healthy.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Consumed by <c>OpcUaEndpointHealthCheck</c> (Ready + Active tiers). The redundancy election and the
|
||||
/// ServiceLevel calculator do not read it yet — they live outside the Host (<c>Cluster</c> /
|
||||
/// <c>Runtime</c>); feeding them is the Wave 3 follow-up the lane report names. Until then a fused
|
||||
/// node whose SDK failed is visible (503) but still Primary-eligible.
|
||||
/// </remarks>
|
||||
public interface IOpcUaEndpointState
|
||||
{
|
||||
/// <summary>The endpoint's current lifecycle status.</summary>
|
||||
OpcUaEndpointStatus Status { get; }
|
||||
|
||||
/// <summary>Why the start failed, when <see cref="Status"/> is <see cref="OpcUaEndpointStatus.Failed"/>.</summary>
|
||||
string? FailureReason { get; }
|
||||
}
|
||||
|
||||
/// <summary>Mutable <see cref="IOpcUaEndpointState"/>; written by the hosted service, read by the health check.</summary>
|
||||
public sealed class OpcUaEndpointState : IOpcUaEndpointState
|
||||
{
|
||||
private volatile OpcUaEndpointStatus _status = OpcUaEndpointStatus.NotStarted;
|
||||
private volatile string? _failureReason;
|
||||
|
||||
/// <inheritdoc />
|
||||
public OpcUaEndpointStatus Status => _status;
|
||||
|
||||
/// <inheritdoc />
|
||||
public string? FailureReason => _failureReason;
|
||||
|
||||
/// <summary>Records a successful SDK start.</summary>
|
||||
public void MarkStarted()
|
||||
{
|
||||
_failureReason = null;
|
||||
_status = OpcUaEndpointStatus.Started;
|
||||
}
|
||||
|
||||
/// <summary>Records a failed SDK start.</summary>
|
||||
/// <param name="reason">The failure, as the health probe should report it.</param>
|
||||
public void MarkFailed(string reason)
|
||||
{
|
||||
_failureReason = reason;
|
||||
_status = OpcUaEndpointStatus.Failed;
|
||||
}
|
||||
|
||||
/// <summary>Records that the hosted service stopped the SDK.</summary>
|
||||
public void MarkStopped()
|
||||
{
|
||||
_failureReason = null;
|
||||
_status = OpcUaEndpointStatus.Stopped;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user