feat(health)!: active node is the oldest Up member, not the leader (0.3.0)

Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.

Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.

Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.

New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.

BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
  That "not applicable => Healthy" made the tier answer 200 on every node and
  silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
  overridable via NoActiveRoleStatus. This case is reachable, not defensive —
  OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
  correctly a different member during the overlap.

Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.

Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.

ZB.MOM.WW.Health/Directory.Build.props

@@ -5,7 +5,7 @@
     <Nullable>enable</Nullable>
     <ImplicitUsings>enable</ImplicitUsings>
     <LangVersion>latest</LangVersion>
-    <Version>0.2.1</Version>
+    <Version>0.3.0</Version>
     <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
     <!-- Emit XML docs so the public API summaries ship inside the packed nupkgs (IntelliSense for
          consumers). CS1591 (missing doc on a public member) is suppressed so undocumented test /

ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ActiveNodeHealthCheck.cs

@@ -9,99 +9,80 @@ using Microsoft.Extensions.Diagnostics.HealthChecks;
 namespace ZB.MOM.WW.Health.Akka;
 
 /// <summary>
-/// Pure decision function for the active / leader probe, factored out of
+/// Active-tier probe: Healthy (200) on the one node in charge, Unhealthy (503) on its standby.
-/// <see cref="ActiveNodeHealthCheck"/> so the role-less and role-filtered matrices are exhaustively
+/// Register to the <see cref="ZbHealthTags.Active"/> tag.
-/// table-testable without forming a real cluster.
-/// </summary>
-internal static class ActiveNodeDecision
-{
-    /// <summary>
-    /// Maps the resolved cluster facts to a <see cref="HealthStatus"/>.
-    /// </summary>
-    /// <param name="selfUp">Whether the local node's member status is <c>Up</c>.</param>
-    /// <param name="isLeader">
-    /// Whether the local node is the leader: the cluster leader in role-less mode, or the
-    /// role-singleton leader in role-filtered mode.
-    /// </param>
-    /// <param name="hasRole">
-    /// Whether the local node carries <paramref name="requiredRole"/>. Ignored when
-    /// <paramref name="requiredRole"/> is <c>null</c>.
-    /// </param>
-    /// <param name="requiredRole">
-    /// The role to scope the check to, or <c>null</c> for the role-less (whole-cluster-leader) mode.
-    /// </param>
-    /// <returns>
-    /// Role-less: Healthy iff the node is Up and the cluster leader, otherwise Unhealthy.
-    /// Role-filtered: Healthy when the node lacks the role (probe irrelevant) or carries the role and
-    /// is the role-singleton leader; <strong>Unhealthy</strong> when it carries the role but is not
-    /// the leader.
-    /// </returns>
-    /// <remarks>
-    /// The role-member-but-not-leader case returns Unhealthy, not Degraded, because the active tier's
-    /// whole purpose is to be a 503 that an orchestrator can act on — the shared spec states it as
-    /// "Fails (503) on a standby or role-member-but-not-leader node", and the same spec maps Degraded
-    /// to 200. Returning Degraded made the tier answer 200 on every node, so a load balancer pointed
-    /// at it (OtOpcUa's Traefik routes the admin UI this way) kept the standby in the pool and its
-    /// leader-pinning silently never worked. Found on a live rig; see lmxopcua#494.
-    /// </remarks>
-    public static HealthStatus Evaluate(bool selfUp, bool isLeader, bool hasRole, string? requiredRole)
-    {
-        if (requiredRole is null)
-            return selfUp && isLeader ? HealthStatus.Healthy : HealthStatus.Unhealthy;
-
-        if (!hasRole)
-            return HealthStatus.Healthy;
-
-        return isLeader ? HealthStatus.Healthy : HealthStatus.Unhealthy;
-    }
-}
-
-/// <summary>
-/// Health check that reports whether this node is the designated active / leader node.
-/// An optional role scopes the check to nodes carrying that role. Register to the
-/// <see cref="ZbHealthTags.Active"/> tag.
 /// </summary>
 /// <remarks>
-/// The <see cref="ActorSystem"/> is resolved lazily from the service provider. If it is not yet
+///     <para>
-/// available — e.g. during startup before Akka is initialised — the check returns
+///         "In charge" means the <b>oldest Up member</b> — see <see cref="ClusterActiveNode"/> for
-/// <see cref="HealthStatus.Degraded"/> rather than throwing, so it is startup-safe.
+///         why leadership is the wrong answer and what breaks when it is used. Scope the competition
+///         to a role with <see cref="ActiveNodeHealthCheckOptions.RolePreference"/>; leave it empty
+///         for a cluster whose nodes all compete for the same single active slot.
+///     </para>
+///     <para>
+///         <b>Behaviour change in 0.3.0.</b> Through 0.2.1 this check selected by
+///         <c>ClusterState.Leader</c> / <c>RoleLeader</c>, and its role-filtered mode reported
+///         Healthy for any node <i>lacking</i> the role. Both were wrong, both were found on live
+///         rigs, and both Akka consumers had already replaced it with a private copy rather than use
+///         it. It now selects by age and reports <see cref="HealthStatus.Unhealthy"/> for a node that
+///         owns no active work. A node whose active tier previously answered 200 unconditionally will
+///         now answer 503 unless it really is the active node — which is the point.
+///     </para>
+///     <para>
+///         Startup-safe: before the <see cref="ActorSystem"/> or the cluster is available the result
+///         is <see cref="HealthStatus.Degraded"/> (a 200 under the tier mapping), so a node that is
+///         merely still booting is not reported as a failed standby and pulled from rotation on the
+///         way up.
+///     </para>
 /// </remarks>
 public sealed class ActiveNodeHealthCheck : IHealthCheck
 {
     private readonly IServiceProvider _serviceProvider;
-    private readonly string? _role;
+    private readonly ActiveNodeHealthCheckOptions _options;
 
     /// <summary>
-    /// Role-less constructor: Healthy when the node is <c>Up</c> and the cluster leader
+    /// Unscoped: the active node is the oldest Up member of the whole cluster.
-    /// (ScadaBridge ActiveNode pattern); Unhealthy otherwise. Degraded when the ActorSystem /
-    /// cluster is not yet ready.
     /// </summary>
     /// <param name="serviceProvider">
     /// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
-    /// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
+    /// check is startup-safe.
     /// </param>
     public ActiveNodeHealthCheck(IServiceProvider serviceProvider)
+        : this(serviceProvider, new ActiveNodeHealthCheckOptions())
     {
-        _serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
-        _role = null;
     }
 
     /// <summary>
-    /// Role-filtered constructor: Healthy when the node lacks <paramref name="role"/> or carries it
+    /// Role-scoped: the active node is the oldest Up member carrying <paramref name="role"/>.
-    /// and is the role-singleton leader; Unhealthy (503) when it carries the role but is not the
-    /// leader (OtOpcUa AdminRoleLeader pattern). Degraded when the ActorSystem / cluster is not yet
-    /// ready.
     /// </summary>
     /// <param name="serviceProvider">
     /// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
-    /// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
+    /// check is startup-safe.
     /// </param>
-    /// <param name="role">The Akka cluster role to scope the check to.</param>
+    /// <param name="role">The cluster role to scope the competition to.</param>
     public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role)
+        : this(serviceProvider, NewOptions(role))
+    {
+    }
+
+    /// <summary>
+    /// Configured with an explicit <see cref="ActiveNodeHealthCheckOptions"/>.
+    /// </summary>
+    /// <param name="serviceProvider">
+    /// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
+    /// check is startup-safe.
+    /// </param>
+    /// <param name="options">The role-scoping options.</param>
+    public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options)
     {
         _serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
+        _options = options ?? throw new ArgumentNullException(nameof(options));
+    }
+
+    private static ActiveNodeHealthCheckOptions NewOptions(string role)
+    {
         ArgumentException.ThrowIfNullOrWhiteSpace(role);
-        _role = role;
+        return new ActiveNodeHealthCheckOptions { RolePreference = new[] { role } };
     }
 
     /// <inheritdoc />
@@ -113,10 +94,8 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
         if (system is null)
             return Task.FromResult(HealthCheckResult.Degraded("ActorSystem not yet available."));
 
-        MemberStatus selfStatus;
+        Member self;
-        bool selfUp;
+        IEnumerable<Member> members;
-        bool hasRole;
-        bool isLeader;
         try
         {
             // Reading cluster membership can throw while the ActorSystem exists but the cluster has
@@ -124,54 +103,49 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
             // ConfigurationException). The spec's startup-safety rule maps this to Degraded rather
             // than letting the exception escape (which the host would record as Unhealthy).
             var cluster = Cluster.Get(system);
-            var self = cluster.SelfMember;
+            self = cluster.SelfMember;
-            selfStatus = self.Status;
+            members = cluster.State.Members;
-            selfUp = selfStatus == MemberStatus.Up;
-
-            if (_role is null)
-            {
-                hasRole = false;
-                var leader = cluster.State.Leader;
-                isLeader = leader is not null && leader == self.Address;
-            }
-            else
-            {
-                hasRole = self.HasRole(_role);
-                var roleLeader = cluster.State.RoleLeader(_role);
-                isLeader = roleLeader is not null && roleLeader == self.Address;
-            }
         }
         catch (Exception ex) when (ex is not OperationCanceledException)
         {
             return Task.FromResult(HealthCheckResult.Degraded("Akka cluster state not yet accessible.", ex));
         }
 
-        var health = ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, _role);
+        var scoped = _options.RolePreference.Count > 0;
-        var description = DescribeResult(health, selfStatus, selfUp, isLeader);
+        var role = ClusterActiveNode.ResolveActiveRole(self.Roles, _options.RolePreference);
-        var result = health switch
-        {
-            HealthStatus.Healthy   => HealthCheckResult.Healthy(description),
-            HealthStatus.Degraded  => HealthCheckResult.Degraded(description),
-            _                      => HealthCheckResult.Unhealthy(description),
-        };
-        return Task.FromResult(result);
-    }
 
-    private string DescribeResult(HealthStatus health, MemberStatus status, bool selfUp, bool isLeader)
+        var data = new Dictionary<string, object>(StringComparer.Ordinal)
-    {
-        if (_role is null)
         {
-            if (health == HealthStatus.Healthy)
+            ["selfAddress"] = self.Address.ToString(),
-                return "Active node (cluster leader).";
+        };
-            return selfUp && !isLeader
+        if (role is not null) data["activeRole"] = role;
-                ? "Standby: node is Up but not the cluster leader."
+
-                : $"Standby: node is not Up (status: {status}).";
+        if (scoped && role is null)
+        {
+            // The node carries none of the preferred roles, so it owns no active work and cannot be
+            // the active node for any of them. Reported per NoActiveRoleStatus — Unhealthy by
+            // default, because the active tier's whole job is to keep such a node out of the pool.
+            var message = $"Node carries none of the active roles: {string.Join(", ", _options.RolePreference)}.";
+            return Task.FromResult(new HealthCheckResult(_options.NoActiveRoleStatus, message, exception: null, data));
         }
 
-        return health switch
+        var active = ClusterActiveNode.OldestUpMember(members, role);
+        if (active is not null) data["activeNode"] = active.Address.ToString();
+
+        var scopeLabel = role is null ? "the cluster" : $"role '{role}'";
+
+        if (active is null)
         {
-            HealthStatus.Healthy => $"Active for role '{_role}' (or not a role member).",
+            // Nobody is Up in the scope — a cold start, or every holder is still Joining or already
-            _ => $"Role '{_role}' member but not leader.",
+            // Leaving. Nobody is active, so nobody may claim to be.
-        };
+            return Task.FromResult(HealthCheckResult.Unhealthy(
+                $"No Up member currently holds the active slot for {scopeLabel}.", exception: null, data));
+        }
+
+        // Compared by UniqueAddress: a node that restarted on the same host and port is a different
+        // member, and during the overlap the cluster briefly holds both incarnations.
+        return Task.FromResult(active.UniqueAddress.Equals(self.UniqueAddress)
+            ? HealthCheckResult.Healthy($"Active node for {scopeLabel} (oldest Up member).", data)
+            : HealthCheckResult.Unhealthy($"Standby: active node for {scopeLabel} is {active.Address}.", exception: null, data));
     }
 }

ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ActiveNodeHealthCheckOptions.cs

@@ -0,0 +1,37 @@
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+
+namespace ZB.MOM.WW.Health.Akka;
+
+/// <summary>
+/// Configures how <see cref="ActiveNodeHealthCheck"/> decides which role's active node this node is
+/// competing to be.
+/// </summary>
+public sealed class ActiveNodeHealthCheckOptions
+{
+    /// <summary>
+    /// Candidate roles in descending precedence. The check answers for the first one the local node
+    /// carries. Empty (the default) means unscoped: the active node is the oldest Up member of the
+    /// whole cluster.
+    /// </summary>
+    /// <remarks>
+    /// A fused node should list the role its singletons are pinned to first — e.g.
+    /// <c>["admin", "driver"]</c> for a node that hosts the admin singletons when fused and competes
+    /// as a plain driver when it is a driver-only node.
+    /// </remarks>
+    public IReadOnlyList<string> RolePreference { get; init; } = Array.Empty<string>();
+
+    /// <summary>
+    /// The status to report when <see cref="RolePreference"/> is non-empty and the node carries none
+    /// of those roles — it owns no active work at all. Defaults to
+    /// <see cref="HealthStatus.Unhealthy"/>.
+    /// </summary>
+    /// <remarks>
+    /// Unhealthy is the safe default and is deliberately <b>not</b> the "not applicable ⇒ Healthy"
+    /// convention used by readiness probes. The active tier exists so a load balancer can pick
+    /// exactly one node; a node that owns no active work must not be in that pool. Answering Healthy
+    /// there is what made the previous role-filtered check report 200 on every node and silently
+    /// broke leader-pinning (lmxopcua#494). Override only when a node genuinely should stay in the
+    /// active pool without holding any of the preferred roles.
+    /// </remarks>
+    public HealthStatus NoActiveRoleStatus { get; init; } = HealthStatus.Unhealthy;
+}

ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/AkkaActiveNodeGate.cs

@@ -6,27 +6,47 @@ namespace ZB.MOM.WW.Health.Akka;
 
 /// <summary>
 /// <see cref="IActiveNodeGate"/> implementation that computes <see cref="IsActiveNode"/> directly
-/// from the Akka cluster state (self member <c>Up</c> and the local node is the cluster leader).
+/// from the Akka cluster state — the local node is the <b>oldest Up member</b>, optionally within a
-/// Register as a singleton.
+/// role scope. Register as a singleton.
 /// </summary>
 /// <remarks>
-/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not yet
+///     <para>
-/// available — e.g. during startup before Akka is initialised — <see cref="IsActiveNode"/> returns
+///         Shares its rule with <see cref="ActiveNodeHealthCheck"/> via
-/// <c>false</c> (the safe default during startup). This gate reads the cluster state directly and
+///         <see cref="ClusterActiveNode"/>, so an endpoint gated by this type and the
-/// does not resolve <see cref="ActiveNodeHealthCheck"/> from DI.
+///         <c>/health/active</c> probe an orchestrator routes by can never disagree about which node
+///         is in charge.
+///     </para>
+///     <para>
+///         <b>Behaviour change in 0.3.0:</b> through 0.2.1 this gate selected by
+///         <c>ClusterState.Leader</c>, which is address-ordered and diverges from singleton placement
+///         after any restart. See <see cref="ClusterActiveNode"/>.
+///     </para>
+///     <para>
+///         The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not
+///         yet available — e.g. during startup before Akka is initialised —
+///         <see cref="IsActiveNode"/> returns <c>false</c> (the safe default during startup, matching
+///         the standby case). This gate reads the cluster state directly and does not resolve
+///         <see cref="ActiveNodeHealthCheck"/> from DI.
+///     </para>
 /// </remarks>
 public sealed class AkkaActiveNodeGate : IActiveNodeGate
 {
     private readonly IServiceProvider _serviceProvider;
+    private readonly string? _role;
 
     /// <summary>Initializes a new <see cref="AkkaActiveNodeGate"/>.</summary>
     /// <param name="serviceProvider">
     /// The application service provider. The <see cref="ActorSystem"/> is resolved lazily; if it is
     /// not yet available <see cref="IsActiveNode"/> returns <c>false</c>.
     /// </param>
-    public AkkaActiveNodeGate(IServiceProvider serviceProvider)
+    /// <param name="role">
+    /// The role to scope the competition to, or <c>null</c> (the default) to consider every Up
+    /// member.
+    /// </param>
+    public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null)
     {
         _serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
+        _role = role;
     }
 
     /// <inheritdoc />
@@ -38,13 +58,16 @@ public sealed class AkkaActiveNodeGate : IActiveNodeGate
             if (system is null)
                 return false;
 
-            var cluster = Cluster.Get(system);
+            try
-            var self = cluster.SelfMember;
+            {
-            if (self.Status != MemberStatus.Up)
+                return ClusterActiveNode.SelfIsActive(Cluster.Get(system), _role);
+            }
+            catch (Exception)
+            {
+                // ActorSystem exists but the cluster is not initialised yet. Same startup-safety
+                // rule as the health check, expressed as the gate's safe default: not active.
                 return false;
-
+            }
-            var leader = cluster.State.Leader;
-            return leader is not null && leader == self.Address;
         }
     }
 }

ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ClusterActiveNode.cs

@@ -0,0 +1,133 @@
+using Akka.Cluster;
+
+namespace ZB.MOM.WW.Health.Akka;
+
+/// <summary>
+/// THE family-wide definition of "the active node": the <b>oldest Up member</b> of the cluster,
+/// optionally scoped to a role — the member <c>ClusterSingletonManager</c> places singletons on.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Cluster <b>leadership</b> (<c>ClusterState.Leader</c> / <c>RoleLeader</c>) is the wrong
+///         answer and must never be used for a product-level active/standby decision. Leadership is
+///         <i>address</i>-ordered (host, then port) and has no relationship to time; singleton
+///         placement is <i>age</i>-ordered. On a freshly-formed cluster the two agree, which is why
+///         the discrepancy hides in unit tests and in a rig that has never been restarted. They
+///         diverge permanently after any restart: the restarted node rejoins as the youngest but
+///         keeps its address, so if it holds the lower address it becomes leader while the
+///         singletons — and all the work they own — stay on the other node.
+///     </para>
+///     <para>
+///         Both Akka consumers in the family independently hit this and each wrote its own
+///         replacement rather than use the shared leader-based check
+///         (ScadaBridge <c>OldestNodeActiveHealthCheck</c>, OtOpcUa <c>ClusterPrimaryHealthCheck</c>).
+///         This type is those two implementations promoted to one, so the health tier, the inbound
+///         API gate, the store-and-forward delivery gate and OtOpcUa's OPC UA <c>ServiceLevel</c>
+///         cannot drift apart about which node is in charge.
+///     </para>
+///     <para>
+///         Only <see cref="MemberStatus.Up"/> members are eligible, matching singleton placement: a
+///         <c>Leaving</c> node is handing its singletons over and must not be named active.
+///     </para>
+/// </remarks>
+public static class ClusterActiveNode
+{
+    /// <summary>
+    /// The oldest <see cref="MemberStatus.Up"/> member, optionally scoped to <paramref name="role"/>.
+    /// </summary>
+    /// <param name="members">The cluster members, in any order.</param>
+    /// <param name="role">
+    /// The role to scope the selection to, or <c>null</c> to consider every Up member.
+    /// </param>
+    /// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
+    public static Member? OldestUpMember(IEnumerable<Member> members, string? role = null)
+    {
+        ArgumentNullException.ThrowIfNull(members);
+
+        Member? oldest = null;
+        foreach (var member in members)
+        {
+            if (member.Status != MemberStatus.Up) continue;
+            if (role is not null && !member.HasRole(role)) continue;
+            if (oldest is null || member.IsOlderThan(oldest)) oldest = member;
+        }
+
+        return oldest;
+    }
+
+    /// <summary>
+    /// The oldest <see cref="MemberStatus.Up"/> member of <paramref name="cluster"/>, optionally
+    /// scoped to <paramref name="role"/>.
+    /// </summary>
+    /// <param name="cluster">The Akka cluster to evaluate.</param>
+    /// <param name="role">
+    /// The role to scope the selection to, or <c>null</c> to consider every Up member.
+    /// </param>
+    /// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
+    public static Member? OldestUpMember(Cluster cluster, string? role = null)
+    {
+        ArgumentNullException.ThrowIfNull(cluster);
+        return OldestUpMember(cluster.State.Members, role);
+    }
+
+    /// <summary>
+    /// Whether the local node is the active node — Up, and the oldest Up member in the role scope.
+    /// </summary>
+    /// <param name="cluster">The Akka cluster to evaluate.</param>
+    /// <param name="role">
+    /// The role to scope the selection to, or <c>null</c> to consider every Up member.
+    /// </param>
+    /// <returns>
+    /// <c>true</c> when the local member is Up, carries <paramref name="role"/> when one is given,
+    /// and no other eligible member is older.
+    /// </returns>
+    /// <remarks>
+    /// Identity is compared by <see cref="Member.UniqueAddress"/>, not <c>Address</c>: a node that
+    /// restarted on the same host and port is a different member, and during the overlap the cluster
+    /// briefly holds both incarnations.
+    /// </remarks>
+    public static bool SelfIsActive(Cluster cluster, string? role = null)
+    {
+        ArgumentNullException.ThrowIfNull(cluster);
+
+        var self = cluster.SelfMember;
+        if (self.Status != MemberStatus.Up) return false;
+        if (role is not null && !self.HasRole(role)) return false;
+
+        var oldest = OldestUpMember(cluster.State.Members, role);
+        return oldest is not null && oldest.UniqueAddress.Equals(self.UniqueAddress);
+    }
+
+    /// <summary>
+    /// The first role in <paramref name="rolePreference"/> that <paramref name="selfRoles"/> carries
+    /// — the role whose oldest Up member owns this node's active work.
+    /// </summary>
+    /// <param name="selfRoles">The local member's cluster roles.</param>
+    /// <param name="rolePreference">
+    /// Candidate roles in descending precedence, e.g. <c>["admin", "driver"]</c> for a node that
+    /// answers for <c>admin</c> when it is fused and for <c>driver</c> when it is a driver-only node.
+    /// </param>
+    /// <returns>
+    /// The first carried role, or <c>null</c> when the node carries none of them.
+    /// </returns>
+    /// <remarks>
+    /// Precedence matters on a fused node: it must answer for the role its singletons — and the
+    /// traffic being routed to it — are pinned to. Answering for a lower-precedence role could name a
+    /// different node the moment a single-role member joins the mesh.
+    /// </remarks>
+    public static string? ResolveActiveRole(IEnumerable<string> selfRoles, IReadOnlyList<string> rolePreference)
+    {
+        ArgumentNullException.ThrowIfNull(selfRoles);
+        ArgumentNullException.ThrowIfNull(rolePreference);
+
+        if (rolePreference.Count == 0) return null;
+
+        var roles = selfRoles as IReadOnlyCollection<string> ?? selfRoles.ToList();
+        foreach (var candidate in rolePreference)
+        {
+            if (roles.Contains(candidate)) return candidate;
+        }
+
+        return null;
+    }
+}

ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveNodeDecisionTests.cs

@@ -1,139 +0,0 @@
-using Akka.Actor;
-using Microsoft.Extensions.DependencyInjection;
-using Microsoft.Extensions.Diagnostics.HealthChecks;
-using ZB.MOM.WW.Health.Akka;
-
-namespace ZB.MOM.WW.Health.Akka.Tests;
-
-/// <summary>
-/// Table-driven tests for the pure <see cref="ActiveNodeDecision.Evaluate"/> helper covering both
-/// the role-less (ScadaBridge ActiveNode) and role-filtered (OtOpcUa AdminRoleLeader) matrices,
-/// plus the startup-safety null-guards on <see cref="ActiveNodeHealthCheck"/> and
-/// <see cref="AkkaActiveNodeGate"/> when no <see cref="ActorSystem"/> is registered.
-/// </summary>
-public sealed class ActiveNodeDecisionTests
-{
-    // Role-less: requiredRole == null. hasRole is irrelevant. Healthy iff (selfUp && isLeader), else Unhealthy.
-    public static IEnumerable<object?[]> RoleLessCases() => new[]
-    {
-        new object?[] { true, true, false, (string?)null, HealthStatus.Healthy },
-        new object?[] { true, false, false, (string?)null, HealthStatus.Unhealthy },
-        new object?[] { false, true, false, (string?)null, HealthStatus.Unhealthy },
-        new object?[] { false, false, false, (string?)null, HealthStatus.Unhealthy },
-    };
-
-    [Theory]
-    [MemberData(nameof(RoleLessCases))]
-    public void Evaluate_RoleLess(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
-    {
-        Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
-    }
-
-    // Role-filtered: requiredRole != null.
-    //   lacks role               -> Healthy (probe irrelevant for this node)
-    //   has role & is leader     -> Healthy (selfUp is ignored — role-filtered mode only cares about leadership)
-    //   has role & not leader    -> Unhealthy (503) — the tier exists to be a 503 an LB can act on
-    public static IEnumerable<object[]> RoleFilteredCases() => new[]
-    {
-        // node lacks the role -> Healthy regardless of selfUp / isLeader
-        new object[] { true, true, false, "admin", HealthStatus.Healthy },
-        new object[] { true, false, false, "admin", HealthStatus.Healthy },
-        new object[] { false, false, false, "admin", HealthStatus.Healthy },
-        // node carries the role and is leader -> Healthy (selfUp=true)
-        new object[] { true, true, true, "admin", HealthStatus.Healthy },
-        // node carries the role and is leader -> Healthy (selfUp=false: role-filtered mode ignores selfUp)
-        new object[] { false, true, true, "admin", HealthStatus.Healthy },
-        // Node carries the role but is not leader -> Unhealthy, i.e. HTTP 503.
-        // Degraded would be a 200 (see the shared health SPEC), which made the active tier answer
-        // 200 on every node and silently broke OtOpcUa's Traefik leader-pinning. lmxopcua#494.
-        new object[] { true, false, true, "admin", HealthStatus.Unhealthy },
-        new object[] { false, false, true, "admin", HealthStatus.Unhealthy },
-    };
-
-    [Theory]
-    [MemberData(nameof(RoleFilteredCases))]
-    public void Evaluate_RoleFiltered(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
-    {
-        Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
-    }
-
-    [Fact]
-    public async Task HealthCheck_RoleLess_NoActorSystem_ReturnsDegraded()
-    {
-        var provider = new ServiceCollection().BuildServiceProvider();
-        var check = new ActiveNodeHealthCheck(provider);
-
-        var result = await check.CheckHealthAsync(NewContext(check));
-
-        Assert.Equal(HealthStatus.Degraded, result.Status);
-    }
-
-    [Fact]
-    public async Task HealthCheck_RoleFiltered_NoActorSystem_ReturnsDegraded()
-    {
-        var provider = new ServiceCollection().BuildServiceProvider();
-        var check = new ActiveNodeHealthCheck(provider, "admin");
-
-        var result = await check.CheckHealthAsync(NewContext(check));
-
-        Assert.Equal(HealthStatus.Degraded, result.Status);
-    }
-
-    [Fact]
-    public void Gate_NoActorSystem_IsActiveNodeFalse()
-    {
-        var provider = new ServiceCollection().BuildServiceProvider();
-        var gate = new AkkaActiveNodeGate(provider);
-
-        Assert.False(gate.IsActiveNode);
-    }
-
-    [Fact]
-    public async Task HealthCheck_RoleLess_ClusterInaccessible_ReturnsDegraded()
-    {
-        // ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
-        // return Degraded (startup-safety rule), not let the exception escape (→ Unhealthy).
-        using var system = ActorSystem.Create("plain-no-cluster-roleless");
-        try
-        {
-            var provider = new ServiceCollection()
-                .AddSingleton(system)
-                .BuildServiceProvider();
-            var check = new ActiveNodeHealthCheck(provider);
-
-            var result = await check.CheckHealthAsync(NewContext(check));
-
-            Assert.Equal(HealthStatus.Degraded, result.Status);
-        }
-        finally
-        {
-            await system.Terminate();
-        }
-    }
-
-    [Fact]
-    public async Task HealthCheck_RoleFiltered_ClusterInaccessible_ReturnsDegraded()
-    {
-        using var system = ActorSystem.Create("plain-no-cluster-rolefiltered");
-        try
-        {
-            var provider = new ServiceCollection()
-                .AddSingleton(system)
-                .BuildServiceProvider();
-            var check = new ActiveNodeHealthCheck(provider, "admin");
-
-            var result = await check.CheckHealthAsync(NewContext(check));
-
-            Assert.Equal(HealthStatus.Degraded, result.Status);
-        }
-        finally
-        {
-            await system.Terminate();
-        }
-    }
-
-    private static HealthCheckContext NewContext(IHealthCheck check) => new()
-    {
-        Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
-    };
-}

ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveNodeStartupSafetyTests.cs

@@ -0,0 +1,113 @@
+using Akka.Actor;
+using Akka.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using ZB.MOM.WW.Health.Akka;
+
+namespace ZB.MOM.WW.Health.Akka.Tests;
+
+/// <summary>
+/// The states a node passes through on the way up, where the active tier must not report a booting
+/// node as a failed standby — now that the tier is a real 503, an Unhealthy here would pull the node
+/// out of an orchestrator's pool while it starts.
+/// </summary>
+public sealed class ActiveNodeStartupSafetyTests
+{
+    [Fact]
+    public async Task NoActorSystem_IsDegraded()
+    {
+        var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider());
+
+        var result = await check.CheckHealthAsync(NewContext(check));
+
+        Assert.Equal(HealthStatus.Degraded, result.Status);
+        // Description-only: an empty data dictionary is what makes the writer omit "data" entirely.
+        Assert.Empty(result.Data);
+    }
+
+    [Fact]
+    public async Task NoActorSystem_RoleScoped_IsDegraded()
+    {
+        var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider(), "admin");
+
+        Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
+    }
+
+    [Fact]
+    public void Gate_NoActorSystem_IsNotActive()
+    {
+        Assert.False(new AkkaActiveNodeGate(new ServiceCollection().BuildServiceProvider()).IsActiveNode);
+    }
+
+    [Fact]
+    public async Task ClusterNotConfigured_IsDegraded()
+    {
+        // ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
+        // return Degraded (the spec's startup-safety rule), not let the exception escape (→ Unhealthy).
+        using var system = ActorSystem.Create("active-node-no-cluster");
+        try
+        {
+            var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
+            var check = new ActiveNodeHealthCheck(provider);
+
+            Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
+        }
+        finally
+        {
+            await system.Terminate();
+        }
+    }
+
+    [Fact]
+    public async Task Gate_ClusterNotConfigured_IsNotActive()
+    {
+        using var system = ActorSystem.Create("active-node-gate-no-cluster");
+        try
+        {
+            var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
+
+            Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
+        }
+        finally
+        {
+            await system.Terminate();
+        }
+    }
+
+    [Fact]
+    public async Task ClusteredButNeverJoined_IsUnhealthy_NobodyIsActive()
+    {
+        // A cold start: the cluster provider is up but no member has reached Up. Nobody is active, so
+        // nobody may claim to be — this is the one case that must NOT be Degraded, because a cluster
+        // that never forms would otherwise sit at 200 forever.
+        var config = ConfigurationFactory.ParseString("""
+            akka {
+              loglevel = "WARNING"
+              actor.provider = cluster
+              remote.dot-netty.tcp { hostname = "127.0.0.1", port = 0 }
+              cluster.seed-nodes = []
+            }
+            """);
+        var system = ActorSystem.Create("active-node-never-joined", config);
+        try
+        {
+            var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
+            var check = new ActiveNodeHealthCheck(provider);
+
+            var result = await check.CheckHealthAsync(NewContext(check));
+
+            Assert.Equal(HealthStatus.Unhealthy, result.Status);
+            Assert.Contains("No Up member", result.Description);
+            Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
+        }
+        finally
+        {
+            await system.Terminate();
+        }
+    }
+
+    private static HealthCheckContext NewContext(IHealthCheck check) => new()
+    {
+        Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
+    };
+}

ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveRoleResolutionTests.cs

@@ -0,0 +1,56 @@
+using ZB.MOM.WW.Health.Akka;
+
+namespace ZB.MOM.WW.Health.Akka.Tests;
+
+/// <summary>
+/// Covers <see cref="ClusterActiveNode.ResolveActiveRole"/> — which role a node competes for. Pure,
+/// so no cluster is needed; the age-ordering half of the rule is pinned in
+/// <see cref="ClusterActiveNodeTests"/> against a real two-node cluster.
+/// </summary>
+public sealed class ActiveRoleResolutionTests
+{
+    [Fact]
+    public void EmptyPreference_IsUnscoped()
+    {
+        // Unscoped: every Up member competes for one active slot (the ScadaBridge central pattern).
+        Assert.Null(ClusterActiveNode.ResolveActiveRole(["admin", "driver"], []));
+    }
+
+    [Fact]
+    public void FusedNode_AnswersForTheHighestPrecedenceRoleItCarries()
+    {
+        Assert.Equal(
+            "admin",
+            ClusterActiveNode.ResolveActiveRole(["admin", "driver", "cluster-MAIN"], ["admin", "driver"]));
+    }
+
+    [Fact]
+    public void SingleRoleNode_AnswersForThatRole()
+    {
+        // The regression the OtOpcUa fix existed for: under the old role-filtered check every
+        // driver-only node reported Healthy — "or not a role member" — so all of them called
+        // themselves active and no consumer could find the active node of a site cluster. lmxopcua#494.
+        Assert.Equal(
+            "driver",
+            ClusterActiveNode.ResolveActiveRole(["driver", "cluster-SITE-A"], ["admin", "driver"]));
+    }
+
+    [Fact]
+    public void PreferenceOrderWins_NotTheOrderTheNodeListsItsRoles()
+    {
+        // Self-role order is incidental (it comes from config parsing); precedence must come from the
+        // caller's list alone, or a fused node's answer would depend on how its roles were spelled.
+        Assert.Equal(
+            "admin",
+            ClusterActiveNode.ResolveActiveRole(["driver", "admin"], ["admin", "driver"]));
+    }
+
+    [Fact]
+    public void NodeCarryingNoneOfThePreferredRoles_ResolvesToNull()
+    {
+        // Reachable, not merely defensive: OtOpcUa's RoleParser admits a dev-only or cluster-role-only
+        // node. Such a node owns no active work — the health check maps this to Unhealthy so it stays
+        // out of the active pool.
+        Assert.Null(ClusterActiveNode.ResolveActiveRole(["dev", "cluster-MAIN"], ["admin", "driver"]));
+    }
+}

ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ClusterActiveNodeTests.cs

@@ -0,0 +1,237 @@
+using Akka.Actor;
+using Akka.Cluster;
+using Akka.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using ZB.MOM.WW.Health.Akka;
+
+namespace ZB.MOM.WW.Health.Akka.Tests;
+
+/// <summary>
+/// Pins <b>which</b> node the active tier names, against a real two-node cluster deliberately built
+/// so the oldest member is <i>not</i> the lowest-addressed one.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The divergence is the whole point. Akka offers two different "first" members:
+///         <c>ClusterState.Leader</c> / <c>RoleLeader</c> is the <b>lowest-addressed</b> Up member
+///         (host, then port — nothing to do with time), while <b>oldest</b> is the member with the
+///         lowest up-number, which is what <c>ClusterSingletonManager</c> uses to place singletons.
+///     </para>
+///     <para>
+///         They coincide on a freshly-formed cluster, which is why every single-node and happy-path
+///         test missed it and why the leader-based check survived to 0.2.1. They diverge the moment a
+///         node restarts: the restarted node rejoins as the youngest while keeping its address, so if
+///         it holds the lower address it becomes leader while the singletons — and every piece of work
+///         they own — stay on the other node. This fixture reproduces that state directly instead of
+///         waiting for a restart.
+///     </para>
+/// </remarks>
+public sealed class ClusterActiveNodeTests : IAsyncLifetime
+{
+    // Fixed, unusual ports: the test needs a deterministic address ordering, which port 0 cannot give.
+    // Distinct from OtOpcUa's equivalent fixture (19_530/19_531) so the two suites can run at once.
+    private const int OldestPort = 19_541;   // joins FIRST  -> oldest, but the HIGHER address
+    private const int YoungestPort = 19_540; // joins SECOND -> cluster leader, the LOWER address
+
+    private const string SystemName = "health-active-node";
+
+    private ActorSystem? _oldest;
+    private ActorSystem? _youngest;
+
+    private static Config NodeConfig(int port, string roles) => ConfigurationFactory.ParseString($$"""
+        akka {
+            loglevel = "WARNING"
+            actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
+            remote.dot-netty.tcp {
+                hostname = "127.0.0.1"
+                port = {{port}}
+            }
+            cluster {
+                seed-nodes = ["akka.tcp://{{SystemName}}@127.0.0.1:{{OldestPort}}"]
+                roles = [{{roles}}]
+                min-nr-of-members = 1
+                run-coordinated-shutdown-when-down = off
+                downing-provider-class = ""
+            }
+        }
+        """);
+
+    public async Task InitializeAsync()
+    {
+        // Order matters: the seed forms the cluster and is therefore the oldest member. The oldest
+        // node is fused (admin + driver); the younger one is driver-only, so the role-scoped
+        // assertions below can distinguish "oldest of this role" from "oldest overall".
+        _oldest = ActorSystem.Create(SystemName, NodeConfig(OldestPort, "\"admin\", \"driver\""));
+        await WaitForUpAsync(_oldest, expectedMembers: 1);
+
+        _youngest = ActorSystem.Create(SystemName, NodeConfig(YoungestPort, "\"driver\""));
+        await WaitForUpAsync(_oldest, expectedMembers: 2);
+        await WaitForUpAsync(_youngest, expectedMembers: 2);
+    }
+
+    public async Task DisposeAsync()
+    {
+        if (_youngest is not null) await _youngest.Terminate();
+        if (_oldest is not null) await _oldest.Terminate();
+    }
+
+    private static async Task WaitForUpAsync(ActorSystem system, int expectedMembers)
+    {
+        var cluster = Cluster.Get(system);
+        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
+        while (DateTime.UtcNow < deadline)
+        {
+            if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expectedMembers) return;
+            await Task.Delay(100);
+        }
+
+        throw new TimeoutException(
+            $"cluster on port {cluster.SelfAddress.Port} never saw {expectedMembers} Up members "
+            + $"(saw {cluster.State.Members.Count(m => m.Status == MemberStatus.Up)})");
+    }
+
+    /// <summary>
+    /// Sanity check on the fixture itself: if the two orderings did not actually diverge, every
+    /// assertion below would pass for the wrong reason.
+    /// </summary>
+    [Fact]
+    public void Fixture_actually_produces_divergent_age_and_address_ordering()
+    {
+        var state = Cluster.Get(_oldest!).State;
+
+        Assert.Equal(YoungestPort, state.Leader!.Port);
+        Assert.Equal(YoungestPort, state.RoleLeader("driver")!.Port);
+    }
+
+    [Fact]
+    public void OldestUpMember_IsTheOldest_NotTheLeader()
+    {
+        var members = Cluster.Get(_oldest!).State.Members;
+
+        Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members)!.Address.Port);
+        Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "driver")!.Address.Port);
+    }
+
+    [Fact]
+    public void OldestUpMember_IsRoleScoped()
+    {
+        var members = Cluster.Get(_oldest!).State.Members;
+
+        // Only the fused node carries admin, so it owns the admin slot even though it is not the
+        // lowest-addressed member.
+        Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "admin")!.Address.Port);
+
+        // A role nobody carries has no owner.
+        Assert.Null(ClusterActiveNode.OldestUpMember(members, "no-such-role"));
+    }
+
+    [Fact]
+    public void SelfIsActive_IsTrueOnlyOnTheOldestNode()
+    {
+        Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!)));
+        Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!)));
+    }
+
+    [Fact]
+    public void SelfIsActive_RoleScoped_YoungestOwnsNoSlot()
+    {
+        // Driver-scoped: the fused node is still the oldest driver, so the driver-only node is standby.
+        Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!), "driver"));
+        Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "driver"));
+
+        // Admin-scoped: the driver-only node does not carry admin, so it can never be the admin active
+        // node — even though it is the cluster leader.
+        Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "admin"));
+    }
+
+    [Fact]
+    public async Task HealthCheck_IsHealthyOnTheOldestAndUnhealthyOnTheLeader()
+    {
+        var onOldest = await CheckAsync(_oldest!, new ActiveNodeHealthCheckOptions());
+        var onYoungest = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
+
+        Assert.Equal(HealthStatus.Healthy, onOldest.Status);
+
+        // The node the old leader-based check would have called active. Exactly one 200 per cluster is
+        // the property an orchestrator's active-tier routing depends on.
+        Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
+    }
+
+    [Fact]
+    public async Task HealthCheck_DataNamesTheActiveNode_EvenFromAStandby()
+    {
+        // A standby reporting *who* is active is what lets a dashboard render the pair from either
+        // node's payload alone (the 0.2.0 per-entry data object).
+        var result = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
+
+        Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{OldestPort}", result.Data["activeNode"]);
+        Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{YoungestPort}", result.Data["selfAddress"]);
+    }
+
+    [Fact]
+    public async Task HealthCheck_RolePreference_FusedNodeAnswersForAdmin()
+    {
+        var options = new ActiveNodeHealthCheckOptions { RolePreference = ["admin", "driver"] };
+
+        var onOldest = await CheckAsync(_oldest!, options);
+        var onYoungest = await CheckAsync(_youngest!, options);
+
+        Assert.Equal(HealthStatus.Healthy, onOldest.Status);
+        Assert.Equal("admin", onOldest.Data["activeRole"]);
+
+        // Resolves to driver (it has no admin role) and loses to the older fused node.
+        Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
+        Assert.Equal("driver", onYoungest.Data["activeRole"]);
+    }
+
+    [Fact]
+    public async Task HealthCheck_NodeOwningNoneOfThePreferredRoles_IsUnhealthyByDefault()
+    {
+        var options = new ActiveNodeHealthCheckOptions { RolePreference = ["no-such-role"] };
+
+        var result = await CheckAsync(_oldest!, options);
+
+        // Not "not applicable ⇒ Healthy": a node that owns no active work must stay out of the pool.
+        Assert.Equal(HealthStatus.Unhealthy, result.Status);
+        Assert.Contains("carries none of the active roles", result.Description);
+        Assert.DoesNotContain("activeRole", result.Data.Keys);
+    }
+
+    [Fact]
+    public async Task HealthCheck_NoActiveRoleStatus_IsOverridable()
+    {
+        var options = new ActiveNodeHealthCheckOptions
+        {
+            RolePreference = ["no-such-role"],
+            NoActiveRoleStatus = HealthStatus.Healthy,
+        };
+
+        Assert.Equal(HealthStatus.Healthy, (await CheckAsync(_oldest!, options)).Status);
+    }
+
+    [Fact]
+    public void Gate_IsActiveOnlyOnTheOldestNode()
+    {
+        Assert.True(new AkkaActiveNodeGate(ProviderFor(_oldest!)).IsActiveNode);
+        Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!)).IsActiveNode);
+
+        // Role-scoped: the driver-only node cannot gate admin work.
+        Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!), "admin").IsActiveNode);
+    }
+
+    private static IServiceProvider ProviderFor(ActorSystem system) =>
+        new ServiceCollection().AddSingleton(system).BuildServiceProvider();
+
+    private static async Task<HealthCheckResult> CheckAsync(
+        ActorSystem system,
+        ActiveNodeHealthCheckOptions options)
+    {
+        var check = new ActiveNodeHealthCheck(ProviderFor(system), options);
+        var context = new HealthCheckContext
+        {
+            Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
+        };
+        return await check.CheckHealthAsync(context);
+    }
+}

akka_failover.md

@@ -0,0 +1,531 @@
+# Akka.NET HA, Failover & Singletons — ScadaBridge + OtOpcUa Deep Dive
+
+*Written 2026-07-22 against ScadaBridge `main @ 69b3ccfc` and OtOpcUa (`lmxopcua`) `master @ 0de1352e`.
+Both trees contain the 2026-07-21 downing/election corrections (ScadaBridge `cf3bd52f`; OtOpcUa `2964361a` auto-down + `50b55ee4` oldest-driver election — note `0de1352e` itself is the adjacent NU1903 dependency pin, not the redundancy fix).*
+
+**Design priority (stated 2026-07-22):** for the 2-node site pairs, a network-partition
+split-brain is *less* risky than failing to bring the second node up quickly. HA goal:
+either node can be launched first, automatic failover, no manual intervention. The
+architecture below is evaluated against that priority throughout.
+
+---
+
+## 1. How Akka.NET handles failover — the fundamentals both apps build on
+
+The deployment unit at every site is a **2-node Akka.NET cluster** (one node per Windows
+VM). Central is also a 2-node cluster, with SQL Server; sites have no SQL Server.
+Everything HA-related decomposes into five Akka mechanisms:
+
+### 1.1 Membership and seed nodes
+A cluster forms by gossip. Each node lists `seed-nodes`; a booting node sends `InitJoin`
+to the seeds. **Only the FIRST seed in the list may join itself to form a *new* cluster** —
+any other node waits until some existing member answers. Both apps set
+`min-nr-of-members = 1`, so a single node (if it's allowed to form) goes `Up` alone and
+hosts all singletons — the pair does not need both members to become operational.
+
+### 1.2 Failure detection (the heartbeat)
+Every node heartbeats its peers and feeds the intervals into a **Phi Accrual failure
+detector**. When the suspicion level crosses the threshold, the peer is marked
+**Unreachable** — a suspicion, not a removal. Both apps run the same values:
+
+| Setting | Value (both apps) | Meaning |
+|---|---|---|
+| `failure-detector.heartbeat-interval` | **2 s** | Cluster-membership heartbeat rate |
+| `failure-detector.threshold` | 10.0 | Phi threshold (suspicion sensitivity) |
+| `failure-detector.acceptable-heartbeat-pause` | **10 s** | Tolerated silence before Unreachable |
+| `transport-failure-detector` (remoting layer) | 2 s / 10 s pause | Detects dead TCP associations |
+
+So a hard-killed peer is declared Unreachable roughly **10–12 s** after death. These are
+exactly the "heartbeat with a small adjustable timeout" — in ScadaBridge they are bound
+from config (`Cluster:HeartbeatInterval`, `Cluster:FailureDetectionThreshold` →
+`ClusterOptions`, rendered into HOCON by `AkkaHostedService.BuildHocon`); in OtOpcUa they
+live in `akka.conf` (lines 68–72) in the Cluster library.
+
+### 1.3 Downing (the decision to act)
+Unreachable alone changes nothing — the leader cannot make membership decisions while any
+member is unreachable, singletons do not move, the cluster is wedged. Something must
+**Down** the unreachable member. This is the strategy choice, and it is the whole
+availability-vs-partition-safety trade:
+
+- **Auto-downing** (`Akka.Cluster.AutoDowning` + `auto-down-unreachable-after`): after the
+  window, the leader *among the reachable members* downs the unreachable one. In a 1-vs-1
+  split, **each side downs the other** — the crash case fails over perfectly; a genuine
+  partition produces dual-active.
+- **Split Brain Resolver `keep-oldest` (+ `down-if-alone`)**: keeps the side containing
+  the oldest member. Partition-safe, but **fatally broken for a 2-node pair**: in
+  Akka.NET 1.5.62, `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side
+  holding ≥ 2 members (`otherSide == 1 && thisSide >= 2`). A 1-vs-1 survivor whose peer
+  (the oldest) crashed takes `DownReachable` and **downs itself** — proven live:
+  `SBR took decision Akka.Cluster.SBR.DownReachable … including myself, [1] unreachable of [2] members`.
+  Net effect: the pair could not survive a crash of the oldest node — total outage.
+
+**Both apps switched their default from `keep-oldest` to `auto-down` on 2026-07-21**
+(ScadaBridge `cf3bd52f`, decision record
+`ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`; OtOpcUa
+`2964361a`, same fix ported). The owner rationale recorded in the decision doc is
+verbatim the priority stated above: *"network partitions are less of a risk than if this
+stops working."*
+
+Rejected alternatives (from the decision record): `static-quorum(1)` → `DownAll` (worse);
+`static-quorum(2)` → survivor self-downs; `keep-majority` → keeps the lowest-address side,
+same fatal crash re-keyed; `lease-majority` → needs a shared lease store (no SQL/K8s at
+sites); a third arbiter node → no third VM at sites; a custom downing provider → would
+reimplement AutoDowning.
+
+### 1.4 Cluster singletons (what actually "fails over")
+`ClusterSingletonManager` runs on every node of the (optionally role-filtered) cluster
+and guarantees exactly one instance of the actor, hosted on the **oldest member** of that
+role. `ClusterSingletonProxy` routes to wherever it currently lives. Failover semantics:
+
+- **Graceful shutdown** (service stop, redeploy): hand-over protocol — the old host
+  transfers to the new oldest with no dead window beyond hand-over latency. Both apps
+  hook `CoordinatedShutdown` (`run-by-clr-shutdown-hook = on`; ScadaBridge additionally
+  drains each singleton via a `PhaseClusterLeave` `GracefulStop` task, 10 s per singleton,
+  `phases.cluster-leave.timeout = 15s`).
+- **Hard crash**: the singleton is simply gone until the dead node is **Downed**; then the
+  new oldest member's manager instantiates a fresh instance. This is why downing is the
+  linchpin — no downing decision, no singleton recovery, ever.
+
+### 1.5 Oldest-member semantics — the family-wide rule
+Both codebases converged on the same insight: **Akka's "leader" (lowest address) and
+"oldest member" (singleton placement) diverge permanently after any node restart** — the
+restarted node rejoins as a *new incarnation*, becoming the youngest, but may still have
+the lowest address and thus be leader. Any "active node" decision keyed on
+leader/lowest-address can therefore point at a node that is *not* hosting the singletons.
+Both apps now derive "active/primary" strictly from **oldest Up member** (ScadaBridge
+`ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa `RedundancyStateActor.SelectDriverPrimary`
+with `Member.AgeOrdering`), which by construction matches singleton placement.
+
+---
+
+## 2. The failover timeline (and the knobs that tune it)
+
+End-to-end failover after a hard node death, with current production values:
+
+```
+t=0        node dies (process crash / VM loss)
+t≈10-12s   failure detector: peer marked Unreachable        ← acceptable-heartbeat-pause (10s)
+t≈25-27s   auto-down window expires; survivor downs peer    ← auto-down-unreachable-after (15s)
+t≈25-30s   survivor is now oldest; singletons instantiate;
+           primary/active gates flip; ServiceLevel 250 moves
+```
+
+Measured, not theoretical: ScadaBridge's 2026-07-21 docker drill measured **28 s**
+(active-node crash → standby takeover) and **27 s** (standby crash → removal, zero routing
+blips); the in-process `FailoverTimingTests` measured 33.7 s for a full cycle.
+
+**Tuning for faster failover** (per the availability-first priority):
+
+| Knob | Where | Current | Effect of lowering |
+|---|---|---|---|
+| `acceptable-heartbeat-pause` | SB: `Cluster:FailureDetectionThreshold`; OtOpcUa: `akka.conf` | 10 s | Faster Unreachable, but < ~5 s risks false positives from GC pauses / Windows VM scheduling stalls / snapshot freezes |
+| `heartbeat-interval` | SB: `Cluster:HeartbeatInterval`; OtOpcUa: `akka.conf` | 2 s | More samples → detector converges faster; cheap to lower to 1 s |
+| Auto-down window | SB: `Cluster:StableAfter`; OtOpcUa: `DowningStableAfter` const (15 s) + emitted HOCON | 15 s | Direct 1:1 reduction of failover time. Every second cut is a second of dual-active risk during a *transient* blip (link flap, VM pause) — the window exists to let transients heal before the irreversible Down |
+| `down-removal-margin` (OtOpcUa akka.conf) | 15 s | Delay before rebalancing after removal | Keep ≈ downing window |
+
+Constraint pinned by an OtOpcUa test (`Downing_window_exceeds_the_acceptable_heartbeat_pause`):
+the downing window **must exceed** `acceptable-heartbeat-pause`, otherwise a node can be
+downed on a pause the detector was configured to tolerate. A realistic aggressive profile
+for the site pairs would be ~5 s pause + ~7–8 s window ≈ **12–13 s total failover**,
+at the cost of downing a peer that stalls (not dies) for > ~12 s — acceptable per the
+stated priority, but worth validating against real Windows VM GC/backup/snapshot pauses
+before rollout. In ScadaBridge these are plain per-deployment config values; in OtOpcUa
+the 15 s window is currently a constant in `ServiceCollectionExtensions.cs` (only the
+*strategy* is config-selectable), so making the window configurable would be a small
+change there.
+
+**What happens to the dead node's side:** both apps set
+`run-coordinated-shutdown-when-down = on` — a node that gets Downed (e.g. it was merely
+partitioned and the other side downed it) terminates its own ActorSystem. Both apps run a
+termination watchdog (ScadaBridge in `AkkaHostedService`; OtOpcUa
+`ActorSystemTerminationWatchdog`) that distinguishes intentional stop from self-down and
+calls `StopApplication()`, so the **process exits and the service supervisor restarts it**
+(Windows: `sc.exe failure … restart/5000/restart/30000/restart/60000`; Docker:
+`restart: unless-stopped`). The restarted process rejoins as a fresh incarnation —
+youngest member, i.e. it comes back as the **standby**, never usurping the survivor.
+
+---
+
+## 3. ScadaBridge
+
+### 3.1 Cluster formation
+All HOCON is generated in code — `AkkaHostedService.BuildHocon(...)`
+(`src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:258–329`) from
+`ClusterOptions`/`NodeOptions` (`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`).
+Key facts:
+
+- **Seed nodes**: central docker pair → `scadabridge-central-a:8081` then
+  `…-central-b:8081` (both nodes list `central-a` first). Site pairs analogous on 8082.
+- **`MinNrOfMembers = 1`** (validator *requires* exactly 1) — one node forms the cluster
+  and brings up all singletons.
+- **Roles**: central node = `["Central"]`; site node = `["Site", "site-{SiteId}"]`
+  (`BuildRoles`, line 406). The role drives the entire composition branch in `Program.cs`.
+- **Ports**: central remoting 8081; site remoting 8082, gRPC 8083, metrics 8084.
+
+### 3.2 Downing configuration
+Configurable via **`Cluster:SplitBrainResolverStrategy`**, allowed values `auto-down` |
+`keep-oldest` (case-insensitive; validator `ClusterOptionsValidator`). **Default and all
+16 appsettings files: `auto-down`.** The HOCON branch (`AkkaHostedService.cs:275–286`):
+
+```csharp
+var downingBlock = string.Equals(strategy, "auto-down", OrdinalIgnoreCase)
+    ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
+        auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
+    : /* SBR provider + split-brain-resolver { active-strategy, stable-after,
+         keep-oldest { down-if-alone } } */;
+```
+
+`StableAfter = 00:00:15` doubles as the SBR `stable-after` and the auto-down window.
+`DownIfAlone` is now validated only under `keep-oldest` (inert under auto-down).
+`keep-oldest` remains supported and SBR-path tests still pin it
+(`SbrFailoverTests.HardCrashOfYoungerNode_SbrDownsIt_AndOldestKeepsSingleton`).
+
+### 3.3 Cluster singletons
+All registered through `SingletonRegistrar.Start(...)` (names `{name}-singleton` /
+`{name}-proxy`, PoisonPill termination, optional role, per-singleton
+`PhaseClusterLeave` drain task of 10 s).
+
+**Central — 7 singletons** (role not further scoped; central nodes are all `Central`):
+
+| Singleton | Actor | Purpose |
+|---|---|---|
+| `notification-outbox` | `NotificationOutboxActor` | Notification ingest/dispatch/purge |
+| `audit-log-ingest` | `AuditLogIngestActor` | Central audit ingest from sites |
+| `site-call-audit` | `SiteCallAuditActor` | SiteCalls upserts + central→site retry/discard relay |
+| `audit-log-purge` | `AuditLogPurgeActor` | Daily partition-switch purge |
+| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Per-site audit reconciliation pull |
+| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI sampling (not readiness-gated) |
+| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Expired staging-row reclaim (not readiness-gated) |
+
+`RequiredSingletonsHealthCheck` probes the first five via bounded `Identify` for
+`/health/ready`; the last two are deliberately best-effort.
+
+**Site — 2 singletons**, role-scoped to `site-{SiteId}` so each site pair elects its own:
+
+| Singleton | Actor | Purpose |
+|---|---|---|
+| `deployment-manager` | `DeploymentManagerActor` | Deployment lifecycle; drains in-flight SQLite writes on hand-over |
+| `event-log-handler` | `EventLogHandlerActor` | Event-log queries always served by the active node (event log is node-local, not replicated) |
+
+Deliberately **not** singletons: `dcl-manager` (DataConnectionManagerActor) and
+`CertStoreActor` run on **every** site node — the deployment singleton fans PKI trust
+changes to both nodes' cert stores so a failover never finds a stale trust store.
+
+### 3.4 Active-node gating
+One definition of "active": `ActiveNodeEvaluator.SelfIsOldestUp`
+(`src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs`) — oldest
+Up member, explicitly **not** `cluster.State.Leader` (the XML docs spell out the
+leader-vs-oldest divergence after restarts). Consumers:
+
+- **Central inbound API**: `IActiveNodeGate` → `InboundApiEndpointFilter` returns
+  **HTTP 503 `{code:"STANDBY_NODE"}`** on the standby; Traefik routes on
+  `/health/active` (`OldestNodeActiveHealthCheck` + `DatabaseHealthCheck`), so the proxy
+  and the API agree on which node is active.
+- **Site store-and-forward**: `storeAndForwardService.SetDeliveryGate(() =>
+  clusterNodeProvider.SelfIsPrimary)` — only the active node runs the delivery sweep;
+  without the gate both nodes would deliver the same replicated `sf_messages` rows
+  (systematic duplicate external calls). Re-evaluated per sweep tick, so delivery resumes
+  within one `RetryTimerInterval` after failover.
+- **Heartbeat to central**: `SiteCommunicationActor` stamps `HeartbeatMessage.IsActive`
+  from the same evaluator (exception-safe, defaults to `false`).
+
+Takeover is purely membership-driven: peer Downed → survivor becomes oldest →
+`SelfIsOldestUp` flips → singletons, S&F gate, `/health/active`, heartbeat all follow.
+
+### 3.5 Site state without SQL Server — LocalDb
+Sites persist everything in **one consolidated SQLite file** (`LocalDb:Path`, required,
+on the data volume) via `ZB.MOM.WW.LocalDb` — 10 replicated tables: `OperationTracking`,
+`site_events`, `sf_messages` (the store-and-forward buffer), and 7 config tables
+(deployed configurations, static overrides, shared scripts, external systems, DB
+connections, DCL definitions, native alarm state).
+
+**Pair replication** (default OFF; opt-in `LocalDb:Replication:PeerAddress` + matching
+`ApiKey` both sides; fail-closed auth interceptor; rides the existing gRPC 8083 listener):
+CDC triggers capture row mutations, shipped bidirectionally under HLC last-writer-wins;
+snapshot resync merges per row and never deletes. **This is what makes site failover
+real**: the standby already holds the S&F buffer and the full config, so on takeover it
+just opens its delivery gate over data it already has — zero cross-node fetch (the old
+notify-and-fetch deploy path is deleted).
+
+Not replicated (by design): the event log (hence the `event-log-handler` singleton),
+in-memory alarm state (re-evaluated from incoming values on the survivor), and
+`notification_lists`/`smtp_configurations` (would ship plaintext SMTP passwords).
+
+**Operational constraints:**
+- **Stop/start a site pair TOGETHER.** Rolling upgrades are unsupported: Phase 2 deleted
+  the legacy replicator and its `SfBufferSnapshot` compat handler, so a mixed-version
+  pair runs but silently does not converge.
+- A node offline longer than `TombstoneRetention` (7 d) can **resurrect deleted rows** on
+  rejoin; within the window rejoin is safe and self-correcting.
+- `MaxBatchSize` counts **rows, not bytes** — rig pins 16 (the 500 default could build a
+  ~35 MB batch against the 4 MB gRPC cap).
+
+### 3.6 Verified behavior + remaining gaps
+- Auto-down failover drill (docker `failover-drill.sh`, SIGKILL): active-crash takeover
+  28 s, standby-crash 27 s, 0 routing blips; both drill modes now *assert* recovery.
+- After a real partition the two mutually-downed halves **do not merge** (quarantined
+  association) — an operator restarts one side, which rejoins as standby. This is the
+  accepted dual-active trade.
+- **The old "restart central-a and central-b together (SBR self-down)" gotcha is closed**
+  by auto-down — but a related constraint survives with a different mechanism: the
+  **seed-node bootstrap constraint** (§6.1).
+- ⚠ Stale doc: `docs/deployment/topology-guide.md:101` still says keep-oldest +
+  down-if-alone. Authoritative: `Component-ClusterInfrastructure.md`, the auto-down
+  decision record, `docker/README.md`.
+
+---
+
+## 4. OtOpcUa
+
+### 4.1 Cluster formation
+Base HOCON in the library (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf`):
+system `otopcua`, remoting port 4053, `min-nr-of-members = 1`, empty seeds/roles overlaid
+at runtime from `AkkaClusterOptions` (`Cluster:*` config; roles restricted to
+`admin` | `driver` | `dev` by `RoleParser`). Bootstrap via Akka.Hosting:
+`WithOtOpcUaClusterBootstrap` (remoting + clustering + the downing HOCON, Prepended).
+Note the dual role plumbing: `OTOPCUA_ROLES` drives DI composition (`hasAdmin`/`hasDriver`
+branches in `Program.cs`), while `Cluster__Roles__*` drives the actual Akka member roles —
+docker-dev sets both (a past production incident came from setting only `OTOPCUA_ROLES`,
+leaving the Akka member role-less).
+
+**Current mesh shape (important):** today OtOpcUa runs **one Akka mesh for the whole
+fleet** — docker-dev is six nodes (central-1/2 = `admin,driver`; site-a-1/2, site-b-1/2 =
+`driver`) all seeded by `central-1`, with logical MAIN/SITE-A/SITE-B separation done by
+`ServerCluster.ClusterId` rows in the shared ConfigDb, not by separate meshes. The
+per-cluster mesh design (§4.4) will change this to one mesh per 2-node pair — the shape
+that matches your site deployment.
+
+### 4.2 Downing configuration
+Same key as ScadaBridge: **`Cluster:SplitBrainResolverStrategy`**, default **`auto-down`**,
+allowed `auto-down` | `keep-oldest`, anything else **fails the host at startup**
+(`ArgumentOutOfRangeException` — no silent fallback; pinned by
+`Unknown_strategy_throws_rather_than_silently_falling_back`). `BuildDowningHocon` emits:
+
+```
+akka.cluster {
+    downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
+    auto-down-unreachable-after = 15000ms
+}
+```
+
+(the comment notes `auto-down-unreachable-after` defaults to `off` upstream and is
+load-bearing — AutoDowning without it never downs anything). Under `keep-oldest` the
+typed `KeepOldestOption { DownIfAlone = true }` installs the SBR instead; the
+`split-brain-resolver` block in akka.conf is inert under the default. Tests read the
+**effective config off a running ActorSystem** (not the input HOCON) — provider class,
+15 s window, case-insensitivity, and the window-exceeds-heartbeat-pause invariant.
+
+**Live gate still open:** the 1-vs-1 oldest-crash pathology cannot be exercised on the
+current six-node docker-dev mesh; the crash-the-oldest drill is deferred to the
+per-cluster mesh work (Phase 6/7). Supporting live evidence exists from a 2-container
+kill test (`archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`) and
+from ScadaBridge's identical fix. Cautionary note recorded in `docs/Redundancy.md`:
+the pre-existing `HardKillFailoverTests` stayed green under broken keep-oldest because it
+shut down the *transport* (ActorSystem alive) rather than killing the process — "a
+standing example of a green test over a fatal defect."
+
+### 4.3 Redundancy primary election + ServiceLevel
+- **`RedundancyStateActor`** (an `admin`-role singleton) computes the fleet snapshot:
+  `SelectDriverPrimary` = oldest **Up** member carrying the `driver` role
+  (`Member.AgeOrdering`; `Leaving` excluded — a leaving node is handing its singletons
+  over and must not be Primary). Debounced 250 ms, republished on a 10 s heartbeat over
+  distributed pub/sub topic `redundancy-state` (DPS doesn't replay to late subscribers).
+- **`NodeRedundancyState`** carries `RedundancyRole { Primary, Secondary, Detached }` and
+  `IsDriverPrimary` (renamed from `IsRoleLeaderForDriver` — "the name now describes what
+  the value means rather than how it used to be computed").
+- **ServiceLevel** (`ServiceLevelCalculator`): health basis 240 (healthy) / 200 (stale) /
+  100 (critically degraded) / 0 (down), **+10 if driver Primary**, so a healthy pair
+  reads **250 / 240**. Published into the OPC UA `Server.ServiceLevel` variable via
+  `SdkServiceLevelPublisher` (deferred publisher until the SDK server exists; first
+  computed value always published so no node lingers at the SDK default 255).
+- **Client-visible failover is OPC UA non-transparent redundancy**: each node has its own
+  `ApplicationUri`; clients read `Server.ServerArray` (self + peers, from
+  `PeerApplicationUris`) and pick the endpoint with the highest ServiceLevel. The shipped
+  client (`OpcUaClientService`) does this: failover URL list, KeepAlive-failure trigger,
+  single-flight failover guard, data + alarm subscription replay after reconnect.
+  **Consumer note:** because Primary = oldest (not lowest address), after any node
+  restart a ServiceLevel-selecting client may legitimately prefer a different node than
+  before — that's correct behavior, not a bug.
+
+**Primary gating of the data plane (`PrimaryGatePolicy`)** — the S4 default-deny fix:
+
+```csharp
+localRole switch {
+    Primary                =&gt; true,
+    Secondary or Detached  =&gt; false,
+    _ /* unknown */        =&gt; driverMemberCount &lt;= 1,  // multi-driver: DENY until proven
+};
+```
+
+During the boot window (role not yet known) a multi-driver node **rejects device writes**
+with `not primary (role unknown)` (denial meter `reason=role-unknown`); a single-driver
+node stays default-allow. Gated surfaces: device writes, native-alarm acks, alarm-alert
+fan-out. A deliberately *different* policy governs alarm-history drain: unknown → drain
+anyway ("a duplicate row beats a silent gap"), keyed on whether the Primary shares this
+node's queue (pair-local). Live-proven on a 2-node docker rig (R2-04 T15): ServiceLevel
+250/240, boot-window write rejected, steady-state secondary rejected + reverted, primary
+write reaches the device.
+
+Failover sequence when the primary dies: peer downed (auto-down) → survivor is oldest
+driver → `SelectDriverPrimary` flips → snapshot republished → gates open on the survivor →
+ServiceLevel 250 moves → OPC UA clients see the old endpoint die (KeepAlive) and re-select
+by ServiceLevel. Singletons migrate to the same node the election names, because both
+derive from member age.
+
+### 4.4 Singletons and the per-cluster mesh gap
+Five control-plane singletons, all role-pinned to `admin`:
+
+| Singleton | Actor | Purpose |
+|---|---|---|
+| `config-publish` | `ConfigPublishCoordinator` | Persists per-node deploy ACKs |
+| `admin-operations` | `AdminOperationsActor` | Operator ack/shelve; TagConfig deploy gate |
+| `audit-writer` | `AuditWriterActor` | Audit persistence |
+| `fleet-status` | `FleetStatusBroadcaster` | AdminUI fleet panel |
+| `redundancy-state` | `RedundancyStateActor` | Primary election + DPS publisher |
+
+**KNOWN LIMITATION:** the Primary election is scoped **per Akka cluster, not per
+application Cluster**. On the one-mesh fleet, `SelectDriverPrimary` names exactly ONE
+Primary among *all* driver nodes (MAIN + SITE-A + SITE-B), while every gated resource is
+pair-local. The per-cluster mesh design
+(`OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`) fixes this by construction:
+one Akka mesh per application Cluster (two nodes max), central↔cluster joined by explicit
+transports (modeled on ScadaBridge's ClusterClient/gRPC/HTTP trio), so "am I the driver
+Primary?" and "the resource I'm gating" have the same scope. Status: Phases 0a (auto-down)
+and 0b (oldest-Up election) DONE and merged; Phases 1–7 (ClusterNode columns, comm actors,
+config fetch-and-cache, cut driver-side ConfigDb, gRPC stream contract, mesh partition,
+failover drill/live gate) not started — **but now sequenced by a program plan
+(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, 2026-07-22), driven by the
+decision that OtOpcUa pairs will run on the SAME Windows VMs as the ScadaBridge nodes** —
+two independent, identically-postured 2-node clusters per site, and driver nodes off the
+ConfigDb entirely (sites have no SQL Server).
+
+### 4.5 Surviving central-SQL loss — availability guarantees
+Two mechanisms make a driver pair independent of central SQL Server availability:
+
+- **Address-space availability guarantee (#485/#486):** an artifact the node cannot read
+  is *no answer*, never *an empty configuration*. Previously a transient ConfigDb error
+  mid-deploy parsed zero bytes into an empty composition → the planner diffed it as a
+  `PureRemove` of every node → served address space emptied, drivers stopped,
+  subscriptions cleared, deploy still reported Applied. Now `OpcUaPublishActor` /
+  `DriverHostActor` hold last-known-good address space, drivers, and subscriptions, and
+  report the deploy **Failed** without advancing the revision so the retry lands.
+- **LocalDb Phase 1 boot-from-cache:** every driver node caches the chunked, SHA-256
+  verified deployed-configuration artifact in a pair-local SQLite DB; a node restarting
+  into a central-SQL outage boots from the last artifact it applied. With opt-in pair
+  replication (`deployment_artifacts` + `deployment_pointer`, HLC last-writer-wins,
+  fail-closed auth) either node holds a byte-identical copy even if it never applied that
+  deploy itself. Scope: config artifact only — no live tags/alarms/historian data.
+
+---
+
+## 5. Dual-active: what it concretely means here
+
+Under auto-down, a *genuine* partition (both nodes alive, link cut) ends with each side
+downing the other and running active until an operator restarts one side (the mutual
+downing quarantines the association — the sides will not self-merge). Accepted per the
+availability-first decision; concrete blast radius:
+
+**ScadaBridge site pair:** both nodes' `SelfIsOldestUp` = true → both open the S&F
+delivery gate → **duplicate external deliveries** of the same replicated `sf_messages`
+rows for the partition's duration; both stamp `IsActive` heartbeats. LocalDb replication
+itself is partition-tolerant (HLC last-writer-wins reconverges on heal, after the
+operator restart). Central pair: both serve `/health/active` = 200 and run duplicate
+singleton sets against the same SQL Server — duplicated sweeps/purges (mostly idempotent
+upserts), and Traefik may alternate between two "active" backends.
+
+**OtOpcUa pair:** both nodes elect themselves driver Primary → both publish
+**ServiceLevel 250** → clients on either side keep their session; both sides accept
+device writes → **two masters against the same field devices** for the duration. This is
+the classic OT dual-primary risk — mitigated by rarity (2 VMs, one LAN segment, real
+partitions ≈ switch failure) and by the alternative (keep-oldest) having been proven to
+turn a plain crash into a total outage.
+
+Detection/ops: the survivor logs the downing decision; OtOpcUa emits
+`otopcua.redundancy.service_level_change` and `primary_gate_denied` meters; two
+simultaneous 250s / two 200-responses on `/health/active` are the alarm signal. Recovery
+is always "restart one side; it rejoins as youngest/standby."
+
+---
+
+## 6. Gaps vs the stated HA goal ("either node first, automatic failover")
+
+### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (differently per repo)
+**Automatic failover: achieved.** Either node crashing is survivable, unattended, in
+~27–30 s. The remaining gap — a non-first seed cold-starting alone loops on `InitJoin`
+forever (the "registered outage gap") — was **closed in both repos on 2026-07-22**, but
+execution overturned the original design and the two repos legitimately diverged:
+
+**⚠️ The original design's safety claim was FALSE — proven independently in both repos.**
+The planned `SelfFormAfter` watchdog assumed "window expires mid-join-handshake → benign,
+Akka ignores `Join` once joined." Wrong: the watchdog sits *outside* Akka's join
+handshake and cannot distinguish *no seed answered* from *a seed answered and the join is
+in flight*. On a routine restart into a live peer, the join stalls briefly behind removal
+of the node's own stale incarnation; a `Join(self)` fired in that window **abandons the
+in-flight join and forms a second cluster at the same address — a permanent island**.
+ScadaBridge proved it with a written test (split still unhealed after 90 s); OtOpcUa hit
+it live when its manual-failover drill bounced a node (InitJoinAck received, no Welcome
+inside the window, fallback fired → second cluster).
+
+**ScadaBridge's fix — self-first seed ordering (no runtime code).** Every node lists
+ITSELF as `seed-nodes[0]`, partner second. Akka's own `FirstSeedNodeProcess` then
+implements exactly the intended semantics *inside* the handshake (InitJoin the other
+seeds; self-join only if nobody answers) — race-free by construction. A `StartupValidator`
+rule hard-gates the ordering at boot. Live-proven behaviors: lone cold-start self-joins
+after ~5 s (`seed-node-timeout`); restart into a live peer rejoins (never islands);
+simultaneous reachable cold-start converges to ONE cluster (the old "two clusters that
+never merge" objection was disproved by test); a genuine boot partition dual-forms — the
+accepted auto-down trade. There is **no `SelfFormAfter` option in ScadaBridge** — the
+whole watchdog was rejected in code review.
+
+**OtOpcUa's fix — converged on self-first ordering the same day.** OtOpcUa first shipped
+the watchdog and hit the race live (its manual-failover drill bounced a node; the fallback
+islanded it), patched it with a TCP reachability guard — and then **retired the whole
+watchdog hours later** (`lmxopcua master @ 3f24d4d6`, a breaking change):
+`ClusterBootstrapFallback` and `Cluster:SelfFormAfter` are **deleted**, docker-dev
+`central-2` now lists itself as `SeedNodes__0`, and a new `AkkaClusterOptionsValidator`
+enforces the ordering at boot. Two subtleties its rule handles that ScadaBridge's didn't
+need: the invariant is **conditional** (self must be `seed[0]` only IF self appears in
+the list at all — site nodes seed off central only and are deliberately not seeds), and
+identity compares on **`PublicHostname` + port** (the address Akka puts in `SelfAddress`),
+because matching the `0.0.0.0` bind address would leave the rule silently inert on the
+whole rig. The retirement was proven by a falsifiability control: the old peer-first
+ordering test failed at 11 s against the watchdog — demonstrating the watchdog could
+never usefully fire for the nodes that needed it.
+
+**Result: both repos now share the identical mechanism** — self-first seed ordering,
+validator-enforced, no bootstrap runtime code. The mesh program's Phase 6 convergence
+item is already done; what remains there is only the per-pair seed topology itself.
+
+**Ops action (outstanding):** the gitignored `ScadaBridge/deploy/wonder-app-vd03/`
+overlay must have its `SeedNodes` reordered self-first before its next deploy — the new
+validator rule is a deliberate hard boot gate.
+
+### 6.2 Other open items
+- **OtOpcUa auto-down live gate** still open (needs a real 1-vs-1 rig; deferred to
+  per-cluster mesh Phase 6/7). ScadaBridge's identical mechanism *is* drill-proven.
+- **OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land
+  (§4.4) — only relevant while multiple pairs share one mesh.
+- **OtOpcUa's 15 s downing window is a code constant**, not config — worth making
+  tunable alongside any heartbeat-timeout reduction (ScadaBridge's already is:
+  `Cluster:StableAfter`).
+- **ScadaBridge site pairs: stop/start together** for upgrades (no mixed-version LocalDb
+  replication); ≥ 7-day-offline nodes risk tombstone resurrection.
+- **Stale doc**: `ScadaBridge/docs/deployment/topology-guide.md:101` still describes
+  keep-oldest.
+
+---
+
+## 7. Quick reference — file map
+
+| Concern | ScadaBridge | OtOpcUa |
+|---|---|---|
+| HOCON / downing emit | `Host/Actors/AkkaHostedService.cs:258–329` | `Cluster/ServiceCollectionExtensions.cs:119–146` + `Cluster/Resources/akka.conf` |
+| Options + validation | `ClusterInfrastructure/ClusterOptions.cs`, `ClusterOptionsValidator.cs` | `Cluster/AkkaClusterOptions.cs`, `RoleParser.cs` |
+| Strategy key | `Cluster:SplitBrainResolverStrategy` (default `auto-down`) | same key, same default; unknown value = startup failure |
+| Active/primary definition | `Communication/ClusterState/ActiveNodeEvaluator.cs` (oldest Up) | `ControlPlane/Redundancy/RedundancyStateActor.cs` (oldest Up `driver`) |
+| Data-plane gating | `InboundApiEndpointFilter` (503 STANDBY_NODE), S&F `SetDeliveryGate` | `Runtime/Drivers/PrimaryGatePolicy.cs` + `DriverHostActor` gates |
+| Singleton registration | `Host/Actors/SingletonRegistrar.cs` (7 central + 2 site) | `ControlPlane/ServiceCollectionExtensions.cs:44–94` (5, role `admin`) |
+| Client-visible failover | Traefik on `/health/active`; gRPC A/B flip | OPC UA ServiceLevel 250/240 + `Server.ServerArray`; client KeepAlive failover |
+| Self-down recovery | watchdog in `AkkaHostedService` → process exit → supervisor | `Host/ActorSystemTerminationWatchdog.cs` → same |
+| Decision records | `docs/plans/2026-07-21-auto-down-availability-decision.md`, `docs/requirements/Component-ClusterInfrastructure.md` | `docs/Redundancy.md` §"Split-brain / downing", `docs/plans/2026-07-21-per-cluster-mesh-design.md` |
+| Failover drills / tests | `docker/failover-drill.sh` (28 s/27 s), `SbrFailoverTests`, `FailoverTimingTests` | `SplitBrainResolverActivationTests`, `RedundancyPrimaryElectionTests`, R2-04 T13/T15 live gate |

docs/plans/2026-07-22-initjoin-selfform-fallback.md

@@ -0,0 +1,104 @@
+# InitJoin Self-Form Fallback (`Cluster:SelfFormAfter`) + Manual Failover Control — Shared Design & Index
+
+> **For Claude:** This is the family-level DESIGN + INDEX document. The executable per-repo plans live in the repos (split 2026-07-22 — the OtOpcUa plan is expected to grow and has a reserved-additions section):
+>
+> - **ScadaBridge:** `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (10 tasks: fallback 1–7, manual-failover UI 8–9, optional site-pair failover 10)
+> - **OtOpcUa:** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (7 tasks + "Reserved for additions" section)
+> - **OtOpcUa mesh-alignment PROGRAM (added 2026-07-22):** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md` — Phases 1–7 moving OtOpcUa to ScadaBridge's mesh shape (one 2-node Akka mesh per application Cluster, ClusterClient/gRPC/fetch transports, driver nodes off the ConfigDb), driven by the co-location decision: OtOpcUa pairs run on the SAME Windows VMs as the ScadaBridge nodes. The selfform plan above is its prerequisite. Design: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`.
+>
+> Execute each with superpowers-extended-cc:executing-plans **in that repo**. The two plans are fully independent and can run in parallel. Only Task C1 below executes from this file.
+
+**Goal:** (1) Either node of a 2-node pair can cold-start alone and become operational, unattended — closing the seed-node bootstrap gap in both ScadaBridge and OtOpcUa via a configurable, fast (default 10 s) InitJoin timeout that falls back to self-forming a cluster. (2) An admin-only "Trigger failover" control on each app's cluster-status page for operator-initiated, graceful role swaps.
+
+> ## ⚠️ EXECUTION OUTCOME (2026-07-22) — design partially overturned
+>
+> Both repo plans executed. **The behavior-spec row "Window expires while a join is mid-handshake → benign race: Akka ignores `Join` once joined" is FALSE** — the watchdog sits outside Akka's join handshake, and a `Join(self)` fired while a join is in flight (routine restart into a live peer, join stalled behind stale-incarnation removal) abandons the join and forms a **permanent second cluster at the same address**. Proven independently by a ScadaBridge test (split unhealed after 90 s) and by an OtOpcUa live island during the manual-failover drill.
+>
+> - **ScadaBridge** rejected the watchdog entirely and shipped **self-first seed ordering** (each node lists itself as `seed-nodes[0]`; Akka's `FirstSeedNodeProcess` provides the intended semantics race-free inside the handshake; `StartupValidator` hard-gates the ordering). No `SelfFormAfter` option exists there. See its plan's "Part 1 as executed".
+> - **OtOpcUa** first kept `SelfFormAfter` behind a TCP reachability guard, then **converged the same day** (`lmxopcua @ 3f24d4d6`): `ClusterBootstrapFallback` + `SelfFormAfter` deleted, self-first ordering shipped with a conditional `AkkaClusterOptionsValidator` (self must be `seed[0]` only IF self is a seed — site nodes exempt; identity on `PublicHostname`+port, not the 0.0.0.0 bind).
+> - **Both repos now share the identical mechanism.** Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
+> - The design sections below are retained as the decision record — read them through this banner. `akka_failover.md` §6.1 carries the corrected family-level account.
+
+**Decision provenance:** `scadaproj/akka_failover.md` §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory `ha-availability-over-partition-safety`). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.
+
+---
+
+## Design
+
+### The defect being fixed
+
+Akka.NET only allows the **first-listed seed** to self-join and form a *new* cluster. Every other node loops on `InitJoin` forever until some existing member answers. Both repos document this as the "registered outage gap": a non-first seed cold-starting while the first seed is down never becomes Up (recovery today = manual env-var override or reviving the first seed). ScadaBridge's `ClusterOptions.SeedNodes` doc comment even claims "either can start first and form the cluster" — which the deployed configs do **not** deliver (both nodes list node-a first). The plans make the claim true and fix the comment.
+
+### Behavior specification
+
+| Scenario | Behavior with fallback |
+|---|---|
+| Peer alive (any boot order) | Normal seed join within ms — fallback never fires (membership arrives before the window) |
+| Lone cold-start, node IS in its own seed list | After `SelfFormAfter` (default 10 s): warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
+| Lone cold-start, node NOT in its own seed list | Fallback inert (info log at arm time); node waits for a seed — self-forming would island it. **This is the OtOpcUa site-node topology** (seeded only by `central-1`). |
+| Peer boots after survivor self-formed | Peer's `InitJoin` reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island. |
+| Both cold-start simultaneously, mutually unreachable (boot-time partition) | Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start. |
+| `SelfFormAfter` = `null` or `<= 0` | Disabled — today's wait-forever behavior |
+| Window expires while a join is mid-handshake | Benign race: Akka ignores `Join` once the node has already joined a cluster |
+
+### Config surface (same key both apps — family convention, like `SplitBrainResolverStrategy`)
+
+- **ScadaBridge:** `ScadaBridge:Cluster:SelfFormAfter` → `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`), validated non-negative-or-null by `ClusterOptionsValidator`.
+- **OtOpcUa:** `Cluster:SelfFormAfter` → `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`); disable-on-`<=0` semantics live in the fallback itself.
+- NOT a code constant (deliberate — see the `DowningStableAfter` lesson, `akka_failover.md` §6.2).
+
+### Mechanism (both apps, deliberately duplicated like the termination watchdogs)
+
+`ClusterBootstrapFallback.Arm(system, options, logger)`, called at ActorSystem creation (ScadaBridge: inside `AkkaHostedService.GetOrCreateActorSystem`; OtOpcUa: an Akka.Hosting `AddStartup` task inside `WithOtOpcUaClusterBootstrap`, so production and test hosts arm identically):
+
+1. `SelfFormAfter` null/`≤0` → log info, return (disabled).
+2. Self address not in own `SeedNodes` (parsed via `Address.Parse`, exact equality) → log info, return (island guard).
+3. `RegisterOnMemberUp` completes a TCS; race it against `Task.Delay(window)`.
+4. On window expiry (and system not terminated): prominent warn log + `Cluster.Join(cluster.SelfAddress)`.
+
+Races are benign — Akka ignores `Join` once joined this incarnation.
+
+### Manual failover control (Part D design)
+
+**Mechanism — graceful `Cluster.Leave`, never `Down`.** "Trigger failover" = issue `Cluster.Leave(<current active/primary member's address>)` from whichever node serves the UI (any member may initiate a leave for any member). The leaving node walks the graceful path: singleton hand-over via the cluster-leave phases (ScadaBridge additionally drains each singleton 10 s), its `CoordinatedShutdown` runs, the existing termination watchdog exits the process, the service supervisor restarts it, and it rejoins as the **youngest** member — a permanent role swap under the oldest-Up election, with no downing window and no dead interval beyond hand-over latency.
+
+Shared rules:
+- **Admin-only:** button renders inside the app's admin-policy `AuthorizeView` (ScadaBridge `AuthorizationPolicies.RequireAdmin` on `/monitoring/health`; OtOpcUa the mutating-pages policy from `AdminUiPolicies` on `/clusters/{id}/redundancy`).
+- **Peer guard:** disabled (with tooltip) when fewer than 2 Up members carry the relevant role.
+- **Confirmation dialog** stating exactly what happens.
+- **Audited** through each app's existing audit seam before the Leave (`cluster.manual-failover`; ScadaBridge uses its CENTRAL audit writer, not the shared seam — dual-seam gotcha).
+- **UX note (ScadaBridge):** Traefik routes the UI to the *active* node, so triggering central failover drops the Blazor circuit; the page reconnects against the new active. The dialog warns about this.
+- **Scope note (OtOpcUa):** until the per-cluster mesh lands, the driver-Primary election is mesh-wide — the button acts on THE Primary of the whole Akka cluster; the panel says so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md`).
+- **Election parity:** each service's target query is intentionally identical to the app's active/primary election (`ActiveNodeEvaluator.SelfIsOldestUp` / `RedundancyStateActor.SelectDriverPrimary`), pinned by a parity test — the node acted on must be exactly the one hosting the singletons.
+- **Self-form interplay:** none — the restarted node rejoins via its peer (alive by definition here), so `SelfFormAfter` never fires on this path.
+
+### Multi-node testing assessment (getakka.net/articles/testing/multi-node-testing.html)
+
+**Verdict: usable in principle, deliberately NOT adopted.** The MultiNode TestKit (`Akka.MultiNode.TestAdapter` + `Akka.Remote.TestKit` + `Akka.Cluster.TestKit`) runs each `RoleName` as a separate process under `dotnet test`, with `TestConductor` able to `Blackhole` links and `Exit` nodes, synchronized by `EnterBarrier`. Reasons not to use it here:
+
+1. **Everything this feature needs is testable in-process** with real ActorSystems built from production bootstrap code (ScadaBridge `TwoNodeClusterFixture`; OtOpcUa `SplitBrainResolverActivationTests` pattern). Same fidelity, no new runner.
+2. **Heavy project-level cost:** dedicated test project, parallelization disabled assembly-wide, single-public-constructor constraints, MNTR log plumbing.
+3. **OtOpcUa xunit constraint:** the Akka TestKit family is xunit-v2-only; OtOpcUa's integration tree is xunit.v3.
+4. **The one MNTR-only scenario** (true boot-time partition → dual self-form) validates an already-documented *accepted* risk; the family's preferred proof is the docker rig drill (`failover-drill.sh` precedent, or `docker network disconnect`).
+
+**Follow-up:** revisit alongside OtOpcUa per-cluster-mesh Phase 7 (failover-drill phase) if a scripted dual-island drill is ever wanted.
+
+### Out of scope (recorded follow-ups)
+
+- Making OtOpcUa's 15 s `DowningStableAfter` configurable — now listed in the OtOpcUa plan's "Reserved for additions".
+- Failure-detector tuning knobs for faster failover (both apps) — see `akka_failover.md` §2 tuning table; OtOpcUa side listed in its reserved-additions section.
+- Staggered Windows service start delays (pure ops; document only).
+- `deploy/wonder-app-vd03/` gitignored production overlay (ops applies `SelfFormAfter` on next deploy).
+- ScadaBridge site-pair failover from the central UI — optional Task 10 in the ScadaBridge plan (new central→site message contract; confirm with the user first).
+
+---
+
+## Task C1: Family docs flip (execute AFTER both repo plans complete and merge)
+
+**Classification:** trivial
+**Estimated implement time:** ~2 min
+
+**Files:**
+- Modify: `~/Desktop/scadaproj/akka_failover.md` §6.1 — "Status: decided, not yet implemented" → implemented (merge commit refs from both repos); leave the §6.2 `DowningStableAfter` line unless the OtOpcUa reserved-addition shipped.
+- Modify: `~/Desktop/scadaproj/CLAUDE.md` — brief addition to the ScadaBridge and OtOpcUa index rows.
+- Update memory `ha-availability-over-partition-safety.md` — mark the decision implemented.