06c56d7719
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on 3a6b2a1d (PrimaryGateFailoverTests, and
DriverProbeRegistrationTests expecting 10 probe types where 12 now register)
plus the AbLegacy integration suite, which needs a docker fixture.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
141 lines
7.6 KiB
C#
141 lines
7.6 KiB
C#
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
|
|
|
public sealed class AkkaClusterOptions
|
|
{
|
|
public const string SectionName = "Cluster";
|
|
|
|
/// <summary>Gets or sets the Akka system name.</summary>
|
|
public string SystemName { get; set; } = "otopcua";
|
|
|
|
/// <summary>Gets or sets the hostname to bind to (default 0.0.0.0).</summary>
|
|
public string Hostname { get; set; } = "0.0.0.0";
|
|
|
|
/// <summary>Gets or sets the port to listen on (default 4053).</summary>
|
|
public int Port { get; set; } = 4053;
|
|
|
|
/// <summary>
|
|
/// Hostname advertised in cluster gossip. Must be reachable by other nodes.
|
|
/// In docker-compose this is the container DNS name; in bare metal it's the
|
|
/// host's stable LAN address.
|
|
/// </summary>
|
|
public string PublicHostname { get; set; } = "127.0.0.1";
|
|
|
|
/// <summary>
|
|
/// Gets or sets the seed nodes for cluster bootstrapping.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
|
|
/// must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
|
|
/// path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
|
|
/// when <c>seed-nodes[0]</c> is this node's own address; any other node runs
|
|
/// <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
|
|
/// cluster. So listing both peers does NOT mean either can cold-start alone: a node that
|
|
/// lists its partner first never comes Up while that partner is down. Self-first ordering
|
|
/// closes that gap inside Akka's own handshake — unlike the retired
|
|
/// <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
|
|
/// seed answered" from "a seed answered and the join is in flight".
|
|
/// </para>
|
|
/// <para>
|
|
/// The rule is <b>conditional</b>: it binds only when this node's own address appears in
|
|
/// this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
|
|
/// which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
|
|
/// at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
|
|
/// <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
|
|
/// <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
|
|
/// </para>
|
|
/// </remarks>
|
|
public string[] SeedNodes { get; set; } = Array.Empty<string>();
|
|
|
|
/// <summary>
|
|
/// Cluster roles for this node. Allowed values: <c>admin</c>, <c>driver</c>, <c>dev</c>, plus one
|
|
/// per-cluster <c>cluster-{ClusterId}</c> role (per-cluster mesh, Phase 6).
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// When this is empty the list falls back to the <c>OTOPCUA_ROLES</c> environment variable,
|
|
/// applied by <c>AddOtOpcUaCluster</c> through
|
|
/// <see cref="RoleParser.ResolveEffectiveRoles(IReadOnlyList{string}?)"/>. That fallback was
|
|
/// described here — and relied upon in the field — for a year before it was implemented; a
|
|
/// node configured only through the env var joined the cluster carrying no roles at all.
|
|
/// See Gitea #532 and <c>docs/deployments/wonder-app-vd03-makino-z-34184.md</c>.
|
|
/// </para>
|
|
/// <para>
|
|
/// The two sources normalize differently: values bound from <c>Cluster:Roles</c> are taken
|
|
/// verbatim, while the environment path is lower-cased by <see cref="RoleParser.Parse"/>.
|
|
/// Akka compares gossiped role names ordinally, so <b>configure both nodes of a redundant
|
|
/// pair through the same source</b> — a pair split across the two with a mixed-case
|
|
/// <c>cluster-{ClusterId}</c> would gossip two different role names and elect two singletons.
|
|
/// </para>
|
|
/// </remarks>
|
|
public string[] Roles { get; set; } = Array.Empty<string>();
|
|
|
|
/// <summary>
|
|
/// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
|
|
/// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
|
|
/// downs the unreachable peer after
|
|
/// <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
|
|
/// node — including the oldest — fails over to the survivor with no operator action.
|
|
/// The trade is that a genuine network partition (both nodes alive, link cut) leaves
|
|
/// both sides running active until an operator restarts one.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
|
|
/// side of a split, so a partition can never run dual-active. <b>The cost is severe for a
|
|
/// two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
|
|
/// <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
|
|
/// >= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
|
|
/// redundancy pair turns a single-node crash into a total outage. Choose this only for
|
|
/// clusters of three or more nodes, or where dual-active is genuinely worse than an
|
|
/// outage.
|
|
/// </para>
|
|
/// </remarks>
|
|
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
|
|
|
|
/// <summary>
|
|
/// The simultaneous-cold-start split-brain guard (<c>Cluster:BootstrapGuard</c>). Default OFF — a
|
|
/// dark switch, so existing deployments and tests keep Akka's config-driven self-first auto-join
|
|
/// unchanged. See <see cref="ClusterBootstrapGuard"/> for the decision logic and
|
|
/// <c>ClusterBootstrapCoordinator</c> for the runtime.
|
|
/// </summary>
|
|
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
|
|
}
|
|
|
|
/// <summary>
|
|
/// Configuration for the simultaneous-cold-start split-brain guard. When
|
|
/// <see cref="Enabled"/>, the node does NOT auto-join from its config seeds; a coordinator picks the
|
|
/// join order (founder self-first / joiner peer-first) after a reachability probe. See
|
|
/// <see cref="ClusterBootstrapGuard"/>.
|
|
/// </summary>
|
|
public sealed class ClusterBootstrapGuardOptions
|
|
{
|
|
/// <summary>
|
|
/// Gets or sets whether the guard is active. Default <see langword="false"/> — Akka auto-joins from
|
|
/// the config seed list exactly as before. Only meaningful on a node that is one of its own two
|
|
/// pair seeds; inert everywhere else.
|
|
/// </summary>
|
|
public bool Enabled { get; set; }
|
|
|
|
/// <summary>
|
|
/// Gets or sets how long the higher-address node probes its partner's Akka endpoint before
|
|
/// concluding the partner is dead and forming alone. Must comfortably exceed the partner's
|
|
/// worst-case process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a
|
|
/// dead one (which would re-open the split). Default 25 s.
|
|
/// </summary>
|
|
public int PartnerProbeSeconds { get; set; } = 25;
|
|
|
|
/// <summary>
|
|
/// Gets or sets the interval between partner reachability probes, in milliseconds. Default 500 ms.
|
|
/// </summary>
|
|
public int PartnerProbeIntervalMs { get; set; } = 500;
|
|
|
|
/// <summary>
|
|
/// Gets or sets the per-probe TCP connect timeout, in milliseconds. Default 1000 ms.
|
|
/// </summary>
|
|
public int ProbeConnectTimeoutMs { get; set; } = 1000;
|
|
}
|