af3b0c3146
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather than falling back to Dps, which on a split-topology node would deliver nothing at all. A new fact pins exactly that. SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays refused; the rule is still an explicit allow-list, so a typo'd mode fails too. Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted (their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the SplitTopology fixtures move to Grpc, and the three-mode Theories in ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
152 lines
8.6 KiB
C#
152 lines
8.6 KiB
C#
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
|
|
|
/// <summary>
|
|
/// Selects and configures the transport carrying central→node commands and node→central
|
|
/// deployment acks (per-cluster mesh Phase 2).
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <see cref="Mode"/> selects between a <b>pair-local</b> transport and a <b>cross-mesh</b>
|
|
/// one. <see cref="ModeDps"/> is the default and cannot cross an application-<c>Cluster</c>
|
|
/// boundary; <see cref="ModeGrpc"/> is the only mode that can, and
|
|
/// <see cref="SplitTopologyTransportValidator"/> requires it on any node carrying a
|
|
/// <c>cluster-{ClusterId}</c> role.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>The DistributedPubSub branch is retained, not scheduled for deletion.</b> An earlier
|
|
/// comment here promised Phase 6 would delete it along with the single mesh; that was
|
|
/// deliberately reversed — see <c>docs/Configuration.md</c> § "The compiled transport-mode
|
|
/// defaults were deliberately NOT flipped". DPS remains the correct and default transport for
|
|
/// a single-mesh topology, which every integration test and every profile that leaves this
|
|
/// key unset still runs.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central
|
|
/// discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c>
|
|
/// rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from
|
|
/// this section</i> (<see cref="CentralGrpcEndpoints"/>): static, restart to change. The
|
|
/// asymmetry follows the rate of change — operators add and retire nodes, but central's own
|
|
/// address is part of the deployment.
|
|
/// </para>
|
|
/// </remarks>
|
|
public sealed class MeshTransportOptions
|
|
{
|
|
/// <summary>Configuration section name.</summary>
|
|
public const string SectionName = "MeshTransport";
|
|
|
|
/// <summary>
|
|
/// Every command stays on DistributedPubSub — pair-local, and the default. Cannot cross an
|
|
/// application-<c>Cluster</c> boundary.
|
|
/// </summary>
|
|
public const string ModeDps = "Dps";
|
|
|
|
/// <summary>
|
|
/// Commands cross the mesh boundary as unary gRPC calls on the <c>mesh_transport.v1</c>
|
|
/// contract — central dials each node's <c>NodeCommandService</c>, a node posts its
|
|
/// deployment ack to central's <c>CentralCommandService</c>. The command plane is
|
|
/// authenticated per application <c>Cluster</c> (<see cref="GrpcPsk"/>).
|
|
/// </summary>
|
|
public const string ModeGrpc = "Grpc";
|
|
|
|
/// <summary>
|
|
/// <see cref="ModeDps"/> (default) or <see cref="ModeGrpc"/>. Any other value fails the host
|
|
/// at startup rather than silently falling back — a transport that is neither delivers
|
|
/// nothing, and the symptom would be commands that vanish. That includes the retired
|
|
/// <c>ClusterClient</c> value: a node still carrying it is refused at boot rather than
|
|
/// quietly demoted to DPS, which on a split-topology node would deliver nothing at all.
|
|
/// </summary>
|
|
public string Mode { get; set; } = ModeDps;
|
|
|
|
/// <summary>
|
|
/// How often central re-reads <c>ClusterNode</c> rows to rebuild its dispatch set.
|
|
/// Refreshing on a timer (rather than only on admin change) means a row edited by any
|
|
/// process — migration, direct SQL, a second admin node — is picked up without a restart.
|
|
/// </summary>
|
|
public int ContactRefreshSeconds { get; set; } = 60;
|
|
|
|
/// <summary>
|
|
/// <c>MeshTransport:CentralGrpcEndpoints</c> — node-side only, and only under
|
|
/// <see cref="ModeGrpc"/>: the gRPC base addresses of the central pair's
|
|
/// <c>CentralCommandService</c>, e.g. <c>http://central-1:4055</c>. Full URLs
|
|
/// (<c>scheme://host:port</c>), not bare <c>host:port</c> — <c>http://</c> is h2c.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Tried <b>in order, stopping at the first success</b>, which is what keeps the ack
|
|
/// one-per-node-per-dispatch: either central node routes it to the same
|
|
/// <c>ConfigPublishCoordinator</c> singleton proxy, so which one answers does not matter,
|
|
/// but two answering would double-count.
|
|
/// </remarks>
|
|
public string[] CentralGrpcEndpoints { get; set; } = [];
|
|
|
|
/// <summary>
|
|
/// <c>MeshTransport:GrpcPsk</c> — this node's pre-shared key for the command plane, one per
|
|
/// application <c>Cluster</c>. Under <see cref="ModeGrpc"/> it is both the bearer this node
|
|
/// presents when posting an ack to central and the bearer it requires on inbound
|
|
/// <c>NodeCommandService</c> calls.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// Production form is a secret reference — <c>${secret:OT-MESH-PSK-<clusterId>}</c>,
|
|
/// e.g. <c>${secret:OT-MESH-PSK-SITE-A}</c> — resolved pre-host by the existing
|
|
/// <c>SecretReferenceExpander</c>, so it works on a driver-only node with no ConfigDb.
|
|
/// Supply a literal only on a rig, via the environment
|
|
/// (<c>MeshTransport__GrpcPsk</c>); never commit one.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>Per-Cluster, not fleet-wide, on purpose.</b> The three existing shared-bearer gRPC
|
|
/// channels are read/telemetry planes; this one carries <c>RestartDriver</c>,
|
|
/// <c>AlarmCommand</c> and deployment dispatch, so a single fleet key would let one
|
|
/// compromised site node command every other site and forge acks fleet-wide.
|
|
/// </para>
|
|
/// <para>
|
|
/// Nullable because <b>unset must be distinguishable from blank at the boundary that
|
|
/// fails closed</b>: the serve-side interceptor rejects every call when no key is
|
|
/// configured rather than defaulting to an empty comparison that any empty bearer would
|
|
/// satisfy.
|
|
/// </para>
|
|
/// </remarks>
|
|
public string? GrpcPsk { get; set; }
|
|
|
|
/// <summary>
|
|
/// <c>MeshTransport:DispatchTimeoutSeconds</c> — central-side deadline, in seconds, for one
|
|
/// <c>NodeCommandService.Execute</c> call to one node. Must be greater than 0 under
|
|
/// <see cref="ModeGrpc"/>.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Bounded per node because a fan-out dispatch dials every enabled node in the cluster: an
|
|
/// unbounded call to one unreachable node would stall the commands queued behind it. Expiry
|
|
/// is logged and dropped — the transport is fire-and-forget and layers no retry, so the
|
|
/// deploy path's 2-minute apply deadline stays the only recovery mechanism.
|
|
/// Validation of this key is scoped to driver-role nodes; that covers the central pair only
|
|
/// because central nodes are always fused (<c>admin,driver,cluster-*</c>). A pure-admin
|
|
/// central node would leave this unvalidated on the node that uses it.
|
|
/// </remarks>
|
|
public int DispatchTimeoutSeconds { get; set; } = 10;
|
|
|
|
/// <summary>
|
|
/// <c>MeshTransport:AckTimeoutSeconds</c> — node-side deadline, in seconds, for one
|
|
/// <c>CentralCommandService.ReportApply</c> call to one central endpoint. Must be greater
|
|
/// than 0 under <see cref="ModeGrpc"/>.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Applied per endpoint, not per ack: the node walks <see cref="CentralGrpcEndpoints"/> in
|
|
/// order, so a dead first endpoint costs this deadline before the second is tried.
|
|
/// </remarks>
|
|
public int AckTimeoutSeconds { get; set; } = 10;
|
|
|
|
/// <summary>
|
|
/// <c>MeshTransport:ClusterPsks:{clusterId}</c> — central-side override map of per-cluster
|
|
/// PSKs, consulted <i>before</i> the Secrets store when central builds a channel to (or
|
|
/// verifies an ack from) a given <c>Cluster</c>. Keyed case-insensitively.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <b>Production leaves this empty</b> and the runtime provider resolves
|
|
/// <c>OT-MESH-PSK-{clusterId}</c> from the Secrets store instead; the map exists so the
|
|
/// docker-dev rig and integration tests can supply literals through the environment
|
|
/// (<c>MeshTransport__ClusterPsks__SITE-A</c>) without standing up a KEK. Central needs a map
|
|
/// rather than a single key because its cluster set is DB-dynamic — one key per site pair it
|
|
/// talks to — whereas a node holds exactly its own (<see cref="GrpcPsk"/>).
|
|
/// </remarks>
|
|
public Dictionary<string, string> ClusterPsks { get; set; } = new(StringComparer.OrdinalIgnoreCase);
|
|
}
|