Files
lmxopcua/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs
T
Joseph Doherty af3b0c3146 refactor(mesh): step 6 — delete the ClusterClient mode value; split topology now requires Grpc
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.

SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.

Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
2026-08-06 15:49:42 -04:00

152 lines
8.6 KiB
C#

namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Selects and configures the transport carrying central→node commands and node→central
/// deployment acks (per-cluster mesh Phase 2).
/// </summary>
/// <remarks>
/// <para>
/// <see cref="Mode"/> selects between a <b>pair-local</b> transport and a <b>cross-mesh</b>
/// one. <see cref="ModeDps"/> is the default and cannot cross an application-<c>Cluster</c>
/// boundary; <see cref="ModeGrpc"/> is the only mode that can, and
/// <see cref="SplitTopologyTransportValidator"/> requires it on any node carrying a
/// <c>cluster-{ClusterId}</c> role.
/// </para>
/// <para>
/// <b>The DistributedPubSub branch is retained, not scheduled for deletion.</b> An earlier
/// comment here promised Phase 6 would delete it along with the single mesh; that was
/// deliberately reversed — see <c>docs/Configuration.md</c> § "The compiled transport-mode
/// defaults were deliberately NOT flipped". DPS remains the correct and default transport for
/// a single-mesh topology, which every integration test and every profile that leaves this
/// key unset still runs.
/// </para>
/// <para>
/// <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central
/// discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c>
/// rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from
/// this section</i> (<see cref="CentralGrpcEndpoints"/>): static, restart to change. The
/// asymmetry follows the rate of change — operators add and retire nodes, but central's own
/// address is part of the deployment.
/// </para>
/// </remarks>
public sealed class MeshTransportOptions
{
/// <summary>Configuration section name.</summary>
public const string SectionName = "MeshTransport";
/// <summary>
/// Every command stays on DistributedPubSub — pair-local, and the default. Cannot cross an
/// application-<c>Cluster</c> boundary.
/// </summary>
public const string ModeDps = "Dps";
/// <summary>
/// Commands cross the mesh boundary as unary gRPC calls on the <c>mesh_transport.v1</c>
/// contract — central dials each node's <c>NodeCommandService</c>, a node posts its
/// deployment ack to central's <c>CentralCommandService</c>. The command plane is
/// authenticated per application <c>Cluster</c> (<see cref="GrpcPsk"/>).
/// </summary>
public const string ModeGrpc = "Grpc";
/// <summary>
/// <see cref="ModeDps"/> (default) or <see cref="ModeGrpc"/>. Any other value fails the host
/// at startup rather than silently falling back — a transport that is neither delivers
/// nothing, and the symptom would be commands that vanish. That includes the retired
/// <c>ClusterClient</c> value: a node still carrying it is refused at boot rather than
/// quietly demoted to DPS, which on a split-topology node would deliver nothing at all.
/// </summary>
public string Mode { get; set; } = ModeDps;
/// <summary>
/// How often central re-reads <c>ClusterNode</c> rows to rebuild its dispatch set.
/// Refreshing on a timer (rather than only on admin change) means a row edited by any
/// process — migration, direct SQL, a second admin node — is picked up without a restart.
/// </summary>
public int ContactRefreshSeconds { get; set; } = 60;
/// <summary>
/// <c>MeshTransport:CentralGrpcEndpoints</c> — node-side only, and only under
/// <see cref="ModeGrpc"/>: the gRPC base addresses of the central pair's
/// <c>CentralCommandService</c>, e.g. <c>http://central-1:4055</c>. Full URLs
/// (<c>scheme://host:port</c>), not bare <c>host:port</c> — <c>http://</c> is h2c.
/// </summary>
/// <remarks>
/// Tried <b>in order, stopping at the first success</b>, which is what keeps the ack
/// one-per-node-per-dispatch: either central node routes it to the same
/// <c>ConfigPublishCoordinator</c> singleton proxy, so which one answers does not matter,
/// but two answering would double-count.
/// </remarks>
public string[] CentralGrpcEndpoints { get; set; } = [];
/// <summary>
/// <c>MeshTransport:GrpcPsk</c> — this node's pre-shared key for the command plane, one per
/// application <c>Cluster</c>. Under <see cref="ModeGrpc"/> it is both the bearer this node
/// presents when posting an ack to central and the bearer it requires on inbound
/// <c>NodeCommandService</c> calls.
/// </summary>
/// <remarks>
/// <para>
/// Production form is a secret reference — <c>${secret:OT-MESH-PSK-&lt;clusterId&gt;}</c>,
/// e.g. <c>${secret:OT-MESH-PSK-SITE-A}</c> — resolved pre-host by the existing
/// <c>SecretReferenceExpander</c>, so it works on a driver-only node with no ConfigDb.
/// Supply a literal only on a rig, via the environment
/// (<c>MeshTransport__GrpcPsk</c>); never commit one.
/// </para>
/// <para>
/// <b>Per-Cluster, not fleet-wide, on purpose.</b> The three existing shared-bearer gRPC
/// channels are read/telemetry planes; this one carries <c>RestartDriver</c>,
/// <c>AlarmCommand</c> and deployment dispatch, so a single fleet key would let one
/// compromised site node command every other site and forge acks fleet-wide.
/// </para>
/// <para>
/// Nullable because <b>unset must be distinguishable from blank at the boundary that
/// fails closed</b>: the serve-side interceptor rejects every call when no key is
/// configured rather than defaulting to an empty comparison that any empty bearer would
/// satisfy.
/// </para>
/// </remarks>
public string? GrpcPsk { get; set; }
/// <summary>
/// <c>MeshTransport:DispatchTimeoutSeconds</c> — central-side deadline, in seconds, for one
/// <c>NodeCommandService.Execute</c> call to one node. Must be greater than 0 under
/// <see cref="ModeGrpc"/>.
/// </summary>
/// <remarks>
/// Bounded per node because a fan-out dispatch dials every enabled node in the cluster: an
/// unbounded call to one unreachable node would stall the commands queued behind it. Expiry
/// is logged and dropped — the transport is fire-and-forget and layers no retry, so the
/// deploy path's 2-minute apply deadline stays the only recovery mechanism.
/// Validation of this key is scoped to driver-role nodes; that covers the central pair only
/// because central nodes are always fused (<c>admin,driver,cluster-*</c>). A pure-admin
/// central node would leave this unvalidated on the node that uses it.
/// </remarks>
public int DispatchTimeoutSeconds { get; set; } = 10;
/// <summary>
/// <c>MeshTransport:AckTimeoutSeconds</c> — node-side deadline, in seconds, for one
/// <c>CentralCommandService.ReportApply</c> call to one central endpoint. Must be greater
/// than 0 under <see cref="ModeGrpc"/>.
/// </summary>
/// <remarks>
/// Applied per endpoint, not per ack: the node walks <see cref="CentralGrpcEndpoints"/> in
/// order, so a dead first endpoint costs this deadline before the second is tried.
/// </remarks>
public int AckTimeoutSeconds { get; set; } = 10;
/// <summary>
/// <c>MeshTransport:ClusterPsks:{clusterId}</c> — central-side override map of per-cluster
/// PSKs, consulted <i>before</i> the Secrets store when central builds a channel to (or
/// verifies an ack from) a given <c>Cluster</c>. Keyed case-insensitively.
/// </summary>
/// <remarks>
/// <b>Production leaves this empty</b> and the runtime provider resolves
/// <c>OT-MESH-PSK-{clusterId}</c> from the Secrets store instead; the map exists so the
/// docker-dev rig and integration tests can supply literals through the environment
/// (<c>MeshTransport__ClusterPsks__SITE-A</c>) without standing up a KEK. Central needs a map
/// rather than a single key because its cluster set is DB-dynamic — one key per site pair it
/// talks to — whereas a node holds exactly its own (<see cref="GrpcPsk"/>).
/// </remarks>
public Dictionary<string, string> ClusterPsks { get; set; } = new(StringComparer.OrdinalIgnoreCase);
}