test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
This commit is contained in:
@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
|
||||
/// <param name="Topic">
|
||||
/// The legacy DPS topic. Carried so the router can publish under
|
||||
/// <c>MeshTransport:Mode = Dps</c> without a second dispatch table; ignored entirely under
|
||||
/// <c>ClusterClient</c> mode, where the destination is an actor path rather than a topic.
|
||||
/// <c>Grpc</c> mode, where the destination is a node endpoint rather than a topic.
|
||||
/// </param>
|
||||
/// <param name="Message">
|
||||
/// The command itself, never wrapped. Node-side handlers see exactly the type they see today,
|
||||
|
||||
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
|
||||
/// Declared once, here, rather than as literals on each side. The <c>deployments</c> /
|
||||
/// <c>deployment-acks</c> topic names are declared twice — once in
|
||||
/// <c>ConfigPublishCoordinator</c> and again in <c>DriverHostActor</c> — and agree only by
|
||||
/// convention; renaming one would compile, deploy, and deliver nothing. A ClusterClient path
|
||||
/// convention; renaming one would compile, deploy, and deliver nothing. A mesh path
|
||||
/// has the same failure mode and no compiler to catch it, so the constant is shared.
|
||||
/// </para>
|
||||
/// <para>
|
||||
|
||||
@@ -9,13 +9,13 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
|
||||
|
||||
/// <summary>
|
||||
/// The one translation between the five cross-mesh command records and their
|
||||
/// <c>mesh_transport.v1</c> wire shapes (ClusterClient→gRPC migration Phase 0). Both directions live
|
||||
/// <c>mesh_transport.v1</c> wire shapes (mesh gRPC command plane, Phase 0). Both directions live
|
||||
/// here so a field can only be added to one side without the other by making this file
|
||||
/// visibly asymmetric.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Nothing here guesses.</b> The transport it replaces — an Akka ClusterClient receptionist —
|
||||
/// <b>Nothing here guesses.</b> The transport it replaced — an Akka ClusterClient receptionist —
|
||||
/// dropped an undeliverable command silently, and the entire point of the migration is to stop
|
||||
/// doing that. So every decode that cannot produce a faithful record throws: an unset oneof, an
|
||||
/// <c>APPLY_OUTCOME_UNSPECIFIED</c> ack, an enum value from a newer peer. A loud failure on one
|
||||
|
||||
@@ -6,9 +6,9 @@ option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1";
|
||||
|
||||
import "google/protobuf/timestamp.proto";
|
||||
|
||||
// ClusterClient→gRPC migration Phase 0: the cross-mesh command plane. The five messages that cross
|
||||
// an application-Cluster boundary today ride an Akka ClusterClient receptionist; this contract is
|
||||
// the replacement carrier. Four are central→node commands, one is the node→central deployment ack.
|
||||
// The cross-mesh command plane (introduced in Phase 0, now the only transport that crosses an
|
||||
// application-Cluster boundary). Five messages ride it: four central→node commands and one
|
||||
// node→central deployment ack.
|
||||
//
|
||||
// The C# namespace deliberately says "MeshTransport", never "MeshCommand" — the latter is already
|
||||
// the CLR envelope record ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh.MeshCommand, and a generated type
|
||||
@@ -20,17 +20,17 @@ import "google/protobuf/timestamp.proto";
|
||||
// MeshTransportProtoContract + its test in the Commons test project.
|
||||
|
||||
// Central → node. Delivered to the node's NodeCommunicationActor, which re-emits the decoded message
|
||||
// on the node's EventStream. ONE-WAY semantics, preserved verbatim from the ClusterClient transport:
|
||||
// the ack means "received and re-emitted", nothing more — not applied, not acknowledged by any
|
||||
// on the node's EventStream. ONE-WAY semantics, matching the node-local DistributedPubSub leg: the
|
||||
// ack means "received and re-emitted", nothing more — not applied, not acknowledged by any
|
||||
// subscriber. No retry is layered on a failure; a failed Execute is logged per node and dropped,
|
||||
// which is an honest upgrade over the receptionist's silent drop rather than a semantic change.
|
||||
// which is an honest upgrade over a pub/sub publish nobody was listening for.
|
||||
service NodeCommandService {
|
||||
rpc Execute (CommandEnvelope) returns (ExecuteAck);
|
||||
}
|
||||
|
||||
// Node → central. Delivered to central's CentralCommunicationActor, which forwards to the
|
||||
// ConfigPublishCoordinator singleton proxy — so an ack landing on either node of the central pair
|
||||
// routes identically, exactly as the receptionist made arrival-node-agnostic. ONE-WAY semantics: the
|
||||
// routes identically — which central node answered is deliberately not observable. ONE-WAY semantics: the
|
||||
// ack means "handed to the coordinator", not "the deployment is complete".
|
||||
service CentralCommandService {
|
||||
rpc ReportApply (ApplyAckMessage) returns (ReportAck);
|
||||
|
||||
@@ -22,7 +22,7 @@ public sealed class ClusterNode
|
||||
/// Akka remoting port (default 4053). <b>This is central's dial target, not the node's own
|
||||
/// binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings,
|
||||
/// and this column duplicates that value for a reader that cannot see the node's config.
|
||||
/// Per-cluster mesh Phase 2 builds its ClusterClient contact points from
|
||||
/// Per-cluster mesh Phase 2 built its Akka contact points from
|
||||
/// <see cref="Host"/> + this port, at which point central and the site node no longer share a
|
||||
/// gossip ring and central has no other way to learn it.
|
||||
/// </summary>
|
||||
|
||||
@@ -17,10 +17,10 @@ using Status = Grpc.Core.Status;
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Central-side ingress for a node's deployment acknowledgment (ClusterClient→gRPC migration
|
||||
/// Central-side ingress for a node's deployment acknowledgment (mesh gRPC command plane
|
||||
/// Phase 1B). Decodes an <see cref="ApplyAckMessage"/> and hands it to the node-local
|
||||
/// <c>CentralCommunicationActor</c> — the same actor an Akka ClusterClient receptionist delivered
|
||||
/// to, whose forward-to-the-coordinator path is untouched. Registered only on admin-role nodes
|
||||
/// <c>CentralCommunicationActor</c> — the same actor the node-local DistributedPubSub path
|
||||
/// delivers to, whose forward-to-the-coordinator path is untouched. Registered only on admin-role nodes
|
||||
/// and only when <c>ConfigServe:GrpcListenPort > 0</c>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
@@ -52,7 +52,7 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
||||
/// <para>
|
||||
/// <b>One-way semantics, preserved.</b> <c>Tell</c>, not <c>Ask</c>: the returned
|
||||
/// <see cref="ReportAck"/> means "handed to the comm actor", never "the deployment is
|
||||
/// complete" — the same promise the receptionist made. The comm actor forwards to the
|
||||
/// complete" — the same promise the DistributedPubSub leg makes. The comm actor forwards to the
|
||||
/// coordinator singleton proxy, so an ack landing on either central node routes identically.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
|
||||
@@ -92,8 +92,7 @@ public sealed class AdminOperationsActor : ReceiveActor
|
||||
/// <b>retained</b>, not scheduled for deletion — an earlier note here promised Phase 6 would
|
||||
/// remove it, which was deliberately reversed (<c>docs/Configuration.md</c> § "The compiled
|
||||
/// transport-mode defaults were deliberately NOT flipped"): DPS is still the correct and
|
||||
/// default transport for a single-mesh topology. What is being replaced is the ClusterClient
|
||||
/// branch, by gRPC.
|
||||
/// default transport for a single-mesh topology; gRPC is what crosses a mesh boundary.
|
||||
/// </remarks>
|
||||
/// <param name="topic">The legacy DistributedPubSub topic.</param>
|
||||
/// <param name="message">The command.</param>
|
||||
|
||||
@@ -10,7 +10,7 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Places one central→node command call on one node's <c>NodeCommandService</c>
|
||||
/// (ClusterClient→gRPC migration Phase 1A). The seam <c>CentralCommunicationActor</c> fans
|
||||
/// (mesh gRPC command plane, Phase 1A). The seam <c>CentralCommunicationActor</c> fans
|
||||
/// across its target set, kept as an interface so the actor's tests never open a socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
|
||||
@@ -72,7 +72,7 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
|
||||
/// not scheduled for deletion — an earlier note here promised Phase 6 would remove it, which
|
||||
/// was deliberately reversed (<c>docs/Configuration.md</c> § "The compiled transport-mode
|
||||
/// defaults were deliberately NOT flipped"): DPS is still the correct and default transport
|
||||
/// for a single-mesh topology. What is being replaced is the ClusterClient branch, by gRPC.
|
||||
/// for a single-mesh topology; gRPC is what crosses a mesh boundary.
|
||||
/// </remarks>
|
||||
private readonly IActorRef? _meshRouter;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
Google.Protobuf + Grpc.Core.Api flow transitively from Commons (the generated client). -->
|
||||
<PackageReference Include="Grpc.Net.Client"/>
|
||||
<PackageReference Include="ZB.MOM.WW.Audit"/>
|
||||
<!-- ClusterClient→gRPC: MeshPskProvider resolves OT-MESH-PSK-{clusterId} from the shared
|
||||
<!-- Mesh gRPC command plane: MeshPskProvider resolves OT-MESH-PSK-{clusterId} from the shared
|
||||
secret store when MeshTransport:ClusterPsks carries no override. Abstractions only —
|
||||
the concrete store is the host's choice (SecretsRegistration), not the control plane's. -->
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions"/>
|
||||
|
||||
@@ -6,7 +6,7 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// Gates the central-side command endpoint (ClusterClient→gRPC migration). The
|
||||
/// Gates the central-side command endpoint (mesh gRPC command plane). The
|
||||
/// <c>CentralCommandService.ReportApply</c> RPC is how a node tells central its deployment
|
||||
/// applied; the coordinator counts those acks, so anything able to reach central's command port
|
||||
/// could otherwise forge a fleet-wide "everything applied". This interceptor is the ONLY inbound
|
||||
|
||||
@@ -9,7 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// Gates the node-side command endpoint (ClusterClient→gRPC migration). The
|
||||
/// Gates the node-side command endpoint (mesh gRPC command plane). The
|
||||
/// <c>NodeCommandService.Execute</c> RPC is how central drives this node — restart a driver,
|
||||
/// acknowledge an alarm, dispatch a deployment — so anything able to reach a node's command port
|
||||
/// could otherwise operate the node. This interceptor is the ONLY inbound auth on that endpoint.
|
||||
|
||||
@@ -15,11 +15,11 @@ using Status = Grpc.Core.Status;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Node-side command ingress (ClusterClient→gRPC migration Phase 1). Central dials a driver node
|
||||
/// Node-side command ingress (mesh gRPC command plane, Phase 1). Central dials a driver node
|
||||
/// and calls <see cref="Execute"/>; the envelope is decoded back into the same CLR record the
|
||||
/// ClusterClient receptionist used to deliver and handed to this node's
|
||||
/// node-local DistributedPubSub path delivers and handed to this node's
|
||||
/// <c>NodeCommunicationActor</c>, which re-emits it on the node's <c>EventStream</c> exactly as
|
||||
/// before. Nothing downstream of that actor can tell which transport carried the command.
|
||||
/// it would otherwise. Nothing downstream of that actor can tell which transport carried it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
|
||||
@@ -527,20 +527,19 @@ if (hasAdmin)
|
||||
// hosted service has built the system).
|
||||
builder.Services.TryAddSingleton<Func<ActorSystem>>(sp => () => sp.GetRequiredService<ActorSystem>());
|
||||
builder.Services.AddSingleton<IManualFailoverService, ManualFailoverService>();
|
||||
// Per-cluster mesh PSKs (ClusterClient→gRPC). Admin-only, and singleton because its whole job is
|
||||
// Per-cluster mesh PSKs (mesh gRPC command plane). Admin-only, and singleton because its whole job is
|
||||
// to hold a cache: central talks to N clusters and would otherwise re-read the secret store on
|
||||
// every inbound ack. A node needs nothing of this — it holds exactly its own MeshTransport:GrpcPsk
|
||||
// — so registering it fleet-wide would give a driver-only node a lookup it can never use.
|
||||
// Depends on ISecretResolver, which AddOtOpcUaSecrets registers unconditionally below (DI
|
||||
// registration order is irrelevant; resolution order is not).
|
||||
builder.Services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
|
||||
// Central's outbound command dial (ClusterClient→gRPC Phase 1A). Admin-only for the same reason
|
||||
// Central's outbound command dial (mesh gRPC command plane, Phase 1A). Admin-only for the same reason
|
||||
// as the provider above — a node never dials another node — and singleton because it owns a
|
||||
// GrpcChannel per node endpoint; per-call channels would leak sockets across a fleet-wide fan.
|
||||
// Registered in every mode: CentralCommunicationActor only reaches it under Mode=Grpc, and an
|
||||
// idle registration costs nothing, whereas gating it here would mean flipping the mode on a
|
||||
// running fleet needs a central restart before any command lands — the same reasoning that
|
||||
// registers the ClusterClient receptionist unconditionally.
|
||||
// running fleet needs a central restart before any command lands.
|
||||
builder.Services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
|
||||
}
|
||||
|
||||
@@ -564,7 +563,7 @@ builder.Services.AddOtOpcUaObservability(builder.Configuration);
|
||||
// Registered whenever either role is present so an admin-only node still serves config and a
|
||||
// driver-only node still gates sync + telemetry.
|
||||
//
|
||||
// The two mesh-command interceptors (ClusterClient→gRPC migration) join on the same terms:
|
||||
// The two mesh-command interceptors (mesh gRPC command plane) join on the same terms:
|
||||
// MeshCommandAuthInterceptor gates the node-side NodeCommandService (driver-role, one key — this
|
||||
// node's own), CentralCommandAuthInterceptor gates the central-side CentralCommandService
|
||||
// (admin-role, one key per cluster via IMeshPskProvider). Both are registered NOW and are pure
|
||||
@@ -745,7 +744,7 @@ if (hasAdmin && configServeGrpcPort > 0)
|
||||
{
|
||||
app.MapGrpcService<DeploymentArtifactGrpcService>();
|
||||
|
||||
// Central command ingress (ClusterClient→gRPC migration): the node→central deployment ack. It
|
||||
// Central command ingress (mesh gRPC command plane): the node→central deployment ack. It
|
||||
// rides the SAME listener rather than claiming a port of its own — a second cleartext h2c port
|
||||
// would buy nothing but another firewall rule, and both endpoints are admin-role and dialled by
|
||||
// the same driver nodes. Deliberately NOT gated on MeshTransport:Mode: mapping it under Dps
|
||||
@@ -759,7 +758,7 @@ if (hasAdmin && configServeGrpcPort > 0)
|
||||
// Driver-node live-telemetry stream (per-cluster mesh Phase 5). Central dials in; gated by
|
||||
// TelemetryStreamAuthInterceptor (fail-closed on empty Telemetry:ApiKey).
|
||||
//
|
||||
// The node-side command endpoint (ClusterClient→gRPC migration Phase 1) rides the SAME driver-role
|
||||
// The node-side command endpoint (mesh gRPC command plane, Phase 1) rides the SAME driver-role
|
||||
// h2c listener rather than claiming a fourth port: central already dials this node here, both
|
||||
// directions are per-cluster-keyed, and a separate port would be one more thing to open on every
|
||||
// site firewall for no isolation gain. Mapping it is what makes MeshCommandAuthInterceptor live —
|
||||
|
||||
@@ -12,14 +12,14 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Posts this node's <see cref="ApplyAck"/> to central's <c>CentralCommandService</c> as a unary
|
||||
/// gRPC call (ClusterClient→gRPC migration Task 1B.2), the <c>MeshTransport:Mode = Grpc</c>
|
||||
/// replacement for the ClusterClient <c>Send</c>.
|
||||
/// gRPC call (mesh gRPC command plane, Task 1B.2), the <c>MeshTransport:Mode = Grpc</c>
|
||||
/// cross-mesh ack transport.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>One ordered pass, stopping at the first success.</b>
|
||||
/// <see cref="MeshTransportOptions.CentralGrpcEndpoints"/> is an availability list, not a
|
||||
/// fan-out list — the semantic of the <c>ClusterClient.Send</c> being replaced (<c>Send</c>,
|
||||
/// fan-out list — the single-target semantic this leg owes (one ack per node per dispatch,
|
||||
/// deliberately not <c>SendToAll</c>).
|
||||
/// </para>
|
||||
/// <para>
|
||||
@@ -35,7 +35,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
||||
/// depth and a bound on wasted work, not a correctness dependency.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>No retry beyond that single pass, and no queue.</b> The receptionist buffered nothing
|
||||
/// <b>No retry beyond that single pass, and no queue.</b> The transport buffers nothing
|
||||
/// and retried nothing, and this is a like-for-like replacement, not an upgrade to delivery
|
||||
/// guarantees. The deploy path's own apply deadline stays the only recovery mechanism; a
|
||||
/// retry loop here would instead pile acks up behind an unreachable central and deliver a
|
||||
@@ -43,7 +43,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>All endpoints failing is a DROP — but a loud one.</b> That is the same outcome the
|
||||
/// ClusterClient path had when it could not reach central; what changes is that the operator
|
||||
/// node has when it cannot reach central; what changes is that the operator
|
||||
/// now gets the node, the deployment, the correlation id and the status each endpoint
|
||||
/// answered with, which is what distinguishes "central is down" from "central rejected this
|
||||
/// node's key".
|
||||
|
||||
@@ -4,15 +4,15 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Posts this node's deployment acknowledgment to central over the <c>mesh_transport.v1</c>
|
||||
/// command plane (ClusterClient→gRPC migration Phase 1B). The seam
|
||||
/// command plane (mesh gRPC command plane, Phase 1B). The seam
|
||||
/// <see cref="NodeCommunicationActor"/> uses under <c>MeshTransport:Mode = Grpc</c>, in place of
|
||||
/// the <c>ClusterClient.Send</c> it uses under <c>ClusterClient</c>.
|
||||
/// the node-local DistributedPubSub publish it uses under <c>Dps</c>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>One ack per node per dispatch.</b> Whatever the implementation does about redundancy, it
|
||||
/// posts to <i>one</i> central endpoint at a time rather than fanning out — the semantic of the
|
||||
/// <c>ClusterClient.Send</c> this replaces (<c>Send</c>, deliberately not <c>SendToAll</c>).
|
||||
/// one-ack-per-node-per-dispatch semantic the deploy coordinator counts on.
|
||||
/// This is <b>defence in depth, not a correctness dependency</b>: the coordinator counts acks
|
||||
/// in a <c>NodeId</c>-keyed dictionary behind an expected-set membership guard
|
||||
/// (<c>ConfigPublishCoordinator.HandleAck</c>), so a duplicate from one node overwrites its own
|
||||
|
||||
@@ -374,9 +374,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
/// <param name="localNode">The local cluster node identifier.</param>
|
||||
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
|
||||
/// <c>MeshTransport:Mode = ClusterClient</c> <b>and</b> <c>= Grpc</c> the node's
|
||||
/// <c>NodeCommunicationActor</c>, which relays across the boundary over whichever transport it was
|
||||
/// built with. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
|
||||
/// <c>MeshTransport:Mode = Grpc</c> the node's <c>NodeCommunicationActor</c>, which relays the ack
|
||||
/// across the mesh boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
|
||||
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
|
||||
/// <param name="localRoles">Optional set of roles assigned to the local node.</param>
|
||||
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
|
||||
@@ -459,9 +458,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
/// <param name="localNode">The local cluster node identifier.</param>
|
||||
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
|
||||
/// <c>MeshTransport:Mode = ClusterClient</c> <b>and</b> <c>= Grpc</c> the node's
|
||||
/// <c>NodeCommunicationActor</c>, which relays across the boundary over whichever transport it was
|
||||
/// built with. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
|
||||
/// <c>MeshTransport:Mode = Grpc</c> the node's <c>NodeCommunicationActor</c>, which relays the ack
|
||||
/// across the mesh boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
|
||||
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
|
||||
/// <param name="localRoles">Optional set of roles assigned to the local node.</param>
|
||||
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
|
||||
@@ -584,11 +582,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
_mediator.Tell(
|
||||
new Subscribe(ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RedundancyStateTopic, Self));
|
||||
|
||||
// Per-cluster mesh Phase 2 dark switch. Under MeshTransport:Mode = ClusterClient these same
|
||||
// commands arrive from NodeCommunicationActor on the node-local EventStream instead of the
|
||||
// mesh-wide DPS topic. Subscribing to BOTH unconditionally is deliberate: exactly one of the
|
||||
// two ever publishes, so the transport flag lives entirely on the publishing side and the
|
||||
// switch can be flipped (or reverted) without touching any subscriber.
|
||||
// Per-cluster mesh Phase 2. Under MeshTransport:Mode = Grpc these same commands arrive from
|
||||
// NodeCommunicationActor on the node-local EventStream instead of the mesh-wide DPS topic.
|
||||
// Subscribing to BOTH unconditionally is deliberate: exactly one of the two ever publishes, so
|
||||
// the transport choice lives entirely on the publishing side and the mode can be flipped (or
|
||||
// reverted) without touching any subscriber.
|
||||
Context.System.EventStream.Subscribe(Self, typeof(DispatchDeployment));
|
||||
Context.System.EventStream.Subscribe(Self, typeof(RestartDriver));
|
||||
Context.System.EventStream.Subscribe(Self, typeof(ReconnectDriver));
|
||||
@@ -2492,11 +2490,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
var ack = new ApplyAck(deploymentId, _localNode, outcome, failureReason, correlation);
|
||||
if (_ackRouter is not null)
|
||||
{
|
||||
// Either a direct coordinator handle (tests) or, under MeshTransport:Mode = ClusterClient
|
||||
// and = Grpc alike, the NodeCommunicationActor that relays this across the boundary (it
|
||||
// owns the choice of transport, not this actor). Renamed from _coordinatorOverride in
|
||||
// per-cluster mesh Phase 2: under those two modes this ref is emphatically NOT a
|
||||
// coordinator, and the old name would send the next reader looking for one.
|
||||
// Either a direct coordinator handle (tests) or, under MeshTransport:Mode = Grpc, the
|
||||
// NodeCommunicationActor that relays this across the mesh boundary (it owns the transport,
|
||||
// not this actor). Renamed from _coordinatorOverride in per-cluster mesh Phase 2: under
|
||||
// Grpc this ref is emphatically NOT a coordinator, and the old name would send the next
|
||||
// reader looking for one.
|
||||
_ackRouter.Tell(ack);
|
||||
}
|
||||
else
|
||||
|
||||
@@ -627,7 +627,7 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
|
||||
// Per-cluster mesh Phase 2 dark switch. An AdminUI-originated ack/shelve arrives from
|
||||
// NodeCommunicationActor on the node-local EventStream under
|
||||
// MeshTransport:Mode = ClusterClient; the DPS subscribe above still carries the node-LOCAL
|
||||
// MeshTransport:Mode = Grpc; the DPS subscribe above still carries the node-LOCAL
|
||||
// publisher (OtOpcUaServerHostedService, the OPC UA Part 9 method calls), which never
|
||||
// crosses the boundary and stays on DPS in both modes. Subscribing to both is safe: exactly
|
||||
// one publisher feeds each path.
|
||||
|
||||
@@ -64,10 +64,10 @@ public class MeshTransportOptionsValidatorTests
|
||||
[Fact]
|
||||
public void Unknown_mode_fails()
|
||||
{
|
||||
var result = Validate(new MeshTransportOptions { Mode = "receptionist" });
|
||||
var result = Validate(new MeshTransportOptions { Mode = "not-a-transport" });
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain("receptionist");
|
||||
result.FailureMessage.ShouldContain("not-a-transport");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -94,7 +94,7 @@ public class SplitTopologyTransportValidatorTests
|
||||
// still a mode that delivers nothing across the boundary. The retired 'ClusterClient' value
|
||||
// lands here too.
|
||||
var result = Validate(
|
||||
meshMode: "receptionist",
|
||||
meshMode: "not-a-transport",
|
||||
telemetryMode: TelemetryOptions.ModeGrpc,
|
||||
telemetryDialMode: null,
|
||||
"driver", "cluster-SITE-A");
|
||||
|
||||
@@ -12,7 +12,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Round-trip goldens for <see cref="MeshTransportDtoMapper"/> — the ClusterClient→gRPC command
|
||||
/// Round-trip goldens for <see cref="MeshTransportDtoMapper"/> — the mesh gRPC command
|
||||
/// plane's only data contract. Every golden goes domain → proto → <b>serialized bytes</b> → proto →
|
||||
/// domain, because the failure class this suite exists to catch is not a mis-assigned property but a
|
||||
/// member that survives an in-memory copy and vanishes on the wire: proto3 explicit presence is a
|
||||
|
||||
@@ -6,7 +6,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Contract-lock for the ClusterClient→gRPC command envelope's oneof. If a fifth command is added
|
||||
/// Contract-lock for the mesh command envelope's oneof. If a fifth command is added
|
||||
/// to <c>mesh_transport.proto</c> without adding a matching entry to
|
||||
/// <see cref="MeshTransportProtoContract.HandledCases"/>, this test goes red — the single guard
|
||||
/// that keeps the wire contract and the handled-case set from silently drifting apart. A drift
|
||||
|
||||
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
|
||||
/// <summary>
|
||||
/// Round-trips <see cref="ClusterNode.AkkaPort"/> / <see cref="ClusterNode.GrpcPort"/> through
|
||||
/// the real schema (per-cluster mesh Phase 1). These are <b>central's dial targets</b>: Phase 2
|
||||
/// builds its ClusterClient contact points from <c>Host</c> + <c>AkkaPort</c>, and Phase 5 dials
|
||||
/// built its Akka contact points from <c>Host</c> + <c>AkkaPort</c>, and Phase 5 dials
|
||||
/// <c>GrpcPort</c> for the telemetry stream. Central cannot see a site node's appsettings once
|
||||
/// the meshes split, so the value has to live in a row central can read.
|
||||
/// </summary>
|
||||
|
||||
@@ -14,9 +14,9 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane;
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// The central-side gRPC ingress for a node's deployment ack (ClusterClient→gRPC migration
|
||||
/// The central-side gRPC ingress for a node's deployment ack (mesh gRPC command plane
|
||||
/// Phase 1B). It decodes the wire message and hands it to the registered
|
||||
/// <c>CentralCommunicationActor</c> — the same actor the ClusterClient receptionist delivered to,
|
||||
/// <c>CentralCommunicationActor</c> — the same actor the DistributedPubSub leg delivers to,
|
||||
/// whose forward-to-the-coordinator path is untouched.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
|
||||
+1
-1
@@ -471,7 +471,7 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
|
||||
{
|
||||
// The refresh is not decoration: operators add and retire nodes, and central must pick them
|
||||
// up without a restart. Under Grpc that is the ONLY thing keeping the dispatch set current —
|
||||
// there is no receptionist to notice a new registration on its own.
|
||||
// nothing on the node side announces itself; the DB row is the only source.
|
||||
//
|
||||
// Driven by telling the actor its own RefreshContacts tick rather than by shortening
|
||||
// ContactRefreshSeconds: a one-second timer would run for the whole fact, adding DB load to
|
||||
|
||||
+1
-1
@@ -13,7 +13,7 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// ClusterClient→gRPC Phase 1A — central's outbound command dial. One node, one call, no retry.
|
||||
/// mesh gRPC command plane, Phase 1A — central's outbound command dial. One node, one call, no retry.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The facts that matter are the credential ones: the bearer must be the TARGET node's cluster's
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ using ZB.MOM.WW.Secrets.Abstractions;
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// ClusterClient→gRPC Phase 0 — central's per-cluster PSK lookup. Central talks to N site
|
||||
/// mesh gRPC command plane, Phase 0 — central's per-cluster PSK lookup. Central talks to N site
|
||||
/// clusters, each with its own key, so every outbound dial and every inbound ack verification
|
||||
/// goes through here.
|
||||
/// </summary>
|
||||
|
||||
@@ -10,11 +10,12 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <c>/user/central-communication</c> and <c>/user/node-communication</c> are the wire
|
||||
/// contract of the central↔node boundary: central sends to a path string, and the node
|
||||
/// registers under one. Nothing else in the codebase asserts they agree. Renaming either
|
||||
/// actor would compile, pass every unit test, deploy — and deliver nothing, with no error
|
||||
/// on either side, because a ClusterClient send to an unregistered path is simply dropped.
|
||||
/// <c>/user/central-communication</c> and <c>/user/node-communication</c> are where each
|
||||
/// side's gRPC ingress expects to find its comm actor: the services resolve them by registry
|
||||
/// key, and these constants are the names the actors are spawned under. Nothing else in the
|
||||
/// codebase asserts they agree. Renaming either actor would compile, pass every unit test,
|
||||
/// deploy — and refuse every inbound call as <c>Unavailable</c> ("the node is still
|
||||
/// starting"), which reads as a boot-window race rather than a permanent misnaming.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Booting the real host is the point. A test that spawned the actors itself would pin the
|
||||
@@ -30,8 +31,8 @@ public class MeshCommActorPathTests
|
||||
.Ask<ActorIdentity>(new Identify(path), TimeSpan.FromSeconds(10));
|
||||
|
||||
identity.Subject.ShouldNotBeNull(
|
||||
$"{path} is the ClusterClient wire contract for the central↔node boundary; nothing else "
|
||||
+ "pins it, and a mismatch delivers nothing without erroring on either side");
|
||||
$"{path} is where the central↔node boundary expects its comm actor; nothing else pins "
|
||||
+ "it, and a mismatch turns every inbound call into a misleading readiness rejection");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -60,7 +61,7 @@ public class MeshCommActorPathTests
|
||||
// is left to live-gate step 8 (stop the OLDEST central; the survivor's comm actor must still
|
||||
// receive a node ack, which a singleton hosted on the stopped node cannot).
|
||||
//
|
||||
// NOT MeshClusterClientBoundaryTests: that class proves a send only arrives if the target is
|
||||
// registered with the receptionist, which is necessary but not discriminating — with one node
|
||||
// NOT MeshGrpcBoundaryTests: that class proves a command only arrives if the target is
|
||||
// registered under its registry key, which is necessary but not discriminating — with one node
|
||||
// per side, per-node and singleton registration look identical.
|
||||
}
|
||||
|
||||
@@ -0,0 +1,422 @@
|
||||
using System.Diagnostics.CodeAnalysis;
|
||||
using System.Net;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Akka.Hosting;
|
||||
using Grpc.Core;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Server.Kestrel.Core;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
||||
using AkkaCluster = Akka.Cluster.Cluster;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Drives the mesh command plane across two genuinely separate <see cref="ActorSystem"/>s over
|
||||
/// a real gRPC wire — the only test in the repo where the central↔node boundary carries a
|
||||
/// message off-process.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Replaces <c>MeshClusterClientBoundaryTests</c>, which did the same job for the Akka
|
||||
/// ClusterClient transport that this program deleted. The reason it exists is unchanged:
|
||||
/// a fake dialer tests actor <i>logic</i> and cannot fail for any transport reason — not a
|
||||
/// service that was never mapped, not an interceptor that refuses the call, not a
|
||||
/// serialization break. Those are the failure modes this class exercises, end to end
|
||||
/// through the REAL <c>CentralCommunicationActor</c>, <c>MeshGrpcCommandDialer</c>,
|
||||
/// <c>MeshCommandAuthInterceptor</c>, <c>NodeCommandGrpcService</c> and
|
||||
/// <c>NodeCommunicationActor</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Three clusters, one ActorSystem name, joined ONLY by gRPC.</b> The shared name is kept
|
||||
/// from the predecessor so the shape is comparable; each system seeds itself and none lists
|
||||
/// another, and
|
||||
/// <see cref="No_cross_mesh_Akka_association_forms_between_central_and_either_site"/>
|
||||
/// asserts the separation outright. That assertion is the whole point of the program — the
|
||||
/// cross-WAN Akka association is gone — so it is kept as a test rather than left to a live
|
||||
/// gate: if the three systems ever gossiped into one mesh, every delivery below could be
|
||||
/// plain local messaging and the class would be testing nothing.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Per-cluster keys, not one fleet key.</b> Each site node's interceptor is configured
|
||||
/// with its OWN <c>MeshTransport:GrpcPsk</c> and central's provider hands the dialer the key
|
||||
/// for the cluster it is dialling. A fan that presented the wrong cluster's key would be
|
||||
/// refused at the far side, so the happy path below also pins that the dialer selects by
|
||||
/// <c>ClusterId</c>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class MeshGrpcBoundaryTests : IAsyncLifetime
|
||||
{
|
||||
private const string SharedSystemName = "otopcua";
|
||||
private const string SiteAKey = "psk-site-a";
|
||||
private const string SiteBKey = "psk-site-b";
|
||||
|
||||
private static readonly TimeSpan Arrival = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// How long the falsifiability control waits before concluding nothing arrived. Deliberately
|
||||
/// shorter than <see cref="Arrival"/> but far longer than a successful delivery takes in
|
||||
/// practice, so a pass means "the boundary refused it", not "the clock ran out first".
|
||||
/// </summary>
|
||||
private static readonly TimeSpan Silence = TimeSpan.FromSeconds(10);
|
||||
|
||||
private ActorSystem _central = null!;
|
||||
private SiteNode _siteA = null!;
|
||||
private SiteNode _siteB = null!;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
_central = StartSelfSeededSystem();
|
||||
_siteA = await SiteNode.StartAsync("SITE-A", SiteAKey);
|
||||
_siteB = await SiteNode.StartAsync("SITE-B", SiteBKey);
|
||||
|
||||
await Task.WhenAll(
|
||||
AwaitSingleMemberUpAsync(_central),
|
||||
AwaitSingleMemberUpAsync(_siteA.System),
|
||||
AwaitSingleMemberUpAsync(_siteB.System));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
await _siteA.DisposeAsync();
|
||||
await _siteB.DisposeAsync();
|
||||
await ShutdownAsync(_central);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void No_cross_mesh_Akka_association_forms_between_central_and_either_site()
|
||||
{
|
||||
// THE assertion of the ClusterClient→gRPC program, kept as a test rather than a live-gate
|
||||
// note. Three systems sharing a name in one process are one gossip accident away from being
|
||||
// one cluster, at which point the gRPC boundary is no longer the thing under test — and,
|
||||
// more importantly, the cross-WAN Akka association this program removed would be back.
|
||||
var central = AkkaCluster.Get(_central);
|
||||
var siteA = AkkaCluster.Get(_siteA.System);
|
||||
var siteB = AkkaCluster.Get(_siteB.System);
|
||||
|
||||
central.State.Members.Count.ShouldBe(1,
|
||||
"central must be its own single-member cluster; commands cross to a site over gRPC only");
|
||||
siteA.State.Members.Count.ShouldBe(1, "site-A must be its own single-member cluster");
|
||||
siteB.State.Members.Count.ShouldBe(1, "site-B must be its own single-member cluster");
|
||||
|
||||
// Stated separately from the counts: a count of 1 on each side would also hold in the instant
|
||||
// before a join completed, whereas naming the addresses says no side ever learned another's.
|
||||
foreach (var (name, cluster) in new[]
|
||||
{
|
||||
("central", central), ("site-A", siteA), ("site-B", siteB),
|
||||
})
|
||||
{
|
||||
var others = new[] { central.SelfAddress, siteA.SelfAddress, siteB.SelfAddress }
|
||||
.Where(a => a != cluster.SelfAddress);
|
||||
foreach (var other in others)
|
||||
{
|
||||
cluster.State.Members.ShouldNotContain(
|
||||
m => m.Address == other,
|
||||
$"{name} must not have {other} as a cluster member — the cross-mesh Akka "
|
||||
+ "association is exactly what the gRPC command plane replaced");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh()
|
||||
{
|
||||
// The direct counterpart of the deleted ClusterClient fan-out test, at the transport level and
|
||||
// driving the REAL CentralCommunicationActor. Two genuinely separate site meshes, each with
|
||||
// its own gRPC command endpoint and its own per-cluster key; one MeshCommand must reach BOTH.
|
||||
var dbFactory = SeedTwoClusterDb(_siteA, _siteB);
|
||||
var actor = _central.ActorOf(CentralCommunicationActor.Props(
|
||||
dbFactory,
|
||||
GrpcOptions(),
|
||||
() => null,
|
||||
() => Dialer(),
|
||||
() => new FixedPskProvider()));
|
||||
|
||||
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
|
||||
var deadline = DateTime.UtcNow + Arrival;
|
||||
while (DateTime.UtcNow < deadline
|
||||
&& !(_siteA.Inbox.Task.IsCompleted && _siteB.Inbox.Task.IsCompleted))
|
||||
{
|
||||
// Re-sent rather than sent once: the actor's first contact load is a piped DB read, and a
|
||||
// command told before it lands is dropped as "no targets" by design. Re-sending weakens
|
||||
// nothing — the wrong-key control below re-sends on the same schedule and must still
|
||||
// receive nothing.
|
||||
actor.Tell(new MeshCommand("deployments", command));
|
||||
await Task.Delay(250, TestContext.Current.CancellationToken);
|
||||
}
|
||||
|
||||
_siteA.Inbox.Task.IsCompleted.ShouldBeTrue(
|
||||
"site-A's mesh must receive the dispatch on its own gRPC command endpoint");
|
||||
_siteB.Inbox.Task.IsCompleted.ShouldBeTrue(
|
||||
"site-B's mesh must receive the SAME dispatch on its own endpoint, under its own "
|
||||
+ "per-cluster key — a fan that stopped at the first cluster would leave site-B silently "
|
||||
+ "on its old configuration while the deployment sealed green");
|
||||
|
||||
(await _siteA.Inbox.Task).ShouldBeOfType<RestartDriver>().DriverInstanceId.ShouldBe("driver-1");
|
||||
(await _siteB.Inbox.Task).ShouldBeOfType<RestartDriver>().DriverInstanceId.ShouldBe("driver-1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing()
|
||||
{
|
||||
// Without this, the test above cannot distinguish "the gRPC boundary carried the command"
|
||||
// from "something else in the process delivered it". Same wire, same service, same comm
|
||||
// actor — and with a key site-A's interceptor does not accept, it must NOT arrive.
|
||||
var envelope = MeshTransportDtoMapper.ToEnvelope(
|
||||
new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid()));
|
||||
var dialer = Dialer(new FixedPskProvider("not-the-key"));
|
||||
|
||||
var deadline = DateTime.UtcNow + Silence;
|
||||
while (DateTime.UtcNow < deadline && !_siteA.Inbox.Task.IsCompleted)
|
||||
{
|
||||
try
|
||||
{
|
||||
await dialer.DispatchAsync(
|
||||
_siteA.Endpoint, envelope, "SITE-A", TestContext.Current.CancellationToken);
|
||||
}
|
||||
catch (RpcException)
|
||||
{
|
||||
// Expected: the node's MeshCommandAuthInterceptor refuses the bearer. The assertion
|
||||
// is what did NOT arrive, so the rejection itself is swallowed rather than asserted
|
||||
// on here — MeshCommandAuthInterceptorTests owns the status-code contract.
|
||||
}
|
||||
|
||||
await Task.Delay(500, TestContext.Current.CancellationToken);
|
||||
}
|
||||
|
||||
_siteA.Inbox.Task.IsCompleted.ShouldBeFalse(
|
||||
"delivery survived presenting the wrong per-cluster key, so the positive test above is "
|
||||
+ "not measuring the authenticated gRPC boundary at all");
|
||||
|
||||
// Same node, same dialer shape, right key: proves the silence above was the auth gate rather
|
||||
// than a dead endpoint or a misaddressed call.
|
||||
await Dialer().DispatchAsync(
|
||||
_siteA.Endpoint, envelope, "SITE-A", TestContext.Current.CancellationToken);
|
||||
await _siteA.Inbox.Task.WaitAsync(Arrival, TestContext.Current.CancellationToken);
|
||||
}
|
||||
|
||||
// ---- harness ---------------------------------------------------------------------------------
|
||||
|
||||
private static MeshGrpcCommandDialer Dialer(IMeshPskProvider? psk = null) =>
|
||||
new(psk ?? new FixedPskProvider(), Options.Create(GrpcOptions()),
|
||||
NullLogger<MeshGrpcCommandDialer>.Instance);
|
||||
|
||||
private static MeshTransportOptions GrpcOptions() => new()
|
||||
{
|
||||
Mode = MeshTransportOptions.ModeGrpc,
|
||||
ContactRefreshSeconds = 1,
|
||||
DispatchTimeoutSeconds = 10,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Hands out the per-cluster key each site node's interceptor expects. Stands in for the real
|
||||
/// <c>MeshPskProvider</c>, whose own subject (secret-store lookup, caching, the known-cluster
|
||||
/// gate) is <c>MeshPskProviderTests</c>' and would add a KEK and a SQLite file to a class
|
||||
/// whose subject is the wire.
|
||||
/// </summary>
|
||||
private sealed class FixedPskProvider : IMeshPskProvider
|
||||
{
|
||||
private readonly string? _override;
|
||||
|
||||
public FixedPskProvider(string? overrideKey = null) => _override = overrideKey;
|
||||
|
||||
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
|
||||
{
|
||||
psk = _override ?? KeyFor(clusterId);
|
||||
return psk is not null;
|
||||
}
|
||||
|
||||
public bool Matches(string clusterId, string presented) =>
|
||||
string.Equals(presented, KeyFor(clusterId), StringComparison.Ordinal);
|
||||
|
||||
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) { }
|
||||
|
||||
public void Invalidate(string clusterId) { }
|
||||
|
||||
public void InvalidateAll() { }
|
||||
|
||||
private static string? KeyFor(string clusterId) => clusterId switch
|
||||
{
|
||||
"SITE-A" => SiteAKey,
|
||||
"SITE-B" => SiteBKey,
|
||||
_ => null,
|
||||
};
|
||||
}
|
||||
|
||||
private static IDbContextFactory<OtOpcUaConfigDbContext> SeedTwoClusterDb(SiteNode a, SiteNode b)
|
||||
{
|
||||
var factory = new InMemoryConfigDbFactory(Guid.NewGuid().ToString("N"));
|
||||
using var db = factory.CreateDbContext();
|
||||
db.ClusterNodes.Add(NodeRow("SITE-A", "node-a", a.Port));
|
||||
db.ClusterNodes.Add(NodeRow("SITE-B", "node-b", b.Port));
|
||||
db.SaveChanges();
|
||||
return factory;
|
||||
}
|
||||
|
||||
private static ClusterNode NodeRow(string clusterId, string nodeId, int grpcPort) => new()
|
||||
{
|
||||
NodeId = nodeId,
|
||||
ClusterId = clusterId,
|
||||
Host = "127.0.0.1",
|
||||
AkkaPort = 4053,
|
||||
GrpcPort = grpcPort,
|
||||
ApplicationUri = $"urn:{nodeId}",
|
||||
Enabled = true,
|
||||
MaintenanceMode = false,
|
||||
CreatedBy = "test",
|
||||
};
|
||||
|
||||
private sealed class InMemoryConfigDbFactory(string dbName)
|
||||
: IDbContextFactory<OtOpcUaConfigDbContext>
|
||||
{
|
||||
public OtOpcUaConfigDbContext CreateDbContext()
|
||||
{
|
||||
var opts = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
|
||||
.UseInMemoryDatabase(dbName)
|
||||
.Options;
|
||||
return new OtOpcUaConfigDbContext(opts);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One site: its own single-member cluster, its own real <c>NodeCommunicationActor</c>, and a
|
||||
/// real h2c Kestrel listener serving the real <c>NodeCommandGrpcService</c> behind the real
|
||||
/// <c>MeshCommandAuthInterceptor</c> — the production shape of a driver node's command port.
|
||||
/// </summary>
|
||||
private sealed class SiteNode : IAsyncDisposable
|
||||
{
|
||||
private WebApplication _server = null!;
|
||||
|
||||
public ActorSystem System { get; private set; } = null!;
|
||||
|
||||
public int Port { get; private set; }
|
||||
|
||||
public string Endpoint => $"http://127.0.0.1:{Port}";
|
||||
|
||||
/// <summary>Completes with the first command re-emitted on this node's EventStream.</summary>
|
||||
public TaskCompletionSource<object> Inbox { get; } =
|
||||
new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
|
||||
public static async Task<SiteNode> StartAsync(string clusterId, string psk)
|
||||
{
|
||||
var node = new SiteNode { System = StartSelfSeededSystem(), Port = GetFreePort() };
|
||||
|
||||
// The REAL comm actor, under the registry key the gRPC service resolves. Nothing else
|
||||
// makes it reachable — there is no receptionist registration any more.
|
||||
var comm = node.System.ActorOf(
|
||||
NodeCommunicationActor.Props(), MeshPaths.NodeCommunicationName);
|
||||
node.System.EventStream.Subscribe(
|
||||
node.System.ActorOf(CaptureActor.Props(node.Inbox)), typeof(RestartDriver));
|
||||
|
||||
var registry = new ActorRegistry();
|
||||
registry.Register<NodeCommunicationActorKey>(comm);
|
||||
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Environment.ApplicationName = typeof(MeshGrpcBoundaryTests).Assembly.GetName().Name!;
|
||||
builder.Logging.ClearProviders();
|
||||
builder.Services.AddSingleton(registry);
|
||||
builder.Services.Configure<MeshTransportOptions>(o =>
|
||||
{
|
||||
o.Mode = MeshTransportOptions.ModeGrpc;
|
||||
o.GrpcPsk = psk;
|
||||
});
|
||||
builder.Services.AddGrpc(o => o.Interceptors.Add<MeshCommandAuthInterceptor>());
|
||||
builder.WebHost.ConfigureKestrel(k =>
|
||||
k.ListenAnyIP(node.Port, o => o.Protocols = HttpProtocols.Http2));
|
||||
|
||||
node._server = builder.Build();
|
||||
node._server.MapGrpcService<NodeCommandGrpcService>();
|
||||
await node._server.StartAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
_ = clusterId; // Named for readability at the call site; the key IS the cluster boundary.
|
||||
return node;
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
await _server.StopAsync(TimeSpan.FromSeconds(10));
|
||||
await _server.DisposeAsync();
|
||||
await ShutdownAsync(System);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Starts a single-member cluster on a dynamic remoting port.</summary>
|
||||
/// <remarks>
|
||||
/// Port 0 + a post-start self-Join, rather than a seed-nodes list: the address is not known
|
||||
/// until the transport binds, and hard-coding ports makes the test collide with itself under
|
||||
/// parallel execution.
|
||||
/// </remarks>
|
||||
/// <returns>The started system.</returns>
|
||||
private static ActorSystem StartSelfSeededSystem()
|
||||
{
|
||||
var config = ConfigurationFactory.ParseString(
|
||||
"akka.remote.dot-netty.tcp.hostname = \"127.0.0.1\"\n"
|
||||
+ "akka.remote.dot-netty.tcp.public-hostname = \"127.0.0.1\"\n"
|
||||
+ "akka.remote.dot-netty.tcp.port = 0\n"
|
||||
+ "akka.cluster.seed-nodes = []\n"
|
||||
+ "akka.loglevel = \"WARNING\"\n")
|
||||
.WithFallback(HoconLoader.LoadBaseConfig());
|
||||
|
||||
var system = ActorSystem.Create(SharedSystemName, config);
|
||||
var cluster = AkkaCluster.Get(system);
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
return system;
|
||||
}
|
||||
|
||||
private static async Task AwaitSingleMemberUpAsync(ActorSystem system)
|
||||
{
|
||||
var cluster = AkkaCluster.Get(system);
|
||||
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up)) return;
|
||||
await Task.Delay(100, TestContext.Current.CancellationToken);
|
||||
}
|
||||
|
||||
cluster.State.Members.Any(m => m.Status == MemberStatus.Up).ShouldBeTrue(
|
||||
$"{cluster.SelfAddress} never reached Up as its own seed");
|
||||
}
|
||||
|
||||
private static async Task ShutdownAsync(ActorSystem? system)
|
||||
{
|
||||
if (system is null) return;
|
||||
await system.Terminate().WaitAsync(TimeSpan.FromSeconds(15));
|
||||
}
|
||||
|
||||
private static int GetFreePort()
|
||||
{
|
||||
using var listener = new System.Net.Sockets.TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
|
||||
/// <summary>Completes an inbox with the first message it receives.</summary>
|
||||
private sealed class CaptureActor : ReceiveActor
|
||||
{
|
||||
public static Props Props(TaskCompletionSource<object> sink) =>
|
||||
Akka.Actor.Props.Create(() => new CaptureActor(sink));
|
||||
|
||||
public CaptureActor(TaskCompletionSource<object> sink) => ReceiveAny(m => sink.TrySetResult(m));
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -9,7 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// ClusterClient→gRPC Phase 0 — the central-side ack endpoint's inbound gate. Mapping
|
||||
/// mesh gRPC command plane, Phase 0 — the central-side ack endpoint's inbound gate. Mapping
|
||||
/// <c>CentralCommandService</c> onto the pipeline is a Phase 1 task; this test covers only the
|
||||
/// interceptor's own authorization behavior.
|
||||
/// </summary>
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// ClusterClient→gRPC Phase 0 — the node-side command endpoint's inbound gate. Mapping
|
||||
/// mesh gRPC command plane, Phase 0 — the node-side command endpoint's inbound gate. Mapping
|
||||
/// <c>NodeCommandService</c> onto the pipeline is a Phase 1 task; this test covers only the
|
||||
/// interceptor's own authorization behavior.
|
||||
/// </summary>
|
||||
|
||||
@@ -16,10 +16,10 @@ using ZB.MOM.WW.OtOpcUa.Runtime;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the node-side command ingress (ClusterClient→gRPC migration Phase 1): each of the four
|
||||
/// Verifies the node-side command ingress (mesh gRPC command plane, Phase 1): each of the four
|
||||
/// central→node commands is decoded off its <see cref="CommandEnvelope"/> and handed to the
|
||||
/// registered <c>NodeCommunicationActor</c> as the same CLR record the ClusterClient transport
|
||||
/// delivered, an undecodable envelope is refused with <see cref="StatusCode.InvalidArgument"/>, and
|
||||
/// registered <c>NodeCommunicationActor</c> as the same CLR record every other transport
|
||||
/// delivers, an undecodable envelope is refused with <see cref="StatusCode.InvalidArgument"/>, and
|
||||
/// a call landing in the boot window before the actor is registered is refused with
|
||||
/// <see cref="StatusCode.Unavailable"/>.
|
||||
/// </summary>
|
||||
|
||||
+3
-3
@@ -13,21 +13,21 @@ using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// The node-side gRPC ack sender (ClusterClient→gRPC migration Task 1B.2). The client is a fake,
|
||||
/// The node-side gRPC ack sender (mesh gRPC command plane, Task 1B.2). The client is a fake,
|
||||
/// so what is under test is the sender's own logic: the ordered single-pass over
|
||||
/// <c>CentralGrpcEndpoints</c>, the per-call credentials, and the drop that used to be silent.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>The load-bearing fact is that ONE endpoint is dialled on success.</b> This replaces an
|
||||
/// Akka <c>ClusterClient.Send</c>, which is single-target at-most-once — not a
|
||||
/// one ack per node per dispatch, single-target at-most-once — not a
|
||||
/// <c>SendToAll</c>. Both central nodes route an ack to the same
|
||||
/// <c>ConfigPublishCoordinator</c> singleton proxy, whose NodeId-keyed upsert absorbs a
|
||||
/// duplicate idempotently — so the single-endpoint rule is defence in depth and a bound
|
||||
/// on wasted dials, not what protects the seal.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The failure side matters just as much: the receptionist dropped an undeliverable ack in
|
||||
/// The failure side matters just as much: an undeliverable ack is dropped in
|
||||
/// silence, and the whole point of the migration is that the same drop is now a Warning naming
|
||||
/// every endpoint and the status it answered with.
|
||||
/// </para>
|
||||
|
||||
Reference in New Issue
Block a user