2ee84af1c0
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
226 lines
9.4 KiB
C#
226 lines
9.4 KiB
C#
using Grpc.Core;
|
|
using Grpc.Net.Client;
|
|
using Microsoft.AspNetCore.Builder;
|
|
using Microsoft.AspNetCore.Hosting;
|
|
using Microsoft.AspNetCore.TestHost;
|
|
using Microsoft.Extensions.DependencyInjection;
|
|
using Microsoft.Extensions.Hosting;
|
|
using Microsoft.Extensions.Logging.Abstractions;
|
|
using Microsoft.Extensions.Options;
|
|
using ZB.MOM.WW.ScadaBridge.Communication;
|
|
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
|
|
|
/// <summary>
|
|
/// End-to-end proof that the two halves of the control-plane PSK actually interoperate:
|
|
/// <see cref="ControlPlaneCredentials"/> on the client and
|
|
/// <see cref="ControlPlaneAuthInterceptor"/> on the server, over a real gRPC stack.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// The unit tests either side of this file each test one half against a hand-built input, and
|
|
/// would both stay green if the halves disagreed — if the credentials never attached to a
|
|
/// streaming call, if the metadata key case differed, or if attaching call credentials to a
|
|
/// plaintext channel were rejected outright (gRPC refuses that by default; the code opts in with
|
|
/// <c>UnsafeUseInsecureChannelCallCredentials</c>, and nothing but a real call proves the opt-in
|
|
/// works). Getting that wrong takes down every site's streaming and audit-pull path at once,
|
|
/// which is a bad thing to discover on the rig.
|
|
/// </para>
|
|
/// <para>
|
|
/// Runs entirely in-process over <see cref="TestServer"/>: no ports, no containers. The service
|
|
/// is a stub rather than the real <c>SiteStreamGrpcServer</c> — this is a test of the auth
|
|
/// pipeline, and the real server would drag in an actor system for no added coverage. The method
|
|
/// paths and message types are the real generated ones.
|
|
/// </para>
|
|
/// </remarks>
|
|
public class ControlPlaneAuthEndToEndTests : IAsyncLifetime
|
|
{
|
|
private IHost _host = null!;
|
|
private TestServer _server = null!;
|
|
|
|
/// <summary>Boots the in-process gRPC host with the real interceptor.</summary>
|
|
public async Task InitializeAsync()
|
|
{
|
|
_host = await new HostBuilder()
|
|
.ConfigureWebHost(web => web
|
|
.UseTestServer()
|
|
.ConfigureServices(services =>
|
|
{
|
|
services.AddGrpc(o => o.Interceptors.Add<ControlPlaneAuthInterceptor>());
|
|
services.AddSingleton(Options.Create(
|
|
new CommunicationOptions { GrpcPsk = SiteKey }));
|
|
services.AddSingleton<ControlPlaneAuthInterceptor>();
|
|
services.AddSingleton<EchoSiteStreamService>();
|
|
})
|
|
.Configure(app =>
|
|
{
|
|
app.UseRouting();
|
|
app.UseEndpoints(e => e.MapGrpcService<EchoSiteStreamService>());
|
|
}))
|
|
.StartAsync();
|
|
|
|
_server = _host.GetTestServer();
|
|
}
|
|
|
|
/// <inheritdoc />
|
|
public async Task DisposeAsync()
|
|
{
|
|
await _host.StopAsync();
|
|
_host.Dispose();
|
|
}
|
|
|
|
private const string SiteKey = "the-site-a-preshared-key";
|
|
|
|
/// <summary>
|
|
/// Builds a channel through the test server, credentialed exactly as production does.
|
|
/// </summary>
|
|
private GrpcChannel Channel(string? key, string siteId = "site-a")
|
|
{
|
|
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() };
|
|
if (key is not null)
|
|
{
|
|
options.WithSiteCredentials(new FixedPskProvider(key), siteId);
|
|
}
|
|
return GrpcChannel.ForAddress(_server.BaseAddress, options);
|
|
}
|
|
|
|
private sealed class FixedPskProvider(string key) : ISitePskProvider
|
|
{
|
|
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
|
|
public void Invalidate(string siteId) { }
|
|
}
|
|
|
|
/// <summary>Stub service: echoes back what the auth pipeline let through.</summary>
|
|
private sealed class EchoSiteStreamService : SiteStreamService.SiteStreamServiceBase
|
|
{
|
|
/// <summary>The site header the last accepted call carried.</summary>
|
|
public string? LastSiteHeader { get; private set; }
|
|
|
|
public override Task<PullAuditEventsResponse> PullAuditEvents(
|
|
PullAuditEventsRequest request, ServerCallContext context)
|
|
{
|
|
LastSiteHeader = context.RequestHeaders
|
|
.FirstOrDefault(h => h.Key == ControlPlaneCredentials.SiteHeader)?.Value;
|
|
return Task.FromResult(new PullAuditEventsResponse { MoreAvailable = false });
|
|
}
|
|
|
|
public override async Task SubscribeInstance(
|
|
InstanceStreamRequest request,
|
|
IServerStreamWriter<SiteStreamEvent> responseStream,
|
|
ServerCallContext context)
|
|
{
|
|
await responseStream.WriteAsync(new SiteStreamEvent { CorrelationId = request.CorrelationId });
|
|
}
|
|
}
|
|
|
|
[Fact]
|
|
public async Task CorrectKey_IsAccepted_OnAUnaryCall()
|
|
{
|
|
using var channel = Channel(SiteKey);
|
|
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
|
|
|
var reply = await client.PullAuditEventsAsync(new PullAuditEventsRequest { BatchSize = 1 });
|
|
|
|
Assert.False(reply.MoreAvailable);
|
|
}
|
|
|
|
[Fact]
|
|
public async Task WrongKey_IsRejected_WithPermissionDenied()
|
|
{
|
|
using var channel = Channel("some-other-sites-key");
|
|
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
|
|
|
var ex = await Assert.ThrowsAsync<RpcException>(
|
|
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
|
|
|
|
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
|
}
|
|
|
|
[Fact]
|
|
public async Task NoCredentialsAtAll_IsRejected()
|
|
{
|
|
// The pre-T0.3 client shape. This is the case that proves the gap is actually closed.
|
|
using var channel = Channel(key: null);
|
|
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
|
|
|
var ex = await Assert.ThrowsAsync<RpcException>(
|
|
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
|
|
|
|
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
|
}
|
|
|
|
[Fact]
|
|
public async Task CredentialsApplyToStreamingCalls_NotJustUnaryOnes()
|
|
{
|
|
// CallCredentials cover every call on the channel; a client interceptor that only
|
|
// handled the unary path would pass the test above and still break every subscription.
|
|
using var channel = Channel(SiteKey);
|
|
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
|
|
|
using var call = client.SubscribeInstance(
|
|
new InstanceStreamRequest { CorrelationId = "c1", InstanceUniqueName = "i1" });
|
|
|
|
Assert.True(await call.ResponseStream.MoveNext(CancellationToken.None));
|
|
Assert.Equal("c1", call.ResponseStream.Current.CorrelationId);
|
|
}
|
|
|
|
[Fact]
|
|
public async Task WrongKey_IsRejected_OnStreamingCallsToo()
|
|
{
|
|
using var channel = Channel("wrong");
|
|
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
|
|
|
using var call = client.SubscribeInstance(new InstanceStreamRequest { CorrelationId = "c1" });
|
|
|
|
var ex = await Assert.ThrowsAsync<RpcException>(
|
|
async () => await call.ResponseStream.MoveNext(CancellationToken.None));
|
|
|
|
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
|
}
|
|
|
|
[Fact]
|
|
public async Task AnUnresolvableKey_FailsTheCall_RatherThanDialingWithoutOne()
|
|
{
|
|
// SitePskProvider throws when a site has no key anywhere. What matters here is that the
|
|
// throw stops the call: the alternative — swallowing it and sending the request
|
|
// unauthenticated — is the exact failure this design exists to prevent. The status code
|
|
// is gRPC's choice, so assert the RpcException and record what it actually is rather
|
|
// than pinning a guess: callers already treat every non-OK status as a failed call, and
|
|
// the diagnosable signal is SitePskProvider's own LogError, not this code.
|
|
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() }
|
|
.WithSiteCredentials(new ThrowingPskProvider(), "site-a");
|
|
using var channel = GrpcChannel.ForAddress(_server.BaseAddress, options);
|
|
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
|
|
|
var ex = await Assert.ThrowsAsync<RpcException>(
|
|
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
|
|
|
|
Assert.NotEqual(StatusCode.OK, ex.StatusCode);
|
|
|
|
// And nothing reached the service.
|
|
Assert.Null(_host.Services.GetRequiredService<EchoSiteStreamService>().LastSiteHeader);
|
|
}
|
|
|
|
private sealed class ThrowingPskProvider : ISitePskProvider
|
|
{
|
|
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
|
|
=> throw new InvalidOperationException($"no key for '{siteId}'");
|
|
public void Invalidate(string siteId) { }
|
|
}
|
|
|
|
[Fact]
|
|
public async Task TheSiteHeaderTravels_SoCentralCanPickAPerSiteKeyInPhase1A()
|
|
{
|
|
// Central's own interceptor (T1A.2) verifies against the key for the site named in this
|
|
// header. Shipping it now means Phase 1A adds a lookup, not a wire change.
|
|
using var channel = Channel(SiteKey, siteId: "site-a");
|
|
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
|
|
|
await client.PullAuditEventsAsync(new PullAuditEventsRequest());
|
|
|
|
var service = _host.Services.GetRequiredService<EchoSiteStreamService>();
|
|
Assert.Equal("site-a", service.LastSiteHeader);
|
|
}
|
|
}
|