feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
This commit is contained in:
Joseph Doherty
+2
-2
@@ -58,7 +58,7 @@ public class GrpcPullAuditEventsClientTests
|
||||
public static FakeInvoker Throwing(Exception ex) => new(null, ex);
|
||||
|
||||
public Task<ProtoPullResponse> InvokeAsync(
|
||||
string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
{
|
||||
CallCount++;
|
||||
Endpoint = endpoint;
|
||||
@@ -84,7 +84,7 @@ public class GrpcPullAuditEventsClientTests
|
||||
_byEndpoint = byEndpoint;
|
||||
|
||||
public Task<ProtoPullResponse> InvokeAsync(
|
||||
string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
{
|
||||
Dialed.Add(endpoint);
|
||||
return Task.FromResult(_byEndpoint[endpoint]());
|
||||
|
||||
@@ -50,7 +50,7 @@ public class GrpcPullSiteCallsClientTests
|
||||
public static FakeInvoker Throwing(Exception ex) => new(null, ex);
|
||||
|
||||
public Task<ProtoPullResponse> InvokeAsync(
|
||||
string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
{
|
||||
CallCount++;
|
||||
Endpoint = endpoint;
|
||||
@@ -77,7 +77,7 @@ public class GrpcPullSiteCallsClientTests
|
||||
_byEndpoint = byEndpoint;
|
||||
|
||||
public Task<ProtoPullResponse> InvokeAsync(
|
||||
string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
string siteId, string endpoint, ProtoPullRequest request, CancellationToken ct)
|
||||
{
|
||||
Dialed.Add(endpoint);
|
||||
// The Func may throw (transport fault) — the client's try/catch handles it.
|
||||
|
||||
+1
-1
@@ -41,7 +41,7 @@ public class SiteStreamGrpcClientFactoryDisposeTests
|
||||
|
||||
public IReadOnlyCollection<TrackingClient> Created => _created.ToList();
|
||||
|
||||
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint)
|
||||
protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
|
||||
{
|
||||
var client = new TrackingClient();
|
||||
_created.Add(client);
|
||||
|
||||
+1
-1
@@ -155,7 +155,7 @@ public class SiteStreamGrpcClientFactoryTests
|
||||
{
|
||||
public TrackingEndpointFactory() : base(NullLoggerFactory.Instance) { }
|
||||
public int CreatedCount { get; private set; }
|
||||
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint)
|
||||
protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
|
||||
{
|
||||
CreatedCount++;
|
||||
return new TrackingEndpointClient(grpcEndpoint);
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
using Grpc.Core;
|
||||
using Grpc.Net.Client;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.TestHost;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// End-to-end proof that the two halves of the control-plane PSK actually interoperate:
|
||||
/// <see cref="ControlPlaneCredentials"/> on the client and
|
||||
/// <see cref="ControlPlaneAuthInterceptor"/> on the server, over a real gRPC stack.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The unit tests either side of this file each test one half against a hand-built input, and
|
||||
/// would both stay green if the halves disagreed — if the credentials never attached to a
|
||||
/// streaming call, if the metadata key case differed, or if attaching call credentials to a
|
||||
/// plaintext channel were rejected outright (gRPC refuses that by default; the code opts in with
|
||||
/// <c>UnsafeUseInsecureChannelCallCredentials</c>, and nothing but a real call proves the opt-in
|
||||
/// works). Getting that wrong takes down every site's streaming and audit-pull path at once,
|
||||
/// which is a bad thing to discover on the rig.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Runs entirely in-process over <see cref="TestServer"/>: no ports, no containers. The service
|
||||
/// is a stub rather than the real <c>SiteStreamGrpcServer</c> — this is a test of the auth
|
||||
/// pipeline, and the real server would drag in an actor system for no added coverage. The method
|
||||
/// paths and message types are the real generated ones.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public class ControlPlaneAuthEndToEndTests : IAsyncLifetime
|
||||
{
|
||||
private IHost _host = null!;
|
||||
private TestServer _server = null!;
|
||||
|
||||
/// <summary>Boots the in-process gRPC host with the real interceptor.</summary>
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
_host = await new HostBuilder()
|
||||
.ConfigureWebHost(web => web
|
||||
.UseTestServer()
|
||||
.ConfigureServices(services =>
|
||||
{
|
||||
services.AddGrpc(o => o.Interceptors.Add<ControlPlaneAuthInterceptor>());
|
||||
services.AddSingleton(Options.Create(
|
||||
new CommunicationOptions { GrpcPsk = SiteKey }));
|
||||
services.AddSingleton<ControlPlaneAuthInterceptor>();
|
||||
services.AddSingleton<EchoSiteStreamService>();
|
||||
})
|
||||
.Configure(app =>
|
||||
{
|
||||
app.UseRouting();
|
||||
app.UseEndpoints(e => e.MapGrpcService<EchoSiteStreamService>());
|
||||
}))
|
||||
.StartAsync();
|
||||
|
||||
_server = _host.GetTestServer();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
await _host.StopAsync();
|
||||
_host.Dispose();
|
||||
}
|
||||
|
||||
private const string SiteKey = "the-site-a-preshared-key";
|
||||
|
||||
/// <summary>
|
||||
/// Builds a channel through the test server, credentialed exactly as production does.
|
||||
/// </summary>
|
||||
private GrpcChannel Channel(string? key, string siteId = "site-a")
|
||||
{
|
||||
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() };
|
||||
if (key is not null)
|
||||
{
|
||||
options.WithSiteCredentials(new FixedPskProvider(key), siteId);
|
||||
}
|
||||
return GrpcChannel.ForAddress(_server.BaseAddress, options);
|
||||
}
|
||||
|
||||
private sealed class FixedPskProvider(string key) : ISitePskProvider
|
||||
{
|
||||
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new(key);
|
||||
public void Invalidate(string siteId) { }
|
||||
}
|
||||
|
||||
/// <summary>Stub service: echoes back what the auth pipeline let through.</summary>
|
||||
private sealed class EchoSiteStreamService : SiteStreamService.SiteStreamServiceBase
|
||||
{
|
||||
/// <summary>The site header the last accepted call carried.</summary>
|
||||
public string? LastSiteHeader { get; private set; }
|
||||
|
||||
public override Task<PullAuditEventsResponse> PullAuditEvents(
|
||||
PullAuditEventsRequest request, ServerCallContext context)
|
||||
{
|
||||
LastSiteHeader = context.RequestHeaders
|
||||
.FirstOrDefault(h => h.Key == ControlPlaneCredentials.SiteHeader)?.Value;
|
||||
return Task.FromResult(new PullAuditEventsResponse { MoreAvailable = false });
|
||||
}
|
||||
|
||||
public override async Task SubscribeInstance(
|
||||
InstanceStreamRequest request,
|
||||
IServerStreamWriter<SiteStreamEvent> responseStream,
|
||||
ServerCallContext context)
|
||||
{
|
||||
await responseStream.WriteAsync(new SiteStreamEvent { CorrelationId = request.CorrelationId });
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CorrectKey_IsAccepted_OnAUnaryCall()
|
||||
{
|
||||
using var channel = Channel(SiteKey);
|
||||
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
||||
|
||||
var reply = await client.PullAuditEventsAsync(new PullAuditEventsRequest { BatchSize = 1 });
|
||||
|
||||
Assert.False(reply.MoreAvailable);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WrongKey_IsRejected_WithPermissionDenied()
|
||||
{
|
||||
using var channel = Channel("some-other-sites-key");
|
||||
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(
|
||||
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
|
||||
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NoCredentialsAtAll_IsRejected()
|
||||
{
|
||||
// The pre-T0.3 client shape. This is the case that proves the gap is actually closed.
|
||||
using var channel = Channel(key: null);
|
||||
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(
|
||||
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
|
||||
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CredentialsApplyToStreamingCalls_NotJustUnaryOnes()
|
||||
{
|
||||
// CallCredentials cover every call on the channel; a client interceptor that only
|
||||
// handled the unary path would pass the test above and still break every subscription.
|
||||
using var channel = Channel(SiteKey);
|
||||
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
||||
|
||||
using var call = client.SubscribeInstance(
|
||||
new InstanceStreamRequest { CorrelationId = "c1", InstanceUniqueName = "i1" });
|
||||
|
||||
Assert.True(await call.ResponseStream.MoveNext(CancellationToken.None));
|
||||
Assert.Equal("c1", call.ResponseStream.Current.CorrelationId);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WrongKey_IsRejected_OnStreamingCallsToo()
|
||||
{
|
||||
using var channel = Channel("wrong");
|
||||
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
||||
|
||||
using var call = client.SubscribeInstance(new InstanceStreamRequest { CorrelationId = "c1" });
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(
|
||||
async () => await call.ResponseStream.MoveNext(CancellationToken.None));
|
||||
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnUnresolvableKey_FailsTheCall_RatherThanDialingWithoutOne()
|
||||
{
|
||||
// SitePskProvider throws when a site has no key anywhere. What matters here is that the
|
||||
// throw stops the call: the alternative — swallowing it and sending the request
|
||||
// unauthenticated — is the exact failure this design exists to prevent. The status code
|
||||
// is gRPC's choice, so assert the RpcException and record what it actually is rather
|
||||
// than pinning a guess: callers already treat every non-OK status as a failed call, and
|
||||
// the diagnosable signal is SitePskProvider's own LogError, not this code.
|
||||
var options = new GrpcChannelOptions { HttpHandler = _server.CreateHandler() }
|
||||
.WithSiteCredentials(new ThrowingPskProvider(), "site-a");
|
||||
using var channel = GrpcChannel.ForAddress(_server.BaseAddress, options);
|
||||
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(
|
||||
async () => await client.PullAuditEventsAsync(new PullAuditEventsRequest()));
|
||||
|
||||
Assert.NotEqual(StatusCode.OK, ex.StatusCode);
|
||||
|
||||
// And nothing reached the service.
|
||||
Assert.Null(_host.Services.GetRequiredService<EchoSiteStreamService>().LastSiteHeader);
|
||||
}
|
||||
|
||||
private sealed class ThrowingPskProvider : ISitePskProvider
|
||||
{
|
||||
public ValueTask<string> GetAsync(string siteId, CancellationToken ct)
|
||||
=> throw new InvalidOperationException($"no key for '{siteId}'");
|
||||
public void Invalidate(string siteId) { }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TheSiteHeaderTravels_SoCentralCanPickAPerSiteKeyInPhase1A()
|
||||
{
|
||||
// Central's own interceptor (T1A.2) verifies against the key for the site named in this
|
||||
// header. Shipping it now means Phase 1A adds a lookup, not a wire change.
|
||||
using var channel = Channel(SiteKey, siteId: "site-a");
|
||||
var client = new SiteStreamService.SiteStreamServiceClient(channel);
|
||||
|
||||
await client.PullAuditEventsAsync(new PullAuditEventsRequest());
|
||||
|
||||
var service = _host.Services.GetRequiredService<EchoSiteStreamService>();
|
||||
Assert.Equal("site-a", service.LastSiteHeader);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The site↔central gRPC control plane's inbound gate (ClusterClient→gRPC migration, T0.3).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <c>SiteStreamService</c> shipped unauthenticated: plaintext h2c with no interceptor, so
|
||||
/// anything that could reach a site node's gRPC port could open a live data stream or pull audit
|
||||
/// rows back with <c>PullAuditEvents</c>/<c>PullSiteCalls</c>. These tests pin the gate that
|
||||
/// closes it, and — just as importantly — pin that it does NOT gate LocalDb sync, which has its
|
||||
/// own interceptor and its own key.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Sibling of <see cref="LocalDbSyncAuthInterceptorTests"/>; the two interceptors share a shape
|
||||
/// deliberately, so the cases mirror each other.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public class ControlPlaneAuthInterceptorTests
|
||||
{
|
||||
// Real method paths: package `sitestream`, service `SiteStreamService` (sitestream.proto).
|
||||
private const string SubscribeMethod = "/sitestream.SiteStreamService/SubscribeInstance";
|
||||
private const string PullAuditMethod = "/sitestream.SiteStreamService/PullAuditEvents";
|
||||
private const string LocalDbSyncMethod = "/localdb_sync.v1.LocalDbSync/Sync";
|
||||
|
||||
private static ControlPlaneAuthInterceptor CreateInterceptor(string? psk)
|
||||
=> new(
|
||||
Options.Create(new CommunicationOptions { GrpcPsk = psk ?? "" }),
|
||||
NullLogger<ControlPlaneAuthInterceptor>.Instance);
|
||||
|
||||
private static ServerCallContext CreateContext(string method, string? authorizationHeader)
|
||||
{
|
||||
var headers = new Metadata();
|
||||
if (authorizationHeader is not null)
|
||||
headers.Add("authorization", authorizationHeader);
|
||||
|
||||
return new FakeServerCallContext(method, headers);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Minimal <see cref="ServerCallContext"/> carrying just a method name and request headers —
|
||||
/// the only two things the interceptor reads. Hand-rolled for the same reason the LocalDb
|
||||
/// sibling hand-rolls one: <c>Grpc.Core.Testing.TestServerCallContext</c> lives in the
|
||||
/// retired native package and does not exist on the grpc-dotnet stack.
|
||||
/// </summary>
|
||||
private sealed class FakeServerCallContext(string method, Metadata requestHeaders)
|
||||
: ServerCallContext
|
||||
{
|
||||
protected override string MethodCore => method;
|
||||
protected override string HostCore => "localhost";
|
||||
protected override string PeerCore => "ipv4:127.0.0.1:12345";
|
||||
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
|
||||
protected override Metadata RequestHeadersCore => requestHeaders;
|
||||
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
|
||||
protected override Metadata ResponseTrailersCore { get; } = [];
|
||||
protected override Status StatusCore { get; set; }
|
||||
protected override WriteOptions? WriteOptionsCore { get; set; }
|
||||
protected override AuthContext AuthContextCore { get; } =
|
||||
new(null, new Dictionary<string, List<AuthProperty>>());
|
||||
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(
|
||||
ContextPropagationOptions? options)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
|
||||
=> Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>Invokes the interceptor's unary path with a trivial continuation.</summary>
|
||||
private static Task<string> Invoke(
|
||||
ControlPlaneAuthInterceptor interceptor, ServerCallContext context)
|
||||
=> interceptor.UnaryServerHandler<string, string>(
|
||||
"request", context, (_, _) => Task.FromResult("ok"));
|
||||
|
||||
[Fact]
|
||||
public async Task LocalDbSyncMethod_PassesThrough_BecauseItHasItsOwnGateAndItsOwnKey()
|
||||
{
|
||||
// Both interceptors sit on the same site AddGrpc pipeline and see every call. If this
|
||||
// one also gated sync, a site would need its central-facing key to equal its pair-replication
|
||||
// key — collapsing two distinct trust relationships into one secret.
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(LocalDbSyncMethod, authorizationHeader: null);
|
||||
|
||||
Assert.Equal("ok", await Invoke(interceptor, context));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GatedMethod_WithNoKeyConfigured_IsDenied_EvenWithABearerToken()
|
||||
{
|
||||
// Fail-closed, and this is the case that differs in consequence from LocalDb's: an
|
||||
// unset key here does not disable an optional feature, it closes the site's entire
|
||||
// central-facing surface. Loud refusal beats silent unauthenticated service.
|
||||
var interceptor = CreateInterceptor(psk: null);
|
||||
var context = CreateContext(SubscribeMethod, "Bearer anything-at-all");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GatedMethod_WithNoBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(SubscribeMethod, authorizationHeader: null);
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GatedMethod_WithWrongBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer some-other-sites-key");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GatedMethod_WithCorrectBearerToken_PassesThrough()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-site-key");
|
||||
|
||||
Assert.Equal("ok", await Invoke(interceptor, context));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GatedMethod_WithCorrectKey_ButNoBearerScheme_IsDenied()
|
||||
{
|
||||
// A raw key with no "Bearer " prefix is not what ControlPlaneCredentials sends;
|
||||
// accepting it would widen the accepted credential shape for nothing.
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(SubscribeMethod, "the-site-key");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GatedMethod_TokenComparison_IsNotAPrefixMatch()
|
||||
{
|
||||
// A StartsWith comparison would accept a truncated key and make the secret recoverable
|
||||
// one character at a time. FixedTimeEquals also rejects on length.
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-site-ke");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ServerStreaming_IsGated_BecauseThatIsHowSubscriptionsActuallyRun()
|
||||
{
|
||||
// SubscribeInstance/SubscribeSite are server-streaming. Gating only the unary path
|
||||
// would leave the live data feed wide open while every unary test still passed.
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() =>
|
||||
interceptor.ServerStreamingServerHandler<string, string>(
|
||||
"request",
|
||||
responseStream: null!,
|
||||
context,
|
||||
(_, _, _) => Task.CompletedTask));
|
||||
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PullRpcs_AreGated_BecauseTheyReturnAuditRows()
|
||||
{
|
||||
// The strongest reason this gate exists: PullAuditEvents/PullSiteCalls hand back audit
|
||||
// content to anyone who asks. Unary, so it would be easy to miss in a streaming-focused
|
||||
// reading of the service.
|
||||
var interceptor = CreateInterceptor("the-site-key");
|
||||
var context = CreateContext(PullAuditMethod, authorizationHeader: null);
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GatedPrefixes_AreConstructorProvided_SoLaterPhasesAddServicesNotInterceptors()
|
||||
{
|
||||
// Phases 1A/1B add CentralControlService and SiteCommandService to this same gate.
|
||||
var interceptor = new ControlPlaneAuthInterceptor(
|
||||
Options.Create(new CommunicationOptions { GrpcPsk = "k" }),
|
||||
NullLogger<ControlPlaneAuthInterceptor>.Instance,
|
||||
new[] { "/scadabridge.sitecommand.v1.SiteCommandService/" });
|
||||
|
||||
var gated = CreateContext("/scadabridge.sitecommand.v1.SiteCommandService/ExecuteQuery", null);
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, gated));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
|
||||
// ...and the default set is no longer implied once an explicit set is supplied.
|
||||
var notGated = CreateContext(SubscribeMethod, authorizationHeader: null);
|
||||
Assert.Equal("ok", await Invoke(interceptor, notGated));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DefaultGatedPrefixes_MatchTheRealSiteStreamServicePath()
|
||||
{
|
||||
// A typo here disables the whole gate silently: every call would simply pass through.
|
||||
// Pin it against a path taken from the generated service, not from the proto text.
|
||||
var method = SiteStreamService.Descriptor.FullName;
|
||||
Assert.Contains(
|
||||
ControlPlaneAuthInterceptor.DefaultGatedPrefixes,
|
||||
p => p == $"/{method}/");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Central's per-site gRPC preshared key resolution (ClusterClient→gRPC migration, T0.3).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The property that matters most here is the negative one: a site whose key cannot be found
|
||||
/// must produce a throw, never an unauthenticated channel. Everything else — the two sources,
|
||||
/// the caching, the invalidation — exists to make that behaviour usable in practice.
|
||||
/// </remarks>
|
||||
public class SitePskProviderTests
|
||||
{
|
||||
private static SitePskProvider Create(
|
||||
ISecretResolver resolver, CommunicationOptions? options = null)
|
||||
=> new(
|
||||
resolver,
|
||||
new StaticOptionsMonitor(options ?? new CommunicationOptions()),
|
||||
NullLogger<SitePskProvider>.Instance);
|
||||
|
||||
private sealed class StaticOptionsMonitor(CommunicationOptions value)
|
||||
: IOptionsMonitor<CommunicationOptions>
|
||||
{
|
||||
public CommunicationOptions CurrentValue => value;
|
||||
public CommunicationOptions Get(string? name) => value;
|
||||
public IDisposable? OnChange(Action<CommunicationOptions, string?> listener) => null;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ResolvesFromTheSecretStore_UnderTheSiteQualifiedName()
|
||||
{
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-a"), Arg.Any<CancellationToken>())
|
||||
.Returns("key-for-a");
|
||||
|
||||
var psk = await Create(resolver).GetAsync("site-a", CancellationToken.None);
|
||||
|
||||
Assert.Equal("key-for-a", psk);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConfiguredMapWins_SoAHostWithNoMasterKeyCanStillDial()
|
||||
{
|
||||
// The development rig runs with no secrets master key at all — every credential
|
||||
// arrives as an environment override. Without this source, the rig could not use the
|
||||
// gated control plane and the fail-closed design would be untestable there.
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns("from-the-store");
|
||||
|
||||
var options = new CommunicationOptions();
|
||||
options.SitePsks["site-a"] = "from-config";
|
||||
|
||||
var psk = await Create(resolver, options).GetAsync("site-a", CancellationToken.None);
|
||||
|
||||
Assert.Equal("from-config", psk);
|
||||
await resolver.DidNotReceive().GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StoreIsStillConsulted_ForASiteMissingFromTheMap()
|
||||
{
|
||||
// Sites are added at runtime from the Central UI, so the map can never be complete.
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-b"), Arg.Any<CancellationToken>())
|
||||
.Returns("key-for-b");
|
||||
|
||||
var options = new CommunicationOptions();
|
||||
options.SitePsks["site-a"] = "from-config";
|
||||
|
||||
var psk = await Create(resolver, options).GetAsync("site-b", CancellationToken.None);
|
||||
|
||||
Assert.Equal("key-for-b", psk);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MissingSecret_Throws_AndNeverYieldsAnEmptyKey()
|
||||
{
|
||||
// Fail-closed. The alternative — returning "" — would build a channel that presents
|
||||
// "Bearer " and gets PermissionDenied anyway, but with a far less diagnosable error.
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns((string?)null);
|
||||
|
||||
var ex = await Assert.ThrowsAsync<InvalidOperationException>(
|
||||
async () => await Create(resolver).GetAsync("site-a", CancellationToken.None));
|
||||
|
||||
// The message must name both sources — either one fixes it.
|
||||
Assert.Contains("SB-GRPC-PSK-site-a", ex.Message);
|
||||
Assert.Contains("SitePsks", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EmptySecret_IsTreatedAsMissing()
|
||||
{
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>()).Returns("");
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(
|
||||
async () => await Create(resolver).GetAsync("site-a", CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AFaultingStore_SurfacesAsKeyNotConfigured_NotAsACryptoError()
|
||||
{
|
||||
// A host with no master key throws from the resolver. The operator's problem is the
|
||||
// same either way — "this site has no key" — so the diagnosis must say that.
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns<string?>(_ => throw new InvalidOperationException("no master key configured"));
|
||||
|
||||
var ex = await Assert.ThrowsAsync<InvalidOperationException>(
|
||||
async () => await Create(resolver).GetAsync("site-a", CancellationToken.None));
|
||||
|
||||
Assert.Contains("SB-GRPC-PSK-site-a", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SuccessfulResolvesAreCached_SoEveryCallDoesNotHitTheStore()
|
||||
{
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>()).Returns("k");
|
||||
var provider = Create(resolver);
|
||||
|
||||
await provider.GetAsync("site-a", CancellationToken.None);
|
||||
await provider.GetAsync("site-a", CancellationToken.None);
|
||||
await provider.GetAsync("site-a", CancellationToken.None);
|
||||
|
||||
await resolver.Received(1).GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FailuresAreNotCached_SoASeededKeyIsPickedUpWithoutARestart()
|
||||
{
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns((string?)null, "seeded-later");
|
||||
var provider = Create(resolver);
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(
|
||||
async () => await provider.GetAsync("site-a", CancellationToken.None));
|
||||
|
||||
Assert.Equal("seeded-later", await provider.GetAsync("site-a", CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Invalidate_DropsTheCachedKey_SoARotatedKeyIsRead()
|
||||
{
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns("old", "rotated");
|
||||
var provider = Create(resolver);
|
||||
|
||||
Assert.Equal("old", await provider.GetAsync("site-a", CancellationToken.None));
|
||||
provider.Invalidate("site-a");
|
||||
|
||||
Assert.Equal("rotated", await provider.GetAsync("site-a", CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task KeysAreScopedPerSite_SoOneSiteNeverPresentsAnothersKey()
|
||||
{
|
||||
// The whole reason the design rejected a single fleet-wide key: blast radius.
|
||||
var resolver = Substitute.For<ISecretResolver>();
|
||||
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-a"), Arg.Any<CancellationToken>())
|
||||
.Returns("key-a");
|
||||
resolver.GetAsync(new SecretName("SB-GRPC-PSK-site-b"), Arg.Any<CancellationToken>())
|
||||
.Returns("key-b");
|
||||
var provider = Create(resolver);
|
||||
|
||||
Assert.Equal("key-a", await provider.GetAsync("site-a", CancellationToken.None));
|
||||
Assert.Equal("key-b", await provider.GetAsync("site-b", CancellationToken.None));
|
||||
}
|
||||
}
|
||||
@@ -38,6 +38,9 @@ public class StartupValidatorTests
|
||||
["ScadaBridge:Database:SiteDbPath"] = "./data/scadabridge.db",
|
||||
["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@site-a-node1:8082",
|
||||
["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@site-a-node2:8082",
|
||||
// T0.3: the gRPC control plane is fail-closed, so a Site node without a preshared
|
||||
// key serves nothing while still looking healthy. Required at boot for that reason.
|
||||
["ScadaBridge:Communication:GrpcPsk"] = "test-site-control-plane-key",
|
||||
};
|
||||
|
||||
[Fact]
|
||||
@@ -56,6 +59,44 @@ public class StartupValidatorTests
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SiteWithoutGrpcPsk_FailsValidation()
|
||||
{
|
||||
// The failure this prevents is silent: ControlPlaneAuthInterceptor refuses every
|
||||
// SiteStream call without a key, so the node boots, joins its pair, reports healthy —
|
||||
// and serves no live streams, no audit pulls and no cached-telemetry ingest. Central
|
||||
// sees a site that is up and answering heartbeats but never sends anything.
|
||||
var values = ValidSiteConfig();
|
||||
values.Remove("ScadaBridge:Communication:GrpcPsk");
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
Assert.Contains("GrpcPsk", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SiteWithBlankGrpcPsk_FailsValidation()
|
||||
{
|
||||
// Whitespace is not a key. An empty-string value would otherwise satisfy a
|
||||
// key-present check while leaving the interceptor in its fail-closed state.
|
||||
var values = ValidSiteConfig();
|
||||
values["ScadaBridge:Communication:GrpcPsk"] = " ";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
Assert.Contains("GrpcPsk", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CentralWithoutGrpcPsk_PassesValidation()
|
||||
{
|
||||
// Central holds one key PER SITE (SitePsks / the secret store), not a single key of
|
||||
// its own, so this setting is meaningless there and must not be required.
|
||||
var config = BuildConfig(ValidCentralConfig());
|
||||
|
||||
Assert.Null(Record.Exception(() => StartupValidator.Validate(config)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MissingRole_FailsValidation()
|
||||
{
|
||||
|
||||
@@ -3643,7 +3643,8 @@ public class ManagementActorTests : TestKit, IDisposable
|
||||
private sealed class TrackingGrpcFactory : SiteStreamGrpcClientFactory
|
||||
{
|
||||
public TrackingGrpcFactory() : base(NullLoggerFactory.Instance) { }
|
||||
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint) => new TrackingClient(grpcEndpoint);
|
||||
protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
|
||||
=> new TrackingClient(grpcEndpoint);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
Reference in New Issue
Block a user