feat(mesh): known-cluster gate + Matches on the PSK provider

Phase 1A security register items 1 and 2, landed ahead of the command
fan-out because they are independently testable and the fan-out is the
first caller of TryGetPsk.

Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an
atomic whole-set swap consulted by TryGetPsk BEFORE the config map and
before the secret store. A cluster id outside the set is refused in O(1)
with no store read and no audit write, which closes two things at once:
the pre-auth store/audit flood an unauthenticated caller could drive with
a forged x-otopcua-cluster header, and the timing oracle that defeated
CentralCommandAuthInterceptor's uniform denial text (unknown clusters are
never cached, so they paid a store read on every probe while known ones
answered from memory). Empty or never-published leaves the gate inactive
— central publishes the set from its DB contact refresh, and 'the DB told
me nothing' must not become a fleet-wide authentication outage during
admin boot.

Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with
FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now
calls it and never holds key material; its two provider-side denial logs
collapse into one (the provider answers both as false) and the operator
hint is restated there.

No production caller publishes a cluster set yet, so behaviour is
unchanged until the fan-out commit wires the contact refresh.

ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
This commit is contained in:
Joseph Doherty
2026-08-06 05:31:08 -04:00
parent 2edef72f6f
commit 14d21496fe
4 changed files with 392 additions and 36 deletions
@@ -1,5 +1,7 @@
using System.Collections.Concurrent;
using System.Diagnostics.CodeAnalysis;
using System.Security.Cryptography;
using System.Text;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.OtOpcUa.Cluster;
@@ -21,10 +23,21 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// proceed unauthenticated; a missing key must cost delivery, not authentication.
/// </para>
/// <para>
/// The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. They are
/// deliberately unconsumed at Phase 0 — Phase 1A wires them to the contact refresh, which is
/// the point at which central learns a cluster's membership (and so possibly its key) has
/// changed.
/// The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. Phase 1A
/// wires them to <c>CentralCommunicationActor</c>'s contact refresh, which is the point at
/// which central learns a cluster's membership (and so possibly its key) has changed — so a
/// rotated key is picked up within one <c>ContactRefreshSeconds</c> without a restart.
/// </para>
/// <para>
/// <b>The known-cluster gate is a real security control, not an optimization</b> (Phase 1A
/// security register item 1). <see cref="SetKnownClusters"/> publishes the cluster ids
/// central just read from <c>ClusterNode</c>; <see cref="TryGetPsk"/> consults that set
/// first and refuses anything outside it in O(1), touching neither the config map nor the
/// secret store. Without it, an unauthenticated caller can pick an arbitrary
/// <c>x-otopcua-cluster</c> value and make central perform a secret-store read plus an audit
/// write per probe — and, because negatives are deliberately never cached, the cost
/// difference between a known and an unknown cluster is measurable from the wire, which
/// defeats the uniform denial text on the central interceptor.
/// </para>
/// <para>
/// <b><c>Invalidate</c> is not a revocation barrier.</b> It drops the cached entry; it does
@@ -43,6 +56,50 @@ public interface IMeshPskProvider
/// <returns><c>true</c> when a usable key was found; otherwise <c>false</c>.</returns>
bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk);
/// <summary>
/// Publishes the set of application <c>Cluster</c> ids central currently knows about.
/// <see cref="TryGetPsk"/> refuses anything outside it before consulting any source.
/// </summary>
/// <param name="clusterIds">
/// The known cluster ids, matched case-insensitively. <b>An empty collection deactivates the
/// gate</b> rather than refusing everything: central publishes this from its DB contact
/// refresh, and "the DB told me nothing" must not become a fleet-wide authentication outage.
/// Never having called this leaves the gate inactive for the same reason — the central
/// interceptor is live from the first request, before the first refresh has run.
/// </param>
/// <remarks>
/// Implementations must take a copy: the caller owns its collection and may reuse it. The
/// swap must also be atomic with respect to concurrent <see cref="TryGetPsk"/> callers,
/// which run on arbitrary gRPC threads.
/// </remarks>
void SetKnownClusters(IReadOnlyCollection<string> clusterIds);
/// <summary>
/// Verifies a presented bearer token against one cluster's key, in constant time with
/// respect to the key's content.
/// </summary>
/// <param name="clusterId">The application <c>Cluster</c> the caller claims to be.</param>
/// <param name="presented">The bearer token the caller supplied.</param>
/// <returns><c>true</c> only when a key exists for that cluster and the token equals it.</returns>
/// <remarks>
/// <para>
/// <b>This is the only comparison on the verify path</b> (Phase 1A security register
/// item 2). Handing the raw key back so a caller can compare it is how <c>==</c>
/// eventually appears in an interceptor and compiles; keeping the comparison behind this
/// member means the verify side never holds key material at all. <see cref="TryGetPsk"/>
/// survives for the <i>dial</i> side, which has to send the key rather than compare it.
/// </para>
/// <para>
/// Declared without a default implementation on purpose: a security member a new
/// implementer can inherit silently is a member a new implementer never considers.
/// </para>
/// <para>
/// Length differences remain observable through timing; that is inherent to comparing
/// byte arrays of different lengths and is not sensitive. Content is not.
/// </para>
/// </remarks>
bool Matches(string clusterId, string presented);
/// <summary>Drops any cached key for one cluster, so the next lookup re-reads the store.</summary>
/// <param name="clusterId">The application <c>Cluster</c> id, matched case-insensitively.</param>
void Invalidate(string clusterId);
@@ -87,6 +144,18 @@ public sealed class MeshPskProvider : IMeshPskProvider
private readonly ConcurrentDictionary<string, Lazy<string?>> _cache =
new(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// The known-cluster gate's membership set, or <see langword="null"/> when the gate is
/// inactive (never published, or published empty — see <see cref="SetKnownClusters"/>).
/// </summary>
/// <remarks>
/// Swapped wholesale, never mutated in place, so a reader either sees the previous set or the
/// next one and never a half-built one. <c>volatile</c> is what makes the publish visible to
/// the gRPC threads doing the reading; no lock is needed because there is exactly one writer
/// (the contact-refresh handler, on the actor's own thread).
/// </remarks>
private volatile IReadOnlySet<string>? _knownClusters;
/// <summary>Creates the provider.</summary>
/// <param name="options">Mesh transport options; <c>ClusterPsks</c> is the override map.</param>
/// <param name="secretResolver">The shared secret store read seam.</param>
@@ -118,6 +187,19 @@ public sealed class MeshPskProvider : IMeshPskProvider
return false;
}
// The known-cluster gate, ahead of BOTH sources. Deliberately silent: the caller that
// rejected the call already logs the denial, and a second line here would be a log flood
// driven by exactly the unauthenticated traffic this gate exists to make cheap.
//
// Ahead of the config map too, not just the store. That has a visible consequence — a
// MeshTransport:ClusterPsks entry for a cluster with no enabled, non-maintenance ClusterNode
// row stops resolving once central publishes its set — and it is the right one: central
// would neither dial such a cluster nor accept an ack from it, so serving a key for it only
// widens what an unauthenticated caller can probe.
var known = _knownClusters;
if (known is not null && !known.Contains(clusterId))
return false;
// Config map first, on every call. Blank counts as absent: an empty
// MeshTransport__ClusterPsks__<id> is an unset environment variable, not an operator
// deciding this cluster has an empty key.
@@ -168,6 +250,34 @@ public sealed class MeshPskProvider : IMeshPskProvider
return true;
}
/// <inheritdoc />
public bool Matches(string clusterId, string presented)
{
// Blank is refused on its own terms rather than by relying on the fact that a blank key
// never survives resolution: this is an authentication gate, and "both sides happened to be
// empty" must never be a pass.
if (string.IsNullOrWhiteSpace(presented))
return false;
if (!TryGetPsk(clusterId, out var expected))
return false;
return CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
/// <inheritdoc />
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds)
{
ArgumentNullException.ThrowIfNull(clusterIds);
// Copied, so a caller reusing its list cannot widen the gate after the fact; empty collapses
// to null so the gate has exactly one "inactive" representation to test for.
_knownClusters = clusterIds.Count == 0
? null
: clusterIds.ToHashSet(StringComparer.OrdinalIgnoreCase);
}
/// <inheritdoc />
public void Invalidate(string clusterId)
{
@@ -1,5 +1,3 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
@@ -38,25 +36,31 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// key this design exists to avoid.
/// </para>
/// <para>
/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so a
/// wrong key cannot be recovered byte-by-byte from response timing. Length differences are
/// unavoidably observable and are not sensitive.
/// <b>This interceptor never handles a key.</b> Verification is
/// <see cref="IMeshPskProvider.Matches"/>, which owns the constant-time comparison, so a
/// wrong key cannot be recovered byte-by-byte from response timing and no future edit here
/// can replace that comparison with <c>==</c> and still compile (Phase 1A security register
/// item 2). Length differences are unavoidably observable and are not sensitive.
/// </para>
/// <para>
/// <b>All three denials return the same status text</b>, unlike the sibling interceptors,
/// which distinguish "no key configured" from "authentication failed". Here that distinction
/// would be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said
/// "no key for that cluster" would let an unauthenticated caller read central's cluster set
/// straight off the wire. The operator-facing detail goes to the log instead.
/// <b>All denials return the same status text</b>, unlike the sibling interceptors, which
/// distinguish "no key configured" from "authentication failed". Here that distinction would
/// be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said "no
/// key for that cluster" would let an unauthenticated caller read central's cluster set
/// straight off the wire. The operator-facing detail goes to the log instead — and for the
/// same reason the log no longer separates "no key for that cluster" from "wrong key": the
/// provider answers both as one <c>false</c>, and collapsing them costs an operator a hint
/// (restated in the one denial message) rather than costing a diagnosis.
/// </para>
/// <para>
/// <b>That is not yet full enumeration resistance, and should not be read as such.</b> A
/// timing oracle survives it: an unknown cluster misses the provider's cache and pays a store
/// read on every probe, while a known one is answered from memory after the first. Closing
/// that needs the lookup to be refused before it reaches the store for any cluster central
/// does not know — which is the Phase 1A known-cluster-set gate, where central learns its
/// cluster set from the DB. Until then, uniform text buys content-level indistinguishability
/// only.
/// <b>The timing oracle that survived Phase 0 is closed by the known-cluster gate.</b> An
/// unknown cluster used to miss the provider's cache and pay a secret-store read on every
/// probe, while a known one was answered from memory after the first — a stable, remotely
/// measurable distinguisher, and a pre-auth store/audit flood besides.
/// <c>CentralCommunicationActor</c> now publishes central's cluster set to the provider on
/// every contact refresh, and a cluster outside it is refused in O(1) with no store touch,
/// so the two cases cost the same. What remains is a known cluster's first-ever lookup,
/// which is one read per cluster per process and not attacker-triggerable.
/// </para>
/// <para>
/// <b>Exactly one public constructor.</b> <c>Grpc.AspNetCore</c> silently stops invoking an
@@ -151,13 +155,15 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
"Central command authentication failed."));
}
if (!_pskProvider.TryGetPsk(clusterId, out var expected))
// Checked here rather than deferred to Matches so the log can still separate "the caller
// presented nothing" — an operator misconfiguration, and the one case the provider cannot
// see — from "the caller presented something that did not verify".
var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
if (presented is null)
{
_logger.LogWarning(
"Rejected a central command call to {Method} claiming cluster {ClusterId}: no "
+ "pre-shared key could be established for it. Seed the secret "
+ "OT-MESH-PSK-<clusterId> on this node, or set "
+ "MeshTransport:ClusterPsks:<clusterId> on a rig.",
"Rejected a central command call to {Method} claiming cluster {ClusterId}: no bearer "
+ "token presented.",
context.Method,
clusterId);
throw new RpcException(new Status(
@@ -165,21 +171,20 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
"Central command authentication failed."));
}
var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
if (!_pskProvider.Matches(clusterId, presented))
{
_logger.LogWarning(
"Rejected a central command call to {Method} claiming cluster {ClusterId}: {Reason}.",
"Rejected a central command call to {Method} claiming cluster {ClusterId}: the "
+ "presented bearer token did not match that cluster's pre-shared key, no key could "
+ "be established for it, or it is not a cluster this node knows. If the cluster is "
+ "legitimate, seed the secret OT-MESH-PSK-<clusterId> on this node, or set "
+ "MeshTransport:ClusterPsks:<clusterId> on a rig, and check it has an enabled, "
+ "non-maintenance ClusterNode row.",
context.Method,
clusterId,
presented is null ? "no bearer token presented" : "bearer token did not match");
clusterId);
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"Central command authentication failed."));
}
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
@@ -367,6 +367,205 @@ public sealed class MeshPskProviderTests
resolver.CallCount.ShouldBe(0);
}
// ------------------------------------------------------------------ known-cluster gate
//
// Phase 1A security register item 1. Central rebuilds its cluster set from ClusterNode rows on
// every contact refresh; a cluster id outside that set must be refused in O(1) BEFORE anything
// touches the secret store. Two things ride on that: an unauthenticated caller can no longer
// make central perform a store read (and an audit write) per forged x-otopcua-cluster header,
// and the timing difference between a known cluster (cached) and an unknown one (never cached,
// because negatives are deliberately not retained) stops being remotely measurable.
[Fact]
public void UnknownCluster_WithTheGatePopulated_IsRefused_WithoutTouchingTheStore()
{
// THE assertion of the register item. CallCount == 0 is the whole point: the refusal must
// cost nothing, or the gate is a comment rather than a control.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A", "SITE-B"]);
provider.TryGetPsk("SITE-Z", out var psk).ShouldBeFalse();
psk.ShouldBeNull();
resolver.CallCount.ShouldBe(0);
}
[Fact]
public void KnownCluster_WithTheGatePopulated_StillResolves()
{
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("SITE-A", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
resolver.CallCount.ShouldBe(1);
}
[Fact]
public void TheKnownClusterSet_IsCaseInsensitive()
{
// ClusterId case is whatever the operator typed into the DB row; the header is whatever the
// node was configured with. A case-sensitive gate would refuse a legitimate cluster.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("site-a", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
}
[Fact]
public void SwappingTheKnownClusterSet_AddsAndRemovesClusters()
{
// The set is rebuilt from the DB every refresh, so both directions happen in production: a
// new site appears, a retired one stops being accepted.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("SITE-A", out _).ShouldBeTrue();
provider.TryGetPsk("SITE-B", out _).ShouldBeFalse();
provider.SetKnownClusters(["SITE-B"]);
provider.TryGetPsk("SITE-B", out _).ShouldBeTrue();
provider.TryGetPsk("SITE-A", out _).ShouldBeFalse();
}
[Fact]
public void AnEmptyKnownClusterSet_LeavesTheGateInactive()
{
// Fail-OPEN, deliberately and narrowly: central publishes the set from its contact refresh,
// and an empty publish means "the DB told me nothing", not "no cluster is legitimate".
// Refusing everything there would make a fleet-wide ack outage the failure mode of an empty
// or unreadable ClusterNode table.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.SetKnownClusters([]);
provider.TryGetPsk("SITE-Z", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
}
[Fact]
public void AnUnsetKnownClusterSet_LeavesTheGateInactive()
{
// Admin boot: the interceptor is live from the first request, but the first contact refresh
// has not run yet. Gating on a set nobody has published would deny every ack in that window.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.TryGetPsk("SITE-Z", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
}
[Fact]
public void TheGate_IsConsultedBeforeTheConfigMap()
{
// Consequence worth pinning rather than discovering: a ClusterPsks entry for a cluster with
// no enabled ClusterNode row is refused once the gate is populated. That is correct — central
// would neither dial such a cluster nor accept an ack from it — but it is surprising enough
// on a rig to deserve a test that names it.
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-X"] = "rig-psk-site-x" });
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("SITE-X", out var psk).ShouldBeFalse();
psk.ShouldBeNull();
}
[Fact]
public void SetKnownClusters_TakesACopy_SoALaterMutationOfTheCallersList_CannotWidenTheGate()
{
var clusters = new List<string> { "SITE-A" };
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(clusters);
clusters.Add("SITE-Z");
provider.TryGetPsk("SITE-Z", out _).ShouldBeFalse();
}
// ------------------------------------------------------------------ Matches
[Fact]
public void Matches_AcceptsTheClustersOwnKey()
{
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
provider.Matches("SITE-A", "psk-site-a").ShouldBeTrue();
}
[Fact]
public void Matches_RejectsAnotherClustersKey()
{
// The isolation property, asserted at the provider rather than only at the interceptor:
// choosing a cluster only changes which key you are graded against.
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver,
new Dictionary<string, string> { ["SITE-A"] = "psk-site-a", ["SITE-B"] = "psk-site-b" });
provider.Matches("SITE-A", "psk-site-b").ShouldBeFalse();
}
[Fact]
public void Matches_IsNotAPrefixMatch()
{
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
provider.Matches("SITE-A", "psk-site-").ShouldBeFalse();
provider.Matches("SITE-A", "psk-site-a-and-more").ShouldBeFalse();
}
[Fact]
public void Matches_FailsClosed_WhenTheClusterHasNoKey()
{
var resolver = new FakeSecretResolver { Value = null };
var provider = CreateProvider(resolver);
provider.Matches("SITE-Z", "anything-at-all").ShouldBeFalse();
}
[Theory]
[InlineData("")]
[InlineData(" ")]
public void Matches_RejectsABlankPresentedToken(string presented)
{
// A blank key never reaches the cache (blank resolves fail closed), but a blank PRESENTED
// token must be refused on its own terms rather than relying on that.
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
provider.Matches("SITE-A", presented).ShouldBeFalse();
}
[Fact]
public void Matches_IsRefusedByTheKnownClusterGate_WithoutTouchingTheStore()
{
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.Matches("SITE-Z", "stored-psk").ShouldBeFalse();
resolver.CallCount.ShouldBe(0);
}
/// <summary>
/// An <see cref="ISecretResolver"/> whose answer the test controls between calls, so cache
/// staleness and invalidation are observable without a real store.
@@ -242,14 +242,56 @@ public sealed class CentralCommandAuthInterceptorTests
publicCtors.Length.ShouldBe(1);
}
[Fact]
public async Task Verification_GoesThroughMatches_AndTheInterceptorNeverHandlesTheKeyItself()
{
// Phase 1A security register item 2. The interceptor must not fetch the raw key and compare
// it — that is how a constant-time comparison eventually becomes '=='. Matches owns the
// comparison; TryGetPsk survives only for the DIAL side, which has to send the key.
// (The provider's own default Matches calls TryGetPsk internally; this fake overrides it, so
// a zero TryGetPsk count here means the INTERCEPTOR never asked for the key.)
var provider = new FakeMeshPskProvider(("SITE-A", "psk-site-a"));
var interceptor = new CentralCommandAuthInterceptor(
provider, NullLogger<CentralCommandAuthInterceptor>.Instance);
await Invoke(interceptor, CreateContext(ReportApplyMethod, "Bearer psk-site-a", "SITE-A"));
provider.MatchesCalls.ShouldBe(1);
provider.TryGetPskCalls.ShouldBe(0);
}
/// <summary>An <see cref="IMeshPskProvider"/> over a fixed key set; unknown clusters fail closed.</summary>
private sealed class FakeMeshPskProvider(params (string Cluster, string Psk)[] keys) : IMeshPskProvider
{
private readonly Dictionary<string, string> _keys =
keys.ToDictionary(k => k.Cluster, k => k.Psk, StringComparer.OrdinalIgnoreCase);
/// <summary>How many times the raw key was asked for — must stay 0 on the verify path.</summary>
public int TryGetPskCalls { get; private set; }
/// <summary>How many times verification went through the provider's comparison.</summary>
public int MatchesCalls { get; private set; }
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
=> _keys.TryGetValue(clusterId, out psk);
{
TryGetPskCalls++;
return _keys.TryGetValue(clusterId, out psk);
}
/// <summary>
/// Overrides the interface's default so the two calls are separately countable. The
/// semantics are the default's: no key, or a key that is not exactly the presented token.
/// </summary>
public bool Matches(string clusterId, string presented)
{
MatchesCalls++;
return !string.IsNullOrWhiteSpace(presented)
&& _keys.TryGetValue(clusterId, out var expected)
&& string.Equals(expected, presented, StringComparison.Ordinal);
}
/// <summary>No-op: the gate is the production provider's behaviour, pinned in its own tests.</summary>
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) { }
public void Invalidate(string clusterId) { }