feat(mesh): known-cluster gate + Matches on the PSK provider
Phase 1A security register items 1 and 2, landed ahead of the command fan-out because they are independently testable and the fan-out is the first caller of TryGetPsk. Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an atomic whole-set swap consulted by TryGetPsk BEFORE the config map and before the secret store. A cluster id outside the set is refused in O(1) with no store read and no audit write, which closes two things at once: the pre-auth store/audit flood an unauthenticated caller could drive with a forged x-otopcua-cluster header, and the timing oracle that defeated CentralCommandAuthInterceptor's uniform denial text (unknown clusters are never cached, so they paid a store read on every probe while known ones answered from memory). Empty or never-published leaves the gate inactive — central publishes the set from its DB contact refresh, and 'the DB told me nothing' must not become a fleet-wide authentication outage during admin boot. Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now calls it and never holds key material; its two provider-side denial logs collapse into one (the provider answers both as false) and the operator hint is restated there. No production caller publishes a cluster set yet, so behaviour is unchanged until the fan-out commit wires the contact refresh. ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
This commit is contained in:
Joseph Doherty
@@ -1,5 +1,7 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Diagnostics.CodeAnalysis;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
@@ -21,10 +23,21 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
/// proceed unauthenticated; a missing key must cost delivery, not authentication.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. They are
|
||||
/// deliberately unconsumed at Phase 0 — Phase 1A wires them to the contact refresh, which is
|
||||
/// the point at which central learns a cluster's membership (and so possibly its key) has
|
||||
/// changed.
|
||||
/// The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. Phase 1A
|
||||
/// wires them to <c>CentralCommunicationActor</c>'s contact refresh, which is the point at
|
||||
/// which central learns a cluster's membership (and so possibly its key) has changed — so a
|
||||
/// rotated key is picked up within one <c>ContactRefreshSeconds</c> without a restart.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The known-cluster gate is a real security control, not an optimization</b> (Phase 1A
|
||||
/// security register item 1). <see cref="SetKnownClusters"/> publishes the cluster ids
|
||||
/// central just read from <c>ClusterNode</c>; <see cref="TryGetPsk"/> consults that set
|
||||
/// first and refuses anything outside it in O(1), touching neither the config map nor the
|
||||
/// secret store. Without it, an unauthenticated caller can pick an arbitrary
|
||||
/// <c>x-otopcua-cluster</c> value and make central perform a secret-store read plus an audit
|
||||
/// write per probe — and, because negatives are deliberately never cached, the cost
|
||||
/// difference between a known and an unknown cluster is measurable from the wire, which
|
||||
/// defeats the uniform denial text on the central interceptor.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b><c>Invalidate</c> is not a revocation barrier.</b> It drops the cached entry; it does
|
||||
@@ -43,6 +56,50 @@ public interface IMeshPskProvider
|
||||
/// <returns><c>true</c> when a usable key was found; otherwise <c>false</c>.</returns>
|
||||
bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk);
|
||||
|
||||
/// <summary>
|
||||
/// Publishes the set of application <c>Cluster</c> ids central currently knows about.
|
||||
/// <see cref="TryGetPsk"/> refuses anything outside it before consulting any source.
|
||||
/// </summary>
|
||||
/// <param name="clusterIds">
|
||||
/// The known cluster ids, matched case-insensitively. <b>An empty collection deactivates the
|
||||
/// gate</b> rather than refusing everything: central publishes this from its DB contact
|
||||
/// refresh, and "the DB told me nothing" must not become a fleet-wide authentication outage.
|
||||
/// Never having called this leaves the gate inactive for the same reason — the central
|
||||
/// interceptor is live from the first request, before the first refresh has run.
|
||||
/// </param>
|
||||
/// <remarks>
|
||||
/// Implementations must take a copy: the caller owns its collection and may reuse it. The
|
||||
/// swap must also be atomic with respect to concurrent <see cref="TryGetPsk"/> callers,
|
||||
/// which run on arbitrary gRPC threads.
|
||||
/// </remarks>
|
||||
void SetKnownClusters(IReadOnlyCollection<string> clusterIds);
|
||||
|
||||
/// <summary>
|
||||
/// Verifies a presented bearer token against one cluster's key, in constant time with
|
||||
/// respect to the key's content.
|
||||
/// </summary>
|
||||
/// <param name="clusterId">The application <c>Cluster</c> the caller claims to be.</param>
|
||||
/// <param name="presented">The bearer token the caller supplied.</param>
|
||||
/// <returns><c>true</c> only when a key exists for that cluster and the token equals it.</returns>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>This is the only comparison on the verify path</b> (Phase 1A security register
|
||||
/// item 2). Handing the raw key back so a caller can compare it is how <c>==</c>
|
||||
/// eventually appears in an interceptor and compiles; keeping the comparison behind this
|
||||
/// member means the verify side never holds key material at all. <see cref="TryGetPsk"/>
|
||||
/// survives for the <i>dial</i> side, which has to send the key rather than compare it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Declared without a default implementation on purpose: a security member a new
|
||||
/// implementer can inherit silently is a member a new implementer never considers.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Length differences remain observable through timing; that is inherent to comparing
|
||||
/// byte arrays of different lengths and is not sensitive. Content is not.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
bool Matches(string clusterId, string presented);
|
||||
|
||||
/// <summary>Drops any cached key for one cluster, so the next lookup re-reads the store.</summary>
|
||||
/// <param name="clusterId">The application <c>Cluster</c> id, matched case-insensitively.</param>
|
||||
void Invalidate(string clusterId);
|
||||
@@ -87,6 +144,18 @@ public sealed class MeshPskProvider : IMeshPskProvider
|
||||
private readonly ConcurrentDictionary<string, Lazy<string?>> _cache =
|
||||
new(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>
|
||||
/// The known-cluster gate's membership set, or <see langword="null"/> when the gate is
|
||||
/// inactive (never published, or published empty — see <see cref="SetKnownClusters"/>).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Swapped wholesale, never mutated in place, so a reader either sees the previous set or the
|
||||
/// next one and never a half-built one. <c>volatile</c> is what makes the publish visible to
|
||||
/// the gRPC threads doing the reading; no lock is needed because there is exactly one writer
|
||||
/// (the contact-refresh handler, on the actor's own thread).
|
||||
/// </remarks>
|
||||
private volatile IReadOnlySet<string>? _knownClusters;
|
||||
|
||||
/// <summary>Creates the provider.</summary>
|
||||
/// <param name="options">Mesh transport options; <c>ClusterPsks</c> is the override map.</param>
|
||||
/// <param name="secretResolver">The shared secret store read seam.</param>
|
||||
@@ -118,6 +187,19 @@ public sealed class MeshPskProvider : IMeshPskProvider
|
||||
return false;
|
||||
}
|
||||
|
||||
// The known-cluster gate, ahead of BOTH sources. Deliberately silent: the caller that
|
||||
// rejected the call already logs the denial, and a second line here would be a log flood
|
||||
// driven by exactly the unauthenticated traffic this gate exists to make cheap.
|
||||
//
|
||||
// Ahead of the config map too, not just the store. That has a visible consequence — a
|
||||
// MeshTransport:ClusterPsks entry for a cluster with no enabled, non-maintenance ClusterNode
|
||||
// row stops resolving once central publishes its set — and it is the right one: central
|
||||
// would neither dial such a cluster nor accept an ack from it, so serving a key for it only
|
||||
// widens what an unauthenticated caller can probe.
|
||||
var known = _knownClusters;
|
||||
if (known is not null && !known.Contains(clusterId))
|
||||
return false;
|
||||
|
||||
// Config map first, on every call. Blank counts as absent: an empty
|
||||
// MeshTransport__ClusterPsks__<id> is an unset environment variable, not an operator
|
||||
// deciding this cluster has an empty key.
|
||||
@@ -168,6 +250,34 @@ public sealed class MeshPskProvider : IMeshPskProvider
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool Matches(string clusterId, string presented)
|
||||
{
|
||||
// Blank is refused on its own terms rather than by relying on the fact that a blank key
|
||||
// never survives resolution: this is an authentication gate, and "both sides happened to be
|
||||
// empty" must never be a pass.
|
||||
if (string.IsNullOrWhiteSpace(presented))
|
||||
return false;
|
||||
|
||||
if (!TryGetPsk(clusterId, out var expected))
|
||||
return false;
|
||||
|
||||
return CryptographicOperations.FixedTimeEquals(
|
||||
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(clusterIds);
|
||||
|
||||
// Copied, so a caller reusing its list cannot widen the gate after the fact; empty collapses
|
||||
// to null so the gate has exactly one "inactive" representation to test for.
|
||||
_knownClusters = clusterIds.Count == 0
|
||||
? null
|
||||
: clusterIds.ToHashSet(StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Invalidate(string clusterId)
|
||||
{
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using Grpc.Core;
|
||||
using Grpc.Core.Interceptors;
|
||||
using Microsoft.Extensions.Logging;
|
||||
@@ -38,25 +36,31 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
/// key this design exists to avoid.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so a
|
||||
/// wrong key cannot be recovered byte-by-byte from response timing. Length differences are
|
||||
/// unavoidably observable and are not sensitive.
|
||||
/// <b>This interceptor never handles a key.</b> Verification is
|
||||
/// <see cref="IMeshPskProvider.Matches"/>, which owns the constant-time comparison, so a
|
||||
/// wrong key cannot be recovered byte-by-byte from response timing and no future edit here
|
||||
/// can replace that comparison with <c>==</c> and still compile (Phase 1A security register
|
||||
/// item 2). Length differences are unavoidably observable and are not sensitive.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>All three denials return the same status text</b>, unlike the sibling interceptors,
|
||||
/// which distinguish "no key configured" from "authentication failed". Here that distinction
|
||||
/// would be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said
|
||||
/// "no key for that cluster" would let an unauthenticated caller read central's cluster set
|
||||
/// straight off the wire. The operator-facing detail goes to the log instead.
|
||||
/// <b>All denials return the same status text</b>, unlike the sibling interceptors, which
|
||||
/// distinguish "no key configured" from "authentication failed". Here that distinction would
|
||||
/// be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said "no
|
||||
/// key for that cluster" would let an unauthenticated caller read central's cluster set
|
||||
/// straight off the wire. The operator-facing detail goes to the log instead — and for the
|
||||
/// same reason the log no longer separates "no key for that cluster" from "wrong key": the
|
||||
/// provider answers both as one <c>false</c>, and collapsing them costs an operator a hint
|
||||
/// (restated in the one denial message) rather than costing a diagnosis.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>That is not yet full enumeration resistance, and should not be read as such.</b> A
|
||||
/// timing oracle survives it: an unknown cluster misses the provider's cache and pays a store
|
||||
/// read on every probe, while a known one is answered from memory after the first. Closing
|
||||
/// that needs the lookup to be refused before it reaches the store for any cluster central
|
||||
/// does not know — which is the Phase 1A known-cluster-set gate, where central learns its
|
||||
/// cluster set from the DB. Until then, uniform text buys content-level indistinguishability
|
||||
/// only.
|
||||
/// <b>The timing oracle that survived Phase 0 is closed by the known-cluster gate.</b> An
|
||||
/// unknown cluster used to miss the provider's cache and pay a secret-store read on every
|
||||
/// probe, while a known one was answered from memory after the first — a stable, remotely
|
||||
/// measurable distinguisher, and a pre-auth store/audit flood besides.
|
||||
/// <c>CentralCommunicationActor</c> now publishes central's cluster set to the provider on
|
||||
/// every contact refresh, and a cluster outside it is refused in O(1) with no store touch,
|
||||
/// so the two cases cost the same. What remains is a known cluster's first-ever lookup,
|
||||
/// which is one read per cluster per process and not attacker-triggerable.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Exactly one public constructor.</b> <c>Grpc.AspNetCore</c> silently stops invoking an
|
||||
@@ -151,13 +155,15 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
|
||||
"Central command authentication failed."));
|
||||
}
|
||||
|
||||
if (!_pskProvider.TryGetPsk(clusterId, out var expected))
|
||||
// Checked here rather than deferred to Matches so the log can still separate "the caller
|
||||
// presented nothing" — an operator misconfiguration, and the one case the provider cannot
|
||||
// see — from "the caller presented something that did not verify".
|
||||
var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
|
||||
if (presented is null)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a central command call to {Method} claiming cluster {ClusterId}: no "
|
||||
+ "pre-shared key could be established for it. Seed the secret "
|
||||
+ "OT-MESH-PSK-<clusterId> on this node, or set "
|
||||
+ "MeshTransport:ClusterPsks:<clusterId> on a rig.",
|
||||
"Rejected a central command call to {Method} claiming cluster {ClusterId}: no bearer "
|
||||
+ "token presented.",
|
||||
context.Method,
|
||||
clusterId);
|
||||
throw new RpcException(new Status(
|
||||
@@ -165,21 +171,20 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
|
||||
"Central command authentication failed."));
|
||||
}
|
||||
|
||||
var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
|
||||
if (presented is null || !FixedTimeEquals(presented, expected))
|
||||
if (!_pskProvider.Matches(clusterId, presented))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a central command call to {Method} claiming cluster {ClusterId}: {Reason}.",
|
||||
"Rejected a central command call to {Method} claiming cluster {ClusterId}: the "
|
||||
+ "presented bearer token did not match that cluster's pre-shared key, no key could "
|
||||
+ "be established for it, or it is not a cluster this node knows. If the cluster is "
|
||||
+ "legitimate, seed the secret OT-MESH-PSK-<clusterId> on this node, or set "
|
||||
+ "MeshTransport:ClusterPsks:<clusterId> on a rig, and check it has an enabled, "
|
||||
+ "non-maintenance ClusterNode row.",
|
||||
context.Method,
|
||||
clusterId,
|
||||
presented is null ? "no bearer token presented" : "bearer token did not match");
|
||||
clusterId);
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.PermissionDenied,
|
||||
"Central command authentication failed."));
|
||||
}
|
||||
}
|
||||
|
||||
private static bool FixedTimeEquals(string presented, string expected)
|
||||
=> CryptographicOperations.FixedTimeEquals(
|
||||
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user