dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 14 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(SEC-31,SEC-32): re-partition the API-key failure limiter on (peer, key id) with probe admission

Browse Source 
The gRPC auth failure limiter partitioned on the key id parsed out of the
*unauthenticated* token and rejected with ResourceExhausted before VerifyAsync
ran. Key ids are not secret — they ride in every token and are listed on the
dashboard — so any network peer could send 10 garbage-secret requests per minute
and deny that key indefinitely: the legitimate holder's correct secret was
refused before it was ever checked, and the success-path Reset that would clear
the block sat behind the verification the block prevented (SEC-31). The tracked
map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the
`mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked
entry and reset the window (SEC-32).

ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a
partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an
ApiKeyThrottleDecision result. Two layers share one sliding window — a composite
(transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id
aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30)
that bounds a source-rotating sprayer. An over-limit state is now a valve rather
than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default
5) is admitted through to the real verifier, so the correct secret always reaches
the constant-time compare and resets both layers. Guarantees preserved: guessing
stays bounded per window, and the failure path still spends no store read per
attempt.

SEC-32 rides the same change set: the interceptor validates token shape (literal
`mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a
key-id partition, each transport peer may mint at most 32 of them before the
overflow collapses onto its fallback partition, and eviction prefers fully
expired windows and never drops an over-limit partition below a 2x transient
overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged
stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key
material may appear there.

Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys,
the Authentication hot-path paragraph, the Authorization SEC-11 section, and the
limiter / SecurityOptions XML remarks (the old NAT rationale described the
defective keying). Tracking rows flipped to Done with a change-log entry.

Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs
aggregate trip points, probe cadence, absolute-block mode, reset across both
layers, junk-spray eviction resistance, the per-peer cap, and expired-window
eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four
SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20
total); GatewayOptionsValidatorTests covers both new keys including 0 as a
supported disable value (66 total).
This commit is contained in:
Joseph Doherty
2026-08-07 05:39:10 -04:00
parent ead921cace
commit df710e18a9
16 changed files with 1148 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs
+243 -24
View File
@@ -22,6 +22,12 @@ namespace ZB.MOM.WW.MxGateway.Tests.Security.Authorization;
public sealed class GatewayGrpcAuthorizationInterceptorTests
{
private const string AttackerPeer = "ipv4:203.0.113.7:5000";
private const string HolderPeer = "ipv4:198.51.100.4:5000";
private static readonly TimeSpan FailureWindow = TimeSpan.FromMinutes(1);
private static readonly TimeSpan ProbeInterval = TimeSpan.FromSeconds(5);
/// <summary>Verifies that missing API key returns unauthenticated status.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -359,21 +365,18 @@ public sealed class GatewayGrpcAuthorizationInterceptorTests
}
/// <summary>
/// Once a peer has exceeded the failure limit, the interceptor short-circuits with
/// <see cref="StatusCode.ResourceExhausted"/> BEFORE calling the verifier, so an online guessing
/// loop stops spending a store read per attempt. A verifier that always fails is used; after the
/// limit is reached the verifier is no longer invoked.
/// SEC-31: once an attacking peer has exceeded the failure limit for a key id, the interceptor
/// short-circuits with <see cref="StatusCode.ResourceExhausted"/> BEFORE calling the verifier, so
/// an online guessing loop stops spending a store read per attempt. The composite
/// <c>(peer, key id)</c> partition keeps that bound per attacking address.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task UnaryServerHandler_ExceedsFailureLimit_ShortCircuitsBeforeVerify()
public async Task BruteForceBound_StillEnforcedPerAttackingPeer()
{
CountingFailureVerifier verifier = new(Failure(ApiKeyFailure.SecretMismatch));
ApiKeyFailureLimiter limiter = new(
limit: 3,
window: TimeSpan.FromMinutes(1),
maxPeers: 16,
clock: TimeProvider.System);
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 3);
GatewayGrpcAuthorizationInterceptor interceptor = CreateInterceptor(
verifier,
new GatewayRequestIdentityAccessor(),
@@ -385,7 +388,7 @@ public sealed class GatewayGrpcAuthorizationInterceptorTests
RpcException failure = await Assert.ThrowsAsync<RpcException>(
() => interceptor.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_bad-secret"),
ContextWithAuthorization("Bearer mxgw_operator01_bad-secret", AttackerPeer),
(_, _) => Task.FromResult(new OpenSessionReply())));
Assert.Equal(StatusCode.Unauthenticated, failure.StatusCode);
}
@@ -396,13 +399,219 @@ public sealed class GatewayGrpcAuthorizationInterceptorTests
RpcException throttled = await Assert.ThrowsAsync<RpcException>(
() => interceptor.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_bad-secret"),
ContextWithAuthorization("Bearer mxgw_operator01_bad-secret", AttackerPeer),
(_, _) => Task.FromResult(new OpenSessionReply())));
Assert.Equal(StatusCode.ResourceExhausted, throttled.StatusCode);
Assert.Equal(3, verifier.CallCount);
}
/// <summary>
/// SEC-31 (the lockout inversion): an attacker who floods failures for a victim's key id from its
/// own address must not deny that key to the legitimate holder. The holder presents the correct
/// secret from a different transport peer and authenticates on the first attempt — the verifier
/// is reached and the RPC succeeds.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AttackerSpamOnVictimKeyId_FromDifferentPeer_DoesNotBlockLegitimateHolderPresentingCorrectSecret()
{
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 3, aggregateLimit: 30);
GatewayGrpcAuthorizationInterceptor attacked = CreateInterceptor(
new CountingFailureVerifier(Failure(ApiKeyFailure.SecretMismatch)),
new GatewayRequestIdentityAccessor(),
failureLimiter: limiter);
// Flood the victim's key id from the attacker's address until that partition is throttled.
StatusCode lastAttackerStatus = StatusCode.OK;
for (int attempt = 0; attempt < 6; attempt++)
{
RpcException failure = await Assert.ThrowsAsync<RpcException>(
() => attacked.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_guess", AttackerPeer),
(_, _) => Task.FromResult(new OpenSessionReply())));
lastAttackerStatus = failure.StatusCode;
}
Assert.Equal(StatusCode.ResourceExhausted, lastAttackerStatus);
// The legitimate holder, on a different address, is verified and admitted immediately.
FakeApiKeyVerifier holderVerifier = new(SuccessWithScopes(GatewayScopes.SessionOpen));
GatewayGrpcAuthorizationInterceptor holder = CreateInterceptor(
holderVerifier,
new GatewayRequestIdentityAccessor(),
failureLimiter: limiter);
OpenSessionReply reply = await holder.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_correct", HolderPeer),
(_, _) => Task.FromResult(new OpenSessionReply { SessionId = "session-1" }));
Assert.True(holderVerifier.WasCalled);
Assert.Equal("session-1", reply.SessionId);
}
/// <summary>
/// SEC-31 layer 2: failures for one key id sprayed across more distinct peers than
/// <c>ApiKeyFailureAggregateLimit</c> put that key id into probe mode globally, so a
/// rotating-source attacker gets at most one verifier call per probe interval.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AggregateSpray_AcrossManyPeers_TripsPerKeyProbeMode()
{
CountingFailureVerifier verifier = new(Failure(ApiKeyFailure.SecretMismatch));
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 100, aggregateLimit: 5);
GatewayGrpcAuthorizationInterceptor interceptor = CreateInterceptor(
verifier,
new GatewayRequestIdentityAccessor(),
failureLimiter: limiter);
for (int peer = 0; peer < 5; peer++)
{
await Assert.ThrowsAsync<RpcException>(
() => interceptor.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_guess", $"ipv4:10.9.0.{peer}:5000"),
(_, _) => Task.FromResult(new OpenSessionReply())));
}
Assert.Equal(5, verifier.CallCount);
// A never-seen peer is now probe-limited: no composite failures of its own, but the key id's
// aggregate is tripped, so the request never reaches the verifier.
RpcException throttled = await Assert.ThrowsAsync<RpcException>(
() => interceptor.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_guess", "ipv4:10.9.1.1:5000"),
(_, _) => Task.FromResult(new OpenSessionReply())));
Assert.Equal(StatusCode.ResourceExhausted, throttled.StatusCode);
Assert.Equal(5, verifier.CallCount);
// One probe slot opens per interval, and it is consumed by the first arrival.
clock.Advance(ProbeInterval);
RpcException probed = await Assert.ThrowsAsync<RpcException>(
() => interceptor.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_guess", "ipv4:10.9.1.2:5000"),
(_, _) => Task.FromResult(new OpenSessionReply())));
Assert.Equal(StatusCode.Unauthenticated, probed.StatusCode);
Assert.Equal(6, verifier.CallCount);
RpcException throttledAgain = await Assert.ThrowsAsync<RpcException>(
() => interceptor.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_guess", "ipv4:10.9.1.3:5000"),
(_, _) => Task.FromResult(new OpenSessionReply())));
Assert.Equal(StatusCode.ResourceExhausted, throttledAgain.StatusCode);
Assert.Equal(6, verifier.CallCount);
}
/// <summary>
/// SEC-31: the success-reset path stays reachable while throttled. A throttled partition admits
/// one probe per interval; the correct secret rides that slot, authenticates, and fully clears
/// both limiter layers, so the next wrong attempt is <see cref="StatusCode.Unauthenticated"/>
/// rather than <see cref="StatusCode.ResourceExhausted"/>.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task CorrectSecret_DuringProbeMode_AuthenticatesViaProbeSlotAndResets()
{
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 3, aggregateLimit: 30);
GatewayGrpcAuthorizationInterceptor failing = CreateInterceptor(
new FakeApiKeyVerifier(Failure(ApiKeyFailure.SecretMismatch)),
new GatewayRequestIdentityAccessor(),
failureLimiter: limiter);
FakeApiKeyVerifier holderVerifier = new(SuccessWithScopes(GatewayScopes.SessionOpen));
GatewayGrpcAuthorizationInterceptor succeeding = CreateInterceptor(
holderVerifier,
new GatewayRequestIdentityAccessor(),
failureLimiter: limiter);
for (int attempt = 0; attempt < 3; attempt++)
{
await Assert.ThrowsAsync<RpcException>(
() => failing.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_bad", HolderPeer),
(_, _) => Task.FromResult(new OpenSessionReply())));
}
RpcException throttled = await Assert.ThrowsAsync<RpcException>(
() => failing.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_bad", HolderPeer),
(_, _) => Task.FromResult(new OpenSessionReply())));
Assert.Equal(StatusCode.ResourceExhausted, throttled.StatusCode);
clock.Advance(ProbeInterval);
OpenSessionReply reply = await succeeding.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_correct", HolderPeer),
(_, _) => Task.FromResult(new OpenSessionReply { SessionId = "session-1" }));
Assert.True(holderVerifier.WasCalled);
Assert.Equal("session-1", reply.SessionId);
RpcException afterReset = await Assert.ThrowsAsync<RpcException>(
() => failing.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization("Bearer mxgw_operator01_bad", HolderPeer),
(_, _) => Task.FromResult(new OpenSessionReply())));
Assert.Equal(StatusCode.Unauthenticated, afterReset.StatusCode);
}
/// <summary>
/// SEC-32: only a validly shaped <c>mxgw_&lt;keyId&gt;_&lt;secret&gt;</c> token mints a key-id
/// partition. Junk tokens of varied shapes all collapse onto the sender's transport-peer fallback
/// partition, so a spray cannot mint one tracked entry per invented token.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task NonMxgwToken_FallsBackToTransportPeerPartition()
{
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 1000);
GatewayGrpcAuthorizationInterceptor interceptor = CreateInterceptor(
new FakeApiKeyVerifier(Failure(ApiKeyFailure.MissingOrMalformed)),
new GatewayRequestIdentityAccessor(),
failureLimiter: limiter);
string[] junkTokens =
[
"Bearer garbage",
"Bearer a_b_c",
"Bearer notmxgw_operator01_secret",
"Bearer MXGW_operator01_secret",
"Bearer mxgw__secret",
"Bearer mxgw_operator01_",
"Bearer mxgw_" + new string('k', 65) + "_secret",
"Bearer mxgw_onlytwo",
];
foreach (string token in junkTokens)
{
await Assert.ThrowsAsync<RpcException>(
() => interceptor.UnaryServerHandler(
new OpenSessionRequest(),
ContextWithAuthorization(token, AttackerPeer),
(_, _) => Task.FromResult(new OpenSessionReply())));
}
Assert.Equal(1, limiter.TrackedPartitionCount);
Assert.True(limiter.IsTracked(new ApiKeyThrottlePartition(AttackerPeer, KeyId: null)));
}
/// <summary>
/// A successful verification resets the peer's failure counter, so accumulated failures
/// from a fat-fingered secret do not lock out a client that subsequently authenticates.
@@ -411,11 +620,7 @@ public sealed class GatewayGrpcAuthorizationInterceptorTests
[Fact]
public async Task UnaryServerHandler_SuccessResetsFailureCounter()
{
ApiKeyFailureLimiter limiter = new(
limit: 3,
window: TimeSpan.FromMinutes(1),
maxPeers: 16,
clock: TimeProvider.System);
ApiKeyFailureLimiter limiter = CreateLimiter(new ManualTimeProvider(DateTimeOffset.UnixEpoch), limit: 3);
// Two failures against the same key id, then a success (which resets), then two more
// failures — without the reset the fifth attempt would be blocked at the limit of 3.
@@ -487,11 +692,23 @@ public sealed class GatewayGrpcAuthorizationInterceptorTests
Mode = authenticationMode
}
}),
failureLimiter ?? new ApiKeyFailureLimiter(
limit: 1000,
window: TimeSpan.FromMinutes(1),
maxPeers: 1000,
clock: TimeProvider.System));
failureLimiter ?? CreateLimiter(TimeProvider.System, limit: 1000),
new GatewayMetrics());
}
private static ApiKeyFailureLimiter CreateLimiter(
TimeProvider clock,
int limit,
int aggregateLimit = 0,
int maxPartitions = 1024)
{
return new ApiKeyFailureLimiter(
limit,
FailureWindow,
maxPartitions,
aggregateLimit,
ProbeInterval,
clock);
}
private static ApiKeyVerification SuccessWithScopes(params string[] scopes)
@@ -511,9 +728,11 @@ public sealed class GatewayGrpcAuthorizationInterceptorTests
return new ApiKeyVerification(Succeeded: false, Identity: null, Failure: failure);
}
private static TestServerCallContext ContextWithAuthorization(string authorizationHeader)
private static TestServerCallContext ContextWithAuthorization(string authorizationHeader, string? peer = null)
{
return new TestServerCallContext([new Metadata.Entry("authorization", authorizationHeader)]);
return new TestServerCallContext(
[new Metadata.Entry("authorization", authorizationHeader)],
peer: peer);
}
/// <summary>Records whether the gateway service ran past the interceptor for composition tests.</summary>