fix(SEC-31,SEC-32): re-partition the API-key failure limiter on (peer, key id) with probe admission

The gRPC auth failure limiter partitioned on the key id parsed out of the
*unauthenticated* token and rejected with ResourceExhausted before VerifyAsync
ran. Key ids are not secret — they ride in every token and are listed on the
dashboard — so any network peer could send 10 garbage-secret requests per minute
and deny that key indefinitely: the legitimate holder's correct secret was
refused before it was ever checked, and the success-path Reset that would clear
the block sat behind the verification the block prevented (SEC-31). The tracked
map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the
`mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked
entry and reset the window (SEC-32).

ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a
partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an
ApiKeyThrottleDecision result. Two layers share one sliding window — a composite
(transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id
aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30)
that bounds a source-rotating sprayer. An over-limit state is now a valve rather
than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default
5) is admitted through to the real verifier, so the correct secret always reaches
the constant-time compare and resets both layers. Guarantees preserved: guessing
stays bounded per window, and the failure path still spends no store read per
attempt.

SEC-32 rides the same change set: the interceptor validates token shape (literal
`mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a
key-id partition, each transport peer may mint at most 32 of them before the
overflow collapses onto its fallback partition, and eviction prefers fully
expired windows and never drops an over-limit partition below a 2x transient
overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged
stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key
material may appear there.

Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys,
the Authentication hot-path paragraph, the Authorization SEC-11 section, and the
limiter / SecurityOptions XML remarks (the old NAT rationale described the
defective keying). Tracking rows flipped to Done with a change-log entry.

Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs
aggregate trip points, probe cadence, absolute-block mode, reset across both
layers, junk-spray eviction resistance, the per-peer cap, and expired-window
eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four
SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20
total); GatewayOptionsValidatorTests covers both new keys including 0 as a
supported disable value (66 total).
This commit is contained in:
Joseph Doherty
2026-08-07 05:39:10 -04:00
parent ead921cace
commit df710e18a9
16 changed files with 1148 additions and 137 deletions
@@ -794,6 +794,45 @@ public sealed class GatewayOptionsValidatorTests
Assert.Contains(result.Failures!, f => f.Contains("ApiKeyFailureTrackedPeers"));
}
/// <summary>Verifies a negative per-key aggregate failure limit fails validation.</summary>
[Fact]
public void Validate_Fails_WhenApiKeyFailureAggregateLimitNegative()
{
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(
null,
WithSecurity(new SecurityOptions { ApiKeyFailureAggregateLimit = -1 }));
Assert.True(result.Failed);
Assert.Contains(result.Failures!, f => f.Contains("ApiKeyFailureAggregateLimit"));
}
/// <summary>Verifies a negative probe interval fails validation.</summary>
[Fact]
public void Validate_Fails_WhenApiKeyFailureProbeIntervalSecondsNegative()
{
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(
null,
WithSecurity(new SecurityOptions { ApiKeyFailureProbeIntervalSeconds = -1 }));
Assert.True(result.Failed);
Assert.Contains(result.Failures!, f => f.Contains("ApiKeyFailureProbeIntervalSeconds"));
}
/// <summary>
/// Zero is a supported (documented) value for both new limiter knobs: it disables the aggregate
/// layer and probe admission respectively, so validation must accept it.
/// </summary>
[Fact]
public void Validate_Succeeds_WhenAggregateLimitAndProbeIntervalAreZero()
{
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(
null,
WithSecurity(new SecurityOptions
{
ApiKeyFailureAggregateLimit = 0,
ApiKeyFailureProbeIntervalSeconds = 0,
}));
Assert.True(result.Succeeded);
}
private static GatewayOptions WithWorkerAndProtocol(WorkerOptions worker, ProtocolOptions protocol)
{
GatewayOptions source = ValidOptions();