fix(SEC-31,SEC-32): make probe admission atomic and stop Reset clearing a shared fallback partition
Two defects found in code review of the limiter rework. Probe admission was check-then-act across two lock scopes: Check() read "probe due" under lock(state), released it, then re-acquired to advance NextProbeAtTicks. A burst of requests arriving together at an interval boundary could therefore all observe the slot as due and all be admitted, handing the verifier the very burst the interval exists to bound. The claim is now a single critical section (TryConsumeProbe). The two layers are still claimed one at a time — holding two per-state locks at once would need a global lock ordering to stay deadlock-free — so a slot claimed on the composite partition is compensated via ReleaseProbe when the aggregate then refuses, which otherwise silently spent the partition's next slot and pushed the legitimate holder out by a full interval. Reset() removed whatever partition the caller resolved to, including the address's shared fallback partition when the caller's key id had been collapsed into it by the per-peer cap (or when the token was junk-shaped). That bucket also carries failures contributed by other key ids from the same address, so one successful authentication became a reset button for an in-progress spray. Reset now clears only a partition the caller owns (effectiveKeyId == presented key id); the shared bucket decays by window expiry instead, and the caller still recovers through probe admission. The key's aggregate is cleared either way, as designed. Also applied from the review: closure-free GetOrAdd overload on _partitions, and a remarks paragraph acknowledging the best-effort O(n) eviction scan under sustained overflow. Threading the resolved partition key from Check through to RecordFailure/Reset was declined: Check resolves with mint:false and RecordFailure with mint:true, and the two can legitimately differ when a concurrent caller fills the per-peer cap in between — reusing Check's key would record into the wrong partition and bypass the cap, which is not worth saving one string concat. Tests (limiter suite 11 -> 14): ProbeAdmission_UnderConcurrentArrivals_ GrantsExactlyOneSlot (200 rounds x 8 barrier-released threads at the boundary), ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot, and Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition. The latter two were confirmed as genuine reds against the unfixed code; the concurrency test is a guard — it is deterministically green on the fixed structure but did not reproduce the original nanosecond-wide window on its own.
This commit is contained in:
Joseph Doherty
@@ -161,4 +161,4 @@ Sequence these together rather than piecemeal — several are one change set spa
|
||||
| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. |
|
||||
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). |
|
||||
| 2026-07-13 | New finding **TST-30** (`Low`/`P2`) added — surfaced during TST-25 acceptance: CI runs on a single shared `gitea-runner` (`maxParallel=1`, co-located `10.100.0.35`) interleaved with `dohertj2/lmxopcua`, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the `run-windev-ci.sh` bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10). |
|
||||
| 2026-08-07 | **SEC-31 + SEC-32 → `Done`** (branch `fix/sec-31-32-limiter`, one change set as planned). `ApiKeyFailureLimiter` reworked from `IsBlocked/RecordFailure/Reset(string peer)` to a partition-pair API (`Check/RecordFailure/Reset(ApiKeyThrottlePartition)` returning `ApiKeyThrottleDecision`): layer 1 is the composite `(transport peer, key id)` partition, layer 2 a per-key-id aggregate across peers (`ApiKeyFailureAggregateLimit`, default 30), and an over-limit state now admits one probe per `ApiKeyFailureProbeIntervalSeconds` (default 5) instead of blocking absolutely — so a success can reset the state while throttled, killing the 10-packets-per-minute lockout. SEC-32 rides along: the interceptor validates token shape (`mxgw` prefix, ≥3 non-empty `_` segments, key id ≤ 64 chars) before minting a key-id partition, each peer may mint at most 32 of them (overflow collapses to its fallback partition), and eviction prefers expired windows, never dropping an over-limit partition below a 2× transient overshoot ceiling. New counter `mxgateway.auth.throttled` tagged `stage=peer\|aggregate` only (no key material — `/metrics` is still unauthenticated per open SEC-14). Docs updated in the same commit (`docs/GatewayConfiguration.md` limiter rows + two new keys, `docs/Authentication.md` hot-path paragraph, `docs/Authorization.md` SEC-11 section, limiter/`SecurityOptions` XML remarks). Evidence: `dotnet build …Server` clean; `--filter ~ApiKeyFailureLimiter` 11/11 passed (new `ApiKeyFailureLimiterTests`), `--filter ~GatewayGrpcAuthorizationInterceptor` 20/20 passed (incl. the four SEC-31 contract tests and `NonMxgwToken_FallsBackToTransportPeerPartition`), `--filter ~GatewayOptionsValidator` 66/66 passed. Full suite on macOS: 804 passed / 44 failed — all 44 are the pre-existing named-pipe fake-worker classes (`WorkerClientTests`, `FakeWorkerHarnessTests`, `SessionWorkerClientFactoryFakeWorkerTests`, `GatewayEndToEnd*`), verified identical (44) on the unmodified tree. Follow-up unchanged: the new `MxGateway:Security` keys belong in old **SEC-24**'s effective-config projection when that is picked up. |
|
||||
| 2026-08-07 | **SEC-31 + SEC-32 → `Done`** (branch `fix/sec-31-32-limiter`, one change set as planned). `ApiKeyFailureLimiter` reworked from `IsBlocked/RecordFailure/Reset(string peer)` to a partition-pair API (`Check/RecordFailure/Reset(ApiKeyThrottlePartition)` returning `ApiKeyThrottleDecision`): layer 1 is the composite `(transport peer, key id)` partition, layer 2 a per-key-id aggregate across peers (`ApiKeyFailureAggregateLimit`, default 30), and an over-limit state now admits one probe per `ApiKeyFailureProbeIntervalSeconds` (default 5) instead of blocking absolutely — so a success can reset the state while throttled, killing the 10-packets-per-minute lockout. SEC-32 rides along: the interceptor validates token shape (`mxgw` prefix, ≥3 non-empty `_` segments, key id ≤ 64 chars) before minting a key-id partition, each peer may mint at most 32 of them (overflow collapses to its fallback partition), and eviction prefers expired windows, never dropping an over-limit partition below a 2× transient overshoot ceiling. New counter `mxgateway.auth.throttled` tagged `stage=peer\|aggregate` only (no key material — `/metrics` is still unauthenticated per open SEC-14). Docs updated in the same commit (`docs/GatewayConfiguration.md` limiter rows + two new keys, `docs/Authentication.md` hot-path paragraph, `docs/Authorization.md` SEC-11 section, limiter/`SecurityOptions` XML remarks). Evidence: `dotnet build …Server` clean; `--filter ~ApiKeyFailureLimiter` 11/11 passed (new `ApiKeyFailureLimiterTests`), `--filter ~GatewayGrpcAuthorizationInterceptor` 20/20 passed (incl. the four SEC-31 contract tests and `NonMxgwToken_FallsBackToTransportPeerPartition`), `--filter ~GatewayOptionsValidator` 66/66 passed. Full suite on macOS: 804 passed / 44 failed — all 44 are the pre-existing named-pipe fake-worker classes (`WorkerClientTests`, `FakeWorkerHarnessTests`, `SessionWorkerClientFactoryFakeWorkerTests`, `GatewayEndToEnd*`), verified identical (44) on the unmodified tree. Follow-up unchanged: the new `MxGateway:Security` keys belong in old **SEC-24**'s effective-config projection when that is picked up. **Code review of the branch found two defects in the first pass, both fixed before merge:** (1) probe admission was check-then-act across two lock scopes, so a burst arriving at an interval boundary could all observe "due" and all be admitted — the claim is now a single critical section (`TryConsumeProbe`), and because the two layers are claimed one at a time, a slot claimed on the partition is compensated (`ReleaseProbe`) when the aggregate then refuses; (2) `Reset` on a success whose key id had been collapsed into the address's shared fallback partition removed that shared partition, letting one authentication wipe an in-progress spray from the same address — it is now left to decay by window expiry, while the key's aggregate is still cleared. Tests added: `ProbeAdmission_UnderConcurrentArrivals_GrantsExactlyOneSlot`, `ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot`, `Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition` (limiter suite 11 → 14). |
|
||||
|
||||
@@ -96,7 +96,7 @@ Before the verification store read, the helper asks a cheap in-process failure c
|
||||
- **Composite `(transport peer, key id)` partitions.** Reaching `MxGateway:Security:ApiKeyFailureLimit` failures binds the throttle to the address that produced them. The key id alone is never the partition: key ids are not secret — they ride in every token and are listed on the dashboard — so keying on them let any network peer deny a key to its legitimate holder. The key id joins the partition only after a token-shape check (literal `mxgw` prefix, at least three non-empty `_` segments, key id of at most 64 characters), and one address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition.
|
||||
- **A per-key-id aggregate** across all peers (`ApiKeyFailureAggregateLimit`, default 30), which bounds a distributed or source-rotating sprayer that never trips any single partition.
|
||||
|
||||
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
|
||||
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
|
||||
|
||||
The dashboard login surface is throttled independently: `POST /auth/login` carries a fixed-window ASP.NET Core rate-limiter policy keyed per remote IP (`MxGateway:Security:LoginRateLimit*`), rejecting a burst with HTTP 429 before the LDAP bind is relayed to the directory. See [GatewayConfiguration](./GatewayConfiguration.md#security-options).
|
||||
|
||||
|
||||
@@ -364,7 +364,7 @@ model requires otherwise.
|
||||
| `MxGateway:Security:ApiKeyFailureLimit` | `10` | Failed API-key verifications, per `(transport peer, key id)` partition, within `ApiKeyFailureWindowSeconds` that trip the in-process short-circuit. Once tripped, the gRPC auth path rejects further attempts from that partition with `ResourceExhausted` **before** the store read — except for the probe admitted every `ApiKeyFailureProbeIntervalSeconds` — and a successful verification resets the partition. The partition always includes the sender's transport address: key ids are public (they ride in every token and are listed on the dashboard), so keying on the key id alone let any peer deny a key to its legitimate holder. Must be greater than zero. |
|
||||
| `MxGateway:Security:ApiKeyFailureWindowSeconds` | `60` | Sliding-window length, in seconds, over which API-key verification failures are counted, for both the per-partition and the per-key-id aggregate layer. Must be greater than zero. |
|
||||
| `MxGateway:Security:ApiKeyFailureAggregateLimit` | `30` | Failed verifications for one key id counted across **all** transport peers within `ApiKeyFailureWindowSeconds` before that key id enters probe mode. This second layer bounds a distributed or source-rotating sprayer that never trips any single `(peer, key id)` partition. `0` disables the aggregate layer, leaving only per-partition counting. Must be zero or greater. |
|
||||
| `MxGateway:Security:ApiKeyFailureProbeIntervalSeconds` | `5` | Minimum interval, in seconds, between probe admissions for an over-limit partition or key-id aggregate. An over-limit state is a valve rather than a wall: one request per interval reaches the real verifier, so the holder of the correct secret always gets through and clears the state, while everything else is still refused before the store read. `0` blocks absolutely instead — **not recommended**, because an unauthenticated peer can then deny the key to its holder for the whole window. Must be zero or greater. |
|
||||
| `MxGateway:Security:ApiKeyFailureProbeIntervalSeconds` | `5` | Minimum interval, in seconds, between probe admissions for an over-limit partition or key-id aggregate. An over-limit state is a valve rather than a wall: one request per interval reaches the real verifier — exactly one, even when a burst arrives together at the interval boundary — so the holder of the correct secret always gets through and clears the state, while everything else is still refused before the store read. `0` blocks absolutely instead — **not recommended**, because an unauthenticated peer can then deny the key to its holder for the whole window. Must be zero or greater. |
|
||||
| `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct partitions tracked by the failure counter (a bounded LRU) so a spray of unique tokens cannot grow memory without limit. It cannot be used to flush an active block either: only a validly shaped `mxgw_<keyId>_<secret>` token mints a key-id partition (everything else lands on the sender's transport-peer partition), each address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition, and eviction prefers fully expired windows, never removing an over-limit partition until the map exceeds twice this cap. Must be greater than zero. |
|
||||
|
||||
## Galaxy Options
|
||||
|
||||
@@ -35,6 +35,13 @@ namespace ZB.MOM.WW.MxGateway.Server.Security.Authorization;
|
||||
/// peers are removed on reset, so in steady state the map holds only partitions with recent failures
|
||||
/// — the common success path is a lock-free dictionary miss.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Eviction is deliberately a best-effort full scan of the over-capacity map, run only on the
|
||||
/// failure path and only once the cap is exceeded. Under a sustained overflow that scan is O(n) per
|
||||
/// recorded failure, which is accepted: n is bounded by the cap, the work lands on attack traffic
|
||||
/// rather than on authenticated calls, and an exact ordered structure would need a second index kept
|
||||
/// consistent with the per-state locks for no security gain.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ApiKeyFailureLimiter
|
||||
{
|
||||
@@ -140,26 +147,29 @@ public sealed class ApiKeyFailureLimiter
|
||||
return peerOver ? ApiKeyThrottleDecision.ThrottledByPeer : ApiKeyThrottleDecision.ThrottledByAggregate;
|
||||
}
|
||||
|
||||
// Both over-limit layers must have a slot before either is consumed, so a request cannot burn
|
||||
// the peer's probe and then be refused by the aggregate.
|
||||
if (peerOver && !IsProbeDue(peerState!, now))
|
||||
{
|
||||
return ApiKeyThrottleDecision.ThrottledByPeer;
|
||||
}
|
||||
|
||||
if (aggregateOver && !IsProbeDue(aggregateState!, now))
|
||||
{
|
||||
return ApiKeyThrottleDecision.ThrottledByAggregate;
|
||||
}
|
||||
|
||||
// Every over-limit layer must yield its probe slot for the request to pass. The slots are
|
||||
// claimed one at a time (holding two per-state locks at once would need a global ordering to
|
||||
// stay deadlock-free), so a claim is reserved and then compensated if a later layer refuses.
|
||||
long peerProbeRestore = 0;
|
||||
bool peerProbeClaimed = false;
|
||||
if (peerOver)
|
||||
{
|
||||
ConsumeProbe(peerState!, now);
|
||||
if (!TryConsumeProbe(peerState!, now, out peerProbeRestore))
|
||||
{
|
||||
return ApiKeyThrottleDecision.ThrottledByPeer;
|
||||
}
|
||||
|
||||
peerProbeClaimed = true;
|
||||
}
|
||||
|
||||
if (aggregateOver)
|
||||
if (aggregateOver && !TryConsumeProbe(aggregateState!, now, out _))
|
||||
{
|
||||
ConsumeProbe(aggregateState!, now);
|
||||
if (peerProbeClaimed)
|
||||
{
|
||||
ReleaseProbe(peerState!, now, peerProbeRestore);
|
||||
}
|
||||
|
||||
return ApiKeyThrottleDecision.ThrottledByAggregate;
|
||||
}
|
||||
|
||||
return ApiKeyThrottleDecision.ProbeAdmitted;
|
||||
@@ -178,7 +188,10 @@ public sealed class ApiKeyFailureLimiter
|
||||
long now = _clock.GetUtcNow().UtcTicks;
|
||||
(string partitionKey, string? effectiveKeyId) = ResolvePartitionKey(peer, partition.KeyId, mint: true);
|
||||
|
||||
WindowState state = _partitions.GetOrAdd(partitionKey, _ => new WindowState(peer, effectiveKeyId));
|
||||
WindowState state = _partitions.GetOrAdd(
|
||||
partitionKey,
|
||||
static (_, owner) => new WindowState(owner.Peer, owner.KeyId),
|
||||
(Peer: peer, KeyId: effectiveKeyId));
|
||||
RecordInto(state, now, _limit);
|
||||
|
||||
// Only a key id that earned its own partition feeds the aggregate: an id squeezed out by the
|
||||
@@ -198,8 +211,17 @@ public sealed class ApiKeyFailureLimiter
|
||||
public void Reset(ApiKeyThrottlePartition partition)
|
||||
{
|
||||
string peer = RequirePeer(partition);
|
||||
(string partitionKey, _) = ResolvePartitionKey(peer, partition.KeyId, mint: false);
|
||||
RemovePartition(partitionKey);
|
||||
(string partitionKey, string? effectiveKeyId) = ResolvePartitionKey(peer, partition.KeyId, mint: false);
|
||||
|
||||
// Clear only a partition this caller actually owns. When its key id was squeezed into the
|
||||
// address's shared fallback bucket by the per-peer cap, that bucket also holds failures
|
||||
// contributed by other key ids — and by junk-shaped tokens — from the same address, so
|
||||
// removing it would let one successful authentication wipe an in-progress spray. The shared
|
||||
// bucket decays by window expiry instead; the caller still recovers through probe admission.
|
||||
if (string.Equals(effectiveKeyId, partition.KeyId, StringComparison.Ordinal))
|
||||
{
|
||||
RemovePartition(partitionKey);
|
||||
}
|
||||
|
||||
// The aggregate is cleared on the presented key id even when the composite partition
|
||||
// collapsed to the fallback: a verified secret is proof the key is not under successful
|
||||
@@ -264,20 +286,41 @@ public sealed class ApiKeyFailureLimiter
|
||||
}
|
||||
}
|
||||
|
||||
private bool IsProbeDue(WindowState state, long now)
|
||||
/// <summary>
|
||||
/// Atomically claims this interval's probe slot. Checking and advancing must happen in one
|
||||
/// critical section: a due-check that released the lock before advancing would let every request
|
||||
/// arriving at the interval boundary observe "due" and all be admitted, which is exactly the
|
||||
/// unbounded-guessing burst the probe interval exists to prevent.
|
||||
/// </summary>
|
||||
private bool TryConsumeProbe(WindowState state, long now, out long previousProbeAtTicks)
|
||||
{
|
||||
lock (state)
|
||||
{
|
||||
return now >= state.NextProbeAtTicks;
|
||||
previousProbeAtTicks = state.NextProbeAtTicks;
|
||||
if (now < previousProbeAtTicks)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
state.NextProbeAtTicks = now + _probeIntervalTicks;
|
||||
state.LastActivityTicks = now;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
private void ConsumeProbe(WindowState state, long now)
|
||||
/// <summary>
|
||||
/// Returns a probe slot claimed for a request that a later layer then refused, so the wasted
|
||||
/// reservation does not cost the next arrival its slot. Only the caller's own reservation is
|
||||
/// undone — a slot re-granted or re-armed in the meantime wins.
|
||||
/// </summary>
|
||||
private void ReleaseProbe(WindowState state, long now, long previousProbeAtTicks)
|
||||
{
|
||||
lock (state)
|
||||
{
|
||||
state.NextProbeAtTicks = now + _probeIntervalTicks;
|
||||
state.LastActivityTicks = now;
|
||||
if (state.NextProbeAtTicks == now + _probeIntervalTicks)
|
||||
{
|
||||
state.NextProbeAtTicks = previousProbeAtTicks;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -125,6 +125,100 @@ public sealed class ApiKeyFailureLimiterTests
|
||||
Assert.Equal(ApiKeyThrottleDecision.ProbeAdmitted, limiter.Check(partition));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The probe slot is claimed atomically: when a crowd of requests arrives at the same interval
|
||||
/// boundary exactly one is admitted and the rest are still refused. A check-then-act grant would
|
||||
/// let every arrival observe "due" and hand the whole burst through to the verifier.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ProbeAdmission_UnderConcurrentArrivals_GrantsExactlyOneSlot()
|
||||
{
|
||||
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
|
||||
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 3, aggregateLimit: 5);
|
||||
ApiKeyThrottlePartition partition = new("ipv4:10.0.0.1:1", "victim");
|
||||
|
||||
// Five failures on one partition trip both layers (limit 3, aggregate 5), so the concurrent
|
||||
// arrivals contend for the composite probe slot AND the aggregate's.
|
||||
RecordFailures(limiter, partition, 5);
|
||||
clock.Advance(ProbeInterval);
|
||||
|
||||
// The unsafe window between reading "probe due" and advancing the slot is nanoseconds wide,
|
||||
// so one burst can miss it by luck. Repeating the boundary makes the red reliable, while the
|
||||
// atomic implementation must yield exactly one admission in every round.
|
||||
const int arrivals = 8;
|
||||
const int rounds = 200;
|
||||
int totalAdmitted = 0;
|
||||
|
||||
for (int round = 0; round < rounds; round++)
|
||||
{
|
||||
// Re-arm: the failures keep both layers over their limits, and recording pushes the next
|
||||
// probe one interval out, which the advance below then reaches.
|
||||
RecordFailures(limiter, partition, 5);
|
||||
clock.Advance(ProbeInterval);
|
||||
|
||||
ApiKeyThrottleDecision[] decisions = new ApiKeyThrottleDecision[arrivals];
|
||||
using (Barrier startLine = new(arrivals))
|
||||
{
|
||||
Thread[] threads = new Thread[arrivals];
|
||||
for (int index = 0; index < arrivals; index++)
|
||||
{
|
||||
int slot = index;
|
||||
threads[slot] = new Thread(() =>
|
||||
{
|
||||
startLine.SignalAndWait();
|
||||
decisions[slot] = limiter.Check(partition);
|
||||
});
|
||||
threads[slot].Start();
|
||||
}
|
||||
|
||||
foreach (Thread thread in threads)
|
||||
{
|
||||
Assert.True(thread.Join(TimeSpan.FromSeconds(30)), "probe-contention thread did not finish");
|
||||
}
|
||||
}
|
||||
|
||||
int admitted = decisions.Count(decision => decision == ApiKeyThrottleDecision.ProbeAdmitted);
|
||||
Assert.Equal(
|
||||
arrivals - admitted,
|
||||
decisions.Count(decision => decision is ApiKeyThrottleDecision.ThrottledByPeer
|
||||
or ApiKeyThrottleDecision.ThrottledByAggregate));
|
||||
totalAdmitted += admitted;
|
||||
}
|
||||
|
||||
Assert.Equal(rounds, totalAdmitted);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The layers claim their probe slots one at a time, so a slot claimed on the composite partition
|
||||
/// must be returned when the aggregate then refuses. Otherwise a refused request would silently
|
||||
/// spend the partition's next slot and push the legitimate holder out by a full interval.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot()
|
||||
{
|
||||
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
|
||||
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 3, aggregateLimit: 5);
|
||||
ApiKeyThrottlePartition holder = new("ipv4:10.0.0.1:1", "victim");
|
||||
|
||||
// Both layers trip together, arming both slots for t0 + interval.
|
||||
RecordFailures(limiter, holder, 5);
|
||||
|
||||
// A failure from a second address re-arms only the aggregate (its own composite is far under
|
||||
// the limit), so the aggregate's slot now opens one interval later than the partition's.
|
||||
clock.Advance(TimeSpan.FromSeconds(3));
|
||||
limiter.RecordFailure(new ApiKeyThrottlePartition("ipv4:10.0.0.2:1", "victim"));
|
||||
|
||||
// t0 + 5s: the partition's slot is due, the aggregate's is not — the request is refused and
|
||||
// the partition's slot must be handed back rather than consumed.
|
||||
clock.Advance(TimeSpan.FromSeconds(2));
|
||||
Assert.Equal(ApiKeyThrottleDecision.ThrottledByAggregate, limiter.Check(holder));
|
||||
|
||||
// t0 + 8s: the aggregate's slot opens. The partition's slot was restored, so this passes; had
|
||||
// it been consumed above it would not reopen until t0 + 10s and this would be ThrottledByPeer.
|
||||
clock.Advance(TimeSpan.FromSeconds(3));
|
||||
Assert.Equal(ApiKeyThrottleDecision.ProbeAdmitted, limiter.Check(holder));
|
||||
}
|
||||
|
||||
/// <summary>A zero probe interval restores absolute blocking (documented as not recommended).</summary>
|
||||
[Fact]
|
||||
public void ProbeIntervalZero_BlocksAbsolutely()
|
||||
@@ -165,6 +259,39 @@ public sealed class ApiKeyFailureLimiterTests
|
||||
limiter.Check(new ApiKeyThrottlePartition("ipv4:10.0.9.9:1", "victim")));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A success whose key id was squeezed into the address's shared fallback bucket must not clear
|
||||
/// that bucket: it carries failures from other key ids at the same address, so clearing it would
|
||||
/// make one successful authentication a reset button for an in-progress spray.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition()
|
||||
{
|
||||
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
|
||||
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 3, aggregateLimit: 100, maxPartitions: 4096);
|
||||
const string peer = "ipv4:10.0.0.1:1";
|
||||
|
||||
// Fill the per-peer key-id cap, then spray past it so the overflow lands on — and trips —
|
||||
// the address's shared fallback partition.
|
||||
for (int i = 0; i < ApiKeyFailureLimiter.MaxKeyIdPartitionsPerPeer; i++)
|
||||
{
|
||||
limiter.RecordFailure(new ApiKeyThrottlePartition(peer, $"key{i}"));
|
||||
}
|
||||
|
||||
for (int i = 0; i < 5; i++)
|
||||
{
|
||||
limiter.RecordFailure(new ApiKeyThrottlePartition(peer, $"overflow{i}"));
|
||||
}
|
||||
|
||||
ApiKeyThrottlePartition overCap = new(peer, "overflow0");
|
||||
Assert.Equal(ApiKeyThrottleDecision.ThrottledByPeer, limiter.Check(overCap));
|
||||
|
||||
limiter.Reset(overCap);
|
||||
|
||||
Assert.True(limiter.IsTracked(overCap));
|
||||
Assert.Equal(ApiKeyThrottleDecision.ThrottledByPeer, limiter.Check(overCap));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// SEC-32: a spray of unique junk partitions (junk tokens resolve to the sender's fallback
|
||||
/// partition, one per address) must not evict a partition that is currently throttled — the
|
||||
|
||||
Reference in New Issue
Block a user