The gRPC auth failure limiter partitioned on the key id parsed out of the
*unauthenticated* token and rejected with ResourceExhausted before VerifyAsync
ran. Key ids are not secret — they ride in every token and are listed on the
dashboard — so any network peer could send 10 garbage-secret requests per minute
and deny that key indefinitely: the legitimate holder's correct secret was
refused before it was ever checked, and the success-path Reset that would clear
the block sat behind the verification the block prevented (SEC-31). The tracked
map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the
`mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked
entry and reset the window (SEC-32).
ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a
partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an
ApiKeyThrottleDecision result. Two layers share one sliding window — a composite
(transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id
aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30)
that bounds a source-rotating sprayer. An over-limit state is now a valve rather
than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default
5) is admitted through to the real verifier, so the correct secret always reaches
the constant-time compare and resets both layers. Guarantees preserved: guessing
stays bounded per window, and the failure path still spends no store read per
attempt.
SEC-32 rides the same change set: the interceptor validates token shape (literal
`mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a
key-id partition, each transport peer may mint at most 32 of them before the
overflow collapses onto its fallback partition, and eviction prefers fully
expired windows and never drops an over-limit partition below a 2x transient
overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged
stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key
material may appear there.
Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys,
the Authentication hot-path paragraph, the Authorization SEC-11 section, and the
limiter / SecurityOptions XML remarks (the old NAT rationale described the
defective keying). Tracking rows flipped to Done with a change-log entry.
Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs
aggregate trip points, probe cadence, absolute-block mode, reset across both
layers, junk-spray eviction resistance, the per-peer cap, and expired-window
eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four
SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20
total); GatewayOptionsValidatorTests covers both new keys including 0 as a
supported disable value (66 total).
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Proto foundation + gateway-side of the size/backpressure pass:
- IPC-02: add GatewayHello.max_frame_bytes (regen Generated/); gateway sends
its negotiated worker-frame max in the handshake so the worker can adopt it
instead of a hard-coded default. Worker read-half lands separately.
- IPC-03: give the pipe frame max envelope-overhead headroom above the public
gRPC cap (WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes = 64 KiB;
default Worker.MaxMessageBytes bumped to 16 MiB + reserve), cross-validate the
headroom at startup, and pre-check command envelope size in WorkerClient so an
oversized command fails only that correlation (ResourceExhausted) instead of
faulting the whole session.
- IPC-04: reject DrainEvents max_events above a public ceiling in the request
validator (worker per-reply cap lands with the worker half).
Docs: GatewayConfiguration, WorkerFrameProtocol, gateway.md.
Tests: headroom validation, DrainEvents bound, oversized-command per-command
failure (pipe-harness, verified on windev).
SEC-07: add QueryActiveAlarmsRequest -> events:read scope arm; fix two tests that
constructed StreamAlarmsRequest instead of QueryActiveAlarmsRequest.
SEC-05: shorten hub-token lifetime 30m -> 5m; document that the ?access_token= query
carriage must never be request-logged.
SEC-08: gateway-side CachingApiKeyVerifier (short TTL, keyed on a hash of the presented
secret) skips the per-call store read+last_used write; CoalescingMarkApiKeyStore bounds
last_used writes to <=1/key/min; identity constraints are cached. Invalidation is wired
at the gateway admin sites (revoke/rotate/delete); short TTL backstops out-of-process CLI.
SEC-11: fixed-window rate limit on POST /auth/login + a per-peer (key-id) failure limiter
checked before VerifyAsync; new MxGateway:Security options bound + validated.
Also fixes regressions from the SEC-01/04/06 commit (c185f62) that a narrow test filter
missed (all now covered by a full-suite checkpoint):
- Rooting check is cross-platform: accepts Windows C:\/UNC forms on Unix so the shipped
appsettings path validates on the macOS dev box, still rejecting bare filenames.
- AddGatewayConfiguration TryAdds a non-production IHostEnvironment fallback so the validator
resolves in minimal test/tooling containers; the real host + apikey CLI register the actual
environment first (TryAdd no-op there).
- Test assembly defaults ASPNETCORE_ENVIRONMENT=Development (ModuleInitializer) so full-host
tests exercise wiring instead of tripping the SEC-04/06 production guards.
- GatewayOptionsTests asserts the SEC-01 CommonApplicationData-derived default (platform-correct).
archreview: SEC-07/05/08/11 (P1). Verified: NonWindows build clean; full gateway suite
747 passed / 42 failed, where all 42 are the pre-existing macOS named-pipe-harness failures
(Unix-socket path limit) and 0 are validation/regression failures.
- SEC-01: derive AuthenticationOptions.SqlitePath / TlsOptions.SelfSignedCertPath
defaults from SpecialFolder.CommonApplicationData (was Windows-literal strings
that became CWD-relative files off-Windows); validator rejects non-rooted
overrides; delete the stray C:\ProgramData\...gateway-auth.db files that
materialized under src/; add a tree-hygiene test failing on any *.db under src/.
- SEC-04: GatewayOptionsValidator fails startup when IsProduction() && DisableLogin.
- SEC-06: validator rejects LDAP Transport=None in Production; document the
MxGateway__Ldap__ServiceAccountPassword env override + dev-credential rotation.
Galaxy SnapshotCachePath rooting could not be validator-enforced (bound by the
external GalaxyRepository package, not GatewayOptions) — documented instead.
archreview: SEC-01/04/06 (P1). Verified on the Auth-0.1.4 baseline: Server build
clean, GatewayOptionsValidator + GatewayTreeHygiene tests 53/53.
Interlocking changes across the gateway server (shared GatewaySession.cs /
SessionManager.cs), committed together:
- GWC-01 (Critical): alarm monitor now attaches as an internal
(non-counted) distributor subscriber instead of a second raw drain of the
single worker event channel; WorkerClient._events -> SingleReader with a
claimed-once guard so a future dual-consumer regression throws loudly.
- GWC-02 (High): faulted sessions are swept in CloseExpiredLeasesAsync
(IsFaultedReapable + FaultedReason); new FaultedGraceSeconds (default 0).
- GWC-03 (High): configurable MaxSparseArrayLength (default 1_000_000)
enforced before allocation.
- TST-02 (High, security): StreamEvents attach now enforces the opening key
id -> PermissionDenied on owner mismatch.
- TST-12 (Medium): CLAUDE.md retention-defaults sentence corrected.
Verified: NonWindows build clean; targeted tests 135/135 on macOS, plus
WorkerClientTests 18/18 on the Windows host.
Server-054/055/056, Contracts-020/021/022, Tests-036/038/039,
IntegrationTests-030/031/032 (+033 deferred to live rig),
Client.Dotnet-026/028/029 (+027 won't-fix), Client.Go-030..034,
Client.Python-032..036, Client.Rust-033..038.
Key fix: SessionEventDistributor orphaned a subscriber that registered after
the pump completed but before disposal (Server-056) -> register paths now
complete late registrants under _lifecycleLock; regression test added. The
racy dashboard-mirror gRPC test made deterministic (Tests-039).
Verified green locally: gateway Tests targeted classes (GatewaySession,
SessionEventDistributor, GatewayOptionsValidator, ProtobufContractRoundTrip,
GatewaySessionDashboardMirror) + dotnet/go/python/rust client suites.
Adds WorkerReadyWaitTimeoutMs to SessionOptions (default 0 = disabled),
validates >= 0 in GatewayOptionsValidator, documents it in
GatewayConfiguration.md, and adds validator + default-value tests.
No wait/poll logic is implemented here (that is Task 8).
Remove the hard-rejection of AllowMultipleEventSubscribers=true in GatewayOptionsValidator
(fan-out is now implemented via SessionEventDistributor). Add MaxEventSubscribersPerSession
(default 8, must be >= 1) to SessionOptions, validate it, expose it in
EffectiveSessionConfiguration / GatewayConfigurationProvider, document it in
GatewayConfiguration.md and appsettings.json. Tests cover the no-error path for
AllowMultipleEventSubscribers=true, the 0/-1 rejection, positive pass, and default pass.
Defect 1: ValidateLdap used AddIfNotPositive for Port, accepting any value
> 0 including 70000. Replaced with builder.Port() from the shared
ZB.MOM.WW.Configuration library, which enforces the 1-65535 TCP range and
emits "MxGateway:Ldap:Port must be between 1 and 65535 (was {value})".
Defect 2: AddIfInvalidPath only caught ArgumentException, NotSupportedException,
and PathTooLongException from Path.GetFullPath. On macOS/Linux a path containing
an embedded null throws IOException, which escaped the catch block and caused
Validate() to throw instead of returning a failure. Added catch (IOException).
Tests: added Validate_Fails_WhenLdapPortIsZero, Validate_Fails_WhenLdapPortExceedsMaximum,
and Validate_Succeeds_WhenLdapEnabledWithValidPort to cover the new range boundary.
Joseph Doherty