feat(security): opt-in authentication for /metrics (SEC-14)

New MxGateway:Metrics:RequireAuthentication (default false). When true,
/metrics requires an API key with the admin scope (verified through the
cached verifier and the shared API-key failure limiter) or a dashboard
login with a real dashboard role; anonymous localhost and
Authentication:Mode=Disabled do not qualify. An anonymous scrape gets a
plain 401 Bearer challenge, a throttled peer 429. /health stays anonymous
because the off-repo Overview dashboard polls /health/active.

MapZbMetrics returns IEndpointConventionBuilder, so the policy is attached
with RequireAuthorization.
This commit is contained in:
Joseph Doherty
2026-10-22 06:37:31 -04:00
parent ed3a2c9c8b
commit 65bca804bb
10 changed files with 619 additions and 3 deletions
+1 -1
View File
@@ -99,7 +99,7 @@ Before the verification store read, the helper asks a cheap in-process failure c
- **Composite `(transport peer, key id)` partitions.** Reaching `MxGateway:Security:ApiKeyFailureLimit` failures binds the throttle to the address that produced them. The key id alone is never the partition: key ids are not secret — they ride in every token and are listed on the dashboard — so keying on them let any network peer deny a key to its legitimate holder. The key id joins the partition only after a token-shape check (literal `mxgw` prefix, at least three non-empty `_` segments, key id of at most 64 characters), and one address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition.
- **A per-key-id aggregate** across all peers (`ApiKeyFailureAggregateLimit`, default 30), which bounds a distributed or source-rotating sprayer that never trips any single partition.
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is anonymous by default (`MxGateway:Metrics:RequireAuthentication` is opt-in), so neither key ids nor peer addresses may appear there.
The dashboard login surface is throttled independently: `POST /auth/login` carries a fixed-window ASP.NET Core rate-limiter policy keyed per remote IP (`MxGateway:Security:LoginRateLimit*`), rejecting a burst with HTTP 429 before the LDAP bind is relayed to the directory. See [GatewayConfiguration](./GatewayConfiguration.md#security-options).
+35
View File
@@ -430,6 +430,41 @@ Suggested mapping:
- Galaxy SQL metadata endpoints if added: `metadata:read`
- worker shutdown diagnostics and key management: `admin`
## Metrics Endpoint Authentication (SEC-14)
Decision: `/metrics` stays anonymous by default and can be put behind
authentication with `MxGateway:Metrics:RequireAuthentication=true`; `/health`
stays anonymous unconditionally.
- **Opt-in rather than on by default.** Existing Prometheus scrapers on the
deployed hosts scrape anonymously; flipping the default would silently break
them on upgrade. Hosts whose network exposure makes the counters sensitive
enable it explicitly.
- **Health is never gated.** The family Overview dashboard (another repo) polls
`/health/active` without credentials, and liveness/readiness probes are
conventionally anonymous. Health responses carry status only.
- **Credentials reuse existing plumbing.** One `MxGateway.Metrics` scheme
accepts an API key with the `admin` scope (verified through the same cached
`IApiKeyVerifier` and the same per-peer failure limiter as gRPC, so `/metrics`
is not a way around the gRPC brute-force throttle) or, when no authorization
header is sent, the dashboard login cookie with a real dashboard role. No new
credential type or store is introduced.
- **`admin` scope, any dashboard role.** Metrics are an operator surface, so a
data-plane client key (`invoke:*`, `events:read`) is not enough. A dashboard
`Viewer` qualifies because every dashboard role already sees the same aggregate
counters on the dashboard; the exported metrics carry no session ids or tag
values.
- **No environment bypass.** The requirement reads no options, so anonymous
localhost and `Authentication:Mode=Disabled` — both of which satisfy the
dashboard Viewer policy — do not satisfy it. Only dev-only
`Dashboard:DisableLogin` (which authenticates every request as an
Administrator, and is rejected in Production) lets an unauthenticated scrape
through.
- **One scheme, plain 401.** Listing the cookie scheme on the policy directly
would challenge an anonymous scrape with a 302 redirect to `/login`; a single
scheme that falls back to the cookie internally keeps the challenge a `401`
with `WWW-Authenticate: Bearer`, which scrapers understand.
## Worker Process Identity
Decision: run workers as the gateway service identity for v1.
+29
View File
@@ -419,6 +419,35 @@ model requires otherwise.
| `MxGateway:Security:AuditRetentionDays` | `90` | Days of canonical audit history kept in the `audit_event` table. The audit drain sweeps once at startup and hourly thereafter, deleting rows older than this window; without it the table grows without bound inside the same SQLite file the authentication hot path reads. Rows whose timestamp SQLite cannot parse are never swept. Must be greater than zero — retention can be widened but not switched off. |
| `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct partitions tracked by the failure counter (a bounded LRU) so a spray of unique tokens cannot grow memory without limit. It cannot be used to flush an active block either: only a validly shaped `mxgw_<keyId>_<secret>` token mints a key-id partition (everything else lands on the sender's transport-peer partition), each address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition, and eviction prefers fully expired windows, never removing an over-limit partition until the map exceeds twice this cap. Must be greater than zero. |
## Metrics Options
Bound from `MxGateway:Metrics`. Governs the Prometheus `/metrics` scrape endpoint
(mounted by the shared `ZB.MOM.WW.Telemetry` package). The health endpoints
(`/healthz`, `/health/ready`, `/health/active`) are not affected by this section and
always stay anonymous: the family Overview dashboard polls `/health/active` from
another repo without credentials, so gating it would break that consumer.
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access and `Authentication:Mode=Disabled` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
Configure a Prometheus scraper against an authenticated endpoint with a dedicated
admin-scoped key, for example:
```yaml
scrape_configs:
- job_name: mxgateway
authorization:
type: Bearer
credentials_file: /etc/prometheus/mxgateway.key # contains mxgw_<keyId>_<secret>
static_configs:
- targets: ["gateway-host:5130"]
```
The setting is read from configuration when endpoints are mapped, so changing it
requires a restart. With dev-only `Dashboard:DisableLogin=true` every request is
auto-authenticated as an Administrator, which also satisfies `/metrics`.
## Galaxy Options
| Option | Default | Description |