fix(metrics-auth): DisableLogin no longer opens /metrics; log correlation pair; host-level tests

- The metrics scheme skips its dashboard-cookie fallback while
  Dashboard:DisableLogin is on, so the dev auto-login handler cannot
  satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
  loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
  non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
  vs gRPC per connection, so only the per-key aggregate is shared.
This commit is contained in:
Joseph Doherty
2026-10-22 06:47:36 -04:00
parent 65bca804bb
commit 82a856b61e
9 changed files with 276 additions and 74 deletions
+7 -4
View File
@@ -456,10 +456,13 @@ stays anonymous unconditionally.
values.
- **No environment bypass.** The requirement reads no options, so anonymous
localhost and `Authentication:Mode=Disabled` — both of which satisfy the
dashboard Viewer policy — do not satisfy it. Only dev-only
`Dashboard:DisableLogin` (which authenticates every request as an
Administrator, and is rejected in Production) lets an unauthenticated scrape
through.
dashboard Viewer policy — do not satisfy it. Nor does dev-only
`Dashboard:DisableLogin`: its auto-login handler sits under the cookie scheme
name, so the metrics scheme skips the cookie fallback while it is on rather than
letting a dev flag silently open an endpoint the operator asked to protect.
- **Throttle sharing is partial.** The `/metrics` per-peer limiter partition is
the remote IP, while the gRPC one is the gRPC peer string (per connection), so
only the per-key-id aggregate is shared between the two paths.
- **One scheme, plain 401.** Listing the cookie scheme on the policy directly
would challenge an anonymous scrape with a 302 redirect to `/login`; a single
scheme that falls back to the cookie internally keeps the challenge a `401`
+13 -3
View File
@@ -429,7 +429,7 @@ another repo without credentials, so gating it would break that consumer.
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access and `Authentication:Mode=Disabled` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access, `Authentication:Mode=Disabled`, and `Dashboard:DisableLogin` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
Configure a Prometheus scraper against an authenticated endpoint with a dedicated
admin-scoped key, for example:
@@ -445,8 +445,18 @@ scrape_configs:
```
The setting is read from configuration when endpoints are mapped, so changing it
requires a restart. With dev-only `Dashboard:DisableLogin=true` every request is
auto-authenticated as an Administrator, which also satisfies `/metrics`.
requires a restart. Dev-only `Dashboard:DisableLogin=true` registers an
always-succeeding auto-login handler under the dashboard cookie scheme; the
metrics scheme deliberately skips its cookie fallback while that flag is on, so
`DisableLogin` never silently opens a `/metrics` endpoint that
`RequireAuthentication` protects — only an admin-scoped API key is accepted then.
The `/metrics` path reuses the gRPC API-key failure limiter
(`MxGateway:Security:ApiKeyFailure*`), but its per-peer partition is keyed on the
remote **IP address**, whereas the gRPC interceptor's is the gRPC peer string
(address and port — effectively per connection). The two paths therefore share
only the per-key-id aggregate layer (`ApiKeyFailureAggregateLimit`); per-peer
failure counts on one path do not throttle the other.
## Galaxy Options
+1 -1
View File
@@ -82,7 +82,7 @@ return mapper.MapCommandReply(workerReply);
Carrying the enqueue timestamp into the worker layer is what lets queue-wait time be measured separately from worker-side execution time when troubleshooting timeouts.
Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies.
Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies. To keep the two ids joinable, `Invoke` logs the pair (command kind, session id, client correlation id, worker envelope correlation id — no values) once per call at `Debug`, so a client error report quoting its correlation id can be matched to gateway and worker log lines.
An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.