fix(metrics-auth): DisableLogin no longer opens /metrics; log correlation pair; host-level tests

- The metrics scheme skips its dashboard-cookie fallback while
  Dashboard:DisableLogin is on, so the dev auto-login handler cannot
  satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
  loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
  non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
  vs gRPC per connection, so only the per-key aggregate is shared.
This commit is contained in:
Joseph Doherty
2026-10-22 06:47:36 -04:00
parent 65bca804bb
commit 82a856b61e
9 changed files with 276 additions and 74 deletions
+7 -4
View File
@@ -456,10 +456,13 @@ stays anonymous unconditionally.
values.
- **No environment bypass.** The requirement reads no options, so anonymous
localhost and `Authentication:Mode=Disabled` — both of which satisfy the
dashboard Viewer policy — do not satisfy it. Only dev-only
`Dashboard:DisableLogin` (which authenticates every request as an
Administrator, and is rejected in Production) lets an unauthenticated scrape
through.
dashboard Viewer policy — do not satisfy it. Nor does dev-only
`Dashboard:DisableLogin`: its auto-login handler sits under the cookie scheme
name, so the metrics scheme skips the cookie fallback while it is on rather than
letting a dev flag silently open an endpoint the operator asked to protect.
- **Throttle sharing is partial.** The `/metrics` per-peer limiter partition is
the remote IP, while the gRPC one is the gRPC peer string (per connection), so
only the per-key-id aggregate is shared between the two paths.
- **One scheme, plain 401.** Listing the cookie scheme on the policy directly
would challenge an anonymous scrape with a 302 redirect to `/login`; a single
scheme that falls back to the cookie internally keeps the challenge a `401`
+13 -3
View File
@@ -429,7 +429,7 @@ another repo without credentials, so gating it would break that consumer.
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access and `Authentication:Mode=Disabled` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access, `Authentication:Mode=Disabled`, and `Dashboard:DisableLogin` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
Configure a Prometheus scraper against an authenticated endpoint with a dedicated
admin-scoped key, for example:
@@ -445,8 +445,18 @@ scrape_configs:
```
The setting is read from configuration when endpoints are mapped, so changing it
requires a restart. With dev-only `Dashboard:DisableLogin=true` every request is
auto-authenticated as an Administrator, which also satisfies `/metrics`.
requires a restart. Dev-only `Dashboard:DisableLogin=true` registers an
always-succeeding auto-login handler under the dashboard cookie scheme; the
metrics scheme deliberately skips its cookie fallback while that flag is on, so
`DisableLogin` never silently opens a `/metrics` endpoint that
`RequireAuthentication` protects — only an admin-scoped API key is accepted then.
The `/metrics` path reuses the gRPC API-key failure limiter
(`MxGateway:Security:ApiKeyFailure*`), but its per-peer partition is keyed on the
remote **IP address**, whereas the gRPC interceptor's is the gRPC peer string
(address and port — effectively per connection). The two paths therefore share
only the per-key-id aggregate layer (`ApiKeyFailureAggregateLimit`); per-peer
failure counts on one path do not throttle the other.
## Galaxy Options
+1 -1
View File
@@ -82,7 +82,7 @@ return mapper.MapCommandReply(workerReply);
Carrying the enqueue timestamp into the worker layer is what lets queue-wait time be measured separately from worker-side execution time when troubleshooting timeouts.
Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies.
Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies. To keep the two ids joinable, `Invoke` logs the pair (command kind, session id, client correlation id, worker envelope correlation id — no values) once per call at `Debug`, so a client error report quoting its correlation id can be matched to gateway and worker log lines.
An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.
@@ -148,6 +148,19 @@ public sealed class MxAccessGatewayService(
// the same value the constraint-denied replies carry — so a client can match every
// reply to its request. The worker reply carries the gateway's internal envelope id,
// which is a gateway↔worker routing detail and is overwritten here, empty included.
// Record the pairing once, before the envelope id is overwritten, so a client error
// report quoting its correlation id can be matched to gateway and worker log lines
// (which carry the envelope id). Ids and the command kind only — never values.
if (logger.IsEnabled(LogLevel.Debug))
{
logger.LogDebug(
"Invoke {CommandKind} for session {SessionId}: client correlation id {ClientCorrelationId} maps to worker envelope correlation id {WorkerCorrelationId}.",
command.Kind,
request.SessionId,
request.ClientCorrelationId,
publicReply.CorrelationId);
}
publicReply.CorrelationId = request.ClientCorrelationId;
session.TrackCommandReply(commandToInvoke, publicReply);
@@ -4,6 +4,7 @@ using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authorization;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
using ZB.MOM.WW.MxGateway.Server.Configuration;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Metrics;
@@ -34,7 +35,9 @@ public static class MetricsAuthorizationDefaults
/// cookie scheme on the policy directly would turn an anonymous scrape into a 302 redirect to the
/// login page. A throttled key partition answers <c>429</c>, mirroring the gRPC
/// <c>ResourceExhausted</c>. Failure reasons are never echoed to the caller and the presented
/// secret is never logged.
/// secret is never logged. The limiter's per-peer partition here is the remote IP, whereas the gRPC
/// interceptor's is the gRPC peer string (address and port, i.e. per connection), so the two paths
/// share only the per-key-id aggregate layer.
/// </remarks>
public sealed class MetricsAuthenticationHandler(
IOptionsMonitor<AuthenticationSchemeOptions> options,
@@ -43,6 +46,7 @@ public sealed class MetricsAuthenticationHandler(
IApiKeyVerifier apiKeyVerifier,
ApiKeyFailureLimiter failureLimiter,
IAuthenticationSchemeProvider schemeProvider,
IOptions<GatewayOptions> gatewayOptions,
GatewayMetrics metrics)
: AuthenticationHandler<AuthenticationSchemeOptions>(options, logger, encoder)
{
@@ -111,6 +115,14 @@ public sealed class MetricsAuthenticationHandler(
// signed in to the dashboard can open /metrics in the same browser.
private async Task<AuthenticateResult> AuthenticateDashboardCookieAsync()
{
// Dashboard:DisableLogin registers an always-succeeding auto-login handler under the cookie
// scheme name. Falling back to it would let every anonymous request satisfy /metrics, so
// with DisableLogin on, RequireAuthentication=true accepts API keys only.
if (gatewayOptions.Value.Dashboard.DisableLogin)
{
return AuthenticateResult.NoResult();
}
if (await schemeProvider.GetSchemeAsync(DashboardAuthenticationDefaults.AuthenticationScheme).ConfigureAwait(false)
is null)
{
@@ -98,70 +98,6 @@ public sealed class GatewayApplicationTests
}
}
/// <summary>
/// With <c>MxGateway:Metrics:RequireAuthentication=true</c> an anonymous scrape is refused with a
/// plain 401 Bearer challenge (not a login redirect), while <c>/health/active</c> — polled by the
/// off-repo Overview dashboard without credentials — stays anonymous.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Build_WhenMetricsRequireAuthentication_RefusesAnonymousScrapeButNotHealth()
{
await using WebApplication app = GatewayApplication.Build(
["--urls=http://127.0.0.1:0", "--MxGateway:Metrics:RequireAuthentication=true"]);
await app.StartAsync();
try
{
using HttpClient client = new() { BaseAddress = new Uri(app.Urls.First()) };
using HttpResponseMessage metrics = await client.GetAsync("/metrics");
using HttpRequestMessage badKeyRequest = new(HttpMethod.Get, "/metrics");
badKeyRequest.Headers.TryAddWithoutValidation("Authorization", "Bearer mxgw_nobody_not-a-real-secret");
using HttpResponseMessage badKey = await client.SendAsync(badKeyRequest);
using HttpResponseMessage health = await client.GetAsync("/health/active");
Assert.Equal(HttpStatusCode.Unauthorized, metrics.StatusCode);
Assert.Contains(metrics.Headers.WwwAuthenticate, value => value.Scheme == "Bearer");
Assert.Equal(HttpStatusCode.Unauthorized, badKey.StatusCode);
Assert.NotEqual(HttpStatusCode.Unauthorized, health.StatusCode);
Assert.NotEqual(HttpStatusCode.Forbidden, health.StatusCode);
}
finally
{
await app.StopAsync();
}
}
/// <summary>
/// The dashboard-login path: with dev-only <c>Dashboard:DisableLogin</c> every request carries
/// the auto-login Administrator principal under the dashboard cookie scheme, which the metrics
/// scheme accepts when no authorization header is sent.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Build_WhenMetricsRequireAuthentication_AcceptsDashboardLogin()
{
await using WebApplication app = GatewayApplication.Build(
[
"--urls=http://127.0.0.1:0",
"--MxGateway:Metrics:RequireAuthentication=true",
"--MxGateway:Dashboard:DisableLogin=true",
]);
await app.StartAsync();
try
{
using HttpClient client = new() { BaseAddress = new Uri(app.Urls.First()) };
using HttpResponseMessage response = await client.GetAsync("/metrics");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
finally
{
await app.StopAsync();
}
}
/// <summary>Verifies that Build maps dashboard and authentication endpoints when the dashboard is enabled.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -505,6 +505,50 @@ public sealed class MxAccessGatewayServiceTests
Assert.Equal(string.Empty, reply.CorrelationId);
}
/// <summary>
/// Invoke logs the client correlation id alongside the worker envelope id once, at Debug, so
/// a client error report can be matched to gateway and worker logs.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Invoke_LogsClientToWorkerCorrelationPairAtDebug()
{
FakeSessionManager sessionManager = new()
{
InvokeReply = new WorkerCommandReply
{
Reply = new MxCommandReply
{
SessionId = "session-1",
CorrelationId = "envelope-7f3a",
Kind = MxCommandKind.AddItem,
ProtocolStatus = MxAccessGrpcMapper.Ok(),
AddItem = new AddItemReply { ItemHandle = 42 },
},
},
};
RecordingLogger<MxAccessGatewayService> logger = new();
MxAccessGatewayService service = CreateService(sessionManager, logger: logger);
MxCommandRequest request = new()
{
SessionId = "session-1",
ClientCorrelationId = "client-trace-9",
Command = new MxCommand
{
Kind = MxCommandKind.AddItem,
AddItem = new AddItemCommand { ServerHandle = 12, ItemDefinition = "Galaxy.Tag.Value" },
},
};
await service.Invoke(request, new TestServerCallContext());
(LogLevel level, string message, _) = Assert.Single(
logger.Entries,
entry => entry.Message.Contains("envelope-7f3a", StringComparison.Ordinal));
Assert.Equal(LogLevel.Debug, level);
Assert.Contains("client-trace-9", message, StringComparison.Ordinal);
}
/// <summary>
/// StreamEvents hands the call's cancellation token to every response write, so a write
/// blocked on a slow or vanished client is released when the call is cancelled.
@@ -0,0 +1,182 @@
using System.Net;
using System.Net.Http;
using System.Security.Claims;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authentication.Cookies;
using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.ApiKeys.Admin;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Security.Authorization;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
/// <summary>
/// Host-level coverage of <c>MxGateway:Metrics:RequireAuthentication=true</c> (SEC-14): each case
/// builds the real gateway with <see cref="GatewayApplication.Build"/> and drives the full
/// middleware pipeline (authentication, authorization, the dashboard cookie scheme, the API-key
/// store) over HTTP on an ephemeral loopback port. Requests therefore arrive from loopback, which
/// is exactly the anonymous-localhost case the dashboard Viewer policy would admit and /metrics
/// must not.
/// </summary>
public sealed class MetricsEndpointAuthenticationTests
{
// A fixed test pepper: the configuration-backed pepper provider reads MxGateway:ApiKeyPepper,
// and the auth store itself is isolated per test process by TestHostEnvironmentInitializer.
private const string TestPepper = "metrics-endpoint-tests-pepper-0123456789abcdef";
/// <summary>
/// An anonymous loopback scrape gets a plain 401 Bearer challenge — not a 302 to /login, and not
/// the anonymous-localhost admission the dashboard Viewer policy grants.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AnonymousLoopbackScrape_Returns401BearerNotLoginRedirect()
{
await using WebApplication app = await StartAsync();
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await client.GetAsync("/metrics");
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
Assert.Null(response.Headers.Location);
Assert.Contains(response.Headers.WwwAuthenticate, value => value.Scheme == "Bearer");
}
/// <summary>/health/active stays anonymous: the off-repo Overview dashboard polls it without credentials.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task HealthActive_StaysAnonymous()
{
await using WebApplication app = await StartAsync();
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await client.GetAsync("/health/active");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
/// <summary>A dashboard login cookie holding the Viewer role is accepted.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DashboardViewerCookie_Returns200()
{
await using WebApplication app = await StartAsync();
using HttpClient client = CreateClient(app);
using HttpRequestMessage request = new(HttpMethod.Get, "/metrics");
request.Headers.Add("Cookie", MintDashboardCookie(app, DashboardRoles.Viewer));
using HttpResponseMessage response = await client.SendAsync(request);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
/// <summary>A real API key with the admin scope is accepted.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AdminScopedApiKey_Returns200()
{
await using WebApplication app = await StartAsync();
string token = await CreateKeyAsync(app, [GatewayScopes.Admin]);
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await SendWithBearerAsync(client, token);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
/// <summary>A real API key without the admin scope authenticates but is forbidden.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task NonAdminApiKey_Returns403()
{
await using WebApplication app = await StartAsync();
string token = await CreateKeyAsync(app, [GatewayScopes.EventsRead, GatewayScopes.InvokeRead]);
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await SendWithBearerAsync(client, token);
Assert.Equal(HttpStatusCode.Forbidden, response.StatusCode);
}
/// <summary>
/// Dev-only <c>Dashboard:DisableLogin</c> registers an always-succeeding auto-login handler under
/// the cookie scheme name; /metrics must not fall back to it, or the flag would silently open
/// an endpoint the operator explicitly asked to protect.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DisableLogin_DoesNotOpenMetrics()
{
await using WebApplication app = await StartAsync("--MxGateway:Dashboard:DisableLogin=true");
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await client.GetAsync("/metrics");
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
}
private static async Task<WebApplication> StartAsync(params string[] extraArgs)
{
// Ephemeral port (:0): xUnit runs collections in parallel, so a fixed port would collide.
WebApplication app = GatewayApplication.Build(
[
"--urls=http://127.0.0.1:0",
"--MxGateway:Metrics:RequireAuthentication=true",
$"--MxGateway:ApiKeyPepper={TestPepper}",
.. extraArgs,
]);
await app.StartAsync();
return app;
}
private static HttpClient CreateClient(WebApplication app) =>
new(new HttpClientHandler { AllowAutoRedirect = false, UseCookies = false })
{
BaseAddress = new Uri(app.Urls.First()),
};
private static async Task<HttpResponseMessage> SendWithBearerAsync(HttpClient client, string token)
{
using HttpRequestMessage request = new(HttpMethod.Get, "/metrics");
request.Headers.TryAddWithoutValidation("Authorization", $"Bearer {token}");
return await client.SendAsync(request);
}
// Creates a key through the same admin command set the CLI and dashboard use. Key ids are unique
// per call because the isolated auth store is shared by every test in the process.
private static async Task<string> CreateKeyAsync(WebApplication app, string[] scopes)
{
ApiKeyAdminCommands commands = app.Services.GetRequiredService<ApiKeyAdminCommands>();
await commands.InitDbAsync(remoteAddress: null, CancellationToken.None);
CreateKeyResult created = await commands.CreateKeyAsync(
$"metrics{Guid.NewGuid():N}",
"Metrics test key",
new HashSet<string>(scopes, StringComparer.Ordinal),
null,
null,
remoteAddress: null,
CancellationToken.None);
return created.Token ?? throw new InvalidOperationException("Key creation returned no token.");
}
// Protects a ticket with the running host's own cookie options (data-protection keys, ticket
// format, cookie name) — the cookie a real dashboard login would issue, minus the LDAP bind.
private static string MintDashboardCookie(WebApplication app, string role)
{
CookieAuthenticationOptions cookieOptions = app.Services
.GetRequiredService<IOptionsMonitor<CookieAuthenticationOptions>>()
.Get(DashboardAuthenticationDefaults.AuthenticationScheme);
ClaimsIdentity identity = new(
[new Claim(ClaimTypes.Name, "viewer-user"), new Claim(ClaimTypes.Role, role)],
DashboardAuthenticationDefaults.AuthenticationScheme,
ClaimTypes.Name,
ClaimTypes.Role);
AuthenticationTicket ticket = new(
new ClaimsPrincipal(identity),
DashboardAuthenticationDefaults.AuthenticationScheme);
return $"{cookieOptions.Cookie.Name}={cookieOptions.TicketDataFormat.Protect(ticket)}";
}
}
@@ -6,6 +6,7 @@ using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
using ZB.MOM.WW.MxGateway.Server.Configuration;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Metrics;
using ZB.MOM.WW.MxGateway.Server.Security.Authorization;
@@ -190,7 +191,8 @@ public sealed class MetricsAuthorizationTests
UrlEncoder.Default,
verifier,
limiter ?? new ApiKeyFailureLimiter(1000, TimeSpan.FromMinutes(1), 1024, 0, TimeSpan.Zero, TimeProvider.System),
new AuthenticationSchemeProvider(Options.Create(new AuthenticationOptions())),
new AuthenticationSchemeProvider(Options.Create(new Microsoft.AspNetCore.Authentication.AuthenticationOptions())),
Options.Create(new GatewayOptions()),
new GatewayMetrics());
await handler.InitializeAsync(
new AuthenticationScheme(