Files
mxaccessgw/src/ZB.MOM.WW.MxGateway.Tests/Gateway/MetricsEndpointAuthenticationTests.cs
T
Joseph Doherty de86f8b80e
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m34s
ci / java (push) Successful in 2m37s
ci / portable (push) Failing after 5m0s
ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):

- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
  New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
  succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
  2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
  second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
  Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
  handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
  harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
  removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
  Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
  Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
  until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
  csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
  window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
  memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
  in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
  classes migrating the one per-process secrets store. Tests now build the gateway through
  TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
  the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
  7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
  burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.

Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:36:34 -05:00

184 lines
8.2 KiB
C#

using System.Net;
using System.Net.Http;
using System.Security.Claims;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authentication.Cookies;
using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.ApiKeys.Admin;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Security.Authorization;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
/// <summary>
/// Host-level coverage of <c>MxGateway:Metrics:RequireAuthentication=true</c> (SEC-14): each case
/// builds the real gateway with <see cref="GatewayApplication.Build"/> and drives the full
/// middleware pipeline (authentication, authorization, the dashboard cookie scheme, the API-key
/// store) over HTTP on an ephemeral loopback port. Requests therefore arrive from loopback, which
/// is exactly the anonymous-localhost case the dashboard Viewer policy would admit and /metrics
/// must not.
/// </summary>
public sealed class MetricsEndpointAuthenticationTests
{
// A fixed test pepper: the configuration-backed pepper provider reads MxGateway:ApiKeyPepper,
// and the auth store itself is isolated per test process by TestHostEnvironmentInitializer.
private const string TestPepper = "metrics-endpoint-tests-pepper-0123456789abcdef";
/// <summary>
/// An anonymous loopback scrape gets a plain 401 Bearer challenge — not a 302 to /login, and not
/// the anonymous-localhost admission the dashboard Viewer policy grants.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AnonymousLoopbackScrape_Returns401BearerNotLoginRedirect()
{
await using WebApplication app = await StartAsync();
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await client.GetAsync("/metrics");
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
Assert.Null(response.Headers.Location);
Assert.Contains(response.Headers.WwwAuthenticate, value => value.Scheme == "Bearer");
}
/// <summary>/health/active stays anonymous: the off-repo Overview dashboard polls it without credentials.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task HealthActive_StaysAnonymous()
{
await using WebApplication app = await StartAsync();
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await client.GetAsync("/health/active");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
/// <summary>A dashboard login cookie holding the Viewer role is accepted.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DashboardViewerCookie_Returns200()
{
await using WebApplication app = await StartAsync();
using HttpClient client = CreateClient(app);
using HttpRequestMessage request = new(HttpMethod.Get, "/metrics");
request.Headers.Add("Cookie", MintDashboardCookie(app, DashboardRoles.Viewer));
using HttpResponseMessage response = await client.SendAsync(request);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
/// <summary>A real API key with the admin scope is accepted.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AdminScopedApiKey_Returns200()
{
await using WebApplication app = await StartAsync();
string token = await CreateKeyAsync(app, [GatewayScopes.Admin]);
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await SendWithBearerAsync(client, token);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
/// <summary>A real API key without the admin scope authenticates but is forbidden.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task NonAdminApiKey_Returns403()
{
await using WebApplication app = await StartAsync();
string token = await CreateKeyAsync(app, [GatewayScopes.EventsRead, GatewayScopes.InvokeRead]);
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await SendWithBearerAsync(client, token);
Assert.Equal(HttpStatusCode.Forbidden, response.StatusCode);
}
/// <summary>
/// Dev-only <c>Dashboard:DisableLogin</c> registers an always-succeeding auto-login handler under
/// the cookie scheme name; /metrics must not fall back to it, or the flag would silently open
/// an endpoint the operator explicitly asked to protect.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DisableLogin_DoesNotOpenMetrics()
{
await using WebApplication app = await StartAsync("--MxGateway:Dashboard:DisableLogin=true");
using HttpClient client = CreateClient(app);
using HttpResponseMessage response = await client.GetAsync("/metrics");
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
}
private static async Task<WebApplication> StartAsync(params string[] extraArgs)
{
// Ephemeral port (:0): xUnit runs collections in parallel, so a fixed port would collide.
WebApplication app = TestGatewayHost.Build(
[
"--urls=http://127.0.0.1:0",
"--MxGateway:Metrics:RequireAuthentication=true",
$"--MxGateway:ApiKeyPepper={TestPepper}",
.. extraArgs,
]);
await app.StartAsync();
return app;
}
private static HttpClient CreateClient(WebApplication app) =>
new(new HttpClientHandler { AllowAutoRedirect = false, UseCookies = false })
{
BaseAddress = new Uri(app.Urls.First()),
};
private static async Task<HttpResponseMessage> SendWithBearerAsync(HttpClient client, string token)
{
using HttpRequestMessage request = new(HttpMethod.Get, "/metrics");
request.Headers.TryAddWithoutValidation("Authorization", $"Bearer {token}");
return await client.SendAsync(request);
}
// Creates a key through the same admin command set the CLI and dashboard use. Key ids are unique
// per call because the isolated auth store is shared by every test in the process.
private static async Task<string> CreateKeyAsync(WebApplication app, string[] scopes)
{
ApiKeyAdminCommands commands = app.Services.GetRequiredService<ApiKeyAdminCommands>();
await commands.InitDbAsync(remoteAddress: null, CancellationToken.None);
CreateKeyResult created = await commands.CreateKeyAsync(
$"metrics{Guid.NewGuid():N}",
"Metrics test key",
new HashSet<string>(scopes, StringComparer.Ordinal),
null,
null,
remoteAddress: null,
CancellationToken.None);
return created.Token ?? throw new InvalidOperationException("Key creation returned no token.");
}
// Protects a ticket with the running host's own cookie options (data-protection keys, ticket
// format, cookie name) — the cookie a real dashboard login would issue, minus the LDAP bind.
private static string MintDashboardCookie(WebApplication app, string role)
{
CookieAuthenticationOptions cookieOptions = app.Services
.GetRequiredService<IOptionsMonitor<CookieAuthenticationOptions>>()
.Get(DashboardAuthenticationDefaults.AuthenticationScheme);
ClaimsIdentity identity = new(
[new Claim(ClaimTypes.Name, "viewer-user"), new Claim(ClaimTypes.Role, role)],
DashboardAuthenticationDefaults.AuthenticationScheme,
ClaimTypes.Name,
ClaimTypes.Role);
AuthenticationTicket ticket = new(
new ClaimsPrincipal(identity),
DashboardAuthenticationDefaults.AuthenticationScheme);
return $"{cookieOptions.Cookie.Name}={cookieOptions.TicketDataFormat.Protect(ticket)}";
}
}