de86f8b80e
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only): - Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive. New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of 2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same second, both bootstraps got in, and the nightly's checkout died on .git/index.lock. Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder removes a leftover .git\index.lock, which can only be debris from a killed git. - windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same Worker build + tests first, so the scheduled copy only doubled the load and created the race. - Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23 until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc, csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical memory, disk and testhost count at start and on failure so the next one is correlatable. - SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50). - Dashboard host build (#143, run 817): two host-building test classes failed at the same instant in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel classes migrating the one per-process secrets store. Tests now build the gateway through TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it. - Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6): 7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and burst-coalescing failures (#126/#127/#139) last failed before50322bac/7da52b65(Aug 2026). - 2026-10-15..20 was the NU1903 package-audit restore failure, fixed infb20997. Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
184 lines
8.2 KiB
C#
184 lines
8.2 KiB
C#
using System.Net;
|
|
using System.Net.Http;
|
|
using System.Security.Claims;
|
|
using Microsoft.AspNetCore.Authentication;
|
|
using Microsoft.AspNetCore.Authentication.Cookies;
|
|
using Microsoft.AspNetCore.Builder;
|
|
using Microsoft.Extensions.DependencyInjection;
|
|
using Microsoft.Extensions.Options;
|
|
using ZB.MOM.WW.Auth.ApiKeys.Admin;
|
|
using ZB.MOM.WW.MxGateway.Server;
|
|
using ZB.MOM.WW.MxGateway.Server.Dashboard;
|
|
using ZB.MOM.WW.MxGateway.Server.Security.Authorization;
|
|
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
|
|
|
|
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
|
|
|
|
/// <summary>
|
|
/// Host-level coverage of <c>MxGateway:Metrics:RequireAuthentication=true</c> (SEC-14): each case
|
|
/// builds the real gateway with <see cref="GatewayApplication.Build"/> and drives the full
|
|
/// middleware pipeline (authentication, authorization, the dashboard cookie scheme, the API-key
|
|
/// store) over HTTP on an ephemeral loopback port. Requests therefore arrive from loopback, which
|
|
/// is exactly the anonymous-localhost case the dashboard Viewer policy would admit and /metrics
|
|
/// must not.
|
|
/// </summary>
|
|
public sealed class MetricsEndpointAuthenticationTests
|
|
{
|
|
// A fixed test pepper: the configuration-backed pepper provider reads MxGateway:ApiKeyPepper,
|
|
// and the auth store itself is isolated per test process by TestHostEnvironmentInitializer.
|
|
private const string TestPepper = "metrics-endpoint-tests-pepper-0123456789abcdef";
|
|
|
|
/// <summary>
|
|
/// An anonymous loopback scrape gets a plain 401 Bearer challenge — not a 302 to /login, and not
|
|
/// the anonymous-localhost admission the dashboard Viewer policy grants.
|
|
/// </summary>
|
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
|
[Fact]
|
|
public async Task AnonymousLoopbackScrape_Returns401BearerNotLoginRedirect()
|
|
{
|
|
await using WebApplication app = await StartAsync();
|
|
using HttpClient client = CreateClient(app);
|
|
|
|
using HttpResponseMessage response = await client.GetAsync("/metrics");
|
|
|
|
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
|
Assert.Null(response.Headers.Location);
|
|
Assert.Contains(response.Headers.WwwAuthenticate, value => value.Scheme == "Bearer");
|
|
}
|
|
|
|
/// <summary>/health/active stays anonymous: the off-repo Overview dashboard polls it without credentials.</summary>
|
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
|
[Fact]
|
|
public async Task HealthActive_StaysAnonymous()
|
|
{
|
|
await using WebApplication app = await StartAsync();
|
|
using HttpClient client = CreateClient(app);
|
|
|
|
using HttpResponseMessage response = await client.GetAsync("/health/active");
|
|
|
|
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
|
}
|
|
|
|
/// <summary>A dashboard login cookie holding the Viewer role is accepted.</summary>
|
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
|
[Fact]
|
|
public async Task DashboardViewerCookie_Returns200()
|
|
{
|
|
await using WebApplication app = await StartAsync();
|
|
using HttpClient client = CreateClient(app);
|
|
using HttpRequestMessage request = new(HttpMethod.Get, "/metrics");
|
|
request.Headers.Add("Cookie", MintDashboardCookie(app, DashboardRoles.Viewer));
|
|
|
|
using HttpResponseMessage response = await client.SendAsync(request);
|
|
|
|
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
|
}
|
|
|
|
/// <summary>A real API key with the admin scope is accepted.</summary>
|
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
|
[Fact]
|
|
public async Task AdminScopedApiKey_Returns200()
|
|
{
|
|
await using WebApplication app = await StartAsync();
|
|
string token = await CreateKeyAsync(app, [GatewayScopes.Admin]);
|
|
using HttpClient client = CreateClient(app);
|
|
|
|
using HttpResponseMessage response = await SendWithBearerAsync(client, token);
|
|
|
|
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
|
}
|
|
|
|
/// <summary>A real API key without the admin scope authenticates but is forbidden.</summary>
|
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
|
[Fact]
|
|
public async Task NonAdminApiKey_Returns403()
|
|
{
|
|
await using WebApplication app = await StartAsync();
|
|
string token = await CreateKeyAsync(app, [GatewayScopes.EventsRead, GatewayScopes.InvokeRead]);
|
|
using HttpClient client = CreateClient(app);
|
|
|
|
using HttpResponseMessage response = await SendWithBearerAsync(client, token);
|
|
|
|
Assert.Equal(HttpStatusCode.Forbidden, response.StatusCode);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Dev-only <c>Dashboard:DisableLogin</c> registers an always-succeeding auto-login handler under
|
|
/// the cookie scheme name; /metrics must not fall back to it, or the flag would silently open
|
|
/// an endpoint the operator explicitly asked to protect.
|
|
/// </summary>
|
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
|
[Fact]
|
|
public async Task DisableLogin_DoesNotOpenMetrics()
|
|
{
|
|
await using WebApplication app = await StartAsync("--MxGateway:Dashboard:DisableLogin=true");
|
|
using HttpClient client = CreateClient(app);
|
|
|
|
using HttpResponseMessage response = await client.GetAsync("/metrics");
|
|
|
|
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
|
}
|
|
|
|
private static async Task<WebApplication> StartAsync(params string[] extraArgs)
|
|
{
|
|
// Ephemeral port (:0): xUnit runs collections in parallel, so a fixed port would collide.
|
|
WebApplication app = TestGatewayHost.Build(
|
|
[
|
|
"--urls=http://127.0.0.1:0",
|
|
"--MxGateway:Metrics:RequireAuthentication=true",
|
|
$"--MxGateway:ApiKeyPepper={TestPepper}",
|
|
.. extraArgs,
|
|
]);
|
|
await app.StartAsync();
|
|
return app;
|
|
}
|
|
|
|
private static HttpClient CreateClient(WebApplication app) =>
|
|
new(new HttpClientHandler { AllowAutoRedirect = false, UseCookies = false })
|
|
{
|
|
BaseAddress = new Uri(app.Urls.First()),
|
|
};
|
|
|
|
private static async Task<HttpResponseMessage> SendWithBearerAsync(HttpClient client, string token)
|
|
{
|
|
using HttpRequestMessage request = new(HttpMethod.Get, "/metrics");
|
|
request.Headers.TryAddWithoutValidation("Authorization", $"Bearer {token}");
|
|
return await client.SendAsync(request);
|
|
}
|
|
|
|
// Creates a key through the same admin command set the CLI and dashboard use. Key ids are unique
|
|
// per call because the isolated auth store is shared by every test in the process.
|
|
private static async Task<string> CreateKeyAsync(WebApplication app, string[] scopes)
|
|
{
|
|
ApiKeyAdminCommands commands = app.Services.GetRequiredService<ApiKeyAdminCommands>();
|
|
await commands.InitDbAsync(remoteAddress: null, CancellationToken.None);
|
|
CreateKeyResult created = await commands.CreateKeyAsync(
|
|
$"metrics{Guid.NewGuid():N}",
|
|
"Metrics test key",
|
|
new HashSet<string>(scopes, StringComparer.Ordinal),
|
|
null,
|
|
null,
|
|
remoteAddress: null,
|
|
CancellationToken.None);
|
|
return created.Token ?? throw new InvalidOperationException("Key creation returned no token.");
|
|
}
|
|
|
|
// Protects a ticket with the running host's own cookie options (data-protection keys, ticket
|
|
// format, cookie name) — the cookie a real dashboard login would issue, minus the LDAP bind.
|
|
private static string MintDashboardCookie(WebApplication app, string role)
|
|
{
|
|
CookieAuthenticationOptions cookieOptions = app.Services
|
|
.GetRequiredService<IOptionsMonitor<CookieAuthenticationOptions>>()
|
|
.Get(DashboardAuthenticationDefaults.AuthenticationScheme);
|
|
ClaimsIdentity identity = new(
|
|
[new Claim(ClaimTypes.Name, "viewer-user"), new Claim(ClaimTypes.Role, role)],
|
|
DashboardAuthenticationDefaults.AuthenticationScheme,
|
|
ClaimTypes.Name,
|
|
ClaimTypes.Role);
|
|
AuthenticationTicket ticket = new(
|
|
new ClaimsPrincipal(identity),
|
|
DashboardAuthenticationDefaults.AuthenticationScheme);
|
|
return $"{cookieOptions.Cookie.Name}={cookieOptions.TicketDataFormat.Protect(ticket)}";
|
|
}
|
|
}
|