diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 7c8ed206..bb1e7826 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -165,6 +165,12 @@ jobs: # actions), and a `runs-on` gate with no matching runner wedges the queue in "queued" forever. # An unreachable windev fails this job RED (loud), never leaves it stuck. A red job can mean the # Windows tier is down rather than the change — see docs/GatewayTesting.md § Continuous Integration. + # + # Skipped on the schedule: nightly-windev's `live` mode already runs this exact build + test as + # its first two tiers, and running both made every scheduled run start two windev sessions in + # the same second, contending for the one CI clone (the 2026-09-22 index.lock failure, run 827, + # mxaccessgw#150) and doubling the nightly's load on a host that was already short of memory. + if: github.event_name != 'schedule' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 diff --git a/docs/GatewayTesting.md b/docs/GatewayTesting.md index 5208bea0..9e12d9a2 100644 --- a/docs/GatewayTesting.md +++ b/docs/GatewayTesting.md @@ -639,7 +639,8 @@ runtime because the x86 Worker cannot build on Linux: client (IPC-24 deleted the old unconditional churn-revert step, which masked exactly that class). The dev Mac has a Homebrew JDK 17, so `gradle generateProto` can be run there to refresh the Java aggregates when a `.proto` changes. -- **`windows-x86`** (Linux runner, per push/PR) — builds the **x86 / net48 Worker and +- **`windows-x86`** (Linux runner, per push/PR; skipped on the schedule, where + `nightly-windev` runs the same tiers) — builds the **x86 / net48 Worker and Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where `scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci` @@ -658,6 +659,25 @@ runtime because the x86 Worker cannot build on Linux: Gated `if: github.event_name == 'schedule'`, so it never gates a push. On failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page. +The windev host lock is an exclusive file handle on `C:\build\mxaccessgw-ci.ci-lock`, held by +the SSH bootstrap for the whole run. It replaced a `New-Item -ItemType Directory` lock that two +runs starting in the same second could both take (PowerShell checks for the directory, then +creates it), which is how the 2026-09-22 nightly collided with the scheduled `windows-x86` run on +`.git/index.lock` (mxaccessgw#150). The OS drops the handle when its holder dies, so a killed run +leaves no stale lock, and a holder may delete a leftover `index.lock` because no other CI git can +be running. Each run also writes a `host (start)` line — and a `host (failure)` line when a tier +fails — to `C:\build\mxaccessgw-ci\ci.log` with free commit, physical memory, disk and the +number of `testhost` processes, so a host-resource failure can be matched to the host's state. + +When a nightly goes red, sort the cause before touching code: a test that fails with an +assertion or test exception is a test or product fault; `OutOfMemoryException` from +MSBuild/csc/protoc, `csc.exe exited with code -1073741502` (0xC0000142, a process that could not +initialize), a remote exit code with no `ci.log` lines at all, or an SSH `kex_exchange_identification` +reset is the host, as is `No route to host`. The 2026-07-24 to 2026-08-03 red streak was the +second kind: windev's System log carries a `Resource-Exhaustion-Detector` event 2004 (low virtual +memory) every few minutes from 2026-07-23 until the box hard-reset on 2026-08-03, with no single +process holding most of the commit. + ### Runner capacity is shared and finite CI runs on two co-located runner containers on docker host `10.100.0.35` — `gitea-runner` diff --git a/scripts/ci/README.md b/scripts/ci/README.md index e7526f9e..df15e932 100644 --- a/scripts/ci/README.md +++ b/scripts/ci/README.md @@ -18,11 +18,18 @@ Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests `live` = test + full-`slnx` build + the net10 gateway suite + live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The `live` order is x86 Worker build, `Worker.Tests`, full solution build, gateway tests, then live smoke. One host lock covers -checkout and every tier, so concurrent runs cannot overlap; each native command is checked +checkout and every tier, so concurrent runs cannot overlap. The lock is an exclusive +(`FileShare.None`) handle on `C:\build\mxaccessgw-ci.ci-lock` held by the SSH bootstrap; the OS +releases it if the run is killed, so it never goes stale. (It replaced a `New-Item -ItemType +Directory` lock that two runs starting in the same second could both take — mxaccessgw#150.) +Runs of SHAs older than that change still use the old directory lock and do not exclude new-style +runs; that window closes as branches rebase. each native command is checked immediately and the first nonzero exit aborts the remaining tiers and propagates to CI. -CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR; `nightly-windev` -runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on failure. +CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR (not on the schedule, +where it would duplicate the nightly's first two tiers and start a second windev session in the +same second); `nightly-windev` runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on +failure. ## One-time bring-up (operator — required before this lands on `main`) diff --git a/scripts/ci/run-windev-ci.sh b/scripts/ci/run-windev-ci.sh index 08b10724..9faddfba 100755 --- a/scripts/ci/run-windev-ci.sh +++ b/scripts/ci/run-windev-ci.sh @@ -84,19 +84,33 @@ echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_S # # The bootstrap fetch/checkout mutates the shared clone's git index, so it MUST hold the # worktree lock — otherwise two concurrent runs collide on .git/index.lock before either -# reaches windev-worker-ci.ps1's lock (the exact race the lock exists to prevent). We take the -# same mkdir lock here (atomic; retry to a 20-min deadline) and export MXGW_CI_LOCK_HELD so the -# handed-off script re-uses this lock instead of re-acquiring/releasing it. A standalone/manual -# run of windev-worker-ci.ps1 (degraded mode) has no such env and self-locks as before. +# reaches windev-worker-ci.ps1 (the exact race the lock exists to prevent). We export +# MXGW_CI_LOCK_HELD so the handed-off script re-uses this lock instead of re-acquiring it; a +# standalone/manual run of windev-worker-ci.ps1 (degraded mode) has no such env and takes the +# same lock itself. +# +# The lock is an exclusive (FileShare.None) handle on '.ci-lock', held by this +# bootstrap process for the whole run (the handle is not inherited by the child powershell). +# It replaced a `New-Item -ItemType Directory` lock that was NOT mutually exclusive: +# PowerShell's directory provider checks for existence and then calls CreateDirectory (which +# succeeds on an existing directory), so two bootstraps starting together both "acquired" it. +# A scheduled run starts windows-x86 and nightly-windev in the same second, and on 2026-09-22 +# (run 827) both got in, ran git concurrently and the nightly died on +# ".git/index.lock: File exists" (mxaccessgw#148/#150). A file handle is also released by the +# OS if the holder is killed (ssh drop, host reboot), so a dead run can never leave a stale +# lock behind; the old directory had to be removed by hand. +# +# Holding the lock also proves no other CI git command can be running in the clone, so a +# leftover .git/index.lock can only be debris from a killed git and is removed. read -r -d '' BOOTSTRAP <>().Value; diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SecretsNavGateTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SecretsNavGateTests.cs index 4bb3cc16..60a9131c 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SecretsNavGateTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SecretsNavGateTests.cs @@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection; using ZB.MOM.WW.Auth.AspNetCore; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Dashboard; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; using ZB.MOM.WW.Secrets.Ui; namespace ZB.MOM.WW.MxGateway.Tests.Dashboard; @@ -35,7 +36,7 @@ public sealed class SecretsNavGateTests [Fact] public async Task ManagePolicy_AdmitsAdministrator() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthorizationService authorization = app.Services.GetRequiredService(); AuthorizationResult result = await authorization.AuthorizeAsync( @@ -56,7 +57,7 @@ public sealed class SecretsNavGateTests [Fact] public async Task ManagePolicy_RefusesViewer() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthorizationService authorization = app.Services.GetRequiredService(); AuthorizationResult result = await authorization.AuthorizeAsync( @@ -75,7 +76,7 @@ public sealed class SecretsNavGateTests [Fact] public async Task ManagePolicy_RefusesUnauthenticated() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthorizationService authorization = app.Services.GetRequiredService(); AuthorizationResult result = await authorization.AuthorizeAsync( diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/AlarmFeedAuthorizationTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/AlarmFeedAuthorizationTests.cs index ba571cc9..b4fac2b7 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/AlarmFeedAuthorizationTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/AlarmFeedAuthorizationTests.cs @@ -11,6 +11,7 @@ using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Configuration; using ZB.MOM.WW.MxGateway.Server.Dashboard; using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; using AuthenticationOptions = ZB.MOM.WW.MxGateway.Server.Configuration.AuthenticationOptions; namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard; @@ -115,7 +116,7 @@ public sealed class AlarmFeedAuthorizationTests [Fact] public async Task AlarmsPolicy_IsRegisteredWithNonBypassableRequirement() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthorizationPolicyProvider provider = app.Services .GetRequiredService(); @@ -147,7 +148,7 @@ public sealed class AlarmFeedAuthorizationTests Assert.Equal(DashboardAuthenticationDefaults.AlarmsPolicy, hubAttribute?.Policy); - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); RouteEndpoint alarmsPage = ((IEndpointRouteBuilder)app).DataSources .SelectMany(source => source.Endpoints) .OfType() diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardCookieOptionsTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardCookieOptionsTests.cs index bbfeb6be..af2fa9ff 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardCookieOptionsTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardCookieOptionsTests.cs @@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Options; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Dashboard; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard; @@ -20,7 +21,7 @@ public sealed class DashboardCookieOptionsTests [Fact] public async Task Build_ConfiguresSecureDashboardCookie() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IOptionsMonitor optionsMonitor = app.Services .GetRequiredService>(); @@ -49,7 +50,7 @@ public sealed class DashboardCookieOptionsTests [Fact] public async Task Build_WithRequireHttpsCookieFalse_UsesSameAsRequestAndPlainName() { - await using WebApplication app = GatewayApplication.Build( + await using WebApplication app = TestGatewayHost.Build( ["--MxGateway:Dashboard:RequireHttpsCookie=false"]); IOptionsMonitor optionsMonitor = app.Services .GetRequiredService>(); @@ -71,7 +72,7 @@ public sealed class DashboardCookieOptionsTests [Fact] public async Task Build_WithCookieNameOverride_UsesConfiguredName() { - await using WebApplication app = GatewayApplication.Build( + await using WebApplication app = TestGatewayHost.Build( ["--MxGateway:Dashboard:CookieName=MxGatewayDashboard.env2"]); IOptionsMonitor optionsMonitor = app.Services .GetRequiredService>(); diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardDisableLoginTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardDisableLoginTests.cs index f40f8630..e18abe06 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardDisableLoginTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardDisableLoginTests.cs @@ -6,6 +6,7 @@ using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.DependencyInjection; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Dashboard; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard; @@ -16,7 +17,7 @@ public sealed class DashboardDisableLoginTests [Fact] public async Task DisableLoginOff_CookieSchemeUsesCookieHandler() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthenticationSchemeProvider provider = app.Services.GetRequiredService(); @@ -32,7 +33,7 @@ public sealed class DashboardDisableLoginTests [Fact] public async Task DisableLoginOn_CookieSchemeUsesAutoLoginHandler() { - await using WebApplication app = GatewayApplication.Build( + await using WebApplication app = TestGatewayHost.Build( ["--MxGateway:Dashboard:DisableLogin=true"]); IAuthenticationSchemeProvider provider = app.Services.GetRequiredService(); @@ -49,7 +50,7 @@ public sealed class DashboardDisableLoginTests [Fact] public async Task DisableLoginOn_AutoLoginPrincipalSatisfiesAdminAndViewerPolicies() { - await using WebApplication app = GatewayApplication.Build( + await using WebApplication app = TestGatewayHost.Build( ["--MxGateway:Dashboard:DisableLogin=true"]); IAuthorizationService authorization = app.Services.GetRequiredService(); diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardHubsRegistrationTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardHubsRegistrationTests.cs index f7e04ab5..8e754535 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardHubsRegistrationTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/DashboardHubsRegistrationTests.cs @@ -7,6 +7,7 @@ using Microsoft.Extensions.Options; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Dashboard; using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard; @@ -17,7 +18,7 @@ public sealed class DashboardHubsRegistrationTests [Fact] public async Task Build_WhenDashboardEnabled_MapsAllThreeHubsAndTokenEndpoint() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IReadOnlyList endpoints = ((IEndpointRouteBuilder)app).DataSources .SelectMany(source => source.Endpoints) .OfType() @@ -40,7 +41,7 @@ public sealed class DashboardHubsRegistrationTests [Fact] public async Task Build_WhenDashboardEnabled_RegistersHubTokenService() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); HubTokenService tokens = app.Services.GetRequiredService(); Assert.NotNull(tokens); @@ -55,7 +56,7 @@ public sealed class DashboardHubsRegistrationTests [Fact] public async Task Build_WhenDashboardEnabled_ResolvesBothEventMirrorInterfacesToOneInstance() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IDashboardEventBroadcaster broadcaster = app.Services.GetRequiredService(); IDashboardSessionEventSubscriber subscriber = app.Services @@ -74,7 +75,7 @@ public sealed class DashboardHubsRegistrationTests [Fact] public async Task Build_WhenDashboardEnabled_ShortensCircuitRetentionAndTracksConnection() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); CircuitOptions circuitOptions = app.Services.GetRequiredService>().Value; Assert.Equal(TimeSpan.FromSeconds(30), circuitOptions.DisconnectedCircuitRetentionPeriod); @@ -96,7 +97,7 @@ public sealed class DashboardHubsRegistrationTests [Fact] public async Task Build_WhenDashboardEnabled_SharesOneBrowseSearchIndex() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); Assert.Same( app.Services.GetRequiredService(), diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/DashboardSecretsMountTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/DashboardSecretsMountTests.cs index 74ae76ab..53214210 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/DashboardSecretsMountTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/DashboardSecretsMountTests.cs @@ -4,6 +4,7 @@ using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.DependencyInjection; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Dashboard; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Gateway; @@ -29,7 +30,7 @@ public sealed class DashboardSecretsMountTests [Fact] public async Task Build_RegistersSecretsAuthorizationPolicies() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthorizationPolicyProvider policyProvider = app.Services.GetRequiredService(); @@ -53,7 +54,7 @@ public sealed class DashboardSecretsMountTests [Fact] public async Task AdministratorPrincipal_SatisfiesSecretsPolicies() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthorizationService authorization = app.Services.GetRequiredService(); ClaimsPrincipal administrator = DashboardAutoLoginAuthenticationHandler.CreatePrincipal("multi-role"); @@ -73,7 +74,7 @@ public sealed class DashboardSecretsMountTests [Fact] public async Task AnonymousPrincipal_IsDeniedSecretsPolicies() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IAuthorizationService authorization = app.Services.GetRequiredService(); ClaimsPrincipal anonymous = new(new ClaimsIdentity()); diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayApplicationTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayApplicationTests.cs index 1ebfbadb..01f101c6 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayApplicationTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayApplicationTests.cs @@ -9,6 +9,7 @@ using Microsoft.Extensions.Options; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Dashboard; using ZB.MOM.WW.MxGateway.Server.Metrics; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Gateway; @@ -19,7 +20,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_MapsCanonicalHealthEndpoints() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); var paths = ((IEndpointRouteBuilder)app).DataSources .SelectMany(dataSource => dataSource.Endpoints) @@ -37,7 +38,7 @@ public sealed class GatewayApplicationTests [Fact] public void Build_UsesSerilogLoggerProvider() { - using var app = GatewayApplication.Build([]); + using var app = TestGatewayHost.Build([]); var factory = app.Services.GetRequiredService(); Assert.Equal("SerilogLoggerFactory", factory.GetType().Name); } @@ -54,7 +55,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_RegistersGatewayBrowseScopeProviderOverLibraryDefault() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); ZB.MOM.WW.GalaxyRepository.Grpc.IGalaxyBrowseScopeProvider scopeProvider = app.Services.GetRequiredService(); @@ -67,7 +68,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_RegistersGatewayMetrics() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); GatewayMetrics metrics = app.Services.GetRequiredService(); @@ -82,7 +83,7 @@ public sealed class GatewayApplicationTests // Bind an ephemeral port (:0) — xUnit runs test collections in parallel, so any // started-host test must avoid a fixed port to prevent a bind collision. The auth SQLite // store path is isolated to a per-process temp file by TestHostEnvironmentInitializer. - await using WebApplication app = GatewayApplication.Build(["--urls=http://127.0.0.1:0"]); + await using WebApplication app = TestGatewayHost.Build(["--urls=http://127.0.0.1:0"]); await app.StartAsync(); try { @@ -103,7 +104,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_WhenDashboardEnabled_MapsBlazorDashboardAndAuthEndpoints() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IReadOnlyList endpoints = GetRouteEndpoints(app); Assert.Contains(endpoints, endpoint => endpoint.RoutePattern.RawText == "/"); @@ -133,7 +134,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_WhenDashboardEnabled_AuthEndpointsAllowAnonymousAccess() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IReadOnlyList endpoints = GetRouteEndpoints(app); string[] anonymousEndpointNames = @@ -163,7 +164,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_WhenDashboardEnabled_ComponentRoutesRequireAuthorization() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IReadOnlyList endpoints = GetRouteEndpoints(app); // Razor-component endpoints are distinguished from minimal-API @@ -192,7 +193,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_WhenDashboardEnabled_RegistersDashboardRoutesAtRoot() { - await using WebApplication app = GatewayApplication.Build([]); + await using WebApplication app = TestGatewayHost.Build([]); IReadOnlyList endpoints = GetRouteEndpoints(app); string[] canonicalRoutes = @@ -224,7 +225,7 @@ public sealed class GatewayApplicationTests [Fact] public async Task Build_WhenDashboardDisabled_DoesNotMapDashboardRoutes() { - await using WebApplication app = GatewayApplication.Build(["--MxGateway:Dashboard:Enabled=false"]); + await using WebApplication app = TestGatewayHost.Build(["--MxGateway:Dashboard:Enabled=false"]); IReadOnlyList endpoints = GetRouteEndpoints(app); Assert.DoesNotContain(endpoints, endpoint => @@ -268,7 +269,7 @@ public sealed class GatewayApplicationTests // WebApplication-building test must avoid a fixed port to prevent a bind collision. The auth // store path is isolated by TestHostEnvironmentInitializer, so startup opens a // writable store and the injected misconfiguration is what fails validation. - await using WebApplication app = GatewayApplication.Build( + await using WebApplication app = TestGatewayHost.Build( [$"--{key}={value}", "--urls=http://127.0.0.1:0"]); OptionsValidationException exception = await Assert.ThrowsAsync( diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayTlsBootstrapTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayTlsBootstrapTests.cs index 810b9cdd..038a07f2 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayTlsBootstrapTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayTlsBootstrapTests.cs @@ -6,6 +6,7 @@ using Microsoft.AspNetCore.Hosting.Server; using Microsoft.AspNetCore.Hosting.Server.Features; using Microsoft.Extensions.DependencyInjection; using ZB.MOM.WW.MxGateway.Server; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Gateway; @@ -36,7 +37,7 @@ public sealed class GatewayTlsBootstrapTests Environment.SetEnvironmentVariable( "MxGateway__Tls__SelfSignedCertPath", Path.Combine(certDir, "gw.pfx")); - WebApplication app = GatewayApplication.Build([]); + WebApplication app = TestGatewayHost.Build([]); await app.StartAsync(); try { diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/MetricsEndpointAuthenticationTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/MetricsEndpointAuthenticationTests.cs index 662c5762..3a1ff075 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/MetricsEndpointAuthenticationTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/MetricsEndpointAuthenticationTests.cs @@ -10,6 +10,7 @@ using ZB.MOM.WW.Auth.ApiKeys.Admin; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Dashboard; using ZB.MOM.WW.MxGateway.Server.Security.Authorization; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Gateway; @@ -121,7 +122,7 @@ public sealed class MetricsEndpointAuthenticationTests private static async Task StartAsync(params string[] extraArgs) { // Ephemeral port (:0): xUnit runs collections in parallel, so a fixed port would collide. - WebApplication app = GatewayApplication.Build( + WebApplication app = TestGatewayHost.Build( [ "--urls=http://127.0.0.1:0", "--MxGateway:Metrics:RequireAuthentication=true", diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/TestGatewayHostTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/TestGatewayHostTests.cs new file mode 100644 index 00000000..59ac9124 --- /dev/null +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/TestGatewayHostTests.cs @@ -0,0 +1,34 @@ +using Microsoft.AspNetCore.Builder; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; + +namespace ZB.MOM.WW.MxGateway.Tests.Gateway; + +/// +/// Guards the isolation exists for: host-building test classes run in +/// parallel, and when they shared one secrets store their concurrent pre-host migrations failed with +/// cannot start a transaction within a transaction on the Windows nightly (mxaccessgw#150). +/// +public sealed class TestGatewayHostTests +{ + /// + /// Two builds get two different secrets stores, and each build's pre-host migration created its + /// own store — so no two hosts in the suite can migrate the same SQLite file. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task Build_GivesEachHostItsOwnMigratedSecretsStore() + { + await using WebApplication first = TestGatewayHost.Build(); + await using WebApplication second = TestGatewayHost.Build(); + + string? firstPath = first.Configuration["Secrets:SqlitePath"]; + string? secondPath = second.Configuration["Secrets:SqlitePath"]; + + Assert.NotNull(firstPath); + Assert.NotNull(secondPath); + Assert.NotEqual(firstPath, secondPath); + Assert.NotEqual(Environment.GetEnvironmentVariable("Secrets__SqlitePath"), firstPath); + Assert.True(File.Exists(firstPath), $"the pre-host migration did not create {firstPath}"); + Assert.True(File.Exists(secondPath), $"the pre-host migration did not create {secondPath}"); + } +} diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Security/Authentication/SqliteAuthStoreTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Security/Authentication/SqliteAuthStoreTests.cs index 9e7fb7b6..1be1edd4 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Security/Authentication/SqliteAuthStoreTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Security/Authentication/SqliteAuthStoreTests.cs @@ -7,6 +7,7 @@ using ZB.MOM.WW.Auth.ApiKeys.Sqlite; using ZB.MOM.WW.MxGateway.Server; using ZB.MOM.WW.MxGateway.Server.Configuration; using ZB.MOM.WW.MxGateway.Server.Security.Authentication; +using ZB.MOM.WW.MxGateway.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Tests.Security.Authentication; @@ -70,7 +71,7 @@ public sealed class SqliteAuthStoreTests : IDisposable string databasePath = CreateTempDatabasePath(); await CreateSchemaVersionDatabaseAsync(databasePath, SqliteAuthSchema.CurrentVersion + 1); - await using WebApplication app = GatewayApplication.Build( + await using WebApplication app = TestGatewayHost.Build( [ $"--MxGateway:Authentication:SqlitePath={databasePath}", "--urls=http://127.0.0.1:0" diff --git a/src/ZB.MOM.WW.MxGateway.Tests/TestSupport/TestGatewayHost.cs b/src/ZB.MOM.WW.MxGateway.Tests/TestSupport/TestGatewayHost.cs new file mode 100644 index 00000000..9ca4c906 --- /dev/null +++ b/src/ZB.MOM.WW.MxGateway.Tests/TestSupport/TestGatewayHost.cs @@ -0,0 +1,39 @@ +using Microsoft.AspNetCore.Builder; +using ZB.MOM.WW.MxGateway.Server; + +namespace ZB.MOM.WW.MxGateway.Tests.TestSupport; + +/// +/// Builds the full gateway host for a test with a secrets store of its own. +/// +/// +/// +/// runs the secrets-store migrator before the host exists +/// (and the runtime registration migrates it again on start). +/// points every host in the test process at one per-process store, so host-building test classes +/// running in parallel migrated that one SQLite file concurrently. On the Windows nightly that +/// surfaced as SqliteException: cannot start a transaction within a transaction from +/// SqliteSecretsStoreMigrator.MigrateAsync in two classes at once (run 817, 2026-09-15, +/// mxaccessgw#143/#150). No host-building test shares or seeds secrets across hosts, so each +/// build gets a fresh store and the concurrent-migration window no longer exists. +/// +/// +/// The path is passed as a command-line argument, which outranks the process-wide +/// Secrets__SqlitePath default. Tests that deliberately set that variable to exercise the +/// store-path guard call directly and are unaffected. +/// +/// +internal static class TestGatewayHost +{ + /// Builds the gateway with and an isolated secrets store. + /// Command-line arguments, as for . + /// The built (not started) gateway application. + public static WebApplication Build(params string[] args) + { + string secretsPath = Path.Combine( + Path.GetTempPath(), + $"mxgw-tests-{Environment.ProcessId}", + $"secrets-{Guid.NewGuid():N}.db"); + return GatewayApplication.Build([.. args, $"--Secrets:SqlitePath={secretsPath}"]); + } +}