ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m34s
ci / java (push) Successful in 2m37s
ci / portable (push) Failing after 5m0s

Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):

- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
  New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
  succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
  2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
  second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
  Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
  handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
  harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
  removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
  Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
  Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
  until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
  csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
  window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
  memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
  in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
  classes migrating the one per-process secrets store. Tests now build the gateway through
  TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
  the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
  7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
  burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.

Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:35:54 -05:00
parent 09b3ecddf2
commit de86f8b80e
18 changed files with 234 additions and 65 deletions
+6
View File
@@ -165,6 +165,12 @@ jobs:
# actions), and a `runs-on` gate with no matching runner wedges the queue in "queued" forever.
# An unreachable windev fails this job RED (loud), never leaves it stuck. A red job can mean the
# Windows tier is down rather than the change — see docs/GatewayTesting.md § Continuous Integration.
#
# Skipped on the schedule: nightly-windev's `live` mode already runs this exact build + test as
# its first two tiers, and running both made every scheduled run start two windev sessions in
# the same second, contending for the one CI clone (the 2026-09-22 index.lock failure, run 827,
# mxaccessgw#150) and doubling the nightly's load on a host that was already short of memory.
if: github.event_name != 'schedule'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
+21 -1
View File
@@ -639,7 +639,8 @@ runtime because the x86 Worker cannot build on Linux:
client (IPC-24 deleted the old unconditional churn-revert step, which masked exactly that
class). The dev Mac has a Homebrew JDK 17, so `gradle generateProto` can be run there to refresh
the Java aggregates when a `.proto` changes.
- **`windows-x86`** (Linux runner, per push/PR) — builds the **x86 / net48 Worker and
- **`windows-x86`** (Linux runner, per push/PR; skipped on the schedule, where
`nightly-windev` runs the same tiers) — builds the **x86 / net48 Worker and
Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a
Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where
`scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci`
@@ -658,6 +659,25 @@ runtime because the x86 Worker cannot build on Linux:
Gated `if: github.event_name == 'schedule'`, so it never gates a push. On failure it opens a
Gitea issue via the Actions token, since nobody watches the Actions page.
The windev host lock is an exclusive file handle on `C:\build\mxaccessgw-ci.ci-lock`, held by
the SSH bootstrap for the whole run. It replaced a `New-Item -ItemType Directory` lock that two
runs starting in the same second could both take (PowerShell checks for the directory, then
creates it), which is how the 2026-09-22 nightly collided with the scheduled `windows-x86` run on
`.git/index.lock` (mxaccessgw#150). The OS drops the handle when its holder dies, so a killed run
leaves no stale lock, and a holder may delete a leftover `index.lock` because no other CI git can
be running. Each run also writes a `host (start)` line — and a `host (failure)` line when a tier
fails — to `C:\build\mxaccessgw-ci\ci.log` with free commit, physical memory, disk and the
number of `testhost` processes, so a host-resource failure can be matched to the host's state.
When a nightly goes red, sort the cause before touching code: a test that fails with an
assertion or test exception is a test or product fault; `OutOfMemoryException` from
MSBuild/csc/protoc, `csc.exe exited with code -1073741502` (0xC0000142, a process that could not
initialize), a remote exit code with no `ci.log` lines at all, or an SSH `kex_exchange_identification`
reset is the host, as is `No route to host`. The 2026-07-24 to 2026-08-03 red streak was the
second kind: windev's System log carries a `Resource-Exhaustion-Detector` event 2004 (low virtual
memory) every few minutes from 2026-07-23 until the box hard-reset on 2026-08-03, with no single
process holding most of the commit.
### Runner capacity is shared and finite
CI runs on two co-located runner containers on docker host `10.100.0.35` — `gitea-runner`
+10 -3
View File
@@ -18,11 +18,18 @@ Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests
`live` = test + full-`slnx` build + the net10 gateway suite + live-MXAccess smoke
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The `live` order is x86 Worker build,
`Worker.Tests`, full solution build, gateway tests, then live smoke. One host lock covers
checkout and every tier, so concurrent runs cannot overlap; each native command is checked
checkout and every tier, so concurrent runs cannot overlap. The lock is an exclusive
(`FileShare.None`) handle on `C:\build\mxaccessgw-ci.ci-lock` held by the SSH bootstrap; the OS
releases it if the run is killed, so it never goes stale. (It replaced a `New-Item -ItemType
Directory` lock that two runs starting in the same second could both take — mxaccessgw#150.)
Runs of SHAs older than that change still use the old directory lock and do not exclude new-style
runs; that window closes as branches rebase. each native command is checked
immediately and the first nonzero exit aborts the remaining tiers and propagates to CI.
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR; `nightly-windev`
runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on failure.
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR (not on the schedule,
where it would duplicate the nightly's first two tiers and start a second windev session in the
same second); `nightly-windev` runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on
failure.
## One-time bring-up (operator — required before this lands on `main`)
+29 -11
View File
@@ -84,19 +84,33 @@ echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_S
#
# The bootstrap fetch/checkout mutates the shared clone's git index, so it MUST hold the
# worktree lock — otherwise two concurrent runs collide on .git/index.lock before either
# reaches windev-worker-ci.ps1's lock (the exact race the lock exists to prevent). We take the
# same mkdir lock here (atomic; retry to a 20-min deadline) and export MXGW_CI_LOCK_HELD so the
# handed-off script re-uses this lock instead of re-acquiring/releasing it. A standalone/manual
# run of windev-worker-ci.ps1 (degraded mode) has no such env and self-locks as before.
# reaches windev-worker-ci.ps1 (the exact race the lock exists to prevent). We export
# MXGW_CI_LOCK_HELD so the handed-off script re-uses this lock instead of re-acquiring it; a
# standalone/manual run of windev-worker-ci.ps1 (degraded mode) has no such env and takes the
# same lock itself.
#
# The lock is an exclusive (FileShare.None) handle on '<clone>.ci-lock', held by this
# bootstrap process for the whole run (the handle is not inherited by the child powershell).
# It replaced a `New-Item -ItemType Directory` lock that was NOT mutually exclusive:
# PowerShell's directory provider checks for existence and then calls CreateDirectory (which
# succeeds on an existing directory), so two bootstraps starting together both "acquired" it.
# A scheduled run starts windows-x86 and nightly-windev in the same second, and on 2026-09-22
# (run 827) both got in, ran git concurrently and the nightly died on
# ".git/index.lock: File exists" (mxaccessgw#148/#150). A file handle is also released by the
# OS if the holder is killed (ssh drop, host reboot), so a dead run can never leave a stale
# lock behind; the old directory had to be removed by hand.
#
# Holding the lock also proves no other CI git command can be running in the clone, so a
# leftover .git/index.lock can only be debris from a killed git and is removed.
read -r -d '' BOOTSTRAP <<PS || true
\$ErrorActionPreference = 'Continue'
\$lockDir = '$REMOTE_CLONE.lock'
\$lockFile = '$REMOTE_CLONE.ci-lock'
\$deadline = (Get-Date).AddMinutes(20)
\$haveLock = \$false
while (-not \$haveLock) {
try { New-Item -Path \$lockDir -ItemType Directory -ErrorAction Stop | Out-Null; \$haveLock = \$true }
\$lock = \$null
while (\$null -eq \$lock) {
try { \$lock = [System.IO.File]::Open(\$lockFile, 'OpenOrCreate', 'ReadWrite', 'None') }
catch {
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockDir"; exit 75 }
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockFile : \$(\$_.Exception.Message)"; exit 75 }
Start-Sleep -Seconds 10
}
}
@@ -104,16 +118,20 @@ while (-not \$haveLock) {
\$rc = 1
try {
Set-Location '$REMOTE_CLONE'
if (Test-Path '.git\\index.lock') {
Write-Warning 'removing stale .git\\index.lock left by a killed git (the CI lock is held, so no CI git is running)'
Remove-Item -Force '.git\\index.lock'
}
git fetch --prune origin
git checkout --force --detach $CI_SHA
if (\$LASTEXITCODE -ne 0) { Write-Error "bootstrap checkout failed"; \$rc = \$LASTEXITCODE }
if (\$LASTEXITCODE -ne 0) { \$rc = \$LASTEXITCODE; Write-Error "bootstrap checkout failed (exit \$rc)" }
else {
powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File '$REMOTE_SCRIPT' -Sha '$CI_SHA' -Mode '$MODE'
\$rc = \$LASTEXITCODE
}
}
finally {
Remove-Item -Path \$lockDir -Recurse -Force -ErrorAction SilentlyContinue
\$lock.Dispose()
}
exit \$rc
PS
+50 -16
View File
@@ -44,7 +44,7 @@ param(
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$lockDir = "$RepoRoot.lock"
$lockFile = "$RepoRoot.ci-lock"
$logFile = Join-Path $RepoRoot 'ci.log'
$lockTimeout = [TimeSpan]::FromMinutes(20)
@@ -67,37 +67,70 @@ function Invoke-Native {
}
}
# --- acquire the worktree lock (mkdir is atomic; retry until timeout) ---
# Logs the host's memory/commit/disk headroom and any leftover test processes, so a
# host-resource failure (OOM in csc/MSBuild/protoc, 0xC0000142 process-start failures, a
# PowerShell that dies before logging) can be correlated with the host's state at the time
# rather than reconstructed afterwards from the event log (mxaccessgw#150). Never fatal.
function Write-HostResources {
param([string]$When)
try {
$os = Get-CimInstance -ClassName Win32_OperatingSystem
$drive = Get-PSDrive -Name ($RepoRoot.Substring(0, 1)) -ErrorAction Stop
$testhosts = @(Get-Process -Name 'testhost*' -ErrorAction SilentlyContinue).Count
Write-Log ('host ({0}): commit free {1:N1} of {2:N1} GB, physical free {3:N1} of {4:N1} GB, {5}: free {6:N1} GB, testhost processes {7}, up since {8:yyyy-MM-dd HH:mm}' -f
$When,
($os.FreeVirtualMemory / 1MB), ($os.TotalVirtualMemorySize / 1MB),
($os.FreePhysicalMemory / 1MB), ($os.TotalVisibleMemorySize / 1MB),
$drive.Name, ($drive.Free / 1GB), $testhosts, $os.LastBootUpTime)
}
catch {
Write-Log "host ($When): resource probe failed: $($_.Exception.Message)"
}
}
# --- acquire the worktree lock ---
# The lock is an exclusive (FileShare.None) handle on "$RepoRoot.ci-lock". It replaced a
# `New-Item -ItemType Directory` lock that was not mutually exclusive (PowerShell checks for
# the directory and then calls CreateDirectory, which succeeds on an existing one), so two
# runs starting together both got in and collided on .git/index.lock (mxaccessgw#148/#150).
# The OS releases the handle if the holder is killed, so a dead run cannot leave it stale.
# run-windev-ci.sh's bootstrap already holds this lock around its pre-hand-off fetch/checkout
# (MXGW_CI_LOCK_HELD=1) so that stage can't race a concurrent run; re-use it rather than
# re-acquire (and leave releasing to the bootstrap). A standalone/manual run (degraded mode)
# has no such env and self-locks here as before.
# (MXGW_CI_LOCK_HELD=1); re-use it rather than re-acquire (and leave releasing to the
# bootstrap). A standalone/manual run (degraded mode) has no such env and self-locks here.
$deadline = (Get-Date).Add($lockTimeout)
$haveLock = $false
$lock = $null
if ($env:MXGW_CI_LOCK_HELD -eq '1') {
Write-Log "Worktree lock already held by bootstrap (mode=$Mode sha=$Sha)"
}
else {
Write-Log "Acquiring worktree lock $lockDir (mode=$Mode sha=$Sha)"
while (-not $haveLock) {
Write-Log "Acquiring worktree lock $lockFile (mode=$Mode sha=$Sha)"
while ($null -eq $lock) {
try {
New-Item -Path $lockDir -ItemType Directory -ErrorAction Stop | Out-Null
$haveLock = $true
$lock = [System.IO.File]::Open($lockFile, 'OpenOrCreate', 'ReadWrite', 'None')
}
catch {
if ((Get-Date) -ge $deadline) {
throw "Timed out after $($lockTimeout.TotalMinutes) min waiting for worktree lock $lockDir. A stuck run may have left it; if no CI job is active, remove it by hand."
throw "Timed out after $($lockTimeout.TotalMinutes) min waiting for worktree lock $lockFile ($($_.Exception.Message)). Another CI run is holding it."
}
Start-Sleep -Seconds 10
}
}
}
Write-HostResources -When 'start'
try {
Push-Location $RepoRoot
try {
# Re-fetch + re-checkout under the lock so a concurrent run's bootstrap checkout
# (done before we held the lock) cannot leave us on the wrong tree.
# Holding the lock proves no other CI git command is running in this clone, so a
# leftover index.lock can only be debris from a killed git; it would fail the checkout.
if (Test-Path '.git\index.lock') {
Write-Log 'Removing stale .git\index.lock left by a killed git'
Remove-Item -Force '.git\index.lock'
}
# Re-fetch + re-checkout under the lock so the tree is the requested SHA even when the
# bootstrap's checkout was skipped (standalone run) or left residue.
Invoke-Native 'git fetch origin' { git fetch --prune origin }
Invoke-Native "git checkout --force $Sha" { git checkout --force --detach $Sha }
# Drop any stray tracked/untracked residue from a prior run so the build is clean.
@@ -151,11 +184,12 @@ try {
}
catch {
Write-Log "FAILURE mode=$Mode sha=$Sha : $($_.Exception.Message)"
Write-HostResources -When 'failure'
throw
}
finally {
if ($haveLock) {
Remove-Item -Path $lockDir -Recurse -Force -ErrorAction SilentlyContinue
Write-Log "Released worktree lock $lockDir"
if ($null -ne $lock) {
$lock.Dispose()
Write-Log "Released worktree lock $lockFile"
}
}
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Options;
using ZB.MOM.WW.GalaxyRepository;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Configuration;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Configuration;
@@ -114,7 +115,7 @@ public sealed class GalaxyRepositoryOptionsValidatorTests
[Fact]
public void Build_DefaultsBlankSnapshotCachePathToRootedDefault()
{
using WebApplication app = GatewayApplication.Build([]);
using WebApplication app = TestGatewayHost.Build([]);
GalaxyRepositoryOptions options =
app.Services.GetRequiredService<IOptions<GalaxyRepositoryOptions>>().Value;
@@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.Auth.AspNetCore;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
using ZB.MOM.WW.Secrets.Ui;
namespace ZB.MOM.WW.MxGateway.Tests.Dashboard;
@@ -35,7 +36,7 @@ public sealed class SecretsNavGateTests
[Fact]
public async Task ManagePolicy_AdmitsAdministrator()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthorizationService authorization = app.Services.GetRequiredService<IAuthorizationService>();
AuthorizationResult result = await authorization.AuthorizeAsync(
@@ -56,7 +57,7 @@ public sealed class SecretsNavGateTests
[Fact]
public async Task ManagePolicy_RefusesViewer()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthorizationService authorization = app.Services.GetRequiredService<IAuthorizationService>();
AuthorizationResult result = await authorization.AuthorizeAsync(
@@ -75,7 +76,7 @@ public sealed class SecretsNavGateTests
[Fact]
public async Task ManagePolicy_RefusesUnauthenticated()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthorizationService authorization = app.Services.GetRequiredService<IAuthorizationService>();
AuthorizationResult result = await authorization.AuthorizeAsync(
@@ -11,6 +11,7 @@ using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Configuration;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
using AuthenticationOptions = ZB.MOM.WW.MxGateway.Server.Configuration.AuthenticationOptions;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
@@ -115,7 +116,7 @@ public sealed class AlarmFeedAuthorizationTests
[Fact]
public async Task AlarmsPolicy_IsRegisteredWithNonBypassableRequirement()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthorizationPolicyProvider provider = app.Services
.GetRequiredService<IAuthorizationPolicyProvider>();
@@ -147,7 +148,7 @@ public sealed class AlarmFeedAuthorizationTests
Assert.Equal(DashboardAuthenticationDefaults.AlarmsPolicy, hubAttribute?.Policy);
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
RouteEndpoint alarmsPage = ((IEndpointRouteBuilder)app).DataSources
.SelectMany(source => source.Endpoints)
.OfType<RouteEndpoint>()
@@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
@@ -20,7 +21,7 @@ public sealed class DashboardCookieOptionsTests
[Fact]
public async Task Build_ConfiguresSecureDashboardCookie()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IOptionsMonitor<CookieAuthenticationOptions> optionsMonitor = app.Services
.GetRequiredService<IOptionsMonitor<CookieAuthenticationOptions>>();
@@ -49,7 +50,7 @@ public sealed class DashboardCookieOptionsTests
[Fact]
public async Task Build_WithRequireHttpsCookieFalse_UsesSameAsRequestAndPlainName()
{
await using WebApplication app = GatewayApplication.Build(
await using WebApplication app = TestGatewayHost.Build(
["--MxGateway:Dashboard:RequireHttpsCookie=false"]);
IOptionsMonitor<CookieAuthenticationOptions> optionsMonitor = app.Services
.GetRequiredService<IOptionsMonitor<CookieAuthenticationOptions>>();
@@ -71,7 +72,7 @@ public sealed class DashboardCookieOptionsTests
[Fact]
public async Task Build_WithCookieNameOverride_UsesConfiguredName()
{
await using WebApplication app = GatewayApplication.Build(
await using WebApplication app = TestGatewayHost.Build(
["--MxGateway:Dashboard:CookieName=MxGatewayDashboard.env2"]);
IOptionsMonitor<CookieAuthenticationOptions> optionsMonitor = app.Services
.GetRequiredService<IOptionsMonitor<CookieAuthenticationOptions>>();
@@ -6,6 +6,7 @@ using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
@@ -16,7 +17,7 @@ public sealed class DashboardDisableLoginTests
[Fact]
public async Task DisableLoginOff_CookieSchemeUsesCookieHandler()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthenticationSchemeProvider provider =
app.Services.GetRequiredService<IAuthenticationSchemeProvider>();
@@ -32,7 +33,7 @@ public sealed class DashboardDisableLoginTests
[Fact]
public async Task DisableLoginOn_CookieSchemeUsesAutoLoginHandler()
{
await using WebApplication app = GatewayApplication.Build(
await using WebApplication app = TestGatewayHost.Build(
["--MxGateway:Dashboard:DisableLogin=true"]);
IAuthenticationSchemeProvider provider =
app.Services.GetRequiredService<IAuthenticationSchemeProvider>();
@@ -49,7 +50,7 @@ public sealed class DashboardDisableLoginTests
[Fact]
public async Task DisableLoginOn_AutoLoginPrincipalSatisfiesAdminAndViewerPolicies()
{
await using WebApplication app = GatewayApplication.Build(
await using WebApplication app = TestGatewayHost.Build(
["--MxGateway:Dashboard:DisableLogin=true"]);
IAuthorizationService authorization =
app.Services.GetRequiredService<IAuthorizationService>();
@@ -7,6 +7,7 @@ using Microsoft.Extensions.Options;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
@@ -17,7 +18,7 @@ public sealed class DashboardHubsRegistrationTests
[Fact]
public async Task Build_WhenDashboardEnabled_MapsAllThreeHubsAndTokenEndpoint()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IReadOnlyList<RouteEndpoint> endpoints = ((IEndpointRouteBuilder)app).DataSources
.SelectMany(source => source.Endpoints)
.OfType<RouteEndpoint>()
@@ -40,7 +41,7 @@ public sealed class DashboardHubsRegistrationTests
[Fact]
public async Task Build_WhenDashboardEnabled_RegistersHubTokenService()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
HubTokenService tokens = app.Services.GetRequiredService<HubTokenService>();
Assert.NotNull(tokens);
@@ -55,7 +56,7 @@ public sealed class DashboardHubsRegistrationTests
[Fact]
public async Task Build_WhenDashboardEnabled_ResolvesBothEventMirrorInterfacesToOneInstance()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IDashboardEventBroadcaster broadcaster = app.Services.GetRequiredService<IDashboardEventBroadcaster>();
IDashboardSessionEventSubscriber subscriber = app.Services
@@ -74,7 +75,7 @@ public sealed class DashboardHubsRegistrationTests
[Fact]
public async Task Build_WhenDashboardEnabled_ShortensCircuitRetentionAndTracksConnection()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
CircuitOptions circuitOptions = app.Services.GetRequiredService<IOptions<CircuitOptions>>().Value;
Assert.Equal(TimeSpan.FromSeconds(30), circuitOptions.DisconnectedCircuitRetentionPeriod);
@@ -96,7 +97,7 @@ public sealed class DashboardHubsRegistrationTests
[Fact]
public async Task Build_WhenDashboardEnabled_SharesOneBrowseSearchIndex()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
Assert.Same(
app.Services.GetRequiredService<DashboardBrowseSearchIndex>(),
@@ -4,6 +4,7 @@ using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
@@ -29,7 +30,7 @@ public sealed class DashboardSecretsMountTests
[Fact]
public async Task Build_RegistersSecretsAuthorizationPolicies()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthorizationPolicyProvider policyProvider =
app.Services.GetRequiredService<IAuthorizationPolicyProvider>();
@@ -53,7 +54,7 @@ public sealed class DashboardSecretsMountTests
[Fact]
public async Task AdministratorPrincipal_SatisfiesSecretsPolicies()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthorizationService authorization =
app.Services.GetRequiredService<IAuthorizationService>();
ClaimsPrincipal administrator = DashboardAutoLoginAuthenticationHandler.CreatePrincipal("multi-role");
@@ -73,7 +74,7 @@ public sealed class DashboardSecretsMountTests
[Fact]
public async Task AnonymousPrincipal_IsDeniedSecretsPolicies()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IAuthorizationService authorization =
app.Services.GetRequiredService<IAuthorizationService>();
ClaimsPrincipal anonymous = new(new ClaimsIdentity());
@@ -9,6 +9,7 @@ using Microsoft.Extensions.Options;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Metrics;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
@@ -19,7 +20,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_MapsCanonicalHealthEndpoints()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
var paths = ((IEndpointRouteBuilder)app).DataSources
.SelectMany(dataSource => dataSource.Endpoints)
@@ -37,7 +38,7 @@ public sealed class GatewayApplicationTests
[Fact]
public void Build_UsesSerilogLoggerProvider()
{
using var app = GatewayApplication.Build([]);
using var app = TestGatewayHost.Build([]);
var factory = app.Services.GetRequiredService<ILoggerFactory>();
Assert.Equal("SerilogLoggerFactory", factory.GetType().Name);
}
@@ -54,7 +55,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_RegistersGatewayBrowseScopeProviderOverLibraryDefault()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
ZB.MOM.WW.GalaxyRepository.Grpc.IGalaxyBrowseScopeProvider scopeProvider =
app.Services.GetRequiredService<ZB.MOM.WW.GalaxyRepository.Grpc.IGalaxyBrowseScopeProvider>();
@@ -67,7 +68,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_RegistersGatewayMetrics()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
GatewayMetrics metrics = app.Services.GetRequiredService<GatewayMetrics>();
@@ -82,7 +83,7 @@ public sealed class GatewayApplicationTests
// Bind an ephemeral port (:0) — xUnit runs test collections in parallel, so any
// started-host test must avoid a fixed port to prevent a bind collision. The auth SQLite
// store path is isolated to a per-process temp file by TestHostEnvironmentInitializer.
await using WebApplication app = GatewayApplication.Build(["--urls=http://127.0.0.1:0"]);
await using WebApplication app = TestGatewayHost.Build(["--urls=http://127.0.0.1:0"]);
await app.StartAsync();
try
{
@@ -103,7 +104,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_WhenDashboardEnabled_MapsBlazorDashboardAndAuthEndpoints()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IReadOnlyList<RouteEndpoint> endpoints = GetRouteEndpoints(app);
Assert.Contains(endpoints, endpoint => endpoint.RoutePattern.RawText == "/");
@@ -133,7 +134,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_WhenDashboardEnabled_AuthEndpointsAllowAnonymousAccess()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IReadOnlyList<RouteEndpoint> endpoints = GetRouteEndpoints(app);
string[] anonymousEndpointNames =
@@ -163,7 +164,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_WhenDashboardEnabled_ComponentRoutesRequireAuthorization()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IReadOnlyList<RouteEndpoint> endpoints = GetRouteEndpoints(app);
// Razor-component endpoints are distinguished from minimal-API
@@ -192,7 +193,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_WhenDashboardEnabled_RegistersDashboardRoutesAtRoot()
{
await using WebApplication app = GatewayApplication.Build([]);
await using WebApplication app = TestGatewayHost.Build([]);
IReadOnlyList<RouteEndpoint> endpoints = GetRouteEndpoints(app);
string[] canonicalRoutes =
@@ -224,7 +225,7 @@ public sealed class GatewayApplicationTests
[Fact]
public async Task Build_WhenDashboardDisabled_DoesNotMapDashboardRoutes()
{
await using WebApplication app = GatewayApplication.Build(["--MxGateway:Dashboard:Enabled=false"]);
await using WebApplication app = TestGatewayHost.Build(["--MxGateway:Dashboard:Enabled=false"]);
IReadOnlyList<RouteEndpoint> endpoints = GetRouteEndpoints(app);
Assert.DoesNotContain(endpoints, endpoint =>
@@ -268,7 +269,7 @@ public sealed class GatewayApplicationTests
// WebApplication-building test must avoid a fixed port to prevent a bind collision. The auth
// store path is isolated by TestHostEnvironmentInitializer, so startup opens a
// writable store and the injected misconfiguration is what fails validation.
await using WebApplication app = GatewayApplication.Build(
await using WebApplication app = TestGatewayHost.Build(
[$"--{key}={value}", "--urls=http://127.0.0.1:0"]);
OptionsValidationException exception = await Assert.ThrowsAsync<OptionsValidationException>(
@@ -6,6 +6,7 @@ using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
@@ -36,7 +37,7 @@ public sealed class GatewayTlsBootstrapTests
Environment.SetEnvironmentVariable(
"MxGateway__Tls__SelfSignedCertPath", Path.Combine(certDir, "gw.pfx"));
WebApplication app = GatewayApplication.Build([]);
WebApplication app = TestGatewayHost.Build([]);
await app.StartAsync();
try
{
@@ -10,6 +10,7 @@ using ZB.MOM.WW.Auth.ApiKeys.Admin;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Dashboard;
using ZB.MOM.WW.MxGateway.Server.Security.Authorization;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
@@ -121,7 +122,7 @@ public sealed class MetricsEndpointAuthenticationTests
private static async Task<WebApplication> StartAsync(params string[] extraArgs)
{
// Ephemeral port (:0): xUnit runs collections in parallel, so a fixed port would collide.
WebApplication app = GatewayApplication.Build(
WebApplication app = TestGatewayHost.Build(
[
"--urls=http://127.0.0.1:0",
"--MxGateway:Metrics:RequireAuthentication=true",
@@ -0,0 +1,34 @@
using Microsoft.AspNetCore.Builder;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway;
/// <summary>
/// Guards the isolation <see cref="TestGatewayHost"/> exists for: host-building test classes run in
/// parallel, and when they shared one secrets store their concurrent pre-host migrations failed with
/// <c>cannot start a transaction within a transaction</c> on the Windows nightly (mxaccessgw#150).
/// </summary>
public sealed class TestGatewayHostTests
{
/// <summary>
/// Two builds get two different secrets stores, and each build's pre-host migration created its
/// own store — so no two hosts in the suite can migrate the same SQLite file.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Build_GivesEachHostItsOwnMigratedSecretsStore()
{
await using WebApplication first = TestGatewayHost.Build();
await using WebApplication second = TestGatewayHost.Build();
string? firstPath = first.Configuration["Secrets:SqlitePath"];
string? secondPath = second.Configuration["Secrets:SqlitePath"];
Assert.NotNull(firstPath);
Assert.NotNull(secondPath);
Assert.NotEqual(firstPath, secondPath);
Assert.NotEqual(Environment.GetEnvironmentVariable("Secrets__SqlitePath"), firstPath);
Assert.True(File.Exists(firstPath), $"the pre-host migration did not create {firstPath}");
Assert.True(File.Exists(secondPath), $"the pre-host migration did not create {secondPath}");
}
}
@@ -7,6 +7,7 @@ using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
using ZB.MOM.WW.MxGateway.Server;
using ZB.MOM.WW.MxGateway.Server.Configuration;
using ZB.MOM.WW.MxGateway.Server.Security.Authentication;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Security.Authentication;
@@ -70,7 +71,7 @@ public sealed class SqliteAuthStoreTests : IDisposable
string databasePath = CreateTempDatabasePath();
await CreateSchemaVersionDatabaseAsync(databasePath, SqliteAuthSchema.CurrentVersion + 1);
await using WebApplication app = GatewayApplication.Build(
await using WebApplication app = TestGatewayHost.Build(
[
$"--MxGateway:Authentication:SqlitePath={databasePath}",
"--urls=http://127.0.0.1:0"
@@ -0,0 +1,39 @@
using Microsoft.AspNetCore.Builder;
using ZB.MOM.WW.MxGateway.Server;
namespace ZB.MOM.WW.MxGateway.Tests.TestSupport;
/// <summary>
/// Builds the full gateway host for a test with a secrets store of its own.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="GatewayApplication.Build"/> runs the secrets-store migrator before the host exists
/// (and the runtime registration migrates it again on start). <see cref="TestHostEnvironmentInitializer"/>
/// points every host in the test process at one per-process store, so host-building test classes
/// running in parallel migrated that one SQLite file concurrently. On the Windows nightly that
/// surfaced as <c>SqliteException: cannot start a transaction within a transaction</c> from
/// <c>SqliteSecretsStoreMigrator.MigrateAsync</c> in two classes at once (run 817, 2026-09-15,
/// mxaccessgw#143/#150). No host-building test shares or seeds secrets across hosts, so each
/// build gets a fresh store and the concurrent-migration window no longer exists.
/// </para>
/// <para>
/// The path is passed as a command-line argument, which outranks the process-wide
/// <c>Secrets__SqlitePath</c> default. Tests that deliberately set that variable to exercise the
/// store-path guard call <see cref="GatewayApplication.CreateBuilder"/> directly and are unaffected.
/// </para>
/// </remarks>
internal static class TestGatewayHost
{
/// <summary>Builds the gateway with <paramref name="args"/> and an isolated secrets store.</summary>
/// <param name="args">Command-line arguments, as for <see cref="GatewayApplication.Build"/>.</param>
/// <returns>The built (not started) gateway application.</returns>
public static WebApplication Build(params string[] args)
{
string secretsPath = Path.Combine(
Path.GetTempPath(),
$"mxgw-tests-{Environment.ProcessId}",
$"secrets-{Guid.NewGuid():N}.db");
return GatewayApplication.Build([.. args, $"--Secrets:SqlitePath={secretsPath}"]);
}
}