30ac3f322568e6923740eaa1a2fdaf5050935b02
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e74918ccfb |
ci: drop the nightly-windev schedule; run the deep windev tier on demand
The scheduled nightly-windev job failed most nights for host reasons and filed a Gitea issue each time. Remove the cron trigger and the job, and add a workflow_dispatch trigger whose windev_mode input (test/build/live) drives the windows-x86 job, so the live tier runs on demand - from the Actions page or via scripts/ci/run-windev-ci.sh live - the same way clients are packed and published on demand. Docs updated to match. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
|
de86f8b80e |
ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only): - Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive. New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of 2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same second, both bootstraps got in, and the nightly's checkout died on .git/index.lock. Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder removes a leftover .git\index.lock, which can only be debris from a killed git. - windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same Worker build + tests first, so the scheduled copy only doubled the load and created the race. - Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23 until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc, csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical memory, disk and testhost count at start and on failure so the next one is correlatable. - SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50). - Dashboard host build (#143, run 817): two host-building test classes failed at the same instant in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel classes migrating the one per-process secrets store. Tests now build the gateway through TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it. - Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 ( |
||
|
|
5c8075996f |
fix(TST-25): serialize the CI bootstrap fetch/checkout under the worktree lock
Acceptance-check finding (concurrency): the worktree lock in windev-worker-ci.ps1 guarded the build stage, but run-windev-ci.sh's bootstrap did git fetch + git checkout on the shared C:\build\mxaccessgw-ci clone BEFORE that lock was taken. Two concurrent runs therefore collided on .git/index.lock at the bootstrap stage (the second failed 'Another git process seems to be running', exit 1) instead of the second waiting — defeating the lock's purpose for manual/degraded-mode overlap or a future second runner. (The single Gitea runner serializes jobs, so CI pushes never actually overlapped; this is defense-in-depth being restored.) Fix: the bootstrap now acquires the same mkdir worktree lock around its fetch/ checkout and exports MXGW_CI_LOCK_HELD; windev-worker-ci.ps1 re-uses that lock (skips re-acquire/release) when the env is set, and still self-locks for a standalone/manual run. Now the second concurrent run waits at the bootstrap. |
||
|
|
6803bab79a |
fix(TST-25,SEC): stop CI SSH key leaking in cleartext CI logs
The windows-x86 acceptance check 'no key material in logs' failed: run #37's job log printed the full WINDEV_SSH_KEY PEM in the step env echo. Gitea's secret masker is line-oriented, so a multiline PEM rendered as one line with literal \n escapes never matches the real-newline secret value and is not redacted. Fix: store WINDEV_SSH_KEY base64-encoded (single line) so the masker redacts it to ***; run-windev-ci.sh auto-decodes a base64 PEM (still accepts a raw PEM for local hand-testing). Drop the redundant WINDEV_SSH_KNOWN_HOSTS from the job env (host keys are public and come from the committed windev.known_hosts pin), removing another cleartext env line. Document the base64 requirement in the bring-up README. Operationally: the previously-exposed CI key has been rotated on windev (old pubkey revoked from administrators_authorized_keys, new key installed) and the Gitea WINDEV_SSH_KEY secret replaced with the new key's base64. |
||
|
|
d769244ac0 |
fix(TST-25,TST-26): restore Windows/x86 CI tier via SSH-driven windev job
The `windows`/`live-mxaccess` CI jobs were removed in
|