Commit Graph

5 Commits

Author SHA1 Message Date
Joseph Doherty e74918ccfb ci: drop the nightly-windev schedule; run the deep windev tier on demand
ci / portable (push) Failing after 10m10s
ci / java (push) Failing after 20m12s
ci / windows-x86 (push) Failing after 22m40s
The scheduled nightly-windev job failed most nights for host reasons and
filed a Gitea issue each time. Remove the cron trigger and the job, and add
a workflow_dispatch trigger whose windev_mode input (test/build/live) drives
the windows-x86 job, so the live tier runs on demand - from the Actions page
or via scripts/ci/run-windev-ci.sh live - the same way clients are packed and
published on demand. Docs updated to match.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-15 09:42:13 -05:00
Joseph Doherty de86f8b80e ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m34s
ci / java (push) Successful in 2m37s
ci / portable (push) Failing after 5m0s
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):

- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
  New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
  succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
  2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
  second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
  Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
  handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
  harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
  removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
  Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
  Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
  until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
  csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
  window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
  memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
  in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
  classes migrating the one per-process secrets store. Tests now build the gateway through
  TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
  the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
  7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
  burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.

Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:36:34 -05:00
Joseph Doherty 5c8075996f fix(TST-25): serialize the CI bootstrap fetch/checkout under the worktree lock
ci / portable (push) Successful in 6m58s
ci / java (push) Successful in 1m52s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m8s
Acceptance-check finding (concurrency): the worktree lock in windev-worker-ci.ps1
guarded the build stage, but run-windev-ci.sh's bootstrap did git fetch + git
checkout on the shared C:\build\mxaccessgw-ci clone BEFORE that lock was taken.
Two concurrent runs therefore collided on .git/index.lock at the bootstrap stage
(the second failed 'Another git process seems to be running', exit 1) instead of
the second waiting — defeating the lock's purpose for manual/degraded-mode overlap
or a future second runner. (The single Gitea runner serializes jobs, so CI pushes
never actually overlapped; this is defense-in-depth being restored.)

Fix: the bootstrap now acquires the same mkdir worktree lock around its fetch/
checkout and exports MXGW_CI_LOCK_HELD; windev-worker-ci.ps1 re-uses that lock
(skips re-acquire/release) when the env is set, and still self-locks for a
standalone/manual run. Now the second concurrent run waits at the bootstrap.
2026-07-13 10:50:22 -04:00
Joseph Doherty 6803bab79a fix(TST-25,SEC): stop CI SSH key leaking in cleartext CI logs
ci / portable (push) Successful in 7m28s
ci / java (push) Successful in 1m59s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m6s
The windows-x86 acceptance check 'no key material in logs' failed: run #37's
job log printed the full WINDEV_SSH_KEY PEM in the step env echo. Gitea's
secret masker is line-oriented, so a multiline PEM rendered as one line with
literal \n escapes never matches the real-newline secret value and is not
redacted.

Fix: store WINDEV_SSH_KEY base64-encoded (single line) so the masker redacts
it to ***; run-windev-ci.sh auto-decodes a base64 PEM (still accepts a raw PEM
for local hand-testing). Drop the redundant WINDEV_SSH_KNOWN_HOSTS from the job
env (host keys are public and come from the committed windev.known_hosts pin),
removing another cleartext env line. Document the base64 requirement in the
bring-up README.

Operationally: the previously-exposed CI key has been rotated on windev
(old pubkey revoked from administrators_authorized_keys, new key installed) and
the Gitea WINDEV_SSH_KEY secret replaced with the new key's base64.
2026-07-13 10:45:16 -04:00
Joseph Doherty d769244ac0 fix(TST-25,TST-26): restore Windows/x86 CI tier via SSH-driven windev job
ci / java (push) Successful in 2m2s
ci / windows-x86 (push) Successful in 1m10s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 8m52s
The `windows`/`live-mxaccess` CI jobs were removed in abb0930 because Gitea
act_runner host-mode on Windows is broken and a runs-on gate with no runner
wedges the queue. This left the entire x86/net48 Worker + Worker.Tests tier
(and the live-MXAccess smoke) unguarded by CI.

Restore it without a native Windows runner: Linux jobs on ubuntu-latest (which
always schedule) SSH to windev (10.100.0.48), fetch+checkout the SHA under test
in an isolated clone C:\build\mxaccessgw-ci, run the x86 build/tests there, and
propagate the remote exit code back — so a Worker regression turns the job red
and an unreachable host fails loud (never stuck).

- scripts/ci/run-windev-ci.sh: Linux driver (key/known-hosts, UTF-16LE base64
  EncodedCommand bootstrap, ssh, exit-code passthrough).
- scripts/ci/windev-worker-ci.ps1: windev stage — worktree lock, re-fetch/
  checkout under lock, build|test|live modes; PS 5.1-safe, checks $LASTEXITCODE.
- scripts/ci/windev.known_hosts: pinned host keys for StrictHostKeyChecking.
- scripts/ci/README.md: operator bring-up + acceptance checklist.
- ci.yml: windows-x86 (per-push `test`) + nightly-windev (scheduled `live` +
  on-failure Gitea issue); drop the removal note; fix header/cron comments.

TST-26 (same commit, by rule): correct docs/scripts that still describe the
removed jobs — GatewayTesting.md, Contracts.md, check-codegen.ps1 — and
reattribute the Generated/-commit guard (primary = check-codegen diff in the
portable job; secondary = windows-x86 net48 compile).

Mechanism hand-verified on windev: build->0, bogus-SHA->nonzero (lock released),
test->356 passed/0 failed in ~50s, ssh exit-code propagation confirmed. Merge
requires operator bring-up first (dedicated ci@ key + Gitea secrets) or the
per-push job is red every push — see scripts/ci/README.md.
2026-07-13 10:09:49 -04:00