Windows nightly: diagnose retained intermittent test, checkout and host-resource failures #150

New Issue

2026-10-13T08:53:09-04:00

dohertj2 commented

2026-10-13 08:53:09 -04:00

The automated nightly incidents #126–#148 (excluding absent #140) have recovered equivalent current Windows/live gates, but historical flake/resource causes are not all proven fixed. This issue retains those unresolved risks instead of calling every old incident one duplicate or a repaired product defect.

Current evidence: run898 on2026-10-07 at exact SHA f2e08617 used by#143–#148 passed Worker529/11skip, Gateway1193/0skip and live8/0skip with SUCCESS mode=live. Run934 on2026-10-13 atf40409a6 independently passed both Windows tiers and same full/live counts; runs925/929 nightly metadata successful. Portable Rust remains separate#149.

Investigate distinct historical causes: Gateway stale-cursor ReplayGapSentinel (#142), handshaking-to-ready (#141), multi-subscriber Worker pipe disconnect (#147), ReconnectAfterFullAgeEvictionResumesWithSentinelFormula (#145/#146), dashboard host/policy (#143); Worker watchdog fault (#126/#127) and burst drain coalescing (#139); Windows memory/pagefile/CLR/compiler/protoc/PowerShell failures (#128/#129/#131–#135/#138), SSH transport failures (#130/#136/#137/#144), and stale git index.lock checkout (#148). Exact per-incident run links remain in their tracker bodies and sanitized triage capture.

Acceptance: correlate first failures with authoritative host/load/job evidence through permitted read-only channels; separate test timing, product and host-resource faults; add deterministic regressions for verified code defects; preserve test assertions; verify repeated full Windows/nightly live passes after changes. Do not claim a current pass proves every historical root cause fixed. No ad-hoc remote execution or host configuration changes authorized by this issue.

The automated nightly incidents #126–#148 (excluding absent #140) have recovered equivalent current Windows/live gates, but historical flake/resource causes are not all proven fixed. This issue retains those unresolved risks instead of calling every old incident one duplicate or a repaired product defect. Current evidence: run898 on2026-10-07 at exact SHA f2e08617 used by#143–#148 passed Worker529/11skip, Gateway1193/0skip and live8/0skip with SUCCESS mode=live. Run934 on2026-10-13 atf40409a6 independently passed both Windows tiers and same full/live counts; runs925/929 nightly metadata successful. Portable Rust remains separate#149. Investigate distinct historical causes: Gateway stale-cursor ReplayGapSentinel (#142), handshaking-to-ready (#141), multi-subscriber Worker pipe disconnect (#147), ReconnectAfterFullAgeEvictionResumesWithSentinelFormula (#145/#146), dashboard host/policy (#143); Worker watchdog fault (#126/#127) and burst drain coalescing (#139); Windows memory/pagefile/CLR/compiler/protoc/PowerShell failures (#128/#129/#131–#135/#138), SSH transport failures (#130/#136/#137/#144), and stale git index.lock checkout (#148). Exact per-incident run links remain in their tracker bodies and sanitized triage capture. Acceptance: correlate first failures with authoritative host/load/job evidence through permitted read-only channels; separate test timing, product and host-resource faults; add deterministic regressions for verified code defects; preserve test assertions; verify repeated full Windows/nightly live passes after changes. Do not claim a current pass proves every historical root cause fixed. No ad-hoc remote execution or host configuration changes authorized by this issue.