Joseph Doherty
fd0e88e74c
A 2026-08-11 investigation treated the windev production binary as having no traceable provenance. Both pieces of evidence were normal output of our own build and deploy procedure, and the binary was fine — but nothing in the repo said so, so it cost a forensics pass to establish. The version stamp is the first trap. SHA stamping landed inec6f82b(TST-11) and was broken on Windows until0152180(NEXT-09) a month later: the trailing backslash in MSBuildThisFileDirectory escaped the Exec's closing quote, and because the target runs ContinueOnError with ConsoleToMSBuild, git's stderr was stamped as the revision. Every Windows build in that window reads `0.1.2+fatal: cannot change to ...`. The point an operator needs is that this is non-diagnostic in *both* directions — it neither incriminates a build nor confirms one — which is not obvious from a stamp that looks like a failure. The second is that the build directory is *supposed* to be gone: deploys build from a detached worktree so the host's checkout stays on its own branch, and remove it afterwards. Records what does identify a build instead, cheapest first — including the wire probe for the worker, since plain Write/Write2 carrying statuses[0] separates53f69cdfromb948e69without host access or symbols — plus the deploys to date. Also records something that went unlogged and is the reason this read as a mystery rather than an improvement: the 2026-08-09 deploy returned the x86 worker to mainline. Production had been runningdd7ca163, contained only by origin/test/client-e2e-coverage and not an ancestor of main, so the worker in production was not rebuildable from any mainline commit.
4.8 KiB
Identifying A Deployed Build (Operator Runbook)
Written 2026-08-11 after a false alarm. An investigation treated the windev production binary as having no traceable provenance, on two pieces of evidence that both turned out to be normal output of our own build and deploy procedure. The binary was fine. This runbook records what those signals actually mean, so the next person spends minutes rather than a forensics pass.
The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10
src/Directory.Build.props appends the git short SHA to InformationalVersion (0.1.2+<sha>) so a
running binary can be mapped back to a commit. That stamping was introduced by ec6f82b
(2026-07-09, TST-11) and was broken on Windows for its first month.
$(MSBuildThisFileDirectory) ends in a path separator. On Windows that trailing backslash escaped
the closing quote of the Exec command, so git rev-parse never ran correctly; because the target
runs with ContinueOnError and ConsoleToMSBuild (which mixes stderr into ConsoleOutput), git's
failure text was stamped as the source revision. The observed form is:
0.1.2+fatal: cannot change to ...
0152180 (2026-08-10 05:49, merged in c46e5bb) fixed it two ways: the quoted path gained a
trailing . so the separator can no longer escape the quote, and SourceRevisionId is now gated on
a short-SHA shape so no future git failure text can become the revision either.
What this means for an operator. A git error string in the version of a binary built on Windows
in that window is the expected result of our own build. It is non-diagnostic in both
directions — it neither incriminates a build nor confirms one, so it should not be treated as
evidence of anything. macOS builds in the same window stamp correctly, as do all builds after
0152180.
An absent C:\build\mxgw-deploy is expected
The deploy procedure builds from a detached worktree (git worktree add C:\build\mxgw-deploy <sha>) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The
worktree is removed once the publish is copied out. Finding that the directory a binary was built
from no longer exists is the normal end state of a correct deploy, not a deleted trail.
What does identify a build
In rough order of cost:
- Behaviour over the wire. Works against a running service, needs no host access, and is the
fastest discriminator for the worker.
53f69cdcorrelatesOnWriteCompleteonto plainWrite/Write2replies;b948e69did so only forWriteSecured/WriteSecured2. So a plainWritewhose reply carriesstatuses[0]proves the worker is at or past53f69cd, and an emptystatusesproves it is not. Keep it non-destructive by writing to a read-only tag — the refusal still exercises the path and returnsOPERATIONAL_ERRORwith detail1007. - PDB source hashes. Slower, needs the deployed symbols, but independent of anything the build stamped. This is what settled the 2026-08-11 investigation.
- Deployment-side naming. Since 2026-08-07 the server deploys to a dated directory
(
Server-YYYYMMDD) with the NSSMApplication/AppDirectoryrepointed at it, and backup directories carry operator-chosen labels naming the work (for exampleWorker.bak-20260809-planwrites). Those conventions place a build in time and intent, and an accidental or off-book deploy tends not to follow them.
Note that mixed Server and Worker SHAs are deliberate, not drift: the two are swapped independently whenever the contracts are wire-identical, so a host legitimately runs one commit for the server and a later one for the worker.
Recorded deploys
| Date | Host | Server | Worker |
|---|---|---|---|
| 2026-08-09 | windev (10.100.0.48) |
b948e69 (Server-20260809) |
53f69cd |
| 2026-08-09 | wonder-app-vd03 |
b948e69 |
53f69cd |
The 2026-08-09 deploy was two separate swaps, which is why a single build time does not describe
it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
the latter two minutes after 53f69cd merged at 19:48. The two worker backup directories from that
day are the second swap's fingerprint, not redundancy.
The 2026-08-09 deploy returned the worker to mainline
Worth stating because it went unrecorded at the time and later read as a mystery rather than as the
improvement it was. Before that deploy, production ran worker dd7ca163 (2026-05-22), which is
contained only by origin/test/client-e2e-coverage and is not an ancestor of main — meaning
the x86 worker in production could not be rebuilt from any mainline commit. 53f69cd is on main,
which closes that. Verified 2026-08-11 with git merge-base --is-ancestor.