Files

T

Joseph Doherty 5fe96b6677 docs(runbook): record the 08-11/08-12 wonder deploys and the backup-chain technique

Adds the two wonder rows that were deliberately withheld on 2026-08-11
while the pre-55f2889 SHA was unsettled. It is settled: b948e69 (08-09)
and 0a9715d (08-11) were never competing claims about one binary, they
are two deploys two days apart.

What settled it is worth recording as a technique in its own right, so
it goes in as a fourth way to identify a build: each Server.bak.<ts>
holds the exe that deploy REPLACED, so a VersionInfo sweep across the
backups reconstructs a host's deploy history from the host alone — no
repo access, no deploy record. The subtlety that makes it readable is
that a backup's timestamp dates the NEXT deploy, not the build inside
it. Reading a file version is non-destructive, unlike opening a SQLite
store in a backup directory.

Also records the full garbage version stamp recovered from the 08-09
binary, because the failure mode is a false positive rather than a
blank: "0.1.2+fatal:..." reads like a version that succeeded and then
picked up noise, when the leading 0.1.2 is just the static base <Version>
every build carries. For a binary in that window the commit is not
recoverable from the binary at all, so finding nothing is the expected
result rather than evidence against a SHA established another way.

Provenance is stated per cell rather than uniformly: the worker SHAs on
the new rows are carried forward and marked unconfirmed, b948e69 rests
on PDB hash plus the contemporaneous record and never on a stamp, and
55f2889 was read from the live stamp, which is trustworthy only because
it postdates 0152180.

2026-08-12 04:20:38 -04:00

8.2 KiB

Raw Blame History

Identifying A Deployed Build (Operator Runbook)

Written 2026-08-11 after a false alarm. An investigation treated the windev production binary as having no traceable provenance, on two pieces of evidence that both turned out to be normal output of our own build and deploy procedure. The binary was fine. This runbook records what those signals actually mean, so the next person spends minutes rather than a forensics pass.

The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10

src/Directory.Build.props appends the git short SHA to InformationalVersion (0.1.2+<sha>) so a running binary can be mapped back to a commit. That stamping was introduced by ec6f82b (2026-07-09, TST-11) and was broken on Windows for its first month.

$(MSBuildThisFileDirectory) ends in a path separator. On Windows that trailing backslash escaped the closing quote of the Exec command, so git rev-parse never ran correctly; because the target runs with ContinueOnError and ConsoleToMSBuild (which mixes stderr into ConsoleOutput), git's failure text was stamped as the source revision. The observed form is:

0.1.2+fatal: cannot change to ...

The full string recovered from wonder's 2026-08-09 server binary (read 2026-08-12) shows the whole failure, including the mismatched ' … " that caused it:

0.1.2+fatal: cannot change to 'C:\build\mxgw-deploy\src" rev-parse --short HEAD': Invalid argument

Do not read the leading 0.1.2 as provenance. It is the static base <Version> every build carries, not a truncated SHA. The hazard is a false positive rather than a blank: 0.1.2+fatal:… reads like a version that succeeded and then picked up noise, when in fact there is no usable identity anywhere in the string. For a binary built in this window the commit is not recoverable from the binary at all — so finding nothing is the expected result, not evidence against a SHA established another way.

0152180 (2026-08-10 05:49, merged in c46e5bb) fixed it two ways: the quoted path gained a trailing . so the separator can no longer escape the quote, and SourceRevisionId is now gated on a short-SHA shape so no future git failure text can become the revision either.

What this means for an operator. A git error string in the version of a binary built on Windows in that window is the expected result of our own build. It is non-diagnostic in both directions — it neither incriminates a build nor confirms one, so it should not be treated as evidence of anything. macOS builds in the same window stamp correctly, as do all builds after 0152180.

An absent `C:\build\mxgw-deploy` is expected

The deploy procedure builds from a detached worktree (git worktree add C:\build\mxgw-deploy <sha>) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The worktree is removed once the publish is copied out. Finding that the directory a binary was built from no longer exists is the normal end state of a correct deploy, not a deleted trail.

What does identify a build

In rough order of cost:

Behaviour over the wire. Works against a running service, needs no host access, and is the fastest discriminator for the worker. 53f69cd correlates OnWriteComplete onto plain Write/Write2 replies; b948e69 did so only for WriteSecured/WriteSecured2. So a plain Write whose reply carries statuses[0] proves the worker is at or past 53f69cd, and an empty statuses proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal still exercises the path and returns OPERATIONAL_ERROR with detail 1007.
PDB source hashes. Slower, needs the deployed symbols, but independent of anything the build stamped. Do not read this as the technique on its own. What settled the 2026-08-11 investigation was two independent derivations agreeing: a PDB source-hash match, and a contemporaneous deploy record written the same evening that named the same two commits. The convergence is the result's strength, not either method alone — a hash match tells you which sources a binary was built from, but not that the build was intentional or which host it went to. A reader with only one of the two available should weight it accordingly and look for a second line of evidence.
Deployment-side naming. Since 2026-08-07 the server deploys to a dated directory (Server-YYYYMMDD) with the NSSM Application/AppDirectory repointed at it, and backup directories carry operator-chosen labels naming the work (for example Worker.bak-20260809-planwrites). Those conventions place a build in time and intent, and an accidental or off-book deploy tends not to follow them.
The host's own backup directories, read as a chain. Each Server.bak.<timestamp> holds the exe that deploy replaced, so a sweep of VersionInfo across them reconstructs the host's deploy history from the host itself, with no repo access and no deploy record. A backup stamped 20260811T060739 containing an exe written 2026-08-09 is the 08-09 build being displaced — the backup's timestamp dates the next deploy, not the build inside it. Reading a file's version is non-destructive, unlike opening a SQLite store in a backup directory, which mutates it. This is what established that wonder's b948e69 and 0a9715d were two deploys two days apart rather than two competing claims about one binary.

Note that mixed Server and Worker SHAs are deliberate, not drift: the two are swapped independently whenever the contracts are wire-identical, so a host legitimately runs one commit for the server and a later one for the worker.

Recorded deploys

Date	Host	Server	Worker
2026-08-09	windev (`10.100.0.48`)	`b948e69` (`Server-20260809`)	`53f69cd`
2026-08-09	`wonder-app-vd03`	`b948e69`	`53f69cd`
2026-08-11	`wonder-app-vd03`	`0a9715d` (this deploy wrote `Server.bak.20260811T060739`, holding the displaced 08-09 build)	carried forward
2026-08-12	`wonder-app-vd03`	`55f2889` (this deploy wrote `Server.bak.20260812T040122`, holding `0a9715d`)	carried forward

The two wonder rows after 08-09 are server swaps; their worker cells are carried forward from the 08-09 entry rather than re-verified, so treat the worker SHA there as unconfirmed. Their server SHAs come from the backup-chain read described above (technique 4), except 55f2889, which was read directly from the live exe's stamp — trustworthy because it postdates 0152180.

b948e69 is confirmed by PDB source-hash match plus the contemporaneous record, never by a version stamp — that build falls in the broken-stamp window and its stamp is structurally unavailable (see the first section). 0a9715d is the first wonder build to stamp cleanly, since 0152180 landed before it.

The 2026-08-09 deploy was two separate swaps, which is why a single build time does not describe it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06, the latter two minutes after 53f69cd merged at 19:48, with a matching service stop/start at 19:50:29/34.

That reading is confirmable from artifacts still on disk, without trusting the narrative: windev carries two worker backup directories from that day (Worker.bak-20260809 and Worker.bak-20260809-planwrites), and wonder carries Worker.bak.20260809-planwrites. Two backups because there were two worker operations. This is easy to misread as redundancy — it is the second swap's fingerprint.

The 2026-08-09 deploy returned the worker to mainline

Worth stating because it went unrecorded at the time and later read as a mystery rather than as the improvement it was. Before that deploy, production ran worker dd7ca163 (2026-05-22), which is contained only by origin/test/client-e2e-coverage and is not an ancestor of main — meaning the x86 worker in production could not be rebuilt from any mainline commit. 53f69cd is on main, which closes that. Verified 2026-08-11 with git merge-base --is-ancestor.

8.2 KiB Raw Blame History