Joseph Doherty
5fe96b6677
Adds the two wonder rows that were deliberately withheld on 2026-08-11 while the pre-55f2889 SHA was unsettled. It is settled:b948e69(08-09) and0a9715d(08-11) were never competing claims about one binary, they are two deploys two days apart. What settled it is worth recording as a technique in its own right, so it goes in as a fourth way to identify a build: each Server.bak.<ts> holds the exe that deploy REPLACED, so a VersionInfo sweep across the backups reconstructs a host's deploy history from the host alone — no repo access, no deploy record. The subtlety that makes it readable is that a backup's timestamp dates the NEXT deploy, not the build inside it. Reading a file version is non-destructive, unlike opening a SQLite store in a backup directory. Also records the full garbage version stamp recovered from the 08-09 binary, because the failure mode is a false positive rather than a blank: "0.1.2+fatal:..." reads like a version that succeeded and then picked up noise, when the leading 0.1.2 is just the static base <Version> every build carries. For a binary in that window the commit is not recoverable from the binary at all, so finding nothing is the expected result rather than evidence against a SHA established another way. Provenance is stated per cell rather than uniformly: the worker SHAs on the new rows are carried forward and marked unconfirmed,b948e69rests on PDB hash plus the contemporaneous record and never on a stamp, and55f2889was read from the live stamp, which is trustworthy only because it postdates0152180.
133 lines
8.2 KiB
Markdown
133 lines
8.2 KiB
Markdown
# Identifying A Deployed Build (Operator Runbook)
|
|
|
|
> **Written 2026-08-11 after a false alarm.** An investigation treated the windev production
|
|
> binary as having no traceable provenance, on two pieces of evidence that both turned out to be
|
|
> normal output of our own build and deploy procedure. The binary was fine. This runbook records
|
|
> what those signals actually mean, so the next person spends minutes rather than a forensics pass.
|
|
|
|
## The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10
|
|
|
|
`src/Directory.Build.props` appends the git short SHA to `InformationalVersion` (`0.1.2+<sha>`) so a
|
|
running binary can be mapped back to a commit. That stamping was introduced by `ec6f82b`
|
|
(2026-07-09, TST-11) and was **broken on Windows for its first month**.
|
|
|
|
`$(MSBuildThisFileDirectory)` ends in a path separator. On Windows that trailing backslash escaped
|
|
the closing quote of the `Exec` command, so `git rev-parse` never ran correctly; because the target
|
|
runs with `ContinueOnError` and `ConsoleToMSBuild` (which mixes stderr into `ConsoleOutput`), git's
|
|
failure text was stamped as the source revision. The observed form is:
|
|
|
|
```
|
|
0.1.2+fatal: cannot change to ...
|
|
```
|
|
|
|
The full string recovered from wonder's 2026-08-09 server binary (read 2026-08-12) shows the whole
|
|
failure, including the mismatched `'` … `"` that caused it:
|
|
|
|
```
|
|
0.1.2+fatal: cannot change to 'C:\build\mxgw-deploy\src" rev-parse --short HEAD': Invalid argument
|
|
```
|
|
|
|
**Do not read the leading `0.1.2` as provenance.** It is the static base `<Version>` every build
|
|
carries, not a truncated SHA. The hazard is a false positive rather than a blank: `0.1.2+fatal:…`
|
|
reads like a version that succeeded and then picked up noise, when in fact there is no usable
|
|
identity anywhere in the string. For a binary built in this window the commit is **not recoverable
|
|
from the binary at all** — so finding nothing is the expected result, not evidence against a SHA
|
|
established another way.
|
|
|
|
`0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a
|
|
trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on
|
|
a short-SHA shape so no future git failure text can become the revision either.
|
|
|
|
**What this means for an operator.** A git error string in the version of a binary built on Windows
|
|
in that window is the *expected* result of our own build. It is non-diagnostic in **both**
|
|
directions — it neither incriminates a build nor confirms one, so it should not be treated as
|
|
evidence of anything. macOS builds in the same window stamp correctly, as do all builds after
|
|
`0152180`.
|
|
|
|
## An absent `C:\build\mxgw-deploy` is expected
|
|
|
|
The deploy procedure builds from a **detached worktree** (`git worktree add C:\build\mxgw-deploy
|
|
<sha>`) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The
|
|
worktree is removed once the publish is copied out. Finding that the directory a binary was built
|
|
from no longer exists is the normal end state of a correct deploy, not a deleted trail.
|
|
|
|
## What does identify a build
|
|
|
|
In rough order of cost:
|
|
|
|
1. **Behaviour over the wire.** Works against a running service, needs no host access, and is the
|
|
fastest discriminator for the worker. `53f69cd` correlates `OnWriteComplete` onto **plain**
|
|
`Write`/`Write2` replies; `b948e69` did so only for `WriteSecured`/`WriteSecured2`. So a plain
|
|
`Write` whose reply carries `statuses[0]` proves the worker is at or past `53f69cd`, and an empty
|
|
`statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal
|
|
still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`.
|
|
2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build
|
|
stamped. **Do not read this as *the* technique on its own.** What settled the 2026-08-11
|
|
investigation was two independent derivations agreeing: a PDB source-hash match, and a
|
|
contemporaneous deploy record written the same evening that named the same two commits. The
|
|
convergence is the result's strength, not either method alone — a hash match tells you which
|
|
sources a binary was built from, but not that the build was intentional or which host it went to.
|
|
A reader with only one of the two available should weight it accordingly and look for a second
|
|
line of evidence.
|
|
3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory
|
|
(`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup
|
|
directories carry operator-chosen labels naming the work (for example
|
|
`Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an
|
|
accidental or off-book deploy tends not to follow them.
|
|
|
|
4. **The host's own backup directories, read as a chain.** Each `Server.bak.<timestamp>` holds the
|
|
exe that deploy *replaced*, so a sweep of `VersionInfo` across them reconstructs the host's deploy
|
|
history from the host itself, with no repo access and no deploy record. A backup stamped
|
|
`20260811T060739` containing an exe written 2026-08-09 is the 08-09 build being displaced — the
|
|
backup's timestamp dates the *next* deploy, not the build inside it. Reading a file's version is
|
|
non-destructive, unlike opening a SQLite store in a backup directory, which mutates it. This is
|
|
what established that wonder's `b948e69` and `0a9715d` were two deploys two days apart rather
|
|
than two competing claims about one binary.
|
|
|
|
Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped
|
|
independently whenever the contracts are wire-identical, so a host legitimately runs one commit for
|
|
the server and a later one for the worker.
|
|
|
|
## Recorded deploys
|
|
|
|
| Date | Host | Server | Worker |
|
|
|---|---|---|---|
|
|
| 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` |
|
|
| 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` |
|
|
| 2026-08-11 | `wonder-app-vd03` | `0a9715d` (this deploy wrote `Server.bak.20260811T060739`, holding the displaced 08-09 build) | *carried forward* |
|
|
| 2026-08-12 | `wonder-app-vd03` | `55f2889` (this deploy wrote `Server.bak.20260812T040122`, holding `0a9715d`) | *carried forward* |
|
|
|
|
The two wonder rows after 08-09 are **server swaps**; their worker cells are carried forward from the
|
|
08-09 entry rather than re-verified, so treat the worker SHA there as unconfirmed. Their server SHAs
|
|
come from the backup-chain read described above (technique 4), except `55f2889`, which was read
|
|
directly from the live exe's stamp — trustworthy because it postdates `0152180`.
|
|
|
|
`b948e69` is **confirmed by PDB source-hash match plus the contemporaneous record, never by a version
|
|
stamp** — that build falls in the broken-stamp window and its stamp is structurally unavailable (see
|
|
the first section). `0a9715d` is the first wonder build to stamp cleanly, since `0152180` landed
|
|
before it.
|
|
|
|
The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
|
|
it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
|
|
the latter two minutes after `53f69cd` merged at 19:48, with a matching service stop/start at
|
|
19:50:29/34.
|
|
|
|
That reading is confirmable from artifacts still on disk, without trusting the narrative: windev
|
|
carries **two** worker backup directories from that day (`Worker.bak-20260809` and
|
|
`Worker.bak-20260809-planwrites`), and wonder carries `Worker.bak.20260809-planwrites`. Two backups
|
|
because there were two worker operations. This is easy to misread as redundancy — it is the second
|
|
swap's fingerprint.
|
|
|
|
## The 2026-08-09 deploy returned the worker to mainline
|
|
|
|
Worth stating because it went unrecorded at the time and later read as a mystery rather than as the
|
|
improvement it was. Before that deploy, production ran worker `dd7ca163` (2026-05-22), which is
|
|
contained **only** by `origin/test/client-e2e-coverage` and is not an ancestor of `main` — meaning
|
|
the x86 worker in production could not be rebuilt from any mainline commit. `53f69cd` is on `main`,
|
|
which closes that. Verified 2026-08-11 with `git merge-base --is-ancestor`.
|
|
|
|
## Related Documentation
|
|
|
|
- [Diagnostics](../Diagnostics.md)
|
|
- [Gateway Configuration](../GatewayConfiguration.md)
|