Merge docs/deployed-build-identification: runbook for mapping a running binary to a commit
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 3m26s
ci / portable (push) Successful in 11m7s

This commit is contained in:
Joseph Doherty
2026-08-11 06:05:23 -04:00
2 changed files with 86 additions and 0 deletions
+1
View File
@@ -219,6 +219,7 @@ The order matters: putting the logging scope first ensures that authentication f
## Related Documentation
- [Identifying A Deployed Build](./runbooks/IdentifyingADeployedBuild.md) — mapping a running binary back to a commit, and why the `InformationalVersion` stamp cannot be trusted on Windows builds from 2026-07-09 to 2026-08-10
- [Sessions](./Sessions.md)
- [gRPC](./Grpc.md)
- [Authentication](./Authentication.md)
@@ -0,0 +1,85 @@
# Identifying A Deployed Build (Operator Runbook)
> **Written 2026-08-11 after a false alarm.** An investigation treated the windev production
> binary as having no traceable provenance, on two pieces of evidence that both turned out to be
> normal output of our own build and deploy procedure. The binary was fine. This runbook records
> what those signals actually mean, so the next person spends minutes rather than a forensics pass.
## The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10
`src/Directory.Build.props` appends the git short SHA to `InformationalVersion` (`0.1.2+<sha>`) so a
running binary can be mapped back to a commit. That stamping was introduced by `ec6f82b`
(2026-07-09, TST-11) and was **broken on Windows for its first month**.
`$(MSBuildThisFileDirectory)` ends in a path separator. On Windows that trailing backslash escaped
the closing quote of the `Exec` command, so `git rev-parse` never ran correctly; because the target
runs with `ContinueOnError` and `ConsoleToMSBuild` (which mixes stderr into `ConsoleOutput`), git's
failure text was stamped as the source revision. The observed form is:
```
0.1.2+fatal: cannot change to ...
```
`0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a
trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on
a short-SHA shape so no future git failure text can become the revision either.
**What this means for an operator.** A git error string in the version of a binary built on Windows
in that window is the *expected* result of our own build. It is non-diagnostic in **both**
directions — it neither incriminates a build nor confirms one, so it should not be treated as
evidence of anything. macOS builds in the same window stamp correctly, as do all builds after
`0152180`.
## An absent `C:\build\mxgw-deploy` is expected
The deploy procedure builds from a **detached worktree** (`git worktree add C:\build\mxgw-deploy
<sha>`) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The
worktree is removed once the publish is copied out. Finding that the directory a binary was built
from no longer exists is the normal end state of a correct deploy, not a deleted trail.
## What does identify a build
In rough order of cost:
1. **Behaviour over the wire.** Works against a running service, needs no host access, and is the
fastest discriminator for the worker. `53f69cd` correlates `OnWriteComplete` onto **plain**
`Write`/`Write2` replies; `b948e69` did so only for `WriteSecured`/`WriteSecured2`. So a plain
`Write` whose reply carries `statuses[0]` proves the worker is at or past `53f69cd`, and an empty
`statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal
still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`.
2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build
stamped. This is what settled the 2026-08-11 investigation.
3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory
(`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup
directories carry operator-chosen labels naming the work (for example
`Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an
accidental or off-book deploy tends not to follow them.
Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped
independently whenever the contracts are wire-identical, so a host legitimately runs one commit for
the server and a later one for the worker.
## Recorded deploys
| Date | Host | Server | Worker |
|---|---|---|---|
| 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` |
| 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` |
The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
the latter two minutes after `53f69cd` merged at 19:48. The two worker backup directories from that
day are the second swap's fingerprint, not redundancy.
## The 2026-08-09 deploy returned the worker to mainline
Worth stating because it went unrecorded at the time and later read as a mystery rather than as the
improvement it was. Before that deploy, production ran worker `dd7ca163` (2026-05-22), which is
contained **only** by `origin/test/client-e2e-coverage` and is not an ancestor of `main` — meaning
the x86 worker in production could not be rebuilt from any mainline commit. `53f69cd` is on `main`,
which closes that. Verified 2026-08-11 with `git merge-base --is-ancestor`.
## Related Documentation
- [Diagnostics](../Diagnostics.md)
- [Gateway Configuration](../GatewayConfiguration.md)