Merge docs/deployed-build-identification: runbook for mapping a running binary to a commit

2026-08-11 06:05:23 -04:00
parent 0a9715d819 fd0e88e74c
commit e5dbcee17c
2 changed files with 86 additions and 0 deletions
@@ -219,6 +219,7 @@ The order matters: putting the logging scope first ensures that authentication f
 ## Related Documentation
 - [Identifying A Deployed Build](./runbooks/IdentifyingADeployedBuild.md) — mapping a running binary back to a commit, and why the `InformationalVersion` stamp cannot be trusted on Windows builds from 2026-07-09 to 2026-08-10
 - [Sessions](./Sessions.md)
 - [gRPC](./Grpc.md)
 - [Authentication](./Authentication.md)
@@ -0,0 +1,85 @@
 # Identifying A Deployed Build (Operator Runbook)
 > **Written 2026-08-11 after a false alarm.** An investigation treated the windev production
 > binary as having no traceable provenance, on two pieces of evidence that both turned out to be
 > normal output of our own build and deploy procedure. The binary was fine. This runbook records
 > what those signals actually mean, so the next person spends minutes rather than a forensics pass.
 ## The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10
 `src/Directory.Build.props` appends the git short SHA to `InformationalVersion` (`0.1.2+<sha>`) so a
 running binary can be mapped back to a commit. That stamping was introduced by `ec6f82b`
 (2026-07-09, TST-11) and was **broken on Windows for its first month**.
 `$(MSBuildThisFileDirectory)` ends in a path separator. On Windows that trailing backslash escaped
 the closing quote of the `Exec` command, so `git rev-parse` never ran correctly; because the target
 runs with `ContinueOnError` and `ConsoleToMSBuild` (which mixes stderr into `ConsoleOutput`), git's
 failure text was stamped as the source revision. The observed form is:
 ```
 0.1.2+fatal: cannot change to ...
 ```
 `0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a
 trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on
 a short-SHA shape so no future git failure text can become the revision either.
 **What this means for an operator.** A git error string in the version of a binary built on Windows
 in that window is the *expected* result of our own build. It is non-diagnostic in **both**
 directions — it neither incriminates a build nor confirms one, so it should not be treated as
 evidence of anything. macOS builds in the same window stamp correctly, as do all builds after
 `0152180`.
 ## An absent `C:\build\mxgw-deploy` is expected
 The deploy procedure builds from a **detached worktree** (`git worktree add C:\build\mxgw-deploy
 <sha>`) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The
 worktree is removed once the publish is copied out. Finding that the directory a binary was built
 from no longer exists is the normal end state of a correct deploy, not a deleted trail.
 ## What does identify a build
 In rough order of cost:
 1. **Behaviour over the wire.** Works against a running service, needs no host access, and is the
   fastest discriminator for the worker. `53f69cd` correlates `OnWriteComplete` onto **plain**
   `Write`/`Write2` replies; `b948e69` did so only for `WriteSecured`/`WriteSecured2`. So a plain
   `Write` whose reply carries `statuses[0]` proves the worker is at or past `53f69cd`, and an empty
   `statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal
   still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`.
 2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build
   stamped. This is what settled the 2026-08-11 investigation.
 3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory
   (`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup
   directories carry operator-chosen labels naming the work (for example
   `Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an
   accidental or off-book deploy tends not to follow them.
 Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped
 independently whenever the contracts are wire-identical, so a host legitimately runs one commit for
 the server and a later one for the worker.
 ## Recorded deploys
 | Date | Host | Server | Worker |
 |---|---|---|---|
 | 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` |
 | 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` |
 The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
 it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
 the latter two minutes after `53f69cd` merged at 19:48. The two worker backup directories from that
 day are the second swap's fingerprint, not redundancy.
 ## The 2026-08-09 deploy returned the worker to mainline
 Worth stating because it went unrecorded at the time and later read as a mystery rather than as the
 improvement it was. Before that deploy, production ran worker `dd7ca163` (2026-05-22), which is
 contained **only** by `origin/test/client-e2e-coverage` and is not an ancestor of `main` — meaning
 the x86 worker in production could not be rebuilt from any mainline commit. `53f69cd` is on `main`,
 which closes that. Verified 2026-08-11 with `git merge-base --is-ancestor`.
 ## Related Documentation
 - [Diagnostics](../Diagnostics.md)
 - [Gateway Configuration](../GatewayConfiguration.md)