Merge docs/deployed-build-identification: runbook for mapping a running binary to a commit
This commit is contained in:
@@ -219,6 +219,7 @@ The order matters: putting the logging scope first ensures that authentication f
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Identifying A Deployed Build](./runbooks/IdentifyingADeployedBuild.md) — mapping a running binary back to a commit, and why the `InformationalVersion` stamp cannot be trusted on Windows builds from 2026-07-09 to 2026-08-10
|
||||
- [Sessions](./Sessions.md)
|
||||
- [gRPC](./Grpc.md)
|
||||
- [Authentication](./Authentication.md)
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# Identifying A Deployed Build (Operator Runbook)
|
||||
|
||||
> **Written 2026-08-11 after a false alarm.** An investigation treated the windev production
|
||||
> binary as having no traceable provenance, on two pieces of evidence that both turned out to be
|
||||
> normal output of our own build and deploy procedure. The binary was fine. This runbook records
|
||||
> what those signals actually mean, so the next person spends minutes rather than a forensics pass.
|
||||
|
||||
## The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10
|
||||
|
||||
`src/Directory.Build.props` appends the git short SHA to `InformationalVersion` (`0.1.2+<sha>`) so a
|
||||
running binary can be mapped back to a commit. That stamping was introduced by `ec6f82b`
|
||||
(2026-07-09, TST-11) and was **broken on Windows for its first month**.
|
||||
|
||||
`$(MSBuildThisFileDirectory)` ends in a path separator. On Windows that trailing backslash escaped
|
||||
the closing quote of the `Exec` command, so `git rev-parse` never ran correctly; because the target
|
||||
runs with `ContinueOnError` and `ConsoleToMSBuild` (which mixes stderr into `ConsoleOutput`), git's
|
||||
failure text was stamped as the source revision. The observed form is:
|
||||
|
||||
```
|
||||
0.1.2+fatal: cannot change to ...
|
||||
```
|
||||
|
||||
`0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a
|
||||
trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on
|
||||
a short-SHA shape so no future git failure text can become the revision either.
|
||||
|
||||
**What this means for an operator.** A git error string in the version of a binary built on Windows
|
||||
in that window is the *expected* result of our own build. It is non-diagnostic in **both**
|
||||
directions — it neither incriminates a build nor confirms one, so it should not be treated as
|
||||
evidence of anything. macOS builds in the same window stamp correctly, as do all builds after
|
||||
`0152180`.
|
||||
|
||||
## An absent `C:\build\mxgw-deploy` is expected
|
||||
|
||||
The deploy procedure builds from a **detached worktree** (`git worktree add C:\build\mxgw-deploy
|
||||
<sha>`) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The
|
||||
worktree is removed once the publish is copied out. Finding that the directory a binary was built
|
||||
from no longer exists is the normal end state of a correct deploy, not a deleted trail.
|
||||
|
||||
## What does identify a build
|
||||
|
||||
In rough order of cost:
|
||||
|
||||
1. **Behaviour over the wire.** Works against a running service, needs no host access, and is the
|
||||
fastest discriminator for the worker. `53f69cd` correlates `OnWriteComplete` onto **plain**
|
||||
`Write`/`Write2` replies; `b948e69` did so only for `WriteSecured`/`WriteSecured2`. So a plain
|
||||
`Write` whose reply carries `statuses[0]` proves the worker is at or past `53f69cd`, and an empty
|
||||
`statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal
|
||||
still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`.
|
||||
2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build
|
||||
stamped. This is what settled the 2026-08-11 investigation.
|
||||
3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory
|
||||
(`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup
|
||||
directories carry operator-chosen labels naming the work (for example
|
||||
`Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an
|
||||
accidental or off-book deploy tends not to follow them.
|
||||
|
||||
Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped
|
||||
independently whenever the contracts are wire-identical, so a host legitimately runs one commit for
|
||||
the server and a later one for the worker.
|
||||
|
||||
## Recorded deploys
|
||||
|
||||
| Date | Host | Server | Worker |
|
||||
|---|---|---|---|
|
||||
| 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` |
|
||||
| 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` |
|
||||
|
||||
The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
|
||||
it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
|
||||
the latter two minutes after `53f69cd` merged at 19:48. The two worker backup directories from that
|
||||
day are the second swap's fingerprint, not redundancy.
|
||||
|
||||
## The 2026-08-09 deploy returned the worker to mainline
|
||||
|
||||
Worth stating because it went unrecorded at the time and later read as a mystery rather than as the
|
||||
improvement it was. Before that deploy, production ran worker `dd7ca163` (2026-05-22), which is
|
||||
contained **only** by `origin/test/client-e2e-coverage` and is not an ancestor of `main` — meaning
|
||||
the x86 worker in production could not be rebuilt from any mainline commit. `53f69cd` is on `main`,
|
||||
which closes that. Verified 2026-08-11 with `git merge-base --is-ancestor`.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Diagnostics](../Diagnostics.md)
|
||||
- [Gateway Configuration](../GatewayConfiguration.md)
|
||||
Reference in New Issue
Block a user