diff --git a/docs/Diagnostics.md b/docs/Diagnostics.md index 5246c88..7ea29d4 100644 --- a/docs/Diagnostics.md +++ b/docs/Diagnostics.md @@ -219,6 +219,7 @@ The order matters: putting the logging scope first ensures that authentication f ## Related Documentation +- [Identifying A Deployed Build](./runbooks/IdentifyingADeployedBuild.md) — mapping a running binary back to a commit, and why the `InformationalVersion` stamp cannot be trusted on Windows builds from 2026-07-09 to 2026-08-10 - [Sessions](./Sessions.md) - [gRPC](./Grpc.md) - [Authentication](./Authentication.md) diff --git a/docs/runbooks/IdentifyingADeployedBuild.md b/docs/runbooks/IdentifyingADeployedBuild.md new file mode 100644 index 0000000..916fc31 --- /dev/null +++ b/docs/runbooks/IdentifyingADeployedBuild.md @@ -0,0 +1,85 @@ +# Identifying A Deployed Build (Operator Runbook) + +> **Written 2026-08-11 after a false alarm.** An investigation treated the windev production +> binary as having no traceable provenance, on two pieces of evidence that both turned out to be +> normal output of our own build and deploy procedure. The binary was fine. This runbook records +> what those signals actually mean, so the next person spends minutes rather than a forensics pass. + +## The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10 + +`src/Directory.Build.props` appends the git short SHA to `InformationalVersion` (`0.1.2+`) so a +running binary can be mapped back to a commit. That stamping was introduced by `ec6f82b` +(2026-07-09, TST-11) and was **broken on Windows for its first month**. + +`$(MSBuildThisFileDirectory)` ends in a path separator. On Windows that trailing backslash escaped +the closing quote of the `Exec` command, so `git rev-parse` never ran correctly; because the target +runs with `ContinueOnError` and `ConsoleToMSBuild` (which mixes stderr into `ConsoleOutput`), git's +failure text was stamped as the source revision. The observed form is: + +``` +0.1.2+fatal: cannot change to ... +``` + +`0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a +trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on +a short-SHA shape so no future git failure text can become the revision either. + +**What this means for an operator.** A git error string in the version of a binary built on Windows +in that window is the *expected* result of our own build. It is non-diagnostic in **both** +directions — it neither incriminates a build nor confirms one, so it should not be treated as +evidence of anything. macOS builds in the same window stamp correctly, as do all builds after +`0152180`. + +## An absent `C:\build\mxgw-deploy` is expected + +The deploy procedure builds from a **detached worktree** (`git worktree add C:\build\mxgw-deploy +`) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The +worktree is removed once the publish is copied out. Finding that the directory a binary was built +from no longer exists is the normal end state of a correct deploy, not a deleted trail. + +## What does identify a build + +In rough order of cost: + +1. **Behaviour over the wire.** Works against a running service, needs no host access, and is the + fastest discriminator for the worker. `53f69cd` correlates `OnWriteComplete` onto **plain** + `Write`/`Write2` replies; `b948e69` did so only for `WriteSecured`/`WriteSecured2`. So a plain + `Write` whose reply carries `statuses[0]` proves the worker is at or past `53f69cd`, and an empty + `statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal + still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`. +2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build + stamped. This is what settled the 2026-08-11 investigation. +3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory + (`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup + directories carry operator-chosen labels naming the work (for example + `Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an + accidental or off-book deploy tends not to follow them. + +Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped +independently whenever the contracts are wire-identical, so a host legitimately runs one commit for +the server and a later one for the worker. + +## Recorded deploys + +| Date | Host | Server | Worker | +|---|---|---|---| +| 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` | +| 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` | + +The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe +it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06, +the latter two minutes after `53f69cd` merged at 19:48. The two worker backup directories from that +day are the second swap's fingerprint, not redundancy. + +## The 2026-08-09 deploy returned the worker to mainline + +Worth stating because it went unrecorded at the time and later read as a mystery rather than as the +improvement it was. Before that deploy, production ran worker `dd7ca163` (2026-05-22), which is +contained **only** by `origin/test/client-e2e-coverage` and is not an ancestor of `main` — meaning +the x86 worker in production could not be rebuilt from any mainline commit. `53f69cd` is on `main`, +which closes that. Verified 2026-08-11 with `git merge-base --is-ancestor`. + +## Related Documentation + +- [Diagnostics](../Diagnostics.md) +- [Gateway Configuration](../GatewayConfiguration.md)