Compare commits

...

4 Commits

Author SHA1 Message Date
Joseph Doherty f6b6184e70 docs(runbook): state what the identification evidence actually rests on
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m55s
ci / portable (push) Successful in 10m39s
Two precision fixes to the runbook landed in fd0e88e, both making it weaker in
the sense that matters.

The PDB source-hash entry read as though hashes settled the 2026-08-09
provenance question by themselves. They did not: a hash match and a
contemporaneous deploy record written that evening independently named the same
two commits, and the agreement is what makes the result trustworthy. A hash
match alone tells you which sources a binary was built from — not that the build
was intentional, nor which host it reached. A future reader holding only one of
the two derivations should know to look for a second.

The two-swap timeline was asserted from the investigation's timestamps, which a
later reader cannot re-derive. It is also confirmable from artifacts still on
disk — windev keeps two worker backup directories from that day and wonder one,
because there were two worker operations. Records that, so the story can be
checked against the boxes rather than believed.
2026-08-11 06:06:48 -04:00
Joseph Doherty e5dbcee17c Merge docs/deployed-build-identification: runbook for mapping a running binary to a commit
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 3m26s
ci / portable (push) Successful in 11m7s
2026-08-11 06:05:23 -04:00
Joseph Doherty fd0e88e74c docs(runbook): how to identify a deployed build, and why the version stamp lies
ci / windows-x86 (push) Successful in 1m19s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 3m24s
ci / portable (push) Failing after 6m14s
A 2026-08-11 investigation treated the windev production binary as having no
traceable provenance. Both pieces of evidence were normal output of our own
build and deploy procedure, and the binary was fine — but nothing in the repo
said so, so it cost a forensics pass to establish.

The version stamp is the first trap. SHA stamping landed in ec6f82b (TST-11)
and was broken on Windows until 0152180 (NEXT-09) a month later: the trailing
backslash in MSBuildThisFileDirectory escaped the Exec's closing quote, and
because the target runs ContinueOnError with ConsoleToMSBuild, git's stderr was
stamped as the revision. Every Windows build in that window reads
`0.1.2+fatal: cannot change to ...`. The point an operator needs is that this
is non-diagnostic in *both* directions — it neither incriminates a build nor
confirms one — which is not obvious from a stamp that looks like a failure.

The second is that the build directory is *supposed* to be gone: deploys build
from a detached worktree so the host's checkout stays on its own branch, and
remove it afterwards.

Records what does identify a build instead, cheapest first — including the wire
probe for the worker, since plain Write/Write2 carrying statuses[0] separates
53f69cd from b948e69 without host access or symbols — plus the deploys to date.

Also records something that went unlogged and is the reason this read as a
mystery rather than an improvement: the 2026-08-09 deploy returned the x86
worker to mainline. Production had been running dd7ca163, contained only by
origin/test/client-e2e-coverage and not an ancestor of main, so the worker in
production was not rebuildable from any mainline commit.
2026-08-11 06:05:19 -04:00
Joseph Doherty 0a9715d819 Merge feat/dashboard-ui-cleanup-sweep: admin-UI cleanup pass + ZB.MOM.WW.Theme 0.4.1
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m28s
ci / portable (push) Successful in 9m25s
The Theme bump is separable from the sweep and lands as its own commit: 0.4.0
upstreamed the button-sizing block the sweep had installed locally, and this
app's local .btn rule was trimmed rather than deleted because it also carried
border-radius/font-weight/white-space that the kit does not ship.
2026-08-11 05:49:11 -04:00
2 changed files with 98 additions and 0 deletions
+1
View File
@@ -219,6 +219,7 @@ The order matters: putting the logging scope first ensures that authentication f
## Related Documentation ## Related Documentation
- [Identifying A Deployed Build](./runbooks/IdentifyingADeployedBuild.md) — mapping a running binary back to a commit, and why the `InformationalVersion` stamp cannot be trusted on Windows builds from 2026-07-09 to 2026-08-10
- [Sessions](./Sessions.md) - [Sessions](./Sessions.md)
- [gRPC](./Grpc.md) - [gRPC](./Grpc.md)
- [Authentication](./Authentication.md) - [Authentication](./Authentication.md)
@@ -0,0 +1,97 @@
# Identifying A Deployed Build (Operator Runbook)
> **Written 2026-08-11 after a false alarm.** An investigation treated the windev production
> binary as having no traceable provenance, on two pieces of evidence that both turned out to be
> normal output of our own build and deploy procedure. The binary was fine. This runbook records
> what those signals actually mean, so the next person spends minutes rather than a forensics pass.
## The version stamp is unreliable on Windows builds from 2026-07-09 to 2026-08-10
`src/Directory.Build.props` appends the git short SHA to `InformationalVersion` (`0.1.2+<sha>`) so a
running binary can be mapped back to a commit. That stamping was introduced by `ec6f82b`
(2026-07-09, TST-11) and was **broken on Windows for its first month**.
`$(MSBuildThisFileDirectory)` ends in a path separator. On Windows that trailing backslash escaped
the closing quote of the `Exec` command, so `git rev-parse` never ran correctly; because the target
runs with `ContinueOnError` and `ConsoleToMSBuild` (which mixes stderr into `ConsoleOutput`), git's
failure text was stamped as the source revision. The observed form is:
```
0.1.2+fatal: cannot change to ...
```
`0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a
trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on
a short-SHA shape so no future git failure text can become the revision either.
**What this means for an operator.** A git error string in the version of a binary built on Windows
in that window is the *expected* result of our own build. It is non-diagnostic in **both**
directions — it neither incriminates a build nor confirms one, so it should not be treated as
evidence of anything. macOS builds in the same window stamp correctly, as do all builds after
`0152180`.
## An absent `C:\build\mxgw-deploy` is expected
The deploy procedure builds from a **detached worktree** (`git worktree add C:\build\mxgw-deploy
<sha>`) so the host's own checkout, which usually sits on a feature branch, is not disturbed. The
worktree is removed once the publish is copied out. Finding that the directory a binary was built
from no longer exists is the normal end state of a correct deploy, not a deleted trail.
## What does identify a build
In rough order of cost:
1. **Behaviour over the wire.** Works against a running service, needs no host access, and is the
fastest discriminator for the worker. `53f69cd` correlates `OnWriteComplete` onto **plain**
`Write`/`Write2` replies; `b948e69` did so only for `WriteSecured`/`WriteSecured2`. So a plain
`Write` whose reply carries `statuses[0]` proves the worker is at or past `53f69cd`, and an empty
`statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal
still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`.
2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build
stamped. **Do not read this as *the* technique on its own.** What settled the 2026-08-11
investigation was two independent derivations agreeing: a PDB source-hash match, and a
contemporaneous deploy record written the same evening that named the same two commits. The
convergence is the result's strength, not either method alone — a hash match tells you which
sources a binary was built from, but not that the build was intentional or which host it went to.
A reader with only one of the two available should weight it accordingly and look for a second
line of evidence.
3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory
(`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup
directories carry operator-chosen labels naming the work (for example
`Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an
accidental or off-book deploy tends not to follow them.
Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped
independently whenever the contracts are wire-identical, so a host legitimately runs one commit for
the server and a later one for the worker.
## Recorded deploys
| Date | Host | Server | Worker |
|---|---|---|---|
| 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` |
| 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` |
The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
the latter two minutes after `53f69cd` merged at 19:48, with a matching service stop/start at
19:50:29/34.
That reading is confirmable from artifacts still on disk, without trusting the narrative: windev
carries **two** worker backup directories from that day (`Worker.bak-20260809` and
`Worker.bak-20260809-planwrites`), and wonder carries `Worker.bak.20260809-planwrites`. Two backups
because there were two worker operations. This is easy to misread as redundancy — it is the second
swap's fingerprint.
## The 2026-08-09 deploy returned the worker to mainline
Worth stating because it went unrecorded at the time and later read as a mystery rather than as the
improvement it was. Before that deploy, production ran worker `dd7ca163` (2026-05-22), which is
contained **only** by `origin/test/client-e2e-coverage` and is not an ancestor of `main` — meaning
the x86 worker in production could not be rebuilt from any mainline commit. `53f69cd` is on `main`,
which closes that. Verified 2026-08-11 with `git merge-base --is-ancestor`.
## Related Documentation
- [Diagnostics](../Diagnostics.md)
- [Gateway Configuration](../GatewayConfiguration.md)