Merge docs/deploy-provenance-rows: 08-11/08-12 wonder deploys recorded; backup dirs read as a provenance chain

docs(runbook): record the 08-11/08-12 wonder deploys and the backup-chain technique
Adds the two wonder rows that were deliberately withheld on 2026-08-11 while the pre-55f2889 SHA was unsettled. It is settled: b948e69 (08-09) and 0a9715d (08-11) were never competing claims about one binary, they are two deploys two days apart. What settled it is worth recording as a technique in its own right, so it goes in as a fourth way to identify a build: each Server.bak.<ts> holds the exe that deploy REPLACED, so a VersionInfo sweep across the backups reconstructs a host's deploy history from the host alone — no repo access, no deploy record. The subtlety that makes it readable is that a backup's timestamp dates the NEXT deploy, not the build inside it. Reading a file version is non-destructive, unlike opening a SQLite store in a backup directory. Also records the full garbage version stamp recovered from the 08-09 binary, because the failure mode is a false positive rather than a blank: "0.1.2+fatal:..." reads like a version that succeeded and then picked up noise, when the leading 0.1.2 is just the static base <Version> every build carries. For a binary in that window the commit is not recoverable from the binary at all, so finding nothing is the expected result rather than evidence against a SHA established another way. Provenance is stated per cell rather than uniformly: the worker SHAs on the new rows are carried forward and marked unconfirmed, b948e69 rests on PDB hash plus the contemporaneous record and never on a stamp, and 55f2889 was read from the live stamp, which is trustworthy only because it postdates 0152180.
2026-08-12 04:20:42 -04:00 · 2026-08-12 04:20:38 -04:00 · 2026-08-12 04:17:34 -04:00 · 2026-08-12 04:16:23 -04:00 · 2026-08-12 02:13:40 -04:00
8 changed files with 80 additions and 12 deletions
@@ -254,6 +254,7 @@ dev/test GLAuth posture (`glauth.md`), not a production posture.
 | `MxGateway:Ldap:UserNameAttribute` | `cn` | LDAP attribute holding the login user name. |
 | `MxGateway:Ldap:DisplayNameAttribute` | `cn` | LDAP attribute holding the display name. |
 | `MxGateway:Ldap:GroupAttribute` | `memberOf` | LDAP attribute enumerating group membership (mapped to dashboard roles via `MxGateway:Dashboard:GroupToRole`). |
+| `MxGateway:Ldap:FallbackServers` | *(empty)* | Ordered backup LDAP endpoints tried when the primary fails with a system-side error (connect/TLS, service-account bind, or search) — **not** when a user's credentials are simply wrong. Each entry is `host` (adopting `Port`) or `host:port`. Empty leaves single-endpoint behaviour exactly as before. Endpoint preference is sticky: the last endpoint that answered keeps being used until it fails. The `Transport` / `AllowInsecure` policy applies to every endpoint — a fallback is not a way to downgrade TLS. Entries are parsed at startup and a malformed one fails the boot, so a typo'd backup DC cannot lie dormant until the outage it exists to survive. Requires ZB.MOM.WW.Auth 0.2.0+. |

 When LDAP is enabled, `Server`, `SearchBase`, `ServiceAccountDn`,
 `ServiceAccountPassword`, and the attribute names must be non-blank, and `Port`
@@ -20,6 +20,20 @@ failure text was stamped as the source revision. The observed form is:
 0.1.2+fatal: cannot change to ...
 ```

+The full string recovered from wonder's 2026-08-09 server binary (read 2026-08-12) shows the whole
+failure, including the mismatched `'` … `"` that caused it:
+
+```
+0.1.2+fatal: cannot change to 'C:\build\mxgw-deploy\src" rev-parse --short HEAD': Invalid argument
+```
+
+**Do not read the leading `0.1.2` as provenance.** It is the static base `<Version>` every build
+carries, not a truncated SHA. The hazard is a false positive rather than a blank: `0.1.2+fatal:…`
+reads like a version that succeeded and then picked up noise, when in fact there is no usable
+identity anywhere in the string. For a binary built in this window the commit is **not recoverable
+from the binary at all** — so finding nothing is the expected result, not evidence against a SHA
+established another way.
+
 `0152180` (2026-08-10 05:49, merged in `c46e5bb`) fixed it two ways: the quoted path gained a
 trailing `.` so the separator can no longer escape the quote, and `SourceRevisionId` is now gated on
 a short-SHA shape so no future git failure text can become the revision either.
@@ -61,6 +75,15 @@ In rough order of cost:
   `Worker.bak-20260809-planwrites`). Those conventions place a build in time and intent, and an
   accidental or off-book deploy tends not to follow them.

+4. **The host's own backup directories, read as a chain.** Each `Server.bak.<timestamp>` holds the
+   exe that deploy *replaced*, so a sweep of `VersionInfo` across them reconstructs the host's deploy
+   history from the host itself, with no repo access and no deploy record. A backup stamped
+   `20260811T060739` containing an exe written 2026-08-09 is the 08-09 build being displaced — the
+   backup's timestamp dates the *next* deploy, not the build inside it. Reading a file's version is
+   non-destructive, unlike opening a SQLite store in a backup directory, which mutates it. This is
+   what established that wonder's `b948e69` and `0a9715d` were two deploys two days apart rather
+   than two competing claims about one binary.
+
 Note that **mixed Server and Worker SHAs are deliberate**, not drift: the two are swapped
 independently whenever the contracts are wire-identical, so a host legitimately runs one commit for
 the server and a later one for the worker.
@@ -71,6 +94,18 @@ the server and a later one for the worker.
 |---|---|---|---|
 | 2026-08-09 | windev (`10.100.0.48`) | `b948e69` (`Server-20260809`) | `53f69cd` |
 | 2026-08-09 | `wonder-app-vd03` | `b948e69` | `53f69cd` |
+| 2026-08-11 | `wonder-app-vd03` | `0a9715d` (this deploy wrote `Server.bak.20260811T060739`, holding the displaced 08-09 build) | *carried forward* |
+| 2026-08-12 | `wonder-app-vd03` | `55f2889` (this deploy wrote `Server.bak.20260812T040122`, holding `0a9715d`) | *carried forward* |
+
+The two wonder rows after 08-09 are **server swaps**; their worker cells are carried forward from the
+08-09 entry rather than re-verified, so treat the worker SHA there as unconfirmed. Their server SHAs
+come from the backup-chain read described above (technique 4), except `55f2889`, which was read
+directly from the live exe's stamp — trustworthy because it postdates `0152180`.
+
+`b948e69` is **confirmed by PDB source-hash match plus the contemporaneous record, never by a version
+stamp** — that build falls in the broken-stamp window and its stamp is structurally unavailable (see
+the first section). `0a9715d` is the first wonder build to stamp cleanly, since `0152180` landed
+before it.

 The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
 it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
@@ -22,8 +22,8 @@
    (IntegrationTests-028).
  -->
  <ItemGroup>
-    <PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.5" />
-    <PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.5" />
+    <PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.2.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="10.0.7" />
  </ItemGroup>
@@ -11,4 +11,5 @@ public sealed record EffectiveLdapConfiguration(
    string ServiceAccountPassword,
    string UserNameAttribute,
    string DisplayNameAttribute,
-    string GroupAttribute);
+    string GroupAttribute,
+    IReadOnlyList<string> FallbackServers);
@@ -30,7 +30,8 @@ public sealed class GatewayConfigurationProvider(IOptions<GatewayOptions> option
                ServiceAccountPassword: RedactedValue,
                UserNameAttribute: value.Ldap.UserNameAttribute,
                DisplayNameAttribute: value.Ldap.DisplayNameAttribute,
-                GroupAttribute: value.Ldap.GroupAttribute),
+                GroupAttribute: value.Ldap.GroupAttribute,
+                FallbackServers: value.Ldap.FallbackServers),
            Worker: new EffectiveWorkerConfiguration(
                ExecutablePath: value.Worker.ExecutablePath,
                WorkingDirectory: value.Worker.WorkingDirectory,
@@ -68,4 +68,19 @@ public sealed class LdapOptions

    /// <summary>Gets the LDAP attribute name for group membership.</summary>
    public string GroupAttribute { get; init; } = "memberOf";
+
+    /// <summary>
+    /// Gets the ordered fallback LDAP endpoints (<c>"host"</c> or <c>"host:port"</c>) the shared
+    /// provider walks when the primary fails with a system-side error. Empty (the default) leaves
+    /// single-endpoint behaviour unchanged. Mirrors
+    /// <see cref="ZB.MOM.WW.Auth.Abstractions.Ldap.LdapOptions.FallbackServers"/>, added in
+    /// ZB.MOM.WW.Auth 0.2.0.
+    /// <para>
+    /// Carried here only so the effective-config display does not hide a configured backup DC —
+    /// nothing on the gateway side reads it. Entry syntax is validated at boot by the shared
+    /// <c>LdapOptionsValidator</c>, which owns the (internal) parser; re-validating here would
+    /// mean a second, drifting copy of that grammar.
+    /// </para>
+    /// </summary>
+    public IReadOnlyList<string> FallbackServers { get; init; } = [];
 }
@@ -26,6 +26,21 @@ else
                    <tr><th scope="row">Run migrations</th><td>@Snapshot.Configuration.Authentication.RunMigrationsOnStartup</td></tr>
                    <tr><th scope="row">LDAP enabled</th><td>@Snapshot.Configuration.Ldap.Enabled</td></tr>
                    <tr><th scope="row">LDAP server</th><td>@Snapshot.Configuration.Ldap.Server:@Snapshot.Configuration.Ldap.Port</td></tr>
+                    <tr>
+                        <th scope="row">LDAP fallback servers</th>
+                        @* Rendered even when empty: "none" is the operationally interesting answer
+                           on a host someone believes has a backup DC configured. *@
+                        <td>
+                            @if (Snapshot.Configuration.Ldap.FallbackServers.Count == 0)
+                            {
+                                <span class="text-muted">none</span>
+                            }
+                            else
+                            {
+                                <code>@string.Join(", ", Snapshot.Configuration.Ldap.FallbackServers)</code>
+                            }
+                        </td>
+                    </tr>
                    <tr><th scope="row">LDAP transport</th><td>@Snapshot.Configuration.Ldap.Transport</td></tr>
                    <tr><th scope="row">LDAP search base</th><td><code>@Snapshot.Configuration.Ldap.SearchBase</code></td></tr>
                    <tr><th scope="row">LDAP service account</th><td><code>@Snapshot.Configuration.Ldap.ServiceAccountDn</code></td></tr>
@@ -10,20 +10,20 @@

  <ItemGroup>
    <PackageReference Include="Grpc.AspNetCore" Version="2.76.0" />
-    <PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.5" />
-    <PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.5" />
-    <PackageReference Include="ZB.MOM.WW.Auth.ApiKeys" Version="0.1.5" />
-    <PackageReference Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
+    <PackageReference Include="ZB.MOM.WW.Auth.Abstractions" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Auth.Ldap" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Auth.ApiKeys" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.2.0" />
    <PackageReference Include="ZB.MOM.WW.Audit" Version="0.1.0" />
    <PackageReference Include="ZB.MOM.WW.Theme" Version="0.4.1" />
    <PackageReference Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
-    <PackageReference Include="ZB.MOM.WW.Health" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Health" Version="0.3.0" />
    <PackageReference Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
    <PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
    <PackageReference Include="ZB.MOM.WW.GalaxyRepository" Version="0.2.0" />
-    <PackageReference Include="ZB.MOM.WW.Secrets" Version="0.6.1" />
-    <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.6.1" />
-    <PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.1" />
+    <PackageReference Include="ZB.MOM.WW.Secrets" Version="0.6.2" />
+    <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.6.2" />
+    <PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
    <PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
    <PackageReference Include="Serilog.Sinks.File" Version="7.0.0" />
Author	SHA1	Message	Date
Joseph Doherty	55bca95ad2	Merge docs/deploy-provenance-rows: 08-11/08-12 wonder deploys recorded; backup dirs read as a provenance chain ci / nightly-windev (push) Has been skipped Details ci / windows-x86 (push) Successful in 1m28s Details ci / java (push) Successful in 2m46s Details ci / portable (push) Successful in 8m27s Details	2026-08-12 04:20:42 -04:00
Joseph Doherty	5fe96b6677	docs(runbook): record the 08-11/08-12 wonder deploys and the backup-chain technique Adds the two wonder rows that were deliberately withheld on 2026-08-11 while the pre-55f2889 SHA was unsettled. It is settled: `b948e69` (08-09) and `0a9715d` (08-11) were never competing claims about one binary, they are two deploys two days apart. What settled it is worth recording as a technique in its own right, so it goes in as a fourth way to identify a build: each Server.bak.<ts> holds the exe that deploy REPLACED, so a VersionInfo sweep across the backups reconstructs a host's deploy history from the host alone — no repo access, no deploy record. The subtlety that makes it readable is that a backup's timestamp dates the NEXT deploy, not the build inside it. Reading a file version is non-destructive, unlike opening a SQLite store in a backup directory. Also records the full garbage version stamp recovered from the 08-09 binary, because the failure mode is a false positive rather than a blank: "0.1.2+fatal:..." reads like a version that succeeded and then picked up noise, when the leading 0.1.2 is just the static base <Version> every build carries. For a binary in that window the commit is not recoverable from the binary at all, so finding nothing is the expected result rather than evidence against a SHA established another way. Provenance is stated per cell rather than uniformly: the worker SHAs on the new rows are carried forward and marked unconfirmed, `b948e69` rests on PDB hash plus the contemporaneous record and never on a stamp, and `55f2889` was read from the live stamp, which is trustworthy only because it postdates `0152180`.	2026-08-12 04:20:38 -04:00
Joseph Doherty	2c0daee481	Merge chore/shared-lib-latest-pins: every ZB.MOM.WW pin to newest published; Auth 0.2.0 obliged the LdapOptions shadow mirror ci / nightly-windev (push) Has been skipped Details ci / windows-x86 (push) Successful in 1m13s Details ci / java (push) Successful in 2m24s Details ci / portable (push) Successful in 9m24s Details	2026-08-12 04:17:34 -04:00
Joseph Doherty	62394f5b85	chore(deps): move every ZB.MOM.WW pin to the newest published version ci / nightly-windev (push) Has been skipped Details ci / windows-x86 (push) Successful in 1m11s Details ci / java (push) Successful in 2m5s Details ci / portable (push) Successful in 9m15s Details Auth 0.1.5 -> 0.2.0, Health 0.2.0 -> 0.3.0, Secrets/.Abstractions/.Ui 0.6.1 -> 0.6.2. Theme, GalaxyRepository, Audit, Configuration, Telemetry and Telemetry.Serilog were already at the newest version on the feed. Checked against the shared-lib source rather than the version numbers, because these packages are versioned as a family and a bump is not by itself evidence that the package changed: - Auth 0.2.0 is the only one carrying content for us: LDAP backup-DC failover (FallbackServers, endpoint walk with sticky preference, boot-time entry validation). Purely additive; the default is empty, which leaves single-endpoint behaviour unchanged. - Health 0.3.0 carries a breaking change, but every line of it is in ZB.MOM.WW.Health.Akka, which we do not reference. No commit touched the core ZB.MOM.WW.Health package between 0.2.0 and 0.3.0. - Secrets 0.6.2 is a message-only change: one validator string literal gains mounted-volume guidance. SecretsStorePathRules is untouched. The four non-csproj files are not a separate feature. Configuration/ LdapOptions is a deliberate shadow of the shared type and carries an explicit warning to mirror any new upstream field, because AddZbLdapAuth binds the whole MxGateway:Ldap section onto the shared options. So FallbackServers is live on our config surface the moment the package lands, and without the mirror an operator could configure a backup DC that works but is invisible on the dashboard's Settings page. The Settings row renders "none" when empty, since that is the answer someone who believes a backup DC is configured actually needs. Entry syntax is deliberately NOT re-validated here: the shared validator already fails the boot on a malformed entry and owns the (internal) parser, so a second copy would drift. Note both validators skip entirely when Ldap:Enabled is false. Verified the binder is non-strict (ErrorOnUnknownConfiguration is unused anywhere in the tree), so the upgrade could not break startup on a newly-recognised key either way. Build 0 warnings / 0 errors; gateway suite 892/892, unchanged. The live LDAP tests are opt-in and were not run, so the failover path itself is covered only by the shared library's own tests.	2026-08-12 04:16:23 -04:00
Joseph Doherty	55f2889c24	Merge fix/secrets-prehost-content-root: run the store-path guard where the store is actually created ci / nightly-windev (push) Has been skipped Details ci / windows-x86 (push) Successful in 1m17s Details ci / java (push) Successful in 3m12s Details ci / portable (push) Successful in 10m36s Details 0.6.0 landed the rules but not the enforcement on this path: the pre-host secrets container has no IHostEnvironment, so the library skipped the content-root check and the migrator created the database before the real host could refuse it. The pin to 0.6.1 alone does not close that — the call site has to pass the content root explicitly, which is why this is a code change and not a version bump.	2026-08-12 02:13:40 -04:00