Compare commits

..

6 Commits

Author SHA1 Message Date
Joseph Doherty 6ba52a68f0 build(secrets): re-pin ZB.MOM.WW.Secrets to 0.6.0
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m18s
ci / java (push) Successful in 2m23s
ci / portable (push) Successful in 8m14s
0.6.0 moves the store-path rules into the shared library — both the rooted check
and the content-root check — with a SecretsOptionsValidator wired into
AddZbSecrets and validated on start, so all four consuming apps get the guard
from one implementation rather than four copies. mxgw therefore adds no local
validator over the Secrets section.

This is a four-minor jump, not a re-pin: mxgw was on 0.2.3 and skips 0.3.0,
0.4.0, 0.4.1, 0.5.0 and 0.5.1 in one step. Verified rather than assumed —
diffing the 0.2.3 and 0.6.0 assemblies shows the added surface is the new path
rules and their plumbing (AddIfNotRooted, AddIfUnderContentRoot,
ComputeDefaultSqlitePath, DefaultSqlitePath, IValidateOptions, IsPathRooted,
GetFullPath) and nothing touching store or delete behaviour. Secrets.Ui is
byte-identical across the range: both assemblies are 39424 bytes and differ only
in the version stamp, because this package family shares one version across
every package even when a release touches only one of them. So the browser gate
run against the /admin/secrets delete modal on 0.2.3 still covers what ships
here.

The library default is LocalApplicationData-derived so the family's
cross-platform apps still boot locally. The gateway keeps its own
CommonApplicationData value, which always wins — see the note on
ApplyDefaultSecretsStorePath for why the difference is deliberate and why
deleting that method as a redundancy would silently move the store.
2026-08-11 08:42:28 -04:00
Joseph Doherty 882c7ca3cd fix(config): reject credential and cache paths inside the application directory
Rooted is not the same as safe, and the gap between the two cost a production
host every one of its API keys on 2026-08-09.

MxGateway:Authentication:SqlitePath was set to an absolute path inside the
directory the upgrade procedure renames to Server.bak.*. That passes the
existing rooted check cleanly. The deploy renamed the directory away, the store
went with it, and the gateway created a fresh empty one at the same path — no
error, no log line. No gRPC consumer could authenticate for two days. The deploy
itself was correct: the binaries were the point of the rename and the store was
collateral.

GatewayConfigPathRules gains AddIfUnderContentRoot, applied to the auth store
and the Galaxy snapshot. Both are written by the running process and both are
lost the same way. The rule compares resolved full paths and requires a
directory-separator boundary, so a sibling directory whose name merely starts
with the content root's ("/srv/app-data" against "/srv/app") is not treated as
inside it — on a fail-closed startup rule, that false positive would be a
gateway that refuses to boot on a legitimate path. Case sensitivity follows the
running OS rather than assuming case-insensitivity everywhere, which would
reject /srv/App as under /srv/app on Linux where they are different directories.

The rule is not exempted in Development. An environment-conditional guard is
never exercised where the mistake is made, and what failed in production was a
config that looked fine.

Secrets:SqlitePath is the same defect one layer down: it shipped as a bare
relative "mxgateway-secrets.db", which is how a stray database landed in
src/…Server/ and tripped the repository's tree-hygiene test. It is bound by the
shared ZB.MOM.WW.Secrets package, so appsettings.json now ships no value and the
default is computed from CommonApplicationData in code — the same mechanism
SEC-33 already used for the Galaxy snapshot, ten lines away, for the same reason.
Setting a default for an unset key is deliberately not the same act as
relocating a value someone configured, which these rules still refuse to do.

Note the migration edge this creates: a host relying on the old repo default now
looks somewhere new, finds nothing, and creates an empty store — this bug
re-introduced by its own fix. Deployed hosts are safe because they set the path
explicitly, in appsettings copied forward or in the service environment. The
latter is the more robust of the two, since it cannot be lost by a missed
preserve step.
2026-08-11 08:42:16 -04:00
Joseph Doherty c69a1c441b feat(diagnostics): report MXAccess session health on the active probe
Adds a `mxaccess-sessions` health check reporting how many MXAccess sessions are
healthy. Each session is one worker process holding one MXAccess COM instance —
a live connection into a Galaxy — so this answers "how many Galaxy connections
are healthy" in the vocabulary the code actually uses.

Zero sessions is Healthy, deliberately, and the rest of the design follows from
that. The gateway opens a session when a client asks and holds none otherwise,
so an idle gateway is working normally. A count threshold ("unhealthy below N")
would sit red forever on a host nothing dials yet, and a permanently red probe
is one operators stop reading — which leaves them worse off than no probe. The
check therefore grades on whether the sessions that exist are usable: nothing
faulted is Healthy, some faulted beside a ready or starting one is Degraded, and
every session faulted is Unhealthy. Counts ride along as entry data for the
family Overview dashboard.

Tagged `active` rather than `ready` for the same reason. Readiness decides
whether the process should be sent traffic, and a gateway with no sessions is
ready to serve — unlike the auth store, which every call depends on. Failing
readiness here would pull a working gateway out of rotation over a condition its
own clients create.

Reads ISessionRegistry, which already exposes Snapshot(); ISessionManager stays
the command surface and grows no enumerator.
2026-08-11 08:41:54 -04:00
Joseph Doherty 22a34f7f31 Merge docs/runbook-evidence-precision: what the deployed-build identification evidence rests on
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m51s
ci / portable (push) Successful in 10m51s
2026-08-11 06:06:48 -04:00
Joseph Doherty f6b6184e70 docs(runbook): state what the identification evidence actually rests on
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m55s
ci / portable (push) Successful in 10m39s
Two precision fixes to the runbook landed in fd0e88e, both making it weaker in
the sense that matters.

The PDB source-hash entry read as though hashes settled the 2026-08-09
provenance question by themselves. They did not: a hash match and a
contemporaneous deploy record written that evening independently named the same
two commits, and the agreement is what makes the result trustworthy. A hash
match alone tells you which sources a binary was built from — not that the build
was intentional, nor which host it reached. A future reader holding only one of
the two derivations should know to look for a second.

The two-swap timeline was asserted from the investigation's timestamps, which a
later reader cannot re-derive. It is also confirmable from artifacts still on
disk — windev keeps two worker backup directories from that day and wonder one,
because there were two worker operations. Records that, so the story can be
checked against the boxes rather than believed.
2026-08-11 06:06:48 -04:00
Joseph Doherty e5dbcee17c Merge docs/deployed-build-identification: runbook for mapping a running binary to a commit
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 3m26s
ci / portable (push) Successful in 11m7s
2026-08-11 06:05:23 -04:00
13 changed files with 626 additions and 32 deletions
+24
View File
@@ -217,6 +217,30 @@ The order matters: putting the logging scope first ensures that authentication f
- `DashboardRedactor.Redact` delegates to `RedactClientIdentity` for any value containing the `mxgw_` marker, then falls back to a marker-keyword check for fields like `password` or `token`. This keeps dashboard renders aligned with log redaction.
- `ZB.MOM.WW.MxGateway.Tests/Diagnostics/GatewayLogRedactorTests.cs` covers each redaction branch, including the assertion that `WriteSecured` values stay redacted even when `valueLoggingEnabled` is true.
## Health Checks
The shared `ZB.MOM.WW.Health` package maps three endpoints — `/healthz` (live), `/health/ready`, and
`/health/active` — and each registered check opts into a tier by tag. The gateway registers two:
| Check | Endpoint tier | Fails when |
|---|---|---|
| `auth-store` | `ready` | The SQLite auth store cannot be opened. Every gRPC call authenticates against it, so its reachability genuinely gates whether the process should receive traffic. |
| `mxaccess-sessions` | `active` | Sessions exist and their workers have faulted. Reports `total` / `ready` / `faulted` / `starting` / `closing` as entry `data`. |
**Zero sessions is Healthy, and the tier choice follows from that.** The gateway opens an MXAccess
session when a client asks for one and holds none otherwise, so an idle gateway is working normally,
not broken. A count threshold ("unhealthy below N") would sit red forever on a host nothing dials
yet, and a permanently red probe is one operators stop reading — which leaves them worse off than no
probe at all. `mxaccess-sessions` is therefore graded on whether the sessions that exist are usable:
- nothing faulted → **Healthy** (including no sessions at all)
- some faulted, some still ready or starting → **Degraded**
- every session faulted → **Unhealthy**
For the same reason it is tagged `active` rather than `ready`. Readiness decides whether the process
should be sent traffic, and a gateway with no sessions is ready to serve; failing readiness there
would pull a working gateway out of rotation over a condition its clients create.
## Related Documentation
- [Identifying A Deployed Build](./runbooks/IdentifyingADeployedBuild.md) — mapping a running binary back to a commit, and why the `InformationalVersion` stamp cannot be trusted on Windows builds from 2026-07-09 to 2026-08-10
+3 -3
View File
@@ -91,7 +91,7 @@ Environment variables use the normal .NET double-underscore form. For example,
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Authentication:Mode` | `ApiKey` | Selects public gRPC authentication. Supported values are `ApiKey` and `Disabled`. `Disabled` bypasses API-key verification and is for local development only. |
| `MxGateway:Authentication:SqlitePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\gateway-auth.db` on Windows, `/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) | SQLite database path for API-key records and audit rows when API-key authentication is enabled. The code default is built from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` so the credential store never lands in the launch working directory on a non-Windows host. `appsettings.json` no longer ships an explicit value (SEC-33): the removed Windows literal was byte-identical to the Windows code default, and a Windows-absolute literal is **not** rooted on a Unix host, so it would have resolved against the CWD there. Deployed hosts still override the path through the NSSM environment (`MxGateway__Authentication__SqlitePath`). |
| `MxGateway:Authentication:SqlitePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\gateway-auth.db` on Windows, `/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) | SQLite database path for API-key records and audit rows when API-key authentication is enabled. The code default is built from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` so the credential store never lands in the launch working directory on a non-Windows host. `appsettings.json` no longer ships an explicit value (SEC-33): the removed Windows literal was byte-identical to the Windows code default, and a Windows-absolute literal is **not** rooted on a Unix host, so it would have resolved against the CWD there. Deployed hosts still override the path through the NSSM environment (`MxGateway__Authentication__SqlitePath`). The validator additionally rejects a path **inside the application content root**, even an absolute one: the upgrade procedure renames that directory to `Server.bak.*`, which takes the credential store with it and silently starts an empty one. That is not hypothetical — it happened on a production host on 2026-08-09 and no gRPC client could authenticate for two days. |
| `MxGateway:Authentication:PepperSecretName` | `MxGateway:ApiKeyPepper` | Configuration key used to read the HMAC pepper for API-key secret hashing. The dashboard effective configuration redacts this value. |
| `MxGateway:Authentication:RunMigrationsOnStartup` | `true` | Runs SQLite auth schema migrations at gateway startup when API-key authentication is enabled. |
@@ -291,7 +291,7 @@ section (a sibling of `MxGateway`, not nested under it):
| Option | Default | Description |
|--------|---------|-------------|
| `Secrets:SqlitePath` | `mxgateway-secrets.db` | Path to the encrypted secrets store, resolved relative to the app content root when not rooted. |
| `Secrets:SqlitePath` | `<CommonApplicationData>/MxGateway/mxgateway-secrets.db` | Path to the encrypted secrets store. The default is supplied in code when the key is unset (`C:\ProgramData\MxGateway\...` on Windows), not from `appsettings.json` — a store inside the application directory is renamed away by the upgrade procedure, taking the secrets with it. On non-Windows hosts the default location is usually not writable by a normal user, so a local run must set `Secrets__SqlitePath` explicitly. |
| `Secrets:MasterKey:Source` | `Environment` | Key-encryption-key (KEK) provider. `Environment` reads a base64-encoded 32-byte key from an env var; `Dpapi` uses a machine-bound key file instead (see below). |
| `Secrets:MasterKey:EnvVarName` | `ZB_SECRETS_MASTER_KEY` | Env var name the `Environment` provider reads the KEK from. |
@@ -402,7 +402,7 @@ model requires otherwise.
| `MxGateway:Galaxy:CommandTimeoutSeconds` | `60` | Per-command SQL timeout for all Galaxy browse RPCs. |
| `MxGateway:Galaxy:DashboardRefreshIntervalSeconds` | `30` | Interval between background refreshes of the dashboard Galaxy summary cache. SQL is hit at most once per interval regardless of dashboard render rate. |
| `MxGateway:Galaxy:PersistSnapshot` | `true` | Persists the latest successful Galaxy browse dataset to disk. When `true`, the cache reloads that snapshot at startup so clients can still browse last-known data while the Galaxy database is unreachable. The restored data is served with `Stale` status until a live query confirms it. |
| `MxGateway:Galaxy:SnapshotCachePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\galaxy-snapshot.json` on Windows, `/usr/share/MxGateway/galaxy-snapshot.json` or the container equivalent elsewhere) | File path for the persisted Galaxy browse snapshot. Ignored when `PersistSnapshot` is `false`. The snapshot is written atomically (temp file plus rename). `appsettings.json` no longer ships an explicit value (SEC-33): the option is bound by the shared `ZB.MOM.WW.GalaxyRepository` package, so the gateway supplies the `CommonApplicationData`-derived default when the bound value is blank and registers `GalaxyRepositoryOptionsValidator` to enforce that — when `PersistSnapshot` is `true` — the path is non-blank, valid, and **rooted on the host running the gateway** (`Path.IsPathRooted`, current OS). A bare filename or a foreign-platform literal fails startup instead of resolving against the launch working directory (SEC-01, SEC-33). |
| `MxGateway:Galaxy:SnapshotCachePath` | derived from `CommonApplicationData` (`C:\ProgramData\MxGateway\galaxy-snapshot.json` on Windows, `/usr/share/MxGateway/galaxy-snapshot.json` or the container equivalent elsewhere) | File path for the persisted Galaxy browse snapshot. Ignored when `PersistSnapshot` is `false`. The snapshot is written atomically (temp file plus rename). `appsettings.json` no longer ships an explicit value (SEC-33): the option is bound by the shared `ZB.MOM.WW.GalaxyRepository` package, so the gateway supplies the `CommonApplicationData`-derived default when the bound value is blank and registers `GalaxyRepositoryOptionsValidator` to enforce that — when `PersistSnapshot` is `true` — the path is non-blank, valid, and **rooted on the host running the gateway** (`Path.IsPathRooted`, current OS). A bare filename or a foreign-platform literal fails startup instead of resolving against the launch working directory (SEC-01, SEC-33). The same validator also rejects a path **inside the application content root**, because the upgrade procedure renames that directory away and the cached snapshot would be discarded on every deploy. |
See [Galaxy Repository Browse](./GalaxyRepository.md) for the RPC surface and
behavior.
+15 -3
View File
@@ -48,7 +48,13 @@ In rough order of cost:
`statuses` proves it is not. Keep it non-destructive by writing to a read-only tag — the refusal
still exercises the path and returns `OPERATIONAL_ERROR` with detail `1007`.
2. **PDB source hashes.** Slower, needs the deployed symbols, but independent of anything the build
stamped. This is what settled the 2026-08-11 investigation.
stamped. **Do not read this as *the* technique on its own.** What settled the 2026-08-11
investigation was two independent derivations agreeing: a PDB source-hash match, and a
contemporaneous deploy record written the same evening that named the same two commits. The
convergence is the result's strength, not either method alone — a hash match tells you which
sources a binary was built from, but not that the build was intentional or which host it went to.
A reader with only one of the two available should weight it accordingly and look for a second
line of evidence.
3. **Deployment-side naming.** Since 2026-08-07 the server deploys to a dated directory
(`Server-YYYYMMDD`) with the NSSM `Application`/`AppDirectory` repointed at it, and backup
directories carry operator-chosen labels naming the work (for example
@@ -68,8 +74,14 @@ the server and a later one for the worker.
The 2026-08-09 deploy was **two separate swaps**, which is why a single build time does not describe
it: the 2026-08-11 investigation dated the server file write to 19:20:24 and the worker to 19:50:06,
the latter two minutes after `53f69cd` merged at 19:48. The two worker backup directories from that
day are the second swap's fingerprint, not redundancy.
the latter two minutes after `53f69cd` merged at 19:48, with a matching service stop/start at
19:50:29/34.
That reading is confirmable from artifacts still on disk, without trusting the narrative: windev
carries **two** worker backup directories from that day (`Worker.bak-20260809` and
`Worker.bak-20260809-planwrites`), and wonder carries `Worker.bak.20260809-planwrites`. Two backups
because there were two worker operations. This is easy to misread as redundancy — it is the second
swap's fingerprint.
## The 2026-08-09 deploy returned the worker to mainline
@@ -13,6 +13,33 @@ namespace ZB.MOM.WW.MxGateway.Server.Configuration;
/// </summary>
public sealed class GalaxyRepositoryOptionsValidator : OptionsValidatorBase<GalaxyRepositoryOptions>
{
// See GatewayOptionsValidator for why this is nullable and what null means.
private readonly string? _contentRootPath;
/// <summary>
/// Initializes a new instance of the <see cref="GalaxyRepositoryOptionsValidator"/> class for
/// the dependency-injection path, taking the content root from the host environment.
/// </summary>
/// <param name="environment">The host environment.</param>
public GalaxyRepositoryOptionsValidator(IHostEnvironment environment)
{
ArgumentNullException.ThrowIfNull(environment);
_contentRootPath = environment.ContentRootPath;
}
/// <summary>
/// Initializes a new instance of the <see cref="GalaxyRepositoryOptionsValidator"/> class for
/// unit tests and non-DI callers.
/// </summary>
/// <param name="contentRootPath">
/// Content root to test the snapshot path against; <see langword="null"/> leaves the
/// content-root rule inactive.
/// </param>
internal GalaxyRepositoryOptionsValidator(string? contentRootPath = null)
{
_contentRootPath = contentRootPath;
}
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, GalaxyRepositoryOptions options)
{
@@ -37,5 +64,10 @@ public sealed class GalaxyRepositoryOptionsValidator : OptionsValidatorBase<Gala
options.SnapshotCachePath,
"MxGateway:Galaxy:SnapshotCachePath must be an absolute (rooted) path so the Galaxy snapshot never lands in the launch working directory.",
builder);
GatewayConfigPathRules.AddIfUnderContentRoot(
options.SnapshotCachePath,
_contentRootPath,
$"MxGateway:Galaxy:SnapshotCachePath must not be inside the application directory ({_contentRootPath}). The upgrade procedure renames that directory, so the cached snapshot is discarded on every deploy and the gateway starts cold.",
builder);
}
}
@@ -53,25 +53,102 @@ internal static class GatewayConfigPathRules
return;
}
try
{
_ = Path.GetFullPath(value);
}
catch (ArgumentException)
{
builder.Add(message);
}
catch (NotSupportedException)
{
builder.Add(message);
}
catch (PathTooLongException)
{
builder.Add(message);
}
catch (IOException)
if (!TryGetFullPath(value, out _))
{
builder.Add(message);
}
}
/// <summary>
/// Fails validation when <paramref name="value"/> resolves to a location inside
/// <paramref name="contentRoot"/> — the directory the application runs from.
/// </summary>
/// <remarks>
/// <para>
/// <b>Rooted is not the same as safe, and this is the rule that closes the gap.</b>
/// <see cref="AddIfNotRooted"/> stops a store drifting with the working directory, but an
/// absolute path <em>inside the app directory</em> passes it cleanly — and that is what failed
/// in production on 2026-08-09. The upgrade procedure renames the app directory to
/// <c>Server.bak.*</c> and unpacks a new one; a store living there is renamed away with it, the
/// process then creates a fresh empty one at the same path, and nothing reports an error. All
/// API keys were lost and no gRPC client could authenticate for two days. The deploy itself was
/// executed correctly — the binaries were the point of the rename, and the store was collateral.
/// </para>
/// <para>
/// The same shape catches the dev-side symptom: a store under the content root lands in the
/// source tree, which is how <c>mxgateway-secrets.db</c> once tripped the repository's
/// tree-hygiene test.
/// </para>
/// <para>
/// Comparison is case-insensitive only on Windows. On a case-insensitive macOS volume this can
/// miss a violation that differs only in case, which is a missed warning in dev; assuming
/// case-insensitivity on Linux would instead reject a legitimate path, and a false startup
/// abort is the worse failure.
/// </para>
/// </remarks>
/// <param name="value">The configured path value.</param>
/// <param name="contentRoot">The application content root to test against.</param>
/// <param name="message">The failure message to record when the value is under the content root.</param>
/// <param name="builder">The validation builder accumulating failures.</param>
public static void AddIfUnderContentRoot(
string? value,
string? contentRoot,
string message,
ValidationBuilder builder)
{
if (string.IsNullOrWhiteSpace(value) || string.IsNullOrWhiteSpace(contentRoot))
{
return;
}
// A malformed path is AddIfInvalidPath's message to report; staying silent here keeps one
// bad value from producing two failures that say different things about the same mistake.
if (!TryGetFullPath(value, out string fullValue) || !TryGetFullPath(contentRoot, out string fullRoot))
{
return;
}
fullRoot = fullRoot.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);
StringComparison comparison = OperatingSystem.IsWindows()
? StringComparison.OrdinalIgnoreCase
: StringComparison.Ordinal;
// The separator is load-bearing: a bare prefix test would also match a sibling directory
// whose name merely starts with the root's ("/srv/app" against "/srv/app-data").
if (string.Equals(fullValue, fullRoot, comparison)
|| fullValue.StartsWith(fullRoot + Path.DirectorySeparatorChar, comparison))
{
builder.Add(message);
}
}
private static bool TryGetFullPath(string value, out string fullPath)
{
try
{
fullPath = Path.GetFullPath(value);
return true;
}
catch (ArgumentException)
{
fullPath = string.Empty;
return false;
}
catch (NotSupportedException)
{
fullPath = string.Empty;
return false;
}
catch (PathTooLongException)
{
fullPath = string.Empty;
return false;
}
catch (IOException)
{
fullPath = string.Empty;
return false;
}
}
}
@@ -15,15 +15,22 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
// rather than merely warn. Non-production hosts keep the permissive dev posture.
private readonly bool _isProduction;
// The application content root. Store paths must not live under it — see
// GatewayConfigPathRules.AddIfUnderContentRoot. Null for non-DI callers that supply no
// environment, which skips the rule rather than inventing a root to test against.
private readonly string? _contentRootPath;
/// <summary>
/// Initializes a new instance of the <see cref="GatewayOptionsValidator"/> class for the
/// dependency-injection path, deriving the production posture from the host environment.
/// dependency-injection path, deriving the production posture and content root from the host
/// environment.
/// </summary>
/// <param name="environment">The host environment.</param>
public GatewayOptionsValidator(IHostEnvironment environment)
{
ArgumentNullException.ThrowIfNull(environment);
_isProduction = environment.IsProduction();
_contentRootPath = environment.ContentRootPath;
}
/// <summary>
@@ -32,15 +39,20 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
/// hard-stops do not fire; pass <see langword="true"/> to exercise them.
/// </summary>
/// <param name="isProduction">Whether to treat the host as running in Production.</param>
internal GatewayOptionsValidator(bool isProduction = false)
/// <param name="contentRootPath">
/// Content root to test store paths against; <see langword="null"/> leaves the content-root
/// rule inactive, which is what a caller with no real host wants.
/// </param>
internal GatewayOptionsValidator(bool isProduction = false, string? contentRootPath = null)
{
_isProduction = isProduction;
_contentRootPath = contentRootPath;
}
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, GatewayOptions options)
{
ValidateAuthentication(options.Authentication, builder);
ValidateAuthentication(options.Authentication, _contentRootPath, builder);
ValidateLdap(options.Ldap, builder, _isProduction);
ValidateWorker(options.Worker, builder);
ValidateSessions(options.Sessions, builder);
@@ -101,7 +113,10 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
builder);
}
private static void ValidateAuthentication(AuthenticationOptions options, ValidationBuilder builder)
private static void ValidateAuthentication(
AuthenticationOptions options,
string? contentRootPath,
ValidationBuilder builder)
{
if (!Enum.IsDefined(options.Mode))
{
@@ -123,6 +138,11 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
options.SqlitePath,
"MxGateway:Authentication:SqlitePath must be an absolute (rooted) path so the credential store never lands in the launch working directory.",
builder);
AddIfUnderContentRoot(
options.SqlitePath,
contentRootPath,
$"MxGateway:Authentication:SqlitePath must not be inside the application directory ({contentRootPath}). The upgrade procedure renames that directory, which abandons the credential store and silently starts an empty one — every API key is lost and no client can authenticate.",
builder);
AddIfBlank(
options.PepperSecretName,
"MxGateway:Authentication:PepperSecretName is required when API-key authentication is enabled.",
@@ -555,4 +575,14 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
private static void AddIfInvalidPath(string? value, string message, ValidationBuilder builder)
=> GatewayConfigPathRules.AddIfInvalidPath(value, message, builder);
// Rooted is not the same as safe: an absolute path inside the app directory passes
// AddIfNotRooted and is still renamed away by the upgrade procedure. See
// GatewayConfigPathRules.AddIfUnderContentRoot.
private static void AddIfUnderContentRoot(
string? value,
string? contentRoot,
string message,
ValidationBuilder builder)
=> GatewayConfigPathRules.AddIfUnderContentRoot(value, contentRoot, message, builder);
}
@@ -0,0 +1,114 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Sessions;
namespace ZB.MOM.WW.MxGateway.Server.Diagnostics;
/// <summary>
/// Reports how many MXAccess sessions are healthy. Each session is one worker process holding one
/// MXAccess COM instance — a live connection into a Galaxy — so this is the "how many Galaxy
/// connections are healthy" probe, expressed in the vocabulary the code actually uses.
/// </summary>
/// <remarks>
/// <para>
/// <b>Zero sessions is healthy, deliberately.</b> The gateway is a server: it opens a session when
/// a client asks and holds none otherwise, so idle-with-no-clients is the normal steady state, not
/// a fault. A count-based rule ("unhealthy below N") would sit red forever on a host nothing dials
/// yet, and a probe that is permanently red is one people learn to ignore — which costs more than
/// having no probe. The status here is therefore false only when a session exists and its worker
/// has actually failed.
/// </para>
/// <para>
/// This is tagged <c>active</c> rather than <c>ready</c> for the same reason. Readiness gates
/// whether the process should receive traffic, and a gateway with no sessions is legitimately ready
/// to serve — unlike the auth store, which every call depends on (see
/// <see cref="AuthStoreHealthCheck"/>). Failing readiness on session state would take a working
/// gateway out of rotation for a condition its own clients cause.
/// </para>
/// </remarks>
public sealed class SessionHealthCheck : IHealthCheck
{
private readonly ISessionRegistry _sessionRegistry;
/// <summary>Initializes a new instance of the <see cref="SessionHealthCheck"/> class.</summary>
/// <param name="sessionRegistry">Registry holding the live sessions.</param>
public SessionHealthCheck(ISessionRegistry sessionRegistry) =>
_sessionRegistry = sessionRegistry ?? throw new ArgumentNullException(nameof(sessionRegistry));
/// <summary>Buckets the live sessions by state and grades the result.</summary>
/// <param name="context">The health check context.</param>
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
/// <returns>
/// Healthy when nothing is faulted (including when no sessions are open), Degraded when some
/// sessions are faulted but others are still usable, and Unhealthy when every session is
/// faulted.
/// </returns>
public Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context,
CancellationToken cancellationToken = default)
{
int ready = 0;
int faulted = 0;
int starting = 0;
int closing = 0;
foreach (GatewaySession session in _sessionRegistry.Snapshot())
{
switch (session.State)
{
case SessionState.Ready:
ready++;
break;
case SessionState.Faulted:
faulted++;
break;
case SessionState.Closing:
case SessionState.Closed:
// Counted but excluded from the verdict: a session on its way out is an
// expected lifecycle stage, not a failure, and Snapshot() still returns
// Closed sessions until they are removed from the registry.
closing++;
break;
default:
// Creating / StartingWorker / WaitingForPipe / Handshaking /
// InitializingWorker — mid-startup, not yet usable but not wrong.
// Unspecified lands here too; it is the proto zero value and should not occur.
starting++;
break;
}
}
int total = ready + faulted + starting + closing;
int usable = ready + starting;
Dictionary<string, object> data = new(StringComparer.Ordinal)
{
["total"] = total,
["ready"] = ready,
["faulted"] = faulted,
["starting"] = starting,
["closing"] = closing,
};
HealthCheckResult result = (faulted, usable) switch
{
(0, _) => HealthCheckResult.Healthy(Describe(total, ready, faulted), data),
(_, 0) => HealthCheckResult.Unhealthy(Describe(total, ready, faulted), data: data),
_ => HealthCheckResult.Degraded(Describe(total, ready, faulted), data: data),
};
return Task.FromResult(result);
}
private static string Describe(int total, int ready, int faulted)
{
if (total == 0)
{
return "No MXAccess sessions are open.";
}
return faulted == 0
? $"{ready} of {total} MXAccess sessions ready."
: $"{ready} of {total} MXAccess sessions ready, {faulted} faulted.";
}
}
@@ -70,6 +70,8 @@ public static class GatewayApplication
});
StaticWebAssetsLoader.UseStaticWebAssets(builder.Environment, builder.Configuration);
ApplyDefaultSecretsStorePath(builder.Configuration);
// Resolve ${secret:...} references in configuration BEFORE any config consumer (TLS, Kestrel,
// GatewayOptions/Ldap/Galaxy validators) reads a value, using a standalone secrets provider
// (envelope-decrypted via the master key). A token referencing a missing secret fails fast
@@ -106,7 +108,13 @@ public static class GatewayApplication
.AddTypeActivatedCheck<AuthStoreHealthCheck>(
"auth-store",
failureStatus: null,
tags: new[] { ZbHealthTags.Ready });
tags: new[] { ZbHealthTags.Ready })
// Active, not Ready: a gateway holding no sessions is legitimately ready to serve.
// See SessionHealthCheck for why zero sessions is healthy.
.AddTypeActivatedCheck<SessionHealthCheck>(
"mxaccess-sessions",
failureStatus: null,
tags: new[] { ZbHealthTags.Active });
builder.Services.AddSingleton<GatewayMetrics>();
builder.AddZbTelemetry(o =>
{
@@ -180,6 +188,60 @@ public static class GatewayApplication
});
}
/// <summary>
/// Supplies the default location of the encrypted secrets store when nothing configured one.
/// </summary>
/// <remarks>
/// <para>
/// The store used to default to a bare relative <c>mxgateway-secrets.db</c>, which resolves
/// against the working directory and therefore normally lands inside the application directory.
/// That is the shape that lost every API key on a production host: the upgrade procedure renames
/// the application directory away, the store goes with it, and a fresh empty one appears in its
/// place with no error. In development the same default writes a database into the source tree.
/// </para>
/// <para>
/// This sets a default for an <em>unset</em> key; it never relocates a value someone configured.
/// That distinction matters — <see cref="Configuration.GatewayConfigPathRules"/> deliberately
/// rejects bad configured paths rather than quietly moving them, because silently relocating a
/// credential store is worse than a boot error. Choosing where to put a value nobody specified
/// is a different act from overriding one they did.
/// </para>
/// <para>
/// The location mirrors <c>AuthenticationOptions.SqlitePath</c> so both gateway stores sit
/// together, and the mechanism is the one SEC-33 already used for
/// <c>MxGateway:Galaxy:SnapshotCachePath</c> below — same problem, same fix, same file. It also
/// matches what <c>docs/GatewayConfiguration.md</c> already tells operators to
/// pass to the <c>secret</c> CLI — an absolute default also removes the CLI/gateway divergence
/// that a working-directory-relative path can cause. On non-Windows hosts
/// <see cref="Environment.SpecialFolder.CommonApplicationData"/> is typically not writable by a
/// normal user, so a local run there must set <c>Secrets__SqlitePath</c> explicitly, exactly as
/// it already must for the auth store.
/// </para>
/// <para>
/// <b>This deliberately differs from the <c>ZB.MOM.WW.Secrets</c> library default</b>, which is
/// <see cref="Environment.SpecialFolder.LocalApplicationData"/>-derived so the family's
/// cross-platform apps still boot locally without an override. The gateway keeps
/// <c>CommonApplicationData</c> because it runs as a machine-wide Windows service and its other
/// two stores — the auth database and the Galaxy snapshot — already live there; splitting them
/// would be the greater inconsistency. The value set here always wins, so the library default is
/// unreachable in this app. Do not "fix" the difference by deleting this method: that would
/// silently move the store, which is the failure this whole rule exists to prevent.
/// </para>
/// </remarks>
/// <param name="configuration">The configuration to supply the default into.</param>
private static void ApplyDefaultSecretsStorePath(IConfiguration configuration)
{
if (!string.IsNullOrWhiteSpace(configuration["Secrets:SqlitePath"]))
{
return;
}
configuration["Secrets:SqlitePath"] = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"MxGateway",
"mxgateway-secrets.db");
}
private static void ConfigureSelfSignedTls(WebApplicationBuilder builder)
{
if (!Security.Tls.KestrelTlsInspector.RequiresGeneratedCertificate(builder.Configuration))
@@ -21,9 +21,9 @@
<PackageReference Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
<PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
<PackageReference Include="ZB.MOM.WW.GalaxyRepository" Version="0.2.0" />
<PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageReference Include="ZB.MOM.WW.Secrets" Version="0.6.0" />
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.6.0" />
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.0" />
<PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
<PackageReference Include="Serilog.Sinks.File" Version="7.0.0" />
@@ -12,7 +12,6 @@
},
"AllowedHosts": "*",
"Secrets": {
"SqlitePath": "mxgateway-secrets.db",
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
"RunMigrationsOnStartup": true,
"ResolveCacheTtl": "00:00:30"
@@ -63,6 +63,49 @@ public sealed class GalaxyRepositoryOptionsValidatorTests
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies an absolute snapshot path inside the application directory fails. Rooted is not the
/// same as safe: the upgrade procedure renames that directory, so a snapshot cached there is
/// discarded on every deploy and the gateway starts cold each time.
/// </summary>
[Fact]
public void Validate_Fails_WhenSnapshotPathIsUnderContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GalaxyRepositoryOptions options = new()
{
PersistSnapshot = true,
SnapshotCachePath = Path.Combine(contentRoot, "galaxy-snapshot.json"),
};
ValidateOptionsResult result =
new GalaxyRepositoryOptionsValidator(contentRoot).Validate(null, options);
Assert.True(result.Failed);
Assert.Contains(
result.Failures!,
f => f.Contains("MxGateway:Galaxy:SnapshotCachePath")
&& f.Contains("must not be inside the application directory"));
}
/// <summary>Verifies a snapshot path outside the application directory still passes.</summary>
[Fact]
public void Validate_Succeeds_WhenSnapshotPathIsOutsideContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GalaxyRepositoryOptions options = new()
{
PersistSnapshot = true,
SnapshotCachePath =
Path.Combine(Path.GetTempPath(), $"mxgw-data-{Guid.NewGuid():N}", "galaxy-snapshot.json"),
};
ValidateOptionsResult result =
new GalaxyRepositoryOptionsValidator(contentRoot).Validate(null, options);
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies the gateway supplies a rooted per-OS default when the shipped config leaves
/// SnapshotCachePath blank, so the removed appsettings literal is not needed and validation
@@ -598,6 +598,69 @@ public sealed class GatewayOptionsValidatorTests
f => f.Contains("MxGateway:Authentication:SqlitePath") && f.Contains("rooted"));
}
/// <summary>
/// Verifies an absolute auth DB path <em>inside</em> the application directory fails. This is
/// the gap the rooted check does not close: the path that lost every API key on a production
/// host on 2026-08-09 was absolute and passed rooting cleanly — it simply lived in the directory
/// the upgrade procedure renames away.
/// </summary>
[Fact]
public void Validate_Fails_WhenSqlitePathIsUnderContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GatewayOptions options = CloneWithAuthentication(
ValidOptions(),
new AuthenticationOptions { SqlitePath = Path.Combine(contentRoot, "gateway-auth.db") });
ValidateOptionsResult result =
new GatewayOptionsValidator(contentRootPath: contentRoot).Validate(null, options);
Assert.True(result.Failed);
Assert.Contains(
result.Failures!,
f => f.Contains("MxGateway:Authentication:SqlitePath")
&& f.Contains("must not be inside the application directory"));
}
/// <summary>
/// Verifies the content-root rule is not a bare string prefix test: a sibling directory whose
/// name merely begins with the content root's must still pass.
/// </summary>
[Fact]
public void Validate_Succeeds_WhenSqlitePathIsSiblingOfContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GatewayOptions options = CloneWithAuthentication(
ValidOptions(),
new AuthenticationOptions { SqlitePath = contentRoot + "-data" + Path.DirectorySeparatorChar + "gateway-auth.db" });
ValidateOptionsResult result =
new GatewayOptionsValidator(contentRootPath: contentRoot).Validate(null, options);
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies a store path outside the application directory passes — the rule must reject only
/// the genuinely unsafe location, not every absolute path.
/// </summary>
[Fact]
public void Validate_Succeeds_WhenSqlitePathIsOutsideContentRoot()
{
string contentRoot = Path.Combine(Path.GetTempPath(), $"mxgw-root-{Guid.NewGuid():N}");
GatewayOptions options = CloneWithAuthentication(
ValidOptions(),
new AuthenticationOptions
{
SqlitePath = Path.Combine(Path.GetTempPath(), $"mxgw-data-{Guid.NewGuid():N}", "gateway-auth.db"),
});
ValidateOptionsResult result =
new GatewayOptionsValidator(contentRootPath: contentRoot).Validate(null, options);
Assert.True(result.Succeeded);
}
/// <summary>
/// Verifies rooting is host-meaningful (SEC-33): a Windows drive-qualified literal fails on a
/// Unix host (where it is not rooted) rather than being blessed and written as a junk-named
@@ -0,0 +1,138 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Diagnostics;
using ZB.MOM.WW.MxGateway.Server.Sessions;
namespace ZB.MOM.WW.MxGateway.Tests.Diagnostics;
public sealed class SessionHealthCheckTests
{
/// <summary>
/// An idle gateway is healthy. This is the load-bearing case: a gateway holding no sessions is
/// the normal steady state on a host nothing dials yet, and a probe that reports red there is
/// one operators learn to ignore.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Healthy_WhenNoSessionsAreOpen()
{
var check = new SessionHealthCheck(new SessionRegistry());
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Healthy, result.Status);
Assert.Equal(0, result.Data["total"]);
Assert.Equal("No MXAccess sessions are open.", result.Description);
}
/// <summary>Every session ready reports healthy, with the counts carried as entry data.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Healthy_WhenAllSessionsReady()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Ready, SessionState.Ready));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Healthy, result.Status);
Assert.Equal(2, result.Data["total"]);
Assert.Equal(2, result.Data["ready"]);
Assert.Equal(0, result.Data["faulted"]);
}
/// <summary>
/// A faulted session alongside a usable one is degraded, not unhealthy — the gateway is still
/// serving the sessions that work.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Degraded_WhenSomeFaultedAndSomeReady()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Ready, SessionState.Faulted));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Degraded, result.Status);
Assert.Equal(1, result.Data["ready"]);
Assert.Equal(1, result.Data["faulted"]);
Assert.Contains("1 faulted", result.Description, StringComparison.Ordinal);
}
/// <summary>
/// A session still starting counts as usable for grading, so a fault beside it is degraded
/// rather than unhealthy — the startup has not failed yet.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Degraded_WhenFaultedBesideAStartingSession()
{
var check = new SessionHealthCheck(
RegistryWith(SessionState.Faulted, SessionState.StartingWorker));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Degraded, result.Status);
Assert.Equal(1, result.Data["starting"]);
}
/// <summary>Every session faulted is the genuinely bad condition, and the only unhealthy one.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Unhealthy_WhenEverySessionIsFaulted()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Faulted, SessionState.Faulted));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Unhealthy, result.Status);
Assert.Equal(2, result.Data["faulted"]);
Assert.Equal(0, result.Data["ready"]);
}
/// <summary>
/// Closed sessions linger in the registry until they are removed. They are counted separately
/// and excluded from the verdict, so a gateway whose sessions all closed cleanly is healthy —
/// not unhealthy for having zero ready ones.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Healthy_WhenOnlyClosedSessionsRemain()
{
var check = new SessionHealthCheck(RegistryWith(SessionState.Closed, SessionState.Closed));
HealthCheckResult result = await check.CheckHealthAsync(new HealthCheckContext());
Assert.Equal(HealthStatus.Healthy, result.Status);
Assert.Equal(2, result.Data["closing"]);
Assert.Equal(0, result.Data["ready"]);
}
private static SessionRegistry RegistryWith(params SessionState[] states)
{
var registry = new SessionRegistry();
for (int i = 0; i < states.Length; i++)
{
GatewaySession session = CreateSession($"session-{i}");
session.TransitionTo(states[i]);
Assert.True(registry.TryAdd(session));
}
return registry;
}
private static GatewaySession CreateSession(string sessionId)
{
return new GatewaySession(
sessionId,
"mxaccess",
$"mxaccess-gateway-1-{sessionId}",
"nonce",
clientIdentity: null,
clientSessionName: "test-session",
clientCorrelationId: "client-correlation",
TimeSpan.FromSeconds(30),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(5),
DateTimeOffset.UnixEpoch);
}
}