Decrement was decrement-first with a single non-retried repair CAS. From zero,
two unmatched decrements (SignalR calls OnDisconnectedAsync for a connection
whose OnConnectedAsync faulted) capture -1 and -2; a real Increment then makes
the count -1, and the first decrementer's stale CompareExchange(0, -1) matches
and resets to zero — erasing a live connection, so the idle gate freezes an open
dashboard. The same lost race also made Decrement report 0 when it had not
written 0.
Clamping now happens inside the compare-and-swap: read, clamp, publish, retry on
loss. A lost race re-reads the fresh value instead of repairing a stale one.
The counter moves to its own file per the one-public-type-per-file convention and
gains direct tests: the zero floor under concurrent unmatched decrements, matched
pairs settling at zero, and an interleaved connect/disconnect stress round. The
stress test asserts the observable invariants only — the specific interleaving
cannot be forced through the public API (verified: the previous implementation
passes it), which its remarks now state rather than implying a reproducer. A hub
wiring test is skipped for the EventsHub reason: driving Hub.OnConnectedAsync
needs caller-clients and connection-context fakes, and the overrides are two
lines of delegation to the tested type.
Also documents that the API-key refresh's pre-gate time check races benignly.
77c5731 committed this file from a working copy that predated 75e3dc2's
advised-set section, silently deleting it. Puts the paragraph back verbatim;
no other content changes.
The snapshot publisher broadcast to Clients.All on every ~1s tick forever, with
zero viewers. Each tick cost a session-registry snapshot and sort, a metrics
snapshot that copies dictionaries under the global metrics lock, a full rebuild
of EffectiveGatewayConfiguration, and a SQLite read of the API key table.
DashboardSnapshotHub now counts live connections into the singleton
DashboardSnapshotHubConnectionCounter (clamped at zero, since SignalR can call
OnDisconnectedAsync for a connection whose OnConnectedAsync faulted). The
publisher drives the snapshot enumerator by hand instead of await foreach: with
no connections it does not call MoveNextAsync at all, so the producing iterator
stays suspended at its yield and no snapshot is built — the gate removes the
build, not just the broadcast. It re-checks once a second, so the first viewer
resumes the tick within about one interval; that viewer is seeded immediately by
DashboardPageBase's synchronous GetSnapshot() and by the hub's OnConnectedAsync.
Two per-tick costs are bounded independently of the gate: the effective
configuration is startup-static (options are bound once at boot and never
reloaded), so it is built once and cached; and the API key summaries refresh at
most every 15s, since the list only changes when an operator creates, rotates,
or revokes a key. Only a successful refresh restarts the interval, so a failed
or timed-out read is still retried on the next tick with the previous summaries
left on screen.
DashboardEventBroadcaster.Publish ran a deep protobuf Clone (redaction is on by
default) and a group SendAsync for every event of every session, before anything
checked whether a dashboard client was actually watching. In the steady state the
session:{id} group is empty, so that work was thrown away per event.
SignalR does not expose group membership, so EventsHub now mirrors its own
add/remove into a singleton EventsHubViewerRegistry, and OnDisconnectedAsync
releases everything a dropped connection held (SessionDetailsPage disposes the
connection rather than unsubscribing). Publish returns early when the session has
no viewers, before the redaction clone. Watched sessions behave exactly as before.
Lazy mirror-lease start/stop was deliberately not attempted — it entangles the
dashboard with SessionEventDistributor subscribe lifetime for no saving beyond
this gate; recorded in docs/GatewayDashboardDesign.md.
The policy tests shipped at 1c30611 could not detect a deleted gate. They
assert that secrets:manage admits an Administrator and refuses a Viewer —
true, and library behaviour this repo did not author. The wiring is the
only thing that change introduced, and nothing covered it.
The point is sharper for repos where the link already existed before
gating, which includes this one: "an Administrator still sees it" is
identical to the pre-change behaviour, so it cannot distinguish a working
gate from an inert AuthorizeView. Only the negative observation proves a
gate is there at all.
SecretsNavRenderTests renders MainLayout through the framework's static
HtmlRenderer — no component-testing package, because the assertion is
about emitted markup rather than interactivity — and asserts:
- absent for a Viewer, and for an anonymous caller (the load-bearing pair)
- present for an Administrator (the control: without it, a rail that
rendered nothing at all would satisfy both absence assertions and the
suite would report a working gate over a blank page)
- the ungated API Keys sibling still present for a Viewer, so a later
"consistency fix" that hides it fails loudly rather than silently
removing read access
Confirmed non-vacuous by mutation rather than by argument: with the
AuthorizeView removed from the layout, both absence tests go red and all
three original policy tests stay green.
Build 0 warnings / 0 errors; suite 899/899 (895 + 4).
Family-wide nav sweep: the Secrets management page should be linked from
each app's UI, visible to Administrator-role users only.
The link already existed in MainLayout's Admin section. The gate did not:
the rail rendered every item for every visitor, including a Viewer and the
anonymous-localhost read-only identity. Not an access hole — the mounted
page carries [Authorize(Policy = "secrets:manage")], so a Viewer clicking
through was denied — but a dead link presented as a live one. There was
also no existing role-gated nav pattern to follow; the rail's only
AuthorizeView was the footer's signed-in/signed-out split.
Gated on the POLICY rather than a role literal, so nav visibility cannot
drift from what the page enforces. In this host the two are equivalent:
GatewayOptionsValidator constrains Dashboard:GroupToRole values to
Administrator or Viewer, so the shared library's other manage-granting
roles (secrets-manager, secrets-reveal) are unreachable. The policy form
stays correct if that ever relaxes, where a role literal would then hide
the link from users who can use the page.
API Keys is deliberately left ungated. It looks like the same case and is
not: ApiKeysPage renders for a Viewer with write affordances hidden, so
hiding its link would remove legitimate read access. The secrets page has
no read-only mode. The rule is "gate the link when the page denies the
role outright", not "gate everything under Admin".
Coverage: three tests pin the policy's verdict per principal
(Administrator admitted, Viewer refused, unauthenticated refused), and
/admin/secrets joins the canonical route list — it is the one nav
destination mounted from an RCL rather than declared here, so a routing
regression could remove it without touching this repo's pages. The
principal helper sets an authentication type deliberately: without one
the role assertions would pass vacuously for the wrong reason.
Not a rendering test — the suite has no component-testing harness, and
adding one to assert a single AuthorizeView would be a large dependency
for a small claim.
Build 0 warnings / 0 errors; suite 895/895.
Adds the two wonder rows that were deliberately withheld on 2026-08-11
while the pre-55f2889 SHA was unsettled. It is settled: b948e69 (08-09)
and 0a9715d (08-11) were never competing claims about one binary, they
are two deploys two days apart.
What settled it is worth recording as a technique in its own right, so
it goes in as a fourth way to identify a build: each Server.bak.<ts>
holds the exe that deploy REPLACED, so a VersionInfo sweep across the
backups reconstructs a host's deploy history from the host alone — no
repo access, no deploy record. The subtlety that makes it readable is
that a backup's timestamp dates the NEXT deploy, not the build inside
it. Reading a file version is non-destructive, unlike opening a SQLite
store in a backup directory.
Also records the full garbage version stamp recovered from the 08-09
binary, because the failure mode is a false positive rather than a
blank: "0.1.2+fatal:..." reads like a version that succeeded and then
picked up noise, when the leading 0.1.2 is just the static base <Version>
every build carries. For a binary in that window the commit is not
recoverable from the binary at all, so finding nothing is the expected
result rather than evidence against a SHA established another way.
Provenance is stated per cell rather than uniformly: the worker SHAs on
the new rows are carried forward and marked unconfirmed, b948e69 rests
on PDB hash plus the contemporaneous record and never on a stamp, and
55f2889 was read from the live stamp, which is trustworthy only because
it postdates 0152180.
Auth 0.1.5 -> 0.2.0, Health 0.2.0 -> 0.3.0, Secrets/.Abstractions/.Ui
0.6.1 -> 0.6.2. Theme, GalaxyRepository, Audit, Configuration, Telemetry
and Telemetry.Serilog were already at the newest version on the feed.
Checked against the shared-lib source rather than the version numbers,
because these packages are versioned as a family and a bump is not by
itself evidence that the package changed:
- Auth 0.2.0 is the only one carrying content for us: LDAP backup-DC
failover (FallbackServers, endpoint walk with sticky preference,
boot-time entry validation). Purely additive; the default is empty,
which leaves single-endpoint behaviour unchanged.
- Health 0.3.0 carries a breaking change, but every line of it is in
ZB.MOM.WW.Health.Akka, which we do not reference. No commit touched
the core ZB.MOM.WW.Health package between 0.2.0 and 0.3.0.
- Secrets 0.6.2 is a message-only change: one validator string literal
gains mounted-volume guidance. SecretsStorePathRules is untouched.
The four non-csproj files are not a separate feature. Configuration/
LdapOptions is a deliberate shadow of the shared type and carries an
explicit warning to mirror any new upstream field, because AddZbLdapAuth
binds the whole MxGateway:Ldap section onto the shared options. So
FallbackServers is live on our config surface the moment the package
lands, and without the mirror an operator could configure a backup DC
that works but is invisible on the dashboard's Settings page. The
Settings row renders "none" when empty, since that is the answer someone
who believes a backup DC is configured actually needs.
Entry syntax is deliberately NOT re-validated here: the shared validator
already fails the boot on a malformed entry and owns the (internal)
parser, so a second copy would drift. Note both validators skip entirely
when Ldap:Enabled is false.
Verified the binder is non-strict (ErrorOnUnknownConfiguration is unused
anywhere in the tree), so the upgrade could not break startup on a
newly-recognised key either way.
Build 0 warnings / 0 errors; gateway suite 892/892, unchanged. The live
LDAP tests are opt-in and were not run, so the failover path itself is
covered only by the shared library's own tests.
0.6.0 landed the rules but not the enforcement on this path: the pre-host secrets
container has no IHostEnvironment, so the library skipped the content-root check
and the migrator created the database before the real host could refuse it. The
pin to 0.6.1 alone does not close that — the call site has to pass the content
root explicitly, which is why this is a code change and not a version bump.
0.6.0 put the store-path rules in the shared library, but the guard was not
running at the moment that matters here.
CreateBuilder resolves ${secret:} references before the host exists, using a
throwaway ServiceCollection that contains no IHostEnvironment — and it runs the
store migrator, which creates the database. The library resolved the content
root from IHostEnvironment alone, so it could not distinguish "no content root"
from "no host registered" and skipped the under-content-root rule entirely. The
store was created at the rejected path; the boot then failed a moment later when
the real host validated. The leftover empty database with its -wal/-shm siblings
is exactly the artifact that made the 2026-08-09 credential loss read as "the
database is there, it's just empty".
The pin alone does not close this. An app with a correctly configured path shows
no symptom and is still unprotected, because the guard simply is not running when
the store is created. 0.6.1 adds a 4-argument AddZbSecrets overload taking the
content root explicitly, and the call site has to use it. The in-host
registration below needs nothing.
Verified by removing the fix rather than by observing a clean boot — which is how
this survived its first release. With the 3-argument overload the new test fails
by finding a created database at
src/ZB.MOM.WW.MxGateway.Server/probe-secrets-*.db: inside the source tree, since
that is what the content root resolves to under test.
Two things about the test itself, both of which it would have been easy to get
subtly wrong:
It asserts no-file-created before asserting that startup threw. "It threw" is the
weaker claim, and asserting it first masks the stronger one — the run that proved
this defect would have reported "no exception was thrown" and said nothing about
the database sitting in the source tree.
The accepting case asserts the database *is* created, not merely that nothing
threw. A not-null builder is close to a tautology once no exception escaped, and
it would still pass if the pre-host container stopped opening the store at all —
which would also quietly void the rejecting case, since that one can only observe
a file the migration would otherwise have written. The two assertions hold each
other up.
Found by HistorianGateway's adoption, which probed the rejected paths instead of
observing a successful boot.
Three separable changes, each building on its own: the mxaccess-sessions health
check on the active tier, the content-root rule that closes the gap the 2026-08-09
credential-store loss went through, and the Secrets re-pin (0.2.3 -> 0.6.0) that
moves the same rules into the shared library.
0.6.0 moves the store-path rules into the shared library — both the rooted check
and the content-root check — with a SecretsOptionsValidator wired into
AddZbSecrets and validated on start, so all four consuming apps get the guard
from one implementation rather than four copies. mxgw therefore adds no local
validator over the Secrets section.
This is a four-minor jump, not a re-pin: mxgw was on 0.2.3 and skips 0.3.0,
0.4.0, 0.4.1, 0.5.0 and 0.5.1 in one step. Verified rather than assumed —
diffing the 0.2.3 and 0.6.0 assemblies shows the added surface is the new path
rules and their plumbing (AddIfNotRooted, AddIfUnderContentRoot,
ComputeDefaultSqlitePath, DefaultSqlitePath, IValidateOptions, IsPathRooted,
GetFullPath) and nothing touching store or delete behaviour. Secrets.Ui is
byte-identical across the range: both assemblies are 39424 bytes and differ only
in the version stamp, because this package family shares one version across
every package even when a release touches only one of them. So the browser gate
run against the /admin/secrets delete modal on 0.2.3 still covers what ships
here.
The library default is LocalApplicationData-derived so the family's
cross-platform apps still boot locally. The gateway keeps its own
CommonApplicationData value, which always wins — see the note on
ApplyDefaultSecretsStorePath for why the difference is deliberate and why
deleting that method as a redundancy would silently move the store.
Rooted is not the same as safe, and the gap between the two cost a production
host every one of its API keys on 2026-08-09.
MxGateway:Authentication:SqlitePath was set to an absolute path inside the
directory the upgrade procedure renames to Server.bak.*. That passes the
existing rooted check cleanly. The deploy renamed the directory away, the store
went with it, and the gateway created a fresh empty one at the same path — no
error, no log line. No gRPC consumer could authenticate for two days. The deploy
itself was correct: the binaries were the point of the rename and the store was
collateral.
GatewayConfigPathRules gains AddIfUnderContentRoot, applied to the auth store
and the Galaxy snapshot. Both are written by the running process and both are
lost the same way. The rule compares resolved full paths and requires a
directory-separator boundary, so a sibling directory whose name merely starts
with the content root's ("/srv/app-data" against "/srv/app") is not treated as
inside it — on a fail-closed startup rule, that false positive would be a
gateway that refuses to boot on a legitimate path. Case sensitivity follows the
running OS rather than assuming case-insensitivity everywhere, which would
reject /srv/App as under /srv/app on Linux where they are different directories.
The rule is not exempted in Development. An environment-conditional guard is
never exercised where the mistake is made, and what failed in production was a
config that looked fine.
Secrets:SqlitePath is the same defect one layer down: it shipped as a bare
relative "mxgateway-secrets.db", which is how a stray database landed in
src/…Server/ and tripped the repository's tree-hygiene test. It is bound by the
shared ZB.MOM.WW.Secrets package, so appsettings.json now ships no value and the
default is computed from CommonApplicationData in code — the same mechanism
SEC-33 already used for the Galaxy snapshot, ten lines away, for the same reason.
Setting a default for an unset key is deliberately not the same act as
relocating a value someone configured, which these rules still refuse to do.
Note the migration edge this creates: a host relying on the old repo default now
looks somewhere new, finds nothing, and creates an empty store — this bug
re-introduced by its own fix. Deployed hosts are safe because they set the path
explicitly, in appsettings copied forward or in the service environment. The
latter is the more robust of the two, since it cannot be lost by a missed
preserve step.
Adds a `mxaccess-sessions` health check reporting how many MXAccess sessions are
healthy. Each session is one worker process holding one MXAccess COM instance —
a live connection into a Galaxy — so this answers "how many Galaxy connections
are healthy" in the vocabulary the code actually uses.
Zero sessions is Healthy, deliberately, and the rest of the design follows from
that. The gateway opens a session when a client asks and holds none otherwise,
so an idle gateway is working normally. A count threshold ("unhealthy below N")
would sit red forever on a host nothing dials yet, and a permanently red probe
is one operators stop reading — which leaves them worse off than no probe. The
check therefore grades on whether the sessions that exist are usable: nothing
faulted is Healthy, some faulted beside a ready or starting one is Degraded, and
every session faulted is Unhealthy. Counts ride along as entry data for the
family Overview dashboard.
Tagged `active` rather than `ready` for the same reason. Readiness decides
whether the process should be sent traffic, and a gateway with no sessions is
ready to serve — unlike the auth store, which every call depends on. Failing
readiness here would pull a working gateway out of rotation over a condition its
own clients create.
Reads ISessionRegistry, which already exposes Snapshot(); ISessionManager stays
the command surface and grows no enumerator.
Two precision fixes to the runbook landed in fd0e88e, both making it weaker in
the sense that matters.
The PDB source-hash entry read as though hashes settled the 2026-08-09
provenance question by themselves. They did not: a hash match and a
contemporaneous deploy record written that evening independently named the same
two commits, and the agreement is what makes the result trustworthy. A hash
match alone tells you which sources a binary was built from — not that the build
was intentional, nor which host it reached. A future reader holding only one of
the two derivations should know to look for a second.
The two-swap timeline was asserted from the investigation's timestamps, which a
later reader cannot re-derive. It is also confirmable from artifacts still on
disk — windev keeps two worker backup directories from that day and wonder one,
because there were two worker operations. Records that, so the story can be
checked against the boxes rather than believed.
A 2026-08-11 investigation treated the windev production binary as having no
traceable provenance. Both pieces of evidence were normal output of our own
build and deploy procedure, and the binary was fine — but nothing in the repo
said so, so it cost a forensics pass to establish.
The version stamp is the first trap. SHA stamping landed in ec6f82b (TST-11)
and was broken on Windows until 0152180 (NEXT-09) a month later: the trailing
backslash in MSBuildThisFileDirectory escaped the Exec's closing quote, and
because the target runs ContinueOnError with ConsoleToMSBuild, git's stderr was
stamped as the revision. Every Windows build in that window reads
`0.1.2+fatal: cannot change to ...`. The point an operator needs is that this
is non-diagnostic in *both* directions — it neither incriminates a build nor
confirms one — which is not obvious from a stamp that looks like a failure.
The second is that the build directory is *supposed* to be gone: deploys build
from a detached worktree so the host's checkout stays on its own branch, and
remove it afterwards.
Records what does identify a build instead, cheapest first — including the wire
probe for the worker, since plain Write/Write2 carrying statuses[0] separates
53f69cd from b948e69 without host access or symbols — plus the deploys to date.
Also records something that went unlogged and is the reason this read as a
mystery rather than an improvement: the 2026-08-09 deploy returned the x86
worker to mainline. Production had been running dd7ca163, contained only by
origin/test/client-e2e-coverage and not an ancestor of main, so the worker in
production was not rebuildable from any mainline commit.
The Theme bump is separable from the sweep and lands as its own commit: 0.4.0
upstreamed the button-sizing block the sweep had installed locally, and this
app's local .btn rule was trimmed rather than deleted because it also carried
border-radius/font-weight/white-space that the kit does not ship.
Family-wide admin-UI cleanup pass (scadaproj admin_ui_cleanup.md) applied to the
Blazor dashboard. Behaviour is unchanged throughout — no @onclick, disabled,
binding, auth gate, or arm->confirm flow was touched.
Uncontrolled error text is now truncated at the render site. Fault messages,
Galaxy load errors, and browse-tree load failures were rendered in full into
fixed-width table cells, where a long exception string blows out the column.
Each site gets DashboardDisplay.Abbreviate plus a title attribute carrying the
untruncated text, so nothing becomes unreachable. Abbreviate is length-checked
rather than a bare range slice: `value[..n]` on a shorter string throws and
takes the whole page render down with it. The two detail views whose entire
purpose is to show one fault in full — SessionDetailsPage and GalaxyPage's
Last Error — are deliberately left untruncated.
Two classes referenced from markup had no definition anywhere in the sheet.
.browse-stale-banner was inert; .tree-load-status was a real visual defect —
loading and failed-to-load rows sit among .tree-row siblings and carry the same
leading .tree-toggle-empty spacer, but that spacer only takes its width as a
flex item, so without a flex container those rows lost their indent.
Confirm/cancel pairs in ConfirmDialog and the API-key create form are now
btn-groups with role="group" and an aria-label, replacing margin-spaced loose
buttons.
Removes a paragraph on GalaxyPage naming internal RPCs (DiscoverHierarchy,
GetLastDeployTime) — implementation detail with no meaning to a dashboard
operator.
Verified in a real browser, not bUnit: full build clean, 879/879 tests, and a
live gate against a running dashboard with a genuine ~250-char SqlClient
exception as the erroring row. Results per check, including the checks that
could NOT be exercised without an x86 worker, are recorded in
docs/plans/2026-08-11-dashboard-ui-sweeps.md.
That plan doc also records a correction: this app is NOT Bootstrap-free. The
sweep brief said it was, citing the scadaproj index; libman.json pins
bootstrap 5.3.3 and App.razor:7 links it ahead of the theme. The stale claim had
already cost this app one skipped family sweep (scadaproj#2, the /admin/secrets
modal), so that modal was live-gated here too and passes.
site.css declared `font-size: 0.82rem` literally on `.btn`. Bootstrap sizes
buttons through `font-size: var(--bs-btn-font-size)`, and `.btn-sm` /
`.btn-group-sm > .btn` do nothing but redefine that variable — so a literal at
equal specificity, loaded after Bootstrap, silently flattened every small button
in the dashboard into a padding-only difference. `btn-sm` was inert app-wide.
The kit now owns the sizing: 0.4.0 shipped the four `--bs-btn-*` overrides in
layout.css, which ThemeHead emits ahead of site.css, so removing the local
literal restores `.btn-sm` without a local copy. 0.4.1 is a pin-only follow-up
(it fixes `.rail-btn-block`, which this app does not use; `theme.css` is
byte-identical and the `.btn` rule is unchanged between the two).
The rest of the local rule is kept deliberately. 0.4.0 upstreamed sizing only,
not `border-radius` / `font-weight` / `white-space`, so deleting the block
wholesale would have dropped three app-specific declarations. `border-radius`
also stays a literal rather than `--bs-btn-border-radius`, because `.btn-sm`
redefines that variable and small buttons would shrink to Bootstrap's radius.
Verified in a browser against the running dashboard: `.btn` 13.6px and `.btn-sm`
12.48px, btn-group seams intact, and `--bs-btn-font-size` now declared in
exactly two sheets (bootstrap.min.css, layout.css) instead of three.
xUnit v2 (2.9.3) never invokes IAsyncDisposable.DisposeAsync on a test
class, so the unobserved-fault safety net this class documents — awaiting
every scripted worker task after each test — has been dead code since it
was written. Found via dumpasync on the wedged-testhost investigation: the
NeverReadyWorkerProcessLauncher's Task.Delay(Infinite, _stop.Token) was
still pending (DelayPromiseWithCancellation, detached) minutes after its
test finished, which is only possible if DisposeAsync never ran.
Proven both ways with a throw-probe in DisposeAsync: under IAsyncDisposable
all 3 tests pass (never called); under IAsyncLifetime all 3 fail from the
probe (called after every test). Sweep confirmed this is the only test
class on IAsyncDisposable — all other implementers are helpers disposed
via await using.
Joseph Doherty