The worker-side half of the review tail. Tests and comments only — nothing here
changes worker behavior, and none of it compiles on the macOS tree (net48/x86),
so it was reviewed line by line against the already-windev-validated files.
- MxAccessHandleRegistryTests gains the multi-candidate case behind
MxAccessSession.TryGetCachedReadFor's fall-through: one tag under two item
handles, the lower registered-but-unadvised and the higher advised. Asserted
at the registry rather than the session because the session's read path needs
a live MXAccess COM instance; what the registry owes the scan is the stable
ascending candidate order and a per-item-handle (not per-tag) advice index,
and both are pinned here along with the fall-through contract in prose.
- A single adversarial lifecycle test — register, advise, re-register the same
item handle under a new tag, unadvise, unregister the server — asserting every
index agrees after each step. The individual transitions were already covered;
what was not was that they compose, and a stale entry in any one index
resurrects a handle MXAccess has already retired.
- StaWaitHelperTests.WaitForSignalOrMessages_PreSignalledHandle_ReturnsImmediately
drains pending messages first, like the other two wait tests. Without it a
stale message can end the wait instead of the handle, failing the
signal-consumed post-condition for an unrelated reason.
- GatewayTesting.md records the two findings from the Task 24 windev gate:
SecretsStorePathGuardTests.CreateBuilder_AcceptsSecretsStoreOutsideContentRoot_AndCreatesIt
fails deterministically on Windows on main too (SQLite pooling holds secrets.db
open across the cleanup's recursive delete; pre-existing, tracked separately),
and the StaWaitHelper timing tests' flake signature on a loaded box is a
message wake — the helper working as designed — not a broken wait.
The remediation reviews approved every task but left a tail of small notes.
This lands the gateway-side half of them.
Hardening (behavior changes, all narrow):
- BuildFilteredWriteBulkCommand's unreachable default case failed OPEN: a
fifth bulk-write kind added upstream without a filter case here would have
shipped the DENIED entries to the worker while reporting them denied to the
caller. It now throws UnreachableException.
- SqliteCanonicalAuditStore.ListRecentAsync no longer throws on a row it
cannot date. The retention sweep deliberately preserves such rows (SQLite's
datetime() yields NULL, so the DELETE never matches), which guaranteed the
dashboard's recent-audit view would meet one eventually and lose the whole
page to it. The row is now reported at DateTimeOffset.MinValue with every
other column intact, behind an optional logger.
- The audit drain loop's finally now completes the channel writer alongside
detaching the drain, so a producer that raced past the attached check takes
the write-through branch instead of stranding its event in a buffer nobody
reads until shutdown. TryComplete is idempotent, so StopAsync is unaffected.
Tests:
- MapCommandReply ownership (Assert.Same on the inner reply), mirroring the
existing MapEvent ownership test.
- Redactor key-id length boundary at exactly 64 and 65 characters, pinning
which way it fails. Nothing validates key-id length at creation, so
docs/Diagnostics.md's "which no issued key id does" is now stated as the
heuristic it is.
- ApiKeyFailureLimiter.Reset with a PartitionResolution whose partition was
evicted between the Check and the Reset: inert, and clears nobody else's
block.
- Constraint-cache concurrency stress: the cap is enforced by the inserting
thread, so overshoot must be transient and proportional to the in-flight
inserters, and the cache must settle at or under the cap.
- ListRecentAsync against a raw-SQL undateable row.
Comment/doc accuracy:
- EventsHubViewerRegistry.ReleaseConnection records that it relies on
SignalR's default sequential per-connection dispatch
(MaximumParallelInvocationsPerClient = 1).
- A PERF(followup) note on Invoke's double session resolve and why removing it
needs a SessionManager overload.
- SessionEventDistributor: the volatile-field comment named the pump as the
lock-free reader, but the pump's single capture point is inside _replayLock;
the genuinely lock-free reader is SubscriberCount. OnSubscriberOverflow's
"cannot be observed here" now excepts the DisposeAsync abandon path. The
churn test names its ConcurrentDictionary bucket-order assumption and that a
violation surfaces as a read timeout, not a silent pass.
- The two "restores the sequential drain's behavior" claims (SessionManager,
docs/Sessions.md) were wrong: the sequential drain leaked too, because
KillWorkerAsync's entry ThrowIfCancellationRequested aborted the whole loop
on the first session for zero kills. Reworded to "fixes a leak the
sequential drain also had", with the sweep-bound/shutdown-unbound
ParallelOptions asymmetry explained.
- ISessionManager.ShutdownAsync's token doc: it degrades the drain to a kill
sweep rather than cancelling it, with the bounded overrun stated.
SessionShutdownHostedService.StopAsync records that its cancellation-logging
branch is now unreachable.
Decrement was decrement-first with a single non-retried repair CAS. From zero,
two unmatched decrements (SignalR calls OnDisconnectedAsync for a connection
whose OnConnectedAsync faulted) capture -1 and -2; a real Increment then makes
the count -1, and the first decrementer's stale CompareExchange(0, -1) matches
and resets to zero — erasing a live connection, so the idle gate freezes an open
dashboard. The same lost race also made Decrement report 0 when it had not
written 0.
Clamping now happens inside the compare-and-swap: read, clamp, publish, retry on
loss. A lost race re-reads the fresh value instead of repairing a stale one.
The counter moves to its own file per the one-public-type-per-file convention and
gains direct tests: the zero floor under concurrent unmatched decrements, matched
pairs settling at zero, and an interleaved connect/disconnect stress round. The
stress test asserts the observable invariants only — the specific interleaving
cannot be forced through the public API (verified: the previous implementation
passes it), which its remarks now state rather than implying a reproducer. A hub
wiring test is skipped for the EventsHub reason: driving Hub.OnConnectedAsync
needs caller-clients and connection-context fakes, and the overrides are two
lines of delegation to the tested type.
Also documents that the API-key refresh's pre-gate time check races benignly.
77c5731 committed this file from a working copy that predated 75e3dc2's
advised-set section, silently deleting it. Puts the paragraph back verbatim;
no other content changes.
The snapshot publisher broadcast to Clients.All on every ~1s tick forever, with
zero viewers. Each tick cost a session-registry snapshot and sort, a metrics
snapshot that copies dictionaries under the global metrics lock, a full rebuild
of EffectiveGatewayConfiguration, and a SQLite read of the API key table.
DashboardSnapshotHub now counts live connections into the singleton
DashboardSnapshotHubConnectionCounter (clamped at zero, since SignalR can call
OnDisconnectedAsync for a connection whose OnConnectedAsync faulted). The
publisher drives the snapshot enumerator by hand instead of await foreach: with
no connections it does not call MoveNextAsync at all, so the producing iterator
stays suspended at its yield and no snapshot is built — the gate removes the
build, not just the broadcast. It re-checks once a second, so the first viewer
resumes the tick within about one interval; that viewer is seeded immediately by
DashboardPageBase's synchronous GetSnapshot() and by the hub's OnConnectedAsync.
Two per-tick costs are bounded independently of the gate: the effective
configuration is startup-static (options are bound once at boot and never
reloaded), so it is built once and cached; and the API key summaries refresh at
most every 15s, since the list only changes when an operator creates, rotates,
or revokes a key. Only a successful refresh restarts the interval, so a failed
or timed-out read is still retried on the next tick with the previous summaries
left on screen.
DashboardEventBroadcaster.Publish ran a deep protobuf Clone (redaction is on by
default) and a group SendAsync for every event of every session, before anything
checked whether a dashboard client was actually watching. In the steady state the
session:{id} group is empty, so that work was thrown away per event.
SignalR does not expose group membership, so EventsHub now mirrors its own
add/remove into a singleton EventsHubViewerRegistry, and OnDisconnectedAsync
releases everything a dropped connection held (SessionDetailsPage disposes the
connection rather than unsubscribing). Publish returns early when the session has
no viewers, before the redaction clone. Watched sessions behave exactly as before.
Lazy mirror-lease start/stop was deliberately not attempted — it entangles the
dashboard with SessionEventDistributor subscribe lifetime for no saving beyond
this gate; recorded in docs/GatewayDashboardDesign.md.
The policy tests shipped at 1c30611 could not detect a deleted gate. They
assert that secrets:manage admits an Administrator and refuses a Viewer —
true, and library behaviour this repo did not author. The wiring is the
only thing that change introduced, and nothing covered it.
The point is sharper for repos where the link already existed before
gating, which includes this one: "an Administrator still sees it" is
identical to the pre-change behaviour, so it cannot distinguish a working
gate from an inert AuthorizeView. Only the negative observation proves a
gate is there at all.
SecretsNavRenderTests renders MainLayout through the framework's static
HtmlRenderer — no component-testing package, because the assertion is
about emitted markup rather than interactivity — and asserts:
- absent for a Viewer, and for an anonymous caller (the load-bearing pair)
- present for an Administrator (the control: without it, a rail that
rendered nothing at all would satisfy both absence assertions and the
suite would report a working gate over a blank page)
- the ungated API Keys sibling still present for a Viewer, so a later
"consistency fix" that hides it fails loudly rather than silently
removing read access
Confirmed non-vacuous by mutation rather than by argument: with the
AuthorizeView removed from the layout, both absence tests go red and all
three original policy tests stay green.
Build 0 warnings / 0 errors; suite 899/899 (895 + 4).
Family-wide nav sweep: the Secrets management page should be linked from
each app's UI, visible to Administrator-role users only.
The link already existed in MainLayout's Admin section. The gate did not:
the rail rendered every item for every visitor, including a Viewer and the
anonymous-localhost read-only identity. Not an access hole — the mounted
page carries [Authorize(Policy = "secrets:manage")], so a Viewer clicking
through was denied — but a dead link presented as a live one. There was
also no existing role-gated nav pattern to follow; the rail's only
AuthorizeView was the footer's signed-in/signed-out split.
Gated on the POLICY rather than a role literal, so nav visibility cannot
drift from what the page enforces. In this host the two are equivalent:
GatewayOptionsValidator constrains Dashboard:GroupToRole values to
Administrator or Viewer, so the shared library's other manage-granting
roles (secrets-manager, secrets-reveal) are unreachable. The policy form
stays correct if that ever relaxes, where a role literal would then hide
the link from users who can use the page.
API Keys is deliberately left ungated. It looks like the same case and is
not: ApiKeysPage renders for a Viewer with write affordances hidden, so
hiding its link would remove legitimate read access. The secrets page has
no read-only mode. The rule is "gate the link when the page denies the
role outright", not "gate everything under Admin".
Coverage: three tests pin the policy's verdict per principal
(Administrator admitted, Viewer refused, unauthenticated refused), and
/admin/secrets joins the canonical route list — it is the one nav
destination mounted from an RCL rather than declared here, so a routing
regression could remove it without touching this repo's pages. The
principal helper sets an authentication type deliberately: without one
the role assertions would pass vacuously for the wrong reason.
Not a rendering test — the suite has no component-testing harness, and
adding one to assert a single AuthorizeView would be a large dependency
for a small claim.
Build 0 warnings / 0 errors; suite 895/895.
Adds the two wonder rows that were deliberately withheld on 2026-08-11
while the pre-55f2889 SHA was unsettled. It is settled: b948e69 (08-09)
and 0a9715d (08-11) were never competing claims about one binary, they
are two deploys two days apart.
What settled it is worth recording as a technique in its own right, so
it goes in as a fourth way to identify a build: each Server.bak.<ts>
holds the exe that deploy REPLACED, so a VersionInfo sweep across the
backups reconstructs a host's deploy history from the host alone — no
repo access, no deploy record. The subtlety that makes it readable is
that a backup's timestamp dates the NEXT deploy, not the build inside
it. Reading a file version is non-destructive, unlike opening a SQLite
store in a backup directory.
Also records the full garbage version stamp recovered from the 08-09
binary, because the failure mode is a false positive rather than a
blank: "0.1.2+fatal:..." reads like a version that succeeded and then
picked up noise, when the leading 0.1.2 is just the static base <Version>
every build carries. For a binary in that window the commit is not
recoverable from the binary at all, so finding nothing is the expected
result rather than evidence against a SHA established another way.
Provenance is stated per cell rather than uniformly: the worker SHAs on
the new rows are carried forward and marked unconfirmed, b948e69 rests
on PDB hash plus the contemporaneous record and never on a stamp, and
55f2889 was read from the live stamp, which is trustworthy only because
it postdates 0152180.
Auth 0.1.5 -> 0.2.0, Health 0.2.0 -> 0.3.0, Secrets/.Abstractions/.Ui
0.6.1 -> 0.6.2. Theme, GalaxyRepository, Audit, Configuration, Telemetry
and Telemetry.Serilog were already at the newest version on the feed.
Checked against the shared-lib source rather than the version numbers,
because these packages are versioned as a family and a bump is not by
itself evidence that the package changed:
- Auth 0.2.0 is the only one carrying content for us: LDAP backup-DC
failover (FallbackServers, endpoint walk with sticky preference,
boot-time entry validation). Purely additive; the default is empty,
which leaves single-endpoint behaviour unchanged.
- Health 0.3.0 carries a breaking change, but every line of it is in
ZB.MOM.WW.Health.Akka, which we do not reference. No commit touched
the core ZB.MOM.WW.Health package between 0.2.0 and 0.3.0.
- Secrets 0.6.2 is a message-only change: one validator string literal
gains mounted-volume guidance. SecretsStorePathRules is untouched.
The four non-csproj files are not a separate feature. Configuration/
LdapOptions is a deliberate shadow of the shared type and carries an
explicit warning to mirror any new upstream field, because AddZbLdapAuth
binds the whole MxGateway:Ldap section onto the shared options. So
FallbackServers is live on our config surface the moment the package
lands, and without the mirror an operator could configure a backup DC
that works but is invisible on the dashboard's Settings page. The
Settings row renders "none" when empty, since that is the answer someone
who believes a backup DC is configured actually needs.
Entry syntax is deliberately NOT re-validated here: the shared validator
already fails the boot on a malformed entry and owns the (internal)
parser, so a second copy would drift. Note both validators skip entirely
when Ldap:Enabled is false.
Verified the binder is non-strict (ErrorOnUnknownConfiguration is unused
anywhere in the tree), so the upgrade could not break startup on a
newly-recognised key either way.
Build 0 warnings / 0 errors; gateway suite 892/892, unchanged. The live
LDAP tests are opt-in and were not run, so the failover path itself is
covered only by the shared library's own tests.
0.6.0 landed the rules but not the enforcement on this path: the pre-host secrets
container has no IHostEnvironment, so the library skipped the content-root check
and the migrator created the database before the real host could refuse it. The
pin to 0.6.1 alone does not close that — the call site has to pass the content
root explicitly, which is why this is a code change and not a version bump.
0.6.0 put the store-path rules in the shared library, but the guard was not
running at the moment that matters here.
CreateBuilder resolves ${secret:} references before the host exists, using a
throwaway ServiceCollection that contains no IHostEnvironment — and it runs the
store migrator, which creates the database. The library resolved the content
root from IHostEnvironment alone, so it could not distinguish "no content root"
from "no host registered" and skipped the under-content-root rule entirely. The
store was created at the rejected path; the boot then failed a moment later when
the real host validated. The leftover empty database with its -wal/-shm siblings
is exactly the artifact that made the 2026-08-09 credential loss read as "the
database is there, it's just empty".
The pin alone does not close this. An app with a correctly configured path shows
no symptom and is still unprotected, because the guard simply is not running when
the store is created. 0.6.1 adds a 4-argument AddZbSecrets overload taking the
content root explicitly, and the call site has to use it. The in-host
registration below needs nothing.
Verified by removing the fix rather than by observing a clean boot — which is how
this survived its first release. With the 3-argument overload the new test fails
by finding a created database at
src/ZB.MOM.WW.MxGateway.Server/probe-secrets-*.db: inside the source tree, since
that is what the content root resolves to under test.
Two things about the test itself, both of which it would have been easy to get
subtly wrong:
It asserts no-file-created before asserting that startup threw. "It threw" is the
weaker claim, and asserting it first masks the stronger one — the run that proved
this defect would have reported "no exception was thrown" and said nothing about
the database sitting in the source tree.
The accepting case asserts the database *is* created, not merely that nothing
threw. A not-null builder is close to a tautology once no exception escaped, and
it would still pass if the pre-host container stopped opening the store at all —
which would also quietly void the rejecting case, since that one can only observe
a file the migration would otherwise have written. The two assertions hold each
other up.
Found by HistorianGateway's adoption, which probed the rejected paths instead of
observing a successful boot.
Joseph Doherty