The remediation reviews approved every task but left a tail of small notes.
This lands the gateway-side half of them.
Hardening (behavior changes, all narrow):
- BuildFilteredWriteBulkCommand's unreachable default case failed OPEN: a
fifth bulk-write kind added upstream without a filter case here would have
shipped the DENIED entries to the worker while reporting them denied to the
caller. It now throws UnreachableException.
- SqliteCanonicalAuditStore.ListRecentAsync no longer throws on a row it
cannot date. The retention sweep deliberately preserves such rows (SQLite's
datetime() yields NULL, so the DELETE never matches), which guaranteed the
dashboard's recent-audit view would meet one eventually and lose the whole
page to it. The row is now reported at DateTimeOffset.MinValue with every
other column intact, behind an optional logger.
- The audit drain loop's finally now completes the channel writer alongside
detaching the drain, so a producer that raced past the attached check takes
the write-through branch instead of stranding its event in a buffer nobody
reads until shutdown. TryComplete is idempotent, so StopAsync is unaffected.
Tests:
- MapCommandReply ownership (Assert.Same on the inner reply), mirroring the
existing MapEvent ownership test.
- Redactor key-id length boundary at exactly 64 and 65 characters, pinning
which way it fails. Nothing validates key-id length at creation, so
docs/Diagnostics.md's "which no issued key id does" is now stated as the
heuristic it is.
- ApiKeyFailureLimiter.Reset with a PartitionResolution whose partition was
evicted between the Check and the Reset: inert, and clears nobody else's
block.
- Constraint-cache concurrency stress: the cap is enforced by the inserting
thread, so overshoot must be transient and proportional to the in-flight
inserters, and the cache must settle at or under the cap.
- ListRecentAsync against a raw-SQL undateable row.
Comment/doc accuracy:
- EventsHubViewerRegistry.ReleaseConnection records that it relies on
SignalR's default sequential per-connection dispatch
(MaximumParallelInvocationsPerClient = 1).
- A PERF(followup) note on Invoke's double session resolve and why removing it
needs a SessionManager overload.
- SessionEventDistributor: the volatile-field comment named the pump as the
lock-free reader, but the pump's single capture point is inside _replayLock;
the genuinely lock-free reader is SubscriberCount. OnSubscriberOverflow's
"cannot be observed here" now excepts the DisposeAsync abandon path. The
churn test names its ConcurrentDictionary bucket-order assumption and that a
violation surfaces as a read timeout, not a silent pass.
- The two "restores the sequential drain's behavior" claims (SessionManager,
docs/Sessions.md) were wrong: the sequential drain leaked too, because
KillWorkerAsync's entry ThrowIfCancellationRequested aborted the whole loop
on the first session for zero kills. Reworded to "fixes a leak the
sequential drain also had", with the sweep-bound/shutdown-unbound
ParallelOptions asymmetry explained.
- ISessionManager.ShutdownAsync's token doc: it degrades the drain to a kill
sweep rather than cancelling it, with the bounded overrun stated.
SessionShutdownHostedService.StopAsync records that its cancellation-logging
branch is now unreachable.
Decrement was decrement-first with a single non-retried repair CAS. From zero,
two unmatched decrements (SignalR calls OnDisconnectedAsync for a connection
whose OnConnectedAsync faulted) capture -1 and -2; a real Increment then makes
the count -1, and the first decrementer's stale CompareExchange(0, -1) matches
and resets to zero — erasing a live connection, so the idle gate freezes an open
dashboard. The same lost race also made Decrement report 0 when it had not
written 0.
Clamping now happens inside the compare-and-swap: read, clamp, publish, retry on
loss. A lost race re-reads the fresh value instead of repairing a stale one.
The counter moves to its own file per the one-public-type-per-file convention and
gains direct tests: the zero floor under concurrent unmatched decrements, matched
pairs settling at zero, and an interleaved connect/disconnect stress round. The
stress test asserts the observable invariants only — the specific interleaving
cannot be forced through the public API (verified: the previous implementation
passes it), which its remarks now state rather than implying a reproducer. A hub
wiring test is skipped for the EventsHub reason: driving Hub.OnConnectedAsync
needs caller-clients and connection-context fakes, and the overrides are two
lines of delegation to the tested type.
Also documents that the API-key refresh's pre-gate time check races benignly.
The snapshot publisher broadcast to Clients.All on every ~1s tick forever, with
zero viewers. Each tick cost a session-registry snapshot and sort, a metrics
snapshot that copies dictionaries under the global metrics lock, a full rebuild
of EffectiveGatewayConfiguration, and a SQLite read of the API key table.
DashboardSnapshotHub now counts live connections into the singleton
DashboardSnapshotHubConnectionCounter (clamped at zero, since SignalR can call
OnDisconnectedAsync for a connection whose OnConnectedAsync faulted). The
publisher drives the snapshot enumerator by hand instead of await foreach: with
no connections it does not call MoveNextAsync at all, so the producing iterator
stays suspended at its yield and no snapshot is built — the gate removes the
build, not just the broadcast. It re-checks once a second, so the first viewer
resumes the tick within about one interval; that viewer is seeded immediately by
DashboardPageBase's synchronous GetSnapshot() and by the hub's OnConnectedAsync.
Two per-tick costs are bounded independently of the gate: the effective
configuration is startup-static (options are bound once at boot and never
reloaded), so it is built once and cached; and the API key summaries refresh at
most every 15s, since the list only changes when an operator creates, rotates,
or revokes a key. Only a successful refresh restarts the interval, so a failed
or timed-out read is still retried on the next tick with the previous summaries
left on screen.
DashboardEventBroadcaster.Publish ran a deep protobuf Clone (redaction is on by
default) and a group SendAsync for every event of every session, before anything
checked whether a dashboard client was actually watching. In the steady state the
session:{id} group is empty, so that work was thrown away per event.
SignalR does not expose group membership, so EventsHub now mirrors its own
add/remove into a singleton EventsHubViewerRegistry, and OnDisconnectedAsync
releases everything a dropped connection held (SessionDetailsPage disposes the
connection rather than unsubscribing). Publish returns early when the session has
no viewers, before the redaction clone. Watched sessions behave exactly as before.
Lazy mirror-lease start/stop was deliberately not attempted — it entangles the
dashboard with SessionEventDistributor subscribe lifetime for no saving beyond
this gate; recorded in docs/GatewayDashboardDesign.md.
Family-wide nav sweep: the Secrets management page should be linked from
each app's UI, visible to Administrator-role users only.
The link already existed in MainLayout's Admin section. The gate did not:
the rail rendered every item for every visitor, including a Viewer and the
anonymous-localhost read-only identity. Not an access hole — the mounted
page carries [Authorize(Policy = "secrets:manage")], so a Viewer clicking
through was denied — but a dead link presented as a live one. There was
also no existing role-gated nav pattern to follow; the rail's only
AuthorizeView was the footer's signed-in/signed-out split.
Gated on the POLICY rather than a role literal, so nav visibility cannot
drift from what the page enforces. In this host the two are equivalent:
GatewayOptionsValidator constrains Dashboard:GroupToRole values to
Administrator or Viewer, so the shared library's other manage-granting
roles (secrets-manager, secrets-reveal) are unreachable. The policy form
stays correct if that ever relaxes, where a role literal would then hide
the link from users who can use the page.
API Keys is deliberately left ungated. It looks like the same case and is
not: ApiKeysPage renders for a Viewer with write affordances hidden, so
hiding its link would remove legitimate read access. The secrets page has
no read-only mode. The rule is "gate the link when the page denies the
role outright", not "gate everything under Admin".
Coverage: three tests pin the policy's verdict per principal
(Administrator admitted, Viewer refused, unauthenticated refused), and
/admin/secrets joins the canonical route list — it is the one nav
destination mounted from an RCL rather than declared here, so a routing
regression could remove it without touching this repo's pages. The
principal helper sets an authentication type deliberately: without one
the role assertions would pass vacuously for the wrong reason.
Not a rendering test — the suite has no component-testing harness, and
adding one to assert a single AuthorizeView would be a large dependency
for a small claim.
Build 0 warnings / 0 errors; suite 895/895.
Auth 0.1.5 -> 0.2.0, Health 0.2.0 -> 0.3.0, Secrets/.Abstractions/.Ui
0.6.1 -> 0.6.2. Theme, GalaxyRepository, Audit, Configuration, Telemetry
and Telemetry.Serilog were already at the newest version on the feed.
Checked against the shared-lib source rather than the version numbers,
because these packages are versioned as a family and a bump is not by
itself evidence that the package changed:
- Auth 0.2.0 is the only one carrying content for us: LDAP backup-DC
failover (FallbackServers, endpoint walk with sticky preference,
boot-time entry validation). Purely additive; the default is empty,
which leaves single-endpoint behaviour unchanged.
- Health 0.3.0 carries a breaking change, but every line of it is in
ZB.MOM.WW.Health.Akka, which we do not reference. No commit touched
the core ZB.MOM.WW.Health package between 0.2.0 and 0.3.0.
- Secrets 0.6.2 is a message-only change: one validator string literal
gains mounted-volume guidance. SecretsStorePathRules is untouched.
The four non-csproj files are not a separate feature. Configuration/
LdapOptions is a deliberate shadow of the shared type and carries an
explicit warning to mirror any new upstream field, because AddZbLdapAuth
binds the whole MxGateway:Ldap section onto the shared options. So
FallbackServers is live on our config surface the moment the package
lands, and without the mirror an operator could configure a backup DC
that works but is invisible on the dashboard's Settings page. The
Settings row renders "none" when empty, since that is the answer someone
who believes a backup DC is configured actually needs.
Entry syntax is deliberately NOT re-validated here: the shared validator
already fails the boot on a malformed entry and owns the (internal)
parser, so a second copy would drift. Note both validators skip entirely
when Ldap:Enabled is false.
Verified the binder is non-strict (ErrorOnUnknownConfiguration is unused
anywhere in the tree), so the upgrade could not break startup on a
newly-recognised key either way.
Build 0 warnings / 0 errors; gateway suite 892/892, unchanged. The live
LDAP tests are opt-in and were not run, so the failover path itself is
covered only by the shared library's own tests.
0.6.0 put the store-path rules in the shared library, but the guard was not
running at the moment that matters here.
CreateBuilder resolves ${secret:} references before the host exists, using a
throwaway ServiceCollection that contains no IHostEnvironment — and it runs the
store migrator, which creates the database. The library resolved the content
root from IHostEnvironment alone, so it could not distinguish "no content root"
from "no host registered" and skipped the under-content-root rule entirely. The
store was created at the rejected path; the boot then failed a moment later when
the real host validated. The leftover empty database with its -wal/-shm siblings
is exactly the artifact that made the 2026-08-09 credential loss read as "the
database is there, it's just empty".
The pin alone does not close this. An app with a correctly configured path shows
no symptom and is still unprotected, because the guard simply is not running when
the store is created. 0.6.1 adds a 4-argument AddZbSecrets overload taking the
content root explicitly, and the call site has to use it. The in-host
registration below needs nothing.
Verified by removing the fix rather than by observing a clean boot — which is how
this survived its first release. With the 3-argument overload the new test fails
by finding a created database at
src/ZB.MOM.WW.MxGateway.Server/probe-secrets-*.db: inside the source tree, since
that is what the content root resolves to under test.
Two things about the test itself, both of which it would have been easy to get
subtly wrong:
It asserts no-file-created before asserting that startup threw. "It threw" is the
weaker claim, and asserting it first masks the stronger one — the run that proved
this defect would have reported "no exception was thrown" and said nothing about
the database sitting in the source tree.
The accepting case asserts the database *is* created, not merely that nothing
threw. A not-null builder is close to a tautology once no exception escaped, and
it would still pass if the pre-host container stopped opening the store at all —
which would also quietly void the rejecting case, since that one can only observe
a file the migration would otherwise have written. The two assertions hold each
other up.
Found by HistorianGateway's adoption, which probed the rejected paths instead of
observing a successful boot.
0.6.0 moves the store-path rules into the shared library — both the rooted check
and the content-root check — with a SecretsOptionsValidator wired into
AddZbSecrets and validated on start, so all four consuming apps get the guard
from one implementation rather than four copies. mxgw therefore adds no local
validator over the Secrets section.
This is a four-minor jump, not a re-pin: mxgw was on 0.2.3 and skips 0.3.0,
0.4.0, 0.4.1, 0.5.0 and 0.5.1 in one step. Verified rather than assumed —
diffing the 0.2.3 and 0.6.0 assemblies shows the added surface is the new path
rules and their plumbing (AddIfNotRooted, AddIfUnderContentRoot,
ComputeDefaultSqlitePath, DefaultSqlitePath, IValidateOptions, IsPathRooted,
GetFullPath) and nothing touching store or delete behaviour. Secrets.Ui is
byte-identical across the range: both assemblies are 39424 bytes and differ only
in the version stamp, because this package family shares one version across
every package even when a release touches only one of them. So the browser gate
run against the /admin/secrets delete modal on 0.2.3 still covers what ships
here.
The library default is LocalApplicationData-derived so the family's
cross-platform apps still boot locally. The gateway keeps its own
CommonApplicationData value, which always wins — see the note on
ApplyDefaultSecretsStorePath for why the difference is deliberate and why
deleting that method as a redundancy would silently move the store.
Rooted is not the same as safe, and the gap between the two cost a production
host every one of its API keys on 2026-08-09.
MxGateway:Authentication:SqlitePath was set to an absolute path inside the
directory the upgrade procedure renames to Server.bak.*. That passes the
existing rooted check cleanly. The deploy renamed the directory away, the store
went with it, and the gateway created a fresh empty one at the same path — no
error, no log line. No gRPC consumer could authenticate for two days. The deploy
itself was correct: the binaries were the point of the rename and the store was
collateral.
GatewayConfigPathRules gains AddIfUnderContentRoot, applied to the auth store
and the Galaxy snapshot. Both are written by the running process and both are
lost the same way. The rule compares resolved full paths and requires a
directory-separator boundary, so a sibling directory whose name merely starts
with the content root's ("/srv/app-data" against "/srv/app") is not treated as
inside it — on a fail-closed startup rule, that false positive would be a
gateway that refuses to boot on a legitimate path. Case sensitivity follows the
running OS rather than assuming case-insensitivity everywhere, which would
reject /srv/App as under /srv/app on Linux where they are different directories.
The rule is not exempted in Development. An environment-conditional guard is
never exercised where the mistake is made, and what failed in production was a
config that looked fine.
Secrets:SqlitePath is the same defect one layer down: it shipped as a bare
relative "mxgateway-secrets.db", which is how a stray database landed in
src/…Server/ and tripped the repository's tree-hygiene test. It is bound by the
shared ZB.MOM.WW.Secrets package, so appsettings.json now ships no value and the
default is computed from CommonApplicationData in code — the same mechanism
SEC-33 already used for the Galaxy snapshot, ten lines away, for the same reason.
Setting a default for an unset key is deliberately not the same act as
relocating a value someone configured, which these rules still refuse to do.
Note the migration edge this creates: a host relying on the old repo default now
looks somewhere new, finds nothing, and creates an empty store — this bug
re-introduced by its own fix. Deployed hosts are safe because they set the path
explicitly, in appsettings copied forward or in the service environment. The
latter is the more robust of the two, since it cannot be lost by a missed
preserve step.
Adds a `mxaccess-sessions` health check reporting how many MXAccess sessions are
healthy. Each session is one worker process holding one MXAccess COM instance —
a live connection into a Galaxy — so this answers "how many Galaxy connections
are healthy" in the vocabulary the code actually uses.
Zero sessions is Healthy, deliberately, and the rest of the design follows from
that. The gateway opens a session when a client asks and holds none otherwise,
so an idle gateway is working normally. A count threshold ("unhealthy below N")
would sit red forever on a host nothing dials yet, and a permanently red probe
is one operators stop reading — which leaves them worse off than no probe. The
check therefore grades on whether the sessions that exist are usable: nothing
faulted is Healthy, some faulted beside a ready or starting one is Degraded, and
every session faulted is Unhealthy. Counts ride along as entry data for the
family Overview dashboard.
Tagged `active` rather than `ready` for the same reason. Readiness decides
whether the process should be sent traffic, and a gateway with no sessions is
ready to serve — unlike the auth store, which every call depends on. Failing
readiness here would pull a working gateway out of rotation over a condition its
own clients create.
Reads ISessionRegistry, which already exposes Snapshot(); ISessionManager stays
the command surface and grows no enumerator.
Family-wide admin-UI cleanup pass (scadaproj admin_ui_cleanup.md) applied to the
Blazor dashboard. Behaviour is unchanged throughout — no @onclick, disabled,
binding, auth gate, or arm->confirm flow was touched.
Uncontrolled error text is now truncated at the render site. Fault messages,
Galaxy load errors, and browse-tree load failures were rendered in full into
fixed-width table cells, where a long exception string blows out the column.
Each site gets DashboardDisplay.Abbreviate plus a title attribute carrying the
untruncated text, so nothing becomes unreachable. Abbreviate is length-checked
rather than a bare range slice: `value[..n]` on a shorter string throws and
takes the whole page render down with it. The two detail views whose entire
purpose is to show one fault in full — SessionDetailsPage and GalaxyPage's
Last Error — are deliberately left untruncated.
Two classes referenced from markup had no definition anywhere in the sheet.
.browse-stale-banner was inert; .tree-load-status was a real visual defect —
loading and failed-to-load rows sit among .tree-row siblings and carry the same
leading .tree-toggle-empty spacer, but that spacer only takes its width as a
flex item, so without a flex container those rows lost their indent.
Confirm/cancel pairs in ConfirmDialog and the API-key create form are now
btn-groups with role="group" and an aria-label, replacing margin-spaced loose
buttons.
Removes a paragraph on GalaxyPage naming internal RPCs (DiscoverHierarchy,
GetLastDeployTime) — implementation detail with no meaning to a dashboard
operator.
Verified in a real browser, not bUnit: full build clean, 879/879 tests, and a
live gate against a running dashboard with a genuine ~250-char SqlClient
exception as the erroring row. Results per check, including the checks that
could NOT be exercised without an x86 worker, are recorded in
docs/plans/2026-08-11-dashboard-ui-sweeps.md.
That plan doc also records a correction: this app is NOT Bootstrap-free. The
sweep brief said it was, citing the scadaproj index; libman.json pins
bootstrap 5.3.3 and App.razor:7 links it ahead of the theme. The stale claim had
already cost this app one skipped family sweep (scadaproj#2, the /admin/secrets
modal), so that modal was live-gated here too and passes.
site.css declared `font-size: 0.82rem` literally on `.btn`. Bootstrap sizes
buttons through `font-size: var(--bs-btn-font-size)`, and `.btn-sm` /
`.btn-group-sm > .btn` do nothing but redefine that variable — so a literal at
equal specificity, loaded after Bootstrap, silently flattened every small button
in the dashboard into a padding-only difference. `btn-sm` was inert app-wide.
The kit now owns the sizing: 0.4.0 shipped the four `--bs-btn-*` overrides in
layout.css, which ThemeHead emits ahead of site.css, so removing the local
literal restores `.btn-sm` without a local copy. 0.4.1 is a pin-only follow-up
(it fixes `.rail-btn-block`, which this app does not use; `theme.css` is
byte-identical and the `.btn` rule is unchanged between the two).
The rest of the local rule is kept deliberately. 0.4.0 upstreamed sizing only,
not `border-radius` / `font-weight` / `white-space`, so deleting the block
wholesale would have dropped three app-specific declarations. `border-radius`
also stays a literal rather than `--bs-btn-border-radius`, because `.btn-sm`
redefines that variable and small buttons would shrink to Bootstrap's radius.
Verified in a browser against the running dashboard: `.btn` 13.6px and `.btn-sm`
12.48px, btn-group seams intact, and `--bs-btn-font-size` now declared in
exactly two sheets (bootstrap.min.css, layout.css) instead of three.
A periodic reconcile can synthesize a repair transition whose matching live
transition is still buffered in the alarm lease; both then broadcast as
indistinguishable duplicates on StreamAlarms and the dashboard hub. Nothing
serializes the two paths, and a correct serialization needs a worker-side
high-water mark on QueryActiveAlarms (proto + worker change + a stall path),
so this closes the common case with a local best-effort dedup instead: both
paths already carry the same worker-derived identity — the worker stamps
record.TransitionTimestampUtc into both OnAlarmTransitionEvent's
transition_timestamp and ActiveAlarmSnapshot.last_transition_timestamp — so
ApplyTransition suppresses a live transition whose (timestamp, resulting
state) the cache already carries from a repair. The Clear leg has no cache
entry left to compare, so ApplyReconcile tombstones each synthesized Clear by
the instance's original_raise_timestamp for one reconcile generation; a
matching live Clear consumes the tombstone, while a new raise/clear cycle
carries a newer raise timestamp and passes. Suppression fires only on a
positive marker match — unset timestamps keep today's behavior — so the
documented at-least-once consumer contract stands (gateway.md, Sessions.md
updated in the same change).
Tests: two new regressions drive the exact race through the GWC-26 harness
(repair-then-buffered-live for Raise and for Clear, each with a genuine
follow-up transition proving no over-suppression); GatewayAlarmMonitor
suites 18/18.
The pipe name mxaccess-gateway-{pid}-session-{32hex} plus .NET's
CoreFxPipe_ prefix overflowed the 104-byte Unix-domain-socket path limit
under the default per-user macOS TMPDIR (~49 chars), so every test that
opened a real pipe threw ArgumentOutOfRangeException at pipe creation
unless TMPDIR=/tmp was exported. Rename to mxgw-{pid}-{sessionUid} (the
session guid hex without the session- prefix; worst-case 43 chars) and
shorten the three test-fixture names the same way. Uniqueness is
unchanged: gateway pid + full session guid. The worker receives the pipe
name via its launch command line, so mixed Server/Worker deploy SHAs are
unaffected. Docs updated in the same change (gateway.md,
GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md); new
regression test pins the format and the length budget.
Verified: SessionManagerTests 39/39; SessionWorkerClientFactory,
GatewayEndToEndFakeWorkerSmoke, WorkerClient, and ReconnectReplay suites
33/33 under the default macOS TMPDIR — this also retires the
previously-misdiagnosed 'macOS pipe-timeout test failures': they were
this path-length throw, not a timeout-message defect.
Repo-side half of SEC-36. The appsettings.json plaintext was already discharged
before this branch (HEAD ships the fail-closed ${secret:ldap/mxgateway/bind}
store reference), so the residual leak was the literal value in glauth.md,
docs/GatewayTesting.md, and the historical archreview SEC-06 evidence -- all
scrubbed to <service-account-password> placeholders pointing at the source of
truth scadaproj/infra/glauth/.
- csproj: add <UserSecretsId>mxaccessgw-server</UserSecretsId> (dev channel)
- GatewayOptionsValidator: blank-password message now names both channels
(dev user-secrets, deployed MxGateway__Ldap__ServiceAccountPassword)
- test: assert the message names both channels
- docs: GatewayConfiguration.md (three channels + rotation note), glauth.md
(placeholders + rotation-required + runbook pointer), GatewayTesting.md
- new operator runbook docs/runbooks/SEC-36-ldap-credential-rotation.md
(live rotation + NSSM staging remain operator-pending)
- tracking: SEC-36 -> Done (repo-side) in both registers + change-log
Deviation: kept the ${secret:} reference in appsettings.json rather than
deleting it (spec step 2 assumed the stale plaintext baseline); deleting it
would regress the shipped/documented/tested secret-store channel.
git grep -i for the old value is empty across all tracked files.
Same-commit docs rule (were missed in the prior commit):
- docs/GalaxyRepository.md: SnapshotCachePath now documents the per-OS derived
default and the GalaxyRepositoryOptionsValidator rooting/validity enforcement.
- A2-galaxyrepository-adoption-handoff.md: correct the now-inaccurate NSSM caveat
(SnapshotCachePath override is optional, not required; blank seeds a rooted host
default, no silent no-op) and repoint the option-validation item at the new
GalaxyRepositoryOptionsValidator.
SEC-34 guard confirmed and documented: TryParseKeyId's '_' split cannot truncate a
key id because both — and the only — gateway key-creation paths
(ApiKeyAdminCommandLineParser.IsValidKeyId, DashboardApiKeyManagementService.ValidateKeyId)
restrict key ids to IsAsciiLetterOrDigit || '.' || '-', and key ids are never
library-generated. Added a citing comment; no behavior change.
Test consolidation: moved the three host-start SqlitePath overrides into
TestHostEnvironmentInitializer (per-process temp store, mirroring Secrets__SqlitePath)
so future host-start tests auto-cover.
SEC-33: make rooting host-meaningful and stop shipping foreign-platform literals.
- Delete IsRootedForAnyPlatform; AddIfNotRooted now uses Path.IsPathRooted (current OS).
- Promote AddIfNotRooted/AddIfInvalidPath to shared GatewayConfigPathRules so the new
Galaxy validator reuses them and the two validators cannot drift.
- Remove Authentication:SqlitePath and Galaxy:SnapshotCachePath Windows literals from
appsettings.json; the CommonApplicationData-derived code defaults take over. The
Galaxy default is seeded as a configuration value before AddZbGalaxyRepository
(SnapshotCachePath is init-only, so a PostConfigure mutation cannot compile).
- New GalaxyRepositoryOptionsValidator (ValidateOnStart) enforces a valid, host-rooted
SnapshotCachePath when PersistSnapshot is true.
- Root-cause the stray junk-named auth DB: host start eagerly builds
AuthSqliteConnectionFactory; under the non-rooted Windows literal on macOS SQLite
wrote it relative to the test bin CWD. The three real-host-start tests now pin
SqlitePath to a temp path.
SEC-34: verification cache Invalidate-vs-in-flight-repopulation race closed with a
per-key generation counter (bump-before-evict, snapshot-then-recheck). The expiry
cap (window 2) takes the documented fallback: the library verification identity
carries no ExpiresUtc, so the cache cannot cap at the key's expiry (donor-library ask).
GWC-24 rider: cap MxGateway:Events:QueueCapacity at int.MaxValue/2 so the derived
checked(2 * EventChannelCapacity) in WorkerClient cannot overflow at session creation.
SEC-35 (doc-only): note IsProduction() env-name semantics in GatewayConfiguration.md.
Docs updated same commit (GatewayConfiguration.md, Authentication.md) and tracking
registers/change-log flipped (00-tracking.md, 40-security-dashboard.md).
ReadAsync allocated a fresh 4-byte array per inbound frame; the GWC-08
pass pooled the payload buffer but left the prefix. Replaced with a
per-instance scratch field — the reader is single-consumer by
construction (one read loop per WorkerClient, handshake reads complete
before the loop starts), so a per-instance buffer is safe and the
non-reentrancy that makes it safe is now stated on the class. Pooling
four bytes via ArrayPool would cost more than the allocation it saves.
Tests: WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_
ParsesEveryFrame reads five frames of differing payload length through
one reader, so a stale prefix carried between calls would misparse.
Invoke deep-cloned the whole MxCommandRequest — including its command
payload, potentially a large bulk-write graph — only to overwrite the
cloned command with commandToInvoke and discard it. MapCommand then did
the one clone actually needed. Net cost: a full wasted command deep-clone
per Invoke, worst for exactly the bulk writes that are largest.
Adds a MapCommand(MxCommand) overload (MapCommand reads nothing else off
the request) and has Invoke pass commandToInvoke directly; the request
overload delegates so other callers are untouched.
The remaining clone inside MapCommand stays and is now documented as
required rather than incidental: commandToInvoke may be the gRPC-owned
request.Command, and the caller reads it again after dispatch via
TrackCommandReply, so ownership transfer (à la GWC-07) is not safe here.
That clone is what keeps WorkerClient.CreateCommandEnvelope's no-aliasing
invariant true.
Tests: MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload
(mutating the input leaves the mapped command untouched; both overloads
produce equal results under a fixed TimeProvider).
CreateEnvelope stamped Sequence with an interlocked increment when the
envelope was built, so two concurrent InvokeAsync callers could take 1
and 2 and then enqueue in the order 2, 1 — non-monotonic on the wire,
breaking gateway.md's "monotonic per sender" contract. Benign today
(neither side validates inbound sequence, old GWC-10 still open) but it
would fault healthy sessions the moment worker-side validation lands.
WriteLoopAsync now stamps immediately before _writer.WriteAsync. It is
the outbound channel's single consumer (SingleReader = true), so wire
order and stamp order are the same thing by construction and
_nextSequence drops to a plain ulong with no interlocking. This mirrors
the worker's WRK-04 fix, which the gateway half never received.
Also adds TST-28: a [Theory] pinning that GatewayHello.MaxFrameBytes
carries the configured worker-frame maximum (default + 2 MiB override).
The adoption half is asserted only in the Windows-only worker suite, so
a regression to sending 0 — "older gateway, use default" to the worker —
would silently downgrade the negotiated IPC-02 limit with every CI test
still green. Mutation-checked (hard-coded 0 fails both cases).
Tests: WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequences
OnTheWire (32 parallel invokes; failed 3/3 pre-fix) and
.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes.
ReleaseProbe recognised its own reservation by comparing NextProbeAtTicks to
now + _probeIntervalTicks. RecordInto's rearm-on-trip writes that identical
expression, so a concurrent RecordFailure on the same WindowState whose `now`
lands on the claimer's tick — routine at ~1 ms clock resolution under load — was
mistaken for the caller's own claim. The release then stomped the legitimate
fresh re-arm back to the stale previousProbeAtTicks, which is already due, handing
the next arrival a free probe the re-arm had just closed.
WindowState gains a monotonic ProbeVersion bumped by every writer of
NextProbeAtTicks (TryConsumeProbe's claim and RecordInto's re-arm alike).
TryConsumeProbe returns the stamp it set as part of a ProbeClaim; ReleaseProbe
restores the previous value only while the state's version still equals that
stamp, checking and restoring in one lock(state) section and bumping the version
again on restore so no other stale release can match either.
Test: ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick, with the clock held
still so the claim and the interleaved failure necessarily share a tick. Making it
deterministic needed a seam — the claim-to-release window is a few nanoseconds and
racing threads do not hit it (an earlier thread-based attempt passed against the
defective guard three runs out of three, and its end state was ordering-dependent
rather than correctness-dependent, so it was dropped rather than shipped as
theatre). The seam is an internal ProbeReleaseInterleaveHook, null in production,
costing one null check on the already-refused path. Verified as a genuine red
against the timestamp guard: Expected ThrottledByPeer, Actual ProbeAdmitted.
Code-review follow-up on fix/gwc-26-27-alarm-attach.
ApplyReconcile's snapshot-derived feed repairs are at-least-once, not
exactly-once: a reconcile reads the worker's current state while the matching
live transition may still be buffered in the monitor's lease, so both broadcast
and the duplicates are indistinguishable on the alarm feed. This pre-dates the
acked-state delta — the Raise/Clear presence repair has always had it, since
nothing serializes a reconcile pass against the in-flight live stream — so
closing it (serialization or timestamp dedup) stays out of scope for a P2 fix.
Documented instead, with the consumer contract stated explicitly (apply
transitions idempotently, never as an increment or toggle):
- ApplyReconcile gains a "Delivery semantics" comment.
- gateway.md softens the "defense in depth" prose to state the semantics.
- docs/Sessions.md carries the same caveat on the alarm-feed description.
- Tracker change-log records it as a known pre-existing characteristic and a
candidate finding for the next review cycle.
Also hoists the ChannelWorkerClient fake — duplicated across the three alarm
test files — into TestSupport/, dropping the usings it took with it.
Joseph Doherty