Commit Graph

257 Commits

Author SHA1 Message Date
Joseph Doherty e23f816bfb feat(dashboard): in-process session event subscription feeds the viewer registry — SessionDetailsPage drops its /hubs/events hop 2026-08-15 20:15:52 -04:00
Joseph Doherty 25f07f89dd fix(tests): clear the SQLite pool before deleting the secrets path-guard temp dir — Windows sharing violation 2026-08-15 20:04:59 -04:00
Joseph Doherty dc2df628e3 chore(followups): reviewer-recommended tests, comments, and hardening from the remediation reviews
The remediation reviews approved every task but left a tail of small notes.
This lands the gateway-side half of them.

Hardening (behavior changes, all narrow):

- BuildFilteredWriteBulkCommand's unreachable default case failed OPEN: a
  fifth bulk-write kind added upstream without a filter case here would have
  shipped the DENIED entries to the worker while reporting them denied to the
  caller. It now throws UnreachableException.
- SqliteCanonicalAuditStore.ListRecentAsync no longer throws on a row it
  cannot date. The retention sweep deliberately preserves such rows (SQLite's
  datetime() yields NULL, so the DELETE never matches), which guaranteed the
  dashboard's recent-audit view would meet one eventually and lose the whole
  page to it. The row is now reported at DateTimeOffset.MinValue with every
  other column intact, behind an optional logger.
- The audit drain loop's finally now completes the channel writer alongside
  detaching the drain, so a producer that raced past the attached check takes
  the write-through branch instead of stranding its event in a buffer nobody
  reads until shutdown. TryComplete is idempotent, so StopAsync is unaffected.

Tests:

- MapCommandReply ownership (Assert.Same on the inner reply), mirroring the
  existing MapEvent ownership test.
- Redactor key-id length boundary at exactly 64 and 65 characters, pinning
  which way it fails. Nothing validates key-id length at creation, so
  docs/Diagnostics.md's "which no issued key id does" is now stated as the
  heuristic it is.
- ApiKeyFailureLimiter.Reset with a PartitionResolution whose partition was
  evicted between the Check and the Reset: inert, and clears nobody else's
  block.
- Constraint-cache concurrency stress: the cap is enforced by the inserting
  thread, so overshoot must be transient and proportional to the in-flight
  inserters, and the cache must settle at or under the cap.
- ListRecentAsync against a raw-SQL undateable row.

Comment/doc accuracy:

- EventsHubViewerRegistry.ReleaseConnection records that it relies on
  SignalR's default sequential per-connection dispatch
  (MaximumParallelInvocationsPerClient = 1).
- A PERF(followup) note on Invoke's double session resolve and why removing it
  needs a SessionManager overload.
- SessionEventDistributor: the volatile-field comment named the pump as the
  lock-free reader, but the pump's single capture point is inside _replayLock;
  the genuinely lock-free reader is SubscriberCount. OnSubscriberOverflow's
  "cannot be observed here" now excepts the DisposeAsync abandon path. The
  churn test names its ConcurrentDictionary bucket-order assumption and that a
  violation surfaces as a read timeout, not a silent pass.
- The two "restores the sequential drain's behavior" claims (SessionManager,
  docs/Sessions.md) were wrong: the sequential drain leaked too, because
  KillWorkerAsync's entry ThrowIfCancellationRequested aborted the whole loop
  on the first session for zero kills. Reworded to "fixes a leak the
  sequential drain also had", with the sweep-bound/shutdown-unbound
  ParallelOptions asymmetry explained.
- ISessionManager.ShutdownAsync's token doc: it degrades the drain to a kill
  sweep rather than cancelling it, with the bounded overrun stated.
  SessionShutdownHostedService.StopAsync records that its cancellation-logging
  branch is now unreachable.
2026-08-15 17:54:31 -04:00
Joseph Doherty b5ea6bb461 fix(alarms): fetch/poll ceilings; truncation-semantics docs; log-format conformance 2026-08-15 17:19:41 -04:00
Joseph Doherty 13583322b5 perf(worker): launcher-configurable event queue capacity 2026-08-15 17:10:54 -04:00
Joseph Doherty f3e1de5f37 fix(alarms): truncation-safe transitions; perf: single-pass alarm parse; configurable poll cadence 2026-08-15 17:04:06 -04:00
Joseph Doherty 95f8ba918d fix(sessions): exception-total shutdown body with non-cancellable kill fallback 2026-08-15 13:19:08 -04:00
Joseph Doherty 0bc13b5292 perf(sessions): bounded-parallel teardown in lease sweep and shutdown 2026-08-15 12:43:30 -04:00
Joseph Doherty a1a38b5538 fix(ipc): cancellation-priority timeout classification; structurally-enforced no-throw cancel send 2026-08-15 12:41:38 -04:00
Joseph Doherty 1742e38c10 fix(events): graceful unregister no longer masquerades as overflow under FailFast 2026-08-15 12:38:20 -04:00
Joseph Doherty 7b2d04605e fix(audit): write-through on completed channel, poison-batch isolation, drain-fault fallback 2026-08-15 12:37:09 -04:00
Joseph Doherty f920b4cbf5 fix(dashboard): clamped CAS retry loop in snapshot hub connection counter + direct counter tests
Decrement was decrement-first with a single non-retried repair CAS. From zero,
two unmatched decrements (SignalR calls OnDisconnectedAsync for a connection
whose OnConnectedAsync faulted) capture -1 and -2; a real Increment then makes
the count -1, and the first decrementer's stale CompareExchange(0, -1) matches
and resets to zero — erasing a live connection, so the idle gate freezes an open
dashboard. The same lost race also made Decrement report 0 when it had not
written 0.

Clamping now happens inside the compare-and-swap: read, clamp, publish, retry on
loss. A lost race re-reads the fresh value instead of repairing a stale one.

The counter moves to its own file per the one-public-type-per-file convention and
gains direct tests: the zero floor under concurrent unmatched decrements, matched
pairs settling at zero, and an interleaved connect/disconnect stress round. The
stress test asserts the observable invariants only — the specific interleaving
cannot be forced through the public API (verified: the previous implementation
passes it), which its remarks now state rather than implying a reproducer. A hub
wiring test is skipped for the EventsHub reason: driving Hub.OnConnectedAsync
needs caller-clients and connection-context fakes, and the overrides are two
lines of delegation to the tested type.

Also documents that the API-key refresh's pre-gate time check races benignly.
2026-08-15 12:32:58 -04:00
Joseph Doherty 44ca7c8623 fix(dashboard): enforce/document the advised-set cap honestly; cover handle-0 and oversize-read paths 2026-08-15 12:31:36 -04:00
Joseph Doherty 7c1ea12331 perf(ipc): WaitAsync command timeouts + forward WorkerCancel so a timed-out COM call frees the STA 2026-08-15 12:28:09 -04:00
Joseph Doherty 7171892984 perf(grpc): O(1) unconstrained bulk fast path, direct filtered-command build, cache eviction, capacity hints 2026-08-15 12:24:17 -04:00
Joseph Doherty 88d38bb900 perf(auth): allocation-free token parsing; single partition-key build per RPC 2026-08-15 12:23:25 -04:00
Joseph Doherty 9735ac3b7c perf(metrics): pull-model worker queue gauge (fixes last-writer-wins), Interlocked command counters 2026-08-15 12:21:51 -04:00
Joseph Doherty 77c5731b7b perf(dashboard): idle-gate the snapshot tick; cache static config; bound key-list refresh
The snapshot publisher broadcast to Clients.All on every ~1s tick forever, with
zero viewers. Each tick cost a session-registry snapshot and sort, a metrics
snapshot that copies dictionaries under the global metrics lock, a full rebuild
of EffectiveGatewayConfiguration, and a SQLite read of the API key table.

DashboardSnapshotHub now counts live connections into the singleton
DashboardSnapshotHubConnectionCounter (clamped at zero, since SignalR can call
OnDisconnectedAsync for a connection whose OnConnectedAsync faulted). The
publisher drives the snapshot enumerator by hand instead of await foreach: with
no connections it does not call MoveNextAsync at all, so the producing iterator
stays suspended at its yield and no snapshot is built — the gate removes the
build, not just the broadcast. It re-checks once a second, so the first viewer
resumes the tick within about one interval; that viewer is seeded immediately by
DashboardPageBase's synchronous GetSnapshot() and by the hub's OnConnectedAsync.

Two per-tick costs are bounded independently of the gate: the effective
configuration is startup-static (options are bound once at boot and never
reloaded), so it is built once and cached; and the API key summaries refresh at
most every 15s, since the list only changes when an operator creates, rotates,
or revokes a key. Only a successful refresh restarts the interval, so a failed
or timed-out read is still retried on the next tick with the previous summaries
left on screen.
2026-08-15 12:21:49 -04:00
Joseph Doherty e04b1c9199 perf(events): copy-on-write subscriber snapshot in fan-out pump 2026-08-15 12:21:42 -04:00
Joseph Doherty 75e3dc2794 perf(dashboard): LRU cap on the shared live-read session's advised set 2026-08-15 12:21:13 -04:00
Joseph Doherty f1e26fed4f perf(alarms): memoize CurrentAlarms projection, invalidate on mutation 2026-08-15 12:20:51 -04:00
Joseph Doherty ca34a2d65d fix(logging): fail-closed bearer redaction; hoist per-request logger creation 2026-08-15 12:17:27 -04:00
Joseph Doherty e2ac5d117a perf(audit): bounded async audit writer with batched inserts, one-time bootstrap, retention sweep 2026-08-15 12:17:22 -04:00
Joseph Doherty 6c5218913b perf(dashboard): gate event mirror on live viewers — no clone, no send for unwatched sessions
DashboardEventBroadcaster.Publish ran a deep protobuf Clone (redaction is on by
default) and a group SendAsync for every event of every session, before anything
checked whether a dashboard client was actually watching. In the steady state the
session:{id} group is empty, so that work was thrown away per event.

SignalR does not expose group membership, so EventsHub now mirrors its own
add/remove into a singleton EventsHubViewerRegistry, and OnDisconnectedAsync
releases everything a dropped connection held (SessionDetailsPage disposes the
connection rather than unsubscribing). Publish returns early when the session has
no viewers, before the redaction clone. Watched sessions behave exactly as before.

Lazy mirror-lease start/stop was deliberately not attempted — it entangles the
dashboard with SessionEventDistributor subscribe lifetime for no saving beyond
this gate; recorded in docs/GatewayDashboardDesign.md.
2026-08-15 12:07:33 -04:00
Joseph Doherty 5744aad028 test(dashboard): prove the Secrets nav gate exists, by its absence
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m15s
ci / java (push) Successful in 2m16s
ci / portable (push) Successful in 9m19s
The policy tests shipped at 1c30611 could not detect a deleted gate. They
assert that secrets:manage admits an Administrator and refuses a Viewer —
true, and library behaviour this repo did not author. The wiring is the
only thing that change introduced, and nothing covered it.

The point is sharper for repos where the link already existed before
gating, which includes this one: "an Administrator still sees it" is
identical to the pre-change behaviour, so it cannot distinguish a working
gate from an inert AuthorizeView. Only the negative observation proves a
gate is there at all.

SecretsNavRenderTests renders MainLayout through the framework's static
HtmlRenderer — no component-testing package, because the assertion is
about emitted markup rather than interactivity — and asserts:

- absent for a Viewer, and for an anonymous caller (the load-bearing pair)
- present for an Administrator (the control: without it, a rail that
  rendered nothing at all would satisfy both absence assertions and the
  suite would report a working gate over a blank page)
- the ungated API Keys sibling still present for a Viewer, so a later
  "consistency fix" that hides it fails loudly rather than silently
  removing read access

Confirmed non-vacuous by mutation rather than by argument: with the
AuthorizeView removed from the layout, both absence tests go red and all
three original policy tests stay green.

Build 0 warnings / 0 errors; suite 899/899 (895 + 4).
2026-08-13 10:02:48 -04:00
Joseph Doherty 1c30611b1e feat(dashboard): gate the side rail's Secrets link on secrets:manage
Family-wide nav sweep: the Secrets management page should be linked from
each app's UI, visible to Administrator-role users only.

The link already existed in MainLayout's Admin section. The gate did not:
the rail rendered every item for every visitor, including a Viewer and the
anonymous-localhost read-only identity. Not an access hole — the mounted
page carries [Authorize(Policy = "secrets:manage")], so a Viewer clicking
through was denied — but a dead link presented as a live one. There was
also no existing role-gated nav pattern to follow; the rail's only
AuthorizeView was the footer's signed-in/signed-out split.

Gated on the POLICY rather than a role literal, so nav visibility cannot
drift from what the page enforces. In this host the two are equivalent:
GatewayOptionsValidator constrains Dashboard:GroupToRole values to
Administrator or Viewer, so the shared library's other manage-granting
roles (secrets-manager, secrets-reveal) are unreachable. The policy form
stays correct if that ever relaxes, where a role literal would then hide
the link from users who can use the page.

API Keys is deliberately left ungated. It looks like the same case and is
not: ApiKeysPage renders for a Viewer with write affordances hidden, so
hiding its link would remove legitimate read access. The secrets page has
no read-only mode. The rule is "gate the link when the page denies the
role outright", not "gate everything under Admin".

Coverage: three tests pin the policy's verdict per principal
(Administrator admitted, Viewer refused, unauthenticated refused), and
/admin/secrets joins the canonical route list — it is the one nav
destination mounted from an RCL rather than declared here, so a routing
regression could remove it without touching this repo's pages. The
principal helper sets an authentication type deliberately: without one
the role assertions would pass vacuously for the wrong reason.

Not a rendering test — the suite has no component-testing harness, and
adding one to assert a single AuthorizeView would be a large dependency
for a small claim.

Build 0 warnings / 0 errors; suite 895/895.
2026-08-13 09:33:04 -04:00
Joseph Doherty 5fdd8a570a fix(secrets): run the store-path guard in the pre-host container (0.6.1)
ci / windows-x86 (push) Successful in 1m19s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m59s
ci / portable (push) Successful in 10m48s
0.6.0 put the store-path rules in the shared library, but the guard was not
running at the moment that matters here.

CreateBuilder resolves ${secret:} references before the host exists, using a
throwaway ServiceCollection that contains no IHostEnvironment — and it runs the
store migrator, which creates the database. The library resolved the content
root from IHostEnvironment alone, so it could not distinguish "no content root"
from "no host registered" and skipped the under-content-root rule entirely. The
store was created at the rejected path; the boot then failed a moment later when
the real host validated. The leftover empty database with its -wal/-shm siblings
is exactly the artifact that made the 2026-08-09 credential loss read as "the
database is there, it's just empty".

The pin alone does not close this. An app with a correctly configured path shows
no symptom and is still unprotected, because the guard simply is not running when
the store is created. 0.6.1 adds a 4-argument AddZbSecrets overload taking the
content root explicitly, and the call site has to use it. The in-host
registration below needs nothing.

Verified by removing the fix rather than by observing a clean boot — which is how
this survived its first release. With the 3-argument overload the new test fails
by finding a created database at
src/ZB.MOM.WW.MxGateway.Server/probe-secrets-*.db: inside the source tree, since
that is what the content root resolves to under test.

Two things about the test itself, both of which it would have been easy to get
subtly wrong:

It asserts no-file-created before asserting that startup threw. "It threw" is the
weaker claim, and asserting it first masks the stronger one — the run that proved
this defect would have reported "no exception was thrown" and said nothing about
the database sitting in the source tree.

The accepting case asserts the database *is* created, not merely that nothing
threw. A not-null builder is close to a tautology once no exception escaped, and
it would still pass if the pre-host container stopped opening the store at all —
which would also quietly void the rejecting case, since that one can only observe
a file the migration would otherwise have written. The two assertions hold each
other up.

Found by HistorianGateway's adoption, which probed the rejected paths instead of
observing a successful boot.
2026-08-11 08:54:06 -04:00
Joseph Doherty 882c7ca3cd fix(config): reject credential and cache paths inside the application directory
Rooted is not the same as safe, and the gap between the two cost a production
host every one of its API keys on 2026-08-09.

MxGateway:Authentication:SqlitePath was set to an absolute path inside the
directory the upgrade procedure renames to Server.bak.*. That passes the
existing rooted check cleanly. The deploy renamed the directory away, the store
went with it, and the gateway created a fresh empty one at the same path — no
error, no log line. No gRPC consumer could authenticate for two days. The deploy
itself was correct: the binaries were the point of the rename and the store was
collateral.

GatewayConfigPathRules gains AddIfUnderContentRoot, applied to the auth store
and the Galaxy snapshot. Both are written by the running process and both are
lost the same way. The rule compares resolved full paths and requires a
directory-separator boundary, so a sibling directory whose name merely starts
with the content root's ("/srv/app-data" against "/srv/app") is not treated as
inside it — on a fail-closed startup rule, that false positive would be a
gateway that refuses to boot on a legitimate path. Case sensitivity follows the
running OS rather than assuming case-insensitivity everywhere, which would
reject /srv/App as under /srv/app on Linux where they are different directories.

The rule is not exempted in Development. An environment-conditional guard is
never exercised where the mistake is made, and what failed in production was a
config that looked fine.

Secrets:SqlitePath is the same defect one layer down: it shipped as a bare
relative "mxgateway-secrets.db", which is how a stray database landed in
src/…Server/ and tripped the repository's tree-hygiene test. It is bound by the
shared ZB.MOM.WW.Secrets package, so appsettings.json now ships no value and the
default is computed from CommonApplicationData in code — the same mechanism
SEC-33 already used for the Galaxy snapshot, ten lines away, for the same reason.
Setting a default for an unset key is deliberately not the same act as
relocating a value someone configured, which these rules still refuse to do.

Note the migration edge this creates: a host relying on the old repo default now
looks somewhere new, finds nothing, and creates an empty store — this bug
re-introduced by its own fix. Deployed hosts are safe because they set the path
explicitly, in appsettings copied forward or in the service environment. The
latter is the more robust of the two, since it cannot be lost by a missed
preserve step.
2026-08-11 08:42:16 -04:00
Joseph Doherty c69a1c441b feat(diagnostics): report MXAccess session health on the active probe
Adds a `mxaccess-sessions` health check reporting how many MXAccess sessions are
healthy. Each session is one worker process holding one MXAccess COM instance —
a live connection into a Galaxy — so this answers "how many Galaxy connections
are healthy" in the vocabulary the code actually uses.

Zero sessions is Healthy, deliberately, and the rest of the design follows from
that. The gateway opens a session when a client asks and holds none otherwise,
so an idle gateway is working normally. A count threshold ("unhealthy below N")
would sit red forever on a host nothing dials yet, and a permanently red probe
is one operators stop reading — which leaves them worse off than no probe. The
check therefore grades on whether the sessions that exist are usable: nothing
faulted is Healthy, some faulted beside a ready or starting one is Degraded, and
every session faulted is Unhealthy. Counts ride along as entry data for the
family Overview dashboard.

Tagged `active` rather than `ready` for the same reason. Readiness decides
whether the process should be sent traffic, and a gateway with no sessions is
ready to serve — unlike the auth store, which every call depends on. Failing
readiness here would pull a working gateway out of rotation over a condition its
own clients create.

Reads ISessionRegistry, which already exposes Snapshot(); ISessionManager stays
the command surface and grows no enumerator.
2026-08-11 08:41:54 -04:00
Joseph Doherty 3e504ca9c4 test(sessions): switch SessionWorkerClientFactoryFakeWorkerTests to IAsyncLifetime so its teardown actually runs
xUnit v2 (2.9.3) never invokes IAsyncDisposable.DisposeAsync on a test
class, so the unobserved-fault safety net this class documents — awaiting
every scripted worker task after each test — has been dead code since it
was written. Found via dumpasync on the wedged-testhost investigation: the
NeverReadyWorkerProcessLauncher's Task.Delay(Infinite, _stop.Token) was
still pending (DelayPromiseWithCancellation, detached) minutes after its
test finished, which is only possible if DisposeAsync never ran.

Proven both ways with a throw-probe in DisposeAsync: under IAsyncDisposable
all 3 tests pass (never called); under IAsyncLifetime all 3 fail from the
probe (called after every test). Sweep confirmed this is the only test
class on IAsyncDisposable — all other implementers are helpers disposed
via await using.
2026-08-10 16:11:54 -04:00
Joseph Doherty bfcf82975c test(windev): buffer the test named pipes so the full-suite testhost exits
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 8m48s
The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.

`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.

That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.

Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.

Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.

The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.

Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.

Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).

Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
2026-08-10 10:01:57 -04:00
Joseph Doherty e2352d1666 test(windev): fix the two Windows-only gateway test failures dismissed as environmental
ci / java (push) Successful in 2m24s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 3m33s
ci / portable (push) Successful in 8m13s
Both failures in the windev baseline were test bugs that reproduce on any Windows
host, not anything missing or misconfigured on windev.

SelfSignedCertificateProviderTests.GenerateCertificate_HasExpectedSansEkuAndValidity
asserted SAN content by substring-matching X509Extension.Format(false). That string
comes from the platform crypto library: Windows' CryptFormatObject renders the IPv6
loopback fully expanded (0000:0000:...:0001) where the managed formatter renders
"::1", so the loopback assertion could never hold on Windows. Decode the extension
with X509SubjectAlternativeNameExtension and compare parsed IPAddress values and DNS
names instead, which removes the platform-dependent formatting from the assertion.

SessionManagerTests.OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession guards
the 104-byte macOS sun_path budget NEXT-01 shortened the pipe name to fit. It padded
the measured name up to a five-digit pid but never substituted that worst case
downward, so Windows' routine six-digit pids over-counted by a character against a
budget that does not constrain the host running the test. Substitute the five-digit
macOS worst case for the running pid's digit count so the check measures the name
format rather than the current pid.

EventStreamServiceTests.WaitUntilAsync now reports the unmet condition on timeout
instead of letting a bare TaskCanceledException escape. Its five-second real-clock
deadline is genuinely load-sensitive on windev (36 logical CPUs, maxParallelThreads
-1), and an opaque cancellation there is exactly what got the previous failures
filed as "environmental" and left unexplained.

Documents the windev run in docs/GatewayTesting.md: the two fixed bugs and their root
causes, the real-pipe suites whose failures are evidence of machine load rather than
of the change under test, and the full-suite testhost that completes every test and
then never exits (filtered runs exit normally; macOS exits cleanly). Corrects the
CLAUDE.md claim that the suite exits cleanly on the Windows dev box.
2026-08-10 09:05:38 -04:00
Joseph Doherty 2c03e0a684 fix(alarms): dedup reconcile/live duplicate broadcasts on the alarm feed (NEXT-03)
A periodic reconcile can synthesize a repair transition whose matching live
transition is still buffered in the alarm lease; both then broadcast as
indistinguishable duplicates on StreamAlarms and the dashboard hub. Nothing
serializes the two paths, and a correct serialization needs a worker-side
high-water mark on QueryActiveAlarms (proto + worker change + a stall path),
so this closes the common case with a local best-effort dedup instead: both
paths already carry the same worker-derived identity — the worker stamps
record.TransitionTimestampUtc into both OnAlarmTransitionEvent's
transition_timestamp and ActiveAlarmSnapshot.last_transition_timestamp — so
ApplyTransition suppresses a live transition whose (timestamp, resulting
state) the cache already carries from a repair. The Clear leg has no cache
entry left to compare, so ApplyReconcile tombstones each synthesized Clear by
the instance's original_raise_timestamp for one reconcile generation; a
matching live Clear consumes the tombstone, while a new raise/clear cycle
carries a newer raise timestamp and passes. Suppression fires only on a
positive marker match — unset timestamps keep today's behavior — so the
documented at-least-once consumer contract stands (gateway.md, Sessions.md
updated in the same change).

Tests: two new regressions drive the exact race through the GWC-26 harness
(repair-then-buffered-live for Raise and for Clear, each with a genuine
follow-up transition proving no over-suppression); GatewayAlarmMonitor
suites 18/18.
2026-08-10 06:06:17 -04:00
Joseph Doherty 8769ee9765 fix(sessions): keep named-pipe socket paths inside the macOS sun_path limit (NEXT-01)
The pipe name mxaccess-gateway-{pid}-session-{32hex} plus .NET's
CoreFxPipe_ prefix overflowed the 104-byte Unix-domain-socket path limit
under the default per-user macOS TMPDIR (~49 chars), so every test that
opened a real pipe threw ArgumentOutOfRangeException at pipe creation
unless TMPDIR=/tmp was exported. Rename to mxgw-{pid}-{sessionUid} (the
session guid hex without the session- prefix; worst-case 43 chars) and
shorten the three test-fixture names the same way. Uniqueness is
unchanged: gateway pid + full session guid. The worker receives the pipe
name via its launch command line, so mixed Server/Worker deploy SHAs are
unaffected. Docs updated in the same change (gateway.md,
GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md); new
regression test pins the format and the length budget.

Verified: SessionManagerTests 39/39; SessionWorkerClientFactory,
GatewayEndToEndFakeWorkerSmoke, WorkerClient, and ReconnectReplay suites
33/33 under the default macOS TMPDIR — this also retires the
previously-misdiagnosed 'macOS pipe-timeout test failures': they were
this path-length throw, not a timeout-message defect.
2026-08-10 05:54:11 -04:00
Joseph Doherty 431a096cab feat(gateway): configurable worker write-completion wait (MxGateway:Worker:WriteCompletionWaitMilliseconds) 2026-08-09 12:27:30 -04:00
Joseph Doherty a346d514dd test(contracts): scope command-reply fixture invariants past the CLI-40/41 authenticate-user malformed-reply fixtures
ci / portable (push) Successful in 14m0s
ci / java (push) Successful in 6m50s
ci / windows-x86 (push) Failing after 1m21s
ci / nightly-windev (push) Has been skipped
The blanket loop asserted HRESULT/Statuses/ReturnValue on every command_replies fixture, but the authenticate-user.* fixtures added for the malformed-reply and credential-redaction contracts deliberately omit them (NRE on ReturnValue.DataType). Keep universal Kind/ProtocolStatus invariants for all; apply the MXAccess-detail block only to fixtures that carry it. Test-only.
2026-08-07 08:48:49 -04:00
Joseph Doherty aba22358f5 Merge branch 'fix/ipc-27-descriptor-test'
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m8s
ci / portable (push) Failing after 4m45s
2026-08-07 07:49:33 -04:00
Joseph Doherty 6060d21995 fix(IPC-27): close descriptor-freshness blind spots for enums, services, and Galaxy
ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField only
compared messages and fields, and only enumerated the gateway/worker
descriptors, so a new enum value, a new RPC, or any galaxy_repository.proto-only
change would not redden the test even though it is documented as the primary
protoc-free CI gate.

Rename to Descriptor_ContainsEveryContractSymbol and extend the reflection walk
on both sides (published protoset and in-process contract) to also collect
enums/enum values ({enumFullName}, {enumFullName}/{valueName}) and
services/methods ({serviceFullName}, {serviceFullName}/{methodName}), and add
GalaxyRepositoryReflection.Descriptor to the enumerated files. The comparison
stays a flat, order-insensitive string-set diff with no protoc dependency.

Update docs/ClientProtoGeneration.md and docs/Contracts.md prose from
"message or field" to the full symbol coverage.

Red-path proof: pointed the test at the pre-IPC-01 stale protoset and confirmed
it failed naming max_frame_bytes, several MxCommandKind/AlarmProviderMode enum
values, MxAccessGateway/StreamAlarms and GalaxyRepository/BrowseChildren, and
the galaxy_repository.v1.* surface; restored the real path and re-ran green.

Flips IPC-27 to Done in the 2026-07-12 remediation tracker and register.
2026-08-07 07:47:18 -04:00
Joseph Doherty 8c312c717c fix(SEC-36): scrub committed dev LDAP service-account password; add user-secrets channel + rotation runbook
Repo-side half of SEC-36. The appsettings.json plaintext was already discharged
before this branch (HEAD ships the fail-closed ${secret:ldap/mxgateway/bind}
store reference), so the residual leak was the literal value in glauth.md,
docs/GatewayTesting.md, and the historical archreview SEC-06 evidence -- all
scrubbed to <service-account-password> placeholders pointing at the source of
truth scadaproj/infra/glauth/.

- csproj: add <UserSecretsId>mxaccessgw-server</UserSecretsId> (dev channel)
- GatewayOptionsValidator: blank-password message now names both channels
  (dev user-secrets, deployed MxGateway__Ldap__ServiceAccountPassword)
- test: assert the message names both channels
- docs: GatewayConfiguration.md (three channels + rotation note), glauth.md
  (placeholders + rotation-required + runbook pointer), GatewayTesting.md
- new operator runbook docs/runbooks/SEC-36-ldap-credential-rotation.md
  (live rotation + NSSM staging remain operator-pending)
- tracking: SEC-36 -> Done (repo-side) in both registers + change-log

Deviation: kept the ${secret:} reference in appsettings.json rather than
deleting it (spec step 2 assumed the stale plaintext baseline); deleting it
would regress the shipped/documented/tested secret-store channel.

git grep -i for the old value is empty across all tracked files.
2026-08-07 07:40:41 -04:00
Joseph Doherty 2aac29618e Merge branch 'fix/sec-33-34'
ci / java (push) Successful in 2m16s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m25s
ci / portable (push) Successful in 8m32s
2026-08-07 06:51:31 -04:00
Joseph Doherty 193daa9ee8 fix(SEC-33,SEC-34): address code review — missed docs, key-id guard comment, test consolidation
Same-commit docs rule (were missed in the prior commit):
- docs/GalaxyRepository.md: SnapshotCachePath now documents the per-OS derived
  default and the GalaxyRepositoryOptionsValidator rooting/validity enforcement.
- A2-galaxyrepository-adoption-handoff.md: correct the now-inaccurate NSSM caveat
  (SnapshotCachePath override is optional, not required; blank seeds a rooted host
  default, no silent no-op) and repoint the option-validation item at the new
  GalaxyRepositoryOptionsValidator.

SEC-34 guard confirmed and documented: TryParseKeyId's '_' split cannot truncate a
key id because both — and the only — gateway key-creation paths
(ApiKeyAdminCommandLineParser.IsValidKeyId, DashboardApiKeyManagementService.ValidateKeyId)
restrict key ids to IsAsciiLetterOrDigit || '.' || '-', and key ids are never
library-generated. Added a citing comment; no behavior change.

Test consolidation: moved the three host-start SqlitePath overrides into
TestHostEnvironmentInitializer (per-process temp store, mirroring Secrets__SqlitePath)
so future host-start tests auto-cover.
2026-08-07 06:49:24 -04:00
Joseph Doherty 7e7f7cad84 fix(SEC-33,SEC-34): host-meaningful path rooting; verification-cache invalidate race
SEC-33: make rooting host-meaningful and stop shipping foreign-platform literals.
- Delete IsRootedForAnyPlatform; AddIfNotRooted now uses Path.IsPathRooted (current OS).
- Promote AddIfNotRooted/AddIfInvalidPath to shared GatewayConfigPathRules so the new
  Galaxy validator reuses them and the two validators cannot drift.
- Remove Authentication:SqlitePath and Galaxy:SnapshotCachePath Windows literals from
  appsettings.json; the CommonApplicationData-derived code defaults take over. The
  Galaxy default is seeded as a configuration value before AddZbGalaxyRepository
  (SnapshotCachePath is init-only, so a PostConfigure mutation cannot compile).
- New GalaxyRepositoryOptionsValidator (ValidateOnStart) enforces a valid, host-rooted
  SnapshotCachePath when PersistSnapshot is true.
- Root-cause the stray junk-named auth DB: host start eagerly builds
  AuthSqliteConnectionFactory; under the non-rooted Windows literal on macOS SQLite
  wrote it relative to the test bin CWD. The three real-host-start tests now pin
  SqlitePath to a temp path.

SEC-34: verification cache Invalidate-vs-in-flight-repopulation race closed with a
per-key generation counter (bump-before-evict, snapshot-then-recheck). The expiry
cap (window 2) takes the documented fallback: the library verification identity
carries no ExpiresUtc, so the cache cannot cap at the key's expiry (donor-library ask).

GWC-24 rider: cap MxGateway:Events:QueueCapacity at int.MaxValue/2 so the derived
checked(2 * EventChannelCapacity) in WorkerClient cannot overflow at session creation.

SEC-35 (doc-only): note IsProduction() env-name semantics in GatewayConfiguration.md.

Docs updated same commit (GatewayConfiguration.md, Authentication.md) and tracking
registers/change-log flipped (00-tracking.md, 40-security-dashboard.md).
2026-08-07 06:36:01 -04:00
Joseph Doherty d2bb32d97b Merge branch 'fix/gwc-28-29-30-polish'
ci / java (push) Successful in 2m44s
ci / windows-x86 (push) Successful in 1m3s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 17m13s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
2026-08-07 06:20:14 -04:00
Joseph Doherty eeee3e48a3 fix(GWC-30): reuse the frame reader's length-prefix scratch buffer
ReadAsync allocated a fresh 4-byte array per inbound frame; the GWC-08
pass pooled the payload buffer but left the prefix. Replaced with a
per-instance scratch field — the reader is single-consumer by
construction (one read loop per WorkerClient, handshake reads complete
before the loop starts), so a per-instance buffer is safe and the
non-reentrancy that makes it safe is now stated on the class. Pooling
four bytes via ArrayPool would cost more than the allocation it saves.

Tests: WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_
ParsesEveryFrame reads five frames of differing payload length through
one reader, so a stale prefix carried between calls would misparse.
2026-08-07 06:15:39 -04:00
Joseph Doherty a044f92c5d fix(GWC-29): drop the wasted request clone on the Invoke hot path
Invoke deep-cloned the whole MxCommandRequest — including its command
payload, potentially a large bulk-write graph — only to overwrite the
cloned command with commandToInvoke and discard it. MapCommand then did
the one clone actually needed. Net cost: a full wasted command deep-clone
per Invoke, worst for exactly the bulk writes that are largest.

Adds a MapCommand(MxCommand) overload (MapCommand reads nothing else off
the request) and has Invoke pass commandToInvoke directly; the request
overload delegates so other callers are untouched.

The remaining clone inside MapCommand stays and is now documented as
required rather than incidental: commandToInvoke may be the gRPC-owned
request.Command, and the caller reads it again after dispatch via
TrackCommandReply, so ownership transfer (à la GWC-07) is not safe here.
That clone is what keeps WorkerClient.CreateCommandEnvelope's no-aliasing
invariant true.

Tests: MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload
(mutating the input leaves the mapped command untouched; both overloads
produce equal results under a fixed TimeProvider).
2026-08-07 06:15:31 -04:00
Joseph Doherty f27eb28063 fix(GWC-28): stamp gateway envelope sequence at write, not construction
CreateEnvelope stamped Sequence with an interlocked increment when the
envelope was built, so two concurrent InvokeAsync callers could take 1
and 2 and then enqueue in the order 2, 1 — non-monotonic on the wire,
breaking gateway.md's "monotonic per sender" contract. Benign today
(neither side validates inbound sequence, old GWC-10 still open) but it
would fault healthy sessions the moment worker-side validation lands.

WriteLoopAsync now stamps immediately before _writer.WriteAsync. It is
the outbound channel's single consumer (SingleReader = true), so wire
order and stamp order are the same thing by construction and
_nextSequence drops to a plain ulong with no interlocking. This mirrors
the worker's WRK-04 fix, which the gateway half never received.

Also adds TST-28: a [Theory] pinning that GatewayHello.MaxFrameBytes
carries the configured worker-frame maximum (default + 2 MiB override).
The adoption half is asserted only in the Windows-only worker suite, so
a regression to sending 0 — "older gateway, use default" to the worker —
would silently downgrade the negotiated IPC-02 limit with every CI test
still green. Mutation-checked (hard-coded 0 fails both cases).

Tests: WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequences
OnTheWire (32 parallel invokes; failed 3/3 pre-fix) and
.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes.
2026-08-07 06:15:20 -04:00
Joseph Doherty 3f854d6cbf Merge branch 'fix/sec-31-32-limiter'
ci / windows-x86 (push) Successful in 1m21s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m23s
ci / portable (push) Successful in 9m4s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
2026-08-07 06:13:14 -04:00
Joseph Doherty 5b681ee59b fix(SEC-31,SEC-32): identify a probe-slot reservation by version, not by timestamp
ReleaseProbe recognised its own reservation by comparing NextProbeAtTicks to
now + _probeIntervalTicks. RecordInto's rearm-on-trip writes that identical
expression, so a concurrent RecordFailure on the same WindowState whose `now`
lands on the claimer's tick — routine at ~1 ms clock resolution under load — was
mistaken for the caller's own claim. The release then stomped the legitimate
fresh re-arm back to the stale previousProbeAtTicks, which is already due, handing
the next arrival a free probe the re-arm had just closed.

WindowState gains a monotonic ProbeVersion bumped by every writer of
NextProbeAtTicks (TryConsumeProbe's claim and RecordInto's re-arm alike).
TryConsumeProbe returns the stamp it set as part of a ProbeClaim; ReleaseProbe
restores the previous value only while the state's version still equals that
stamp, checking and restoring in one lock(state) section and bumping the version
again on restore so no other stale release can match either.

Test: ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick, with the clock held
still so the claim and the interleaved failure necessarily share a tick. Making it
deterministic needed a seam — the claim-to-release window is a few nanoseconds and
racing threads do not hit it (an earlier thread-based attempt passed against the
defective guard three runs out of three, and its end state was ordering-dependent
rather than correctness-dependent, so it was dropped rather than shipped as
theatre). The seam is an internal ProbeReleaseInterleaveHook, null in production,
costing one null check on the already-refused path. Verified as a genuine red
against the timestamp guard: Expected ThrottledByPeer, Actual ProbeAdmitted.
2026-08-07 06:10:14 -04:00
Joseph Doherty 6092172694 Merge branch 'fix/gwc-26-27-alarm-attach'
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m4s
ci / portable (push) Successful in 7m8s
# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
#	archreview/2026-07-12/remediation/10-gateway-core.md
2026-08-07 06:02:17 -04:00
Joseph Doherty 09ccd9561f docs(GWC-26): record alarm feed repairs as at-least-once; share the channel worker fake
Code-review follow-up on fix/gwc-26-27-alarm-attach.

ApplyReconcile's snapshot-derived feed repairs are at-least-once, not
exactly-once: a reconcile reads the worker's current state while the matching
live transition may still be buffered in the monitor's lease, so both broadcast
and the duplicates are indistinguishable on the alarm feed. This pre-dates the
acked-state delta — the Raise/Clear presence repair has always had it, since
nothing serializes a reconcile pass against the in-flight live stream — so
closing it (serialization or timestamp dedup) stays out of scope for a P2 fix.
Documented instead, with the consumer contract stated explicitly (apply
transitions idempotently, never as an increment or toggle):

- ApplyReconcile gains a "Delivery semantics" comment.
- gateway.md softens the "defense in depth" prose to state the semantics.
- docs/Sessions.md carries the same caveat on the alarm-feed description.
- Tracker change-log records it as a known pre-existing characteristic and a
  candidate finding for the next review cycle.

Also hoists the ChannelWorkerClient fake — duplicated across the three alarm
test files — into TestSupport/, dropping the usings it took with it.
2026-08-07 06:00:16 -04:00