R21-O32: SessionLeaseMonitorHostedService gets FakeTimeProvider tests (no sweep
before the interval, one sweep per interval with the clock's now, a failing
sweep does not stop the loop, interval floor, stop ends the loop), and
OrphanWorkerCleanupHostedService gets startup tests (sweeps once, failure
swallowed) plus a registration-order test (orphan cleanup before the session
hosted services). TickObservingTimeProvider gains WaitForWatchedTimersAsync so
Advance never races a service that has not armed its timer yet.
R21-O29: the 250 ms negative-assertion sleep in the stopping-WorkerClient test
is replaced by disposing the client and awaiting the pipe read-ahead to EOF.
Remaining real-clock sleeps (dashboard feed/publisher, alarm monitor tests)
need TimeProvider seams in production code and are left open.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds WorkerExecutableHealthCheck (configured worker .exe exists) and
GalaxySqlHealthCheck (IGalaxyRepository.TestConnectionAsync bounded by a 5 s
TimeProvider-driven timeout). Both are tagged ready only, never live, so an
outage takes the node out of rotation without triggering a restart; sessions
stay on the active tier so zero sessions is still healthy for Overview.
Registrations move into AddGatewayHealthChecks() so the tier tagging is
unit-tested. LDAP is not added (sign-in already surfaces bind failures).
docs/Diagnostics.md lists the two new checks.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HeartbeatLoopAsync waited with a plain Task.Delay, so heartbeat grace/stuck
detection could not be driven with a fake clock. It now uses
Task.Delay(interval, _timeProvider, ct).
Adds fake-time tests (grace boundary then fault; in-flight command within the
stuck ceiling) built on a new TickObservingTimeProvider, and replaces the
150 ms real-clock sleep in the within-ceiling negative test (mxaccessgw#205).
Both new tests fail without the fix.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The replay ring was bounded only by ReplayBufferCapacity (1024 events) and
ReplayRetentionSeconds. Event size is unbounded up to the worker frame size,
so a burst of large-array events could pin gigabytes per session while the
count looked small.
New MxGateway:Events:ReplayBufferMaxBytes (default 64 MiB, 0 disables) caps
the summed serialized size (MxEvent.CalculateSize, measured only when the cap
is on). Over the cap the oldest entries are evicted, exactly like the count
cap, so a resuming subscriber sees the existing ReplayGap path; an event bigger
than the whole cap is not retained. Every eviction (count, age or byte) now
clears its slot, so an evicted event is released immediately instead of
staying referenced until the slot is reused.
The 64 MiB default is a judgement call: it never binds for scalar traffic
(1024 x a few hundred bytes) and only trims large-array bursts.
Tests: SessionEventDistributorTests byte-cap eviction, single oversized event,
cap 0 = count only (the first two fail with the cap enforcement removed);
validator rejects a negative value. Docs: GatewayConfiguration.md, Sessions.md,
DesignDecisions.md; appsettings.json carries the key.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-O14: MxGateway:Worker:HeartbeatIntervalSeconds was documented as the
worker heartbeat send interval, but it never reached the worker (which sends
on its own fixed 5 s) and only set the gateway's HeartbeatCheckInterval. It is
renamed HeartbeatCheckIntervalSeconds (default 5, unchanged). The old key stays
as a deprecated alias with the same meaning so existing appsettings keep
working; setting both is a validation error. The grace-vs-interval check now
compares against the resolved check interval.
R21-O15: WorkerClientOptions.EventChannelFullModeTimeout and
HeartbeatStuckCeiling were always their compile-time defaults. They are now
MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds (default 5000, > 0)
and HeartbeatStuckCeilingSeconds (default 75, >= HeartbeatGraceSeconds). The
factory's mapping moved into SessionWorkerClientFactory.CreateWorkerClientOptions
so every field is set from configuration and is testable. The effective-config
record shows the resolved check interval and both new values.
Docs: GatewayConfiguration.md, GatewayProcessDesign.md sample,
MxAccessWorkerInstanceDesign.md heartbeat section; appsettings.json uses the
new keys.
Tests: CreateWorkerClientOptions mapping + defaults (fail under a mutated
mapping), alias honoured, both-keys conflict, and bounds for the new options.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
GetReadyWorkerClientAsync re-read the session and worker states every 25 ms
until the WorkerReadyWaitTimeoutMs deadline. It now waits on two
TaskCompletionSource signals - one completed by every GatewaySession _state /
_workerClient write, one by every WorkerClient _state write - and re-evaluates
only when one fires or the remaining budget elapses. Both signals are captured
under the lock before the state is evaluated, so a transition between the
evaluation and the await is not lost.
IWorkerClient gains GetStateChangedSignal(), a default interface member that
returns null; a client without a signal (the test fakes) is re-read on the old
25 ms interval, so the fakes need no change. The production WorkerClient always
publishes one. Default behaviour (WorkerReadyWaitTimeoutMs = 0, fail fast) is
unchanged.
Regression tests (SessionManagerTests): ...WakesWithoutPollTimer and
...FailsFastWithoutPollTimer never advance the FakeTimeProvider, so they hang
and fail against the poll loop; WorkerClientTests.GetStateChangedSignal_
CompletesOnEachTransition pins the worker-side signal.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-O12: GatewaySession already owned an async DisposeAsync teardown but did not
declare IAsyncDisposable, so anything holding it as IAsyncDisposable (or a DI
container) could not find the teardown. Declaring it changes no call site;
the existing await-using sites bound to the same method by pattern.
R21-O13: KillWorker(string) had no callers and was an ungated kill that
bypassed _closeLock. Deleted it; docs/Sessions.md and the test comments now
name KillWorkerWithCloseGateAsync, the real kill path.
Regression test: GatewaySessionTests.GatewaySession_ImplementsIAsyncDisposable
(fails before the interface is declared).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The launcher's Polly startup-probe pipeline was its only recorder and was
removed in mxaccessgw#188, so the counter, the RetryAttempted method and the
snapshot's RetryAttempts / RetryAttemptsByArea fields carried a permanent
zero. No dashboard page or in-repo alert rule references them. Metrics.md
drops the row and the snapshot shape (which also gains the missing
AlarmProviderSwitchCount field).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
D11: BrowsePage search CTS is cancelled+disposed exactly once under a lock.
D12: the hub refreshes the cached health status before seeding a new connection.
D13: DashboardEventBroadcaster gains DisposeAsync that awaits the pumps (bounded).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
O19: reject a __Host- CookieName with RequireHttpsCookie=false at startup.
O20: project DisableLogin, AutoLoginUser, RequireHttpsCookie and CookieName into
EffectiveDashboardConfiguration and show them on the Settings page.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
--pepper on the command line lands in shell history and the process list.
Add --pepper-stdin (first stdin line), warn on stderr when --pepper is used,
and reject both together. Documented in GatewayProcessDesign.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Set retainedFileCountLimit=30, fileSizeLimitBytes=50 MiB and rollOnFileSizeLimit=true on the
file sink in appsettings.json and appsettings.Development.json; document under a new Logging
section in docs/GatewayConfiguration.md; add a guard test over the shipped config.
Adds SparseArrayException so the Sessions layer no longer throws Grpc.Core.RpcException;
MxAccessGatewayService.MapException maps it to StatusCode.InvalidArgument (client-visible
behaviour unchanged). Adds a service-level mapping test.
Auth.ApiKeys 0.3.0 throws a plain InvalidOperationException("pepper unavailable")
with no typed exception to catch, so take the issue's second option: check
IApiKeyPepperProvider before Create/Rotate and drop the message-matching catch.
Adds create/rotate missing-pepper tests.
The generation counter and key-id eviction index used ordinal comparison, so an
Invalidate spelled with different case than the presented token missed the guard.
Both now use OrdinalIgnoreCase; two tests added (fail without the fix).
WorkerProcessStartedProbe only checks HasExited, so a retry could never change
the outcome. Remove StartupProbeRetryAttempts/StartupProbeRetryDelayMilliseconds,
their validation, the launcher's Polly retry pipeline and the docs; the probe now
runs once under StartupTimeoutSeconds. Stale config keys are ignored by the binder.
0.2.1 on the feed predates the 488649c AcknowledgeAlarmRequest
operator_password/operator_domain fields; 0.3.0 ships them so OtOpcUa can
adopt operator-authenticated alarm ack (lmxopcua#749).
NT AUTHORITY\NETWORK marks a network logon, not a remote caller. On windev
a gateway run from an SSH session carries it in its own token, so the deny
entry locked the gateway's same-machine workers out (UnauthorizedAccess on
connect; 15 fake-worker tests failed). The ACL now holds exactly one entry:
FullControl for the gateway's user.
On Windows each worker pipe is created via NamedPipeServerStreamAcl with a
protected DACL granting FullControl only to the gateway process user (the
worker is launched under the same account) and denying NETWORK logons. On
non-Windows hosts (test matrix) the pipe uses PipeOptions.CurrentUserOnly.
Tests assert the factory ACL and the ACL read back from the created pipe
(Windows), and that a same-user client still connects.
Adds both secured bulk kinds to GatewayLogRedactor's credential-bearing set,
plus a contract cross-check test that walks every MxCommand payload descriptor
and fails if a secured or password-bearing command kind is not redacted.
Stream E made the worker wait once for all uncached tags. Fix the Go, Python, Rust, Java
and .NET library docs and CLI help text, the proto comment (default is the worker's
1000 ms), and the worker design doc. Regenerated Contracts Generated/ and the Java
generated protos; both diffs are comment-only.
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
- TryAttachWorkerClientAndMarkReadyAsync returns SessionAttachOutcome and refuses a
worker client that is already Faulted/Closed; the open discards it and fails OpenFailed.
- MapWorkerEventsAsync calls ObserveWorkerTerminalState when its first readiness check
fails, so a worker that dies between attach and the first pump read faults the session
and the faulted sweep reaps it.
- A caller-cancelled open rolls back, records no fault and rethrows the cancellation
(gRPC Cancelled); an attach refused because the drain closed the session reports
GatewayStopping.
- docs/Sessions.md updated.
- AddItem/AddItem2/AddBufferedItem drop stale cache entries for a reused handle id
- ReadBulk: dedupe handles within a batch (advise/teardown once, fan out result);
pump every 16 items in add/advise and teardown phases
- alarm poll stops without recording a fault once StaRuntime.LoopFault is set
- StaRuntime skips the loop-fault handler during a requested shutdown
- StaHung loop fault diagnostic carries type + HRESULT only
- gateway.md documents ReadBulk's shared deadline
Live windev finding: after a client sent the ShutdownWorker control command
the worker exited 0 and WorkerClient moved to Closed on the shutdown ack,
but GatewaySession.State stayed Ready (even with an event stream attached)
until its lease expired.
IWorkerClient has no state-change callback, so the session observes the
worker from its own worker-event pump source: WorkerClient enters
Closed/Faulted before completing the event channel, so when the enumeration
ends ObserveWorkerTerminalState sees the terminal state and faults a Ready
session. Faulted rather than Closed: Closed means the gateway closed it and
would linger in the registry; a faulted session stays observable with the
reason and the faulted-session sweep reaps it, accounting SessionClosed
exactly once. Only a Ready session follows its worker, so a gateway-
initiated close is unaffected.
FakeWorkerHarness-based tests cover the ShutdownWorker exit and a broken
worker pipe.
- The metrics scheme skips its dashboard-cookie fallback while
Dashboard:DisableLogin is on, so the dev auto-login handler cannot
satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
vs gRPC per connection, so only the per-key aggregate is shared.
- Open path: the attach gate now waits on CancellationToken.None and the
manager discards (kills + disposes) the late worker client on ANY
exception out of the attach step, including the caller's own cancellation
(checked first, so an abandoned open never yields a session). Previously
an OperationCanceledException from the gate wait escaped before the
discard and orphaned the x86 worker. The gate Release is guarded against
ObjectDisposedException.
- Unattached grace: AttachInternalEventSubscriber sets a separate latch so
the gateway's own alarm-monitor session (internal subscriber only) is
never reaped; the idle clock starts at Ready (MarkReady stamps
_readyAtUtc) instead of at open, so a slow worker start does not eat the
window.
- KillWorkerAsync / CloseSessionCoreAsync: an ObjectDisposedException from
the close gate on a session that is no longer registered means a
concurrent close already removed, accounted and disposed it; both now
return AlreadyClosed with no fault, no second gauge decrement and no sweep
failure. The general kill catch only decrements the gauge when the
session is still registered. The sweep no longer counts AlreadyClosed
results among the sessions it closed.
New MxGateway:Metrics:RequireAuthentication (default false). When true,
/metrics requires an API key with the admin scope (verified through the
cached verifier and the shared API-key failure limiter) or a dashboard
login with a real dashboard role; anonymous localhost and
Authentication:Mode=Disabled do not qualify. An anonymous scrape gets a
plain 401 Bearer challenge, a throttled peer 429. /health stays anonymous
because the off-repo Overview dashboard polls /health/active.
MapZbMetrics returns IEndpointConventionBuilder, so the policy is attached
with RequireAuthorization.
CachingApiKeyVerifier's key-id index was only pruned by Invalidate, so it
grew with every token ever verified. A post-eviction callback now drops the
slot (under the same lock as Set + index, so a late callback cannot
unindex a re-cached entry). Concurrent misses for the same token and key
generation now share one inner verification, so a TTL expiry under load no
longer stampedes SQLite; keying the flight on the generation keeps the
revoke guarantee for joiners.
A client that opens a session and vanishes before it ever streams (e.g.
OtOpcUa's GalaxyDriver on a failed start) pinned a session slot and a live
x86 worker for the full DefaultLeaseSeconds (1800 s), because detach-grace
only applies to sessions that once had a subscriber.
MxGateway:Sessions:UnattachedSessionGraceSeconds (int, default 0 = off,
validated >= 0) makes the existing lease sweep reap a Ready session that
has never attached an event subscriber once it has been idle (no unary
activity) for the window, with the close reason unattached-grace-expired.
Ready-only, idle-from-last-activity and first-subscriber-cancels keep it
from catching slow starts or unary-only clients.
GatewaySession.ExtendLease had no production callers and did not fit the
rule, so it is removed; tests move leases through TouchClientActivity.
WorkerClient.StageWorkerEvent and StreamEvents called Enum.ToString per
event (per subscriber on the stream path). MxEventFamilyNames serves the
contract-fixed names from a frozen lookup, falling back to ToString for
undefined values so exported tag values are unchanged.
StreamDisconnected always recorded "Detached". The reason tag is now a
bounded EventStreamDisconnectReasons constant for how the stream ended:
SessionClosed, Overflow, WorkerFault, ClientCancelled, Error, or Detached.
- StreamAlarms: slow subscribers are completed with
AlarmFeedSubscriberOverflowException and surface as ResourceExhausted
(like a StreamEvents overflow) instead of warning-with-stack + Unavailable.
- GatewayAlarmMonitor.StreamAsync reuses the memoized alarm-set projection
instead of cloning the whole set under the broadcast lock per attach; the
prefix filter runs outside the lock.
- StreamEvents/StreamAlarms pass the call token to response writes; a write
failure after client cancel maps to Cancelled without a warning.
- Every MxCommandReply.correlation_id now echoes the request's
client_correlation_id (previously only constraint-denied replies did;
normal replies leaked the internal envelope id).
CloseAsync rethrew the graceful-shutdown exception after a successful
Kill, so the manager faulted the session, recorded a CloseFailed fault and
surfaced CloseFailed to the caller for the ordinary hung-worker case.
docs/Sessions.md reserves CloseFailed for both shutdown and kill failing.
A successful kill now completes the close as Closed; the shutdown failure
rides back on SessionCloseResult.WorkerShutdownFailure and the manager
logs it at warning level. Both-fail still raises CloseFailed.
- Search: the oninput setter scanned the whole Galaxy synchronously per
keystroke. It now debounces 150ms, runs on the thread pool with
cancellation (newer keystroke supersedes), and reads a shared singleton
DashboardBrowseSearchIndex of pre-lowercased references keyed on the
Galaxy cache Sequence.
- Disposal: DisposeAsync awaited both loops unbounded and the loops caught
only OCE. Loops now handle faults per iteration (poll faults surface in
_readError, deploy-notifier faults in a retrying warning), mutate the tree
on the dispatcher, drain concurrently with a 5s bound, and the CTSs are
disposed in finally.
- Circuits: DisconnectedCircuitRetentionPeriod is 30s (was the 3 min
default) and a scoped DashboardCircuitConnectionTracker (CircuitHandler)
lets the 2s ReadBulk poll skip ticks while the browser is disconnected.
- Reads pass a per-page viewer id to the live-data per-viewer share.
ShutdownAsync iterated a single registry snapshot while Kestrel still
accepted OpenSession calls (ApplicationStopping fires before hosted
services stop), so a session opened behind the snapshot had nobody left to
close it and its x86 worker outlived the gateway as an orphan.
SessionManager now observes IHostApplicationLifetime.ApplicationStopping
(and sets the same flag at the start of ShutdownAsync) and rejects
OpenSessionAsync with the new SessionManagerErrorCode.GatewayStopping,
re-checked after registration so the flag write before the drain's
snapshot closes the race. GatewayStopping maps to gRPC Unavailable and is
not a fault. ShutdownAsync re-snapshots until the registry is empty,
bounded at three passes.
The 256-tag LRU was global, so two Browse circuits on different areas evicted
each other's whole sets every poll. Tags now record the viewer that read them
last; each active viewer (read within 30s) owns MaxSubscribedTags/active
(floor 32). Eviction takes other viewers' excess over their share first
(inactive viewers' share is zero), then the reader's own untouched tags, and
never strips a neighbour within its share (soft cap). Single-viewer behaviour
is unchanged. IDashboardLiveDataService gains a viewerId ReadAsync overload.