R21-K12: WorkerFrameWriter.WriteBatchAsync allocated a TaskCompletionSource
(plus its Task) per frame on the event drain hot path. PendingFrame now holds
its terminal state itself (Interlocked pending/succeeded/faulted/canceled) and
only single-frame WriteAsync frames, which race their completion against the
write lock, get a TCS. A batch caller reads each frame's outcome after its own
drain pass: every frame a pass claims is terminal before that pass releases the
lock, so nothing is left to await. First-failure-in-batch-order, tombstoning,
claim/cancel interlock and flush coalescing are unchanged.
R21-K13: WnWrapAlarmConsumer.ParseSnapshotXml streams the GetXmlCurrentAlarms2
payload with an XmlTextReader (same defaults XmlDocument.LoadXml uses:
Normalization off, DTDs parsed, no resolver; Significant whitespace only)
instead of building a full XmlDocument every 500 ms poll.
Behaviour is pinned by new tests: batch first-failure ordering (own drain and
drained by another lock holder), batch cancellation tombstoning, batch flush
failure, and a differential test running the replaced XmlDocument parser as an
oracle over a 48-payload corpus (CRLF, CDATA, entities, DTD, namespaces,
nesting, malformed). Mutation checks on windev: 8 of 8 injected regressions
(whitespace, normalization, DTD, namespace, CDATA, batch last-failure, batch
ignored failures, ignored tombstone) fail the new tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
WorkerPipeSession had seven CreateEnvelope overloads that each forwarded to a
matching CreateBaseEnvelope overload; they are now one overload per body type
that stamps the header and sets the body. WorkerPipeClient's six chained
constructors collapse to one with optional arguments (null attempt timeout =
env/default, null session factory = production WorkerPipeSession). Adds
constructor-validation tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Pure move: cli definition, per-group handlers (session, bulk, bench, events,
galaxy, smoke, batch) and shared output/parse/connect helpers now live in
their own modules. No behaviour or CLI surface change; --help of all 33
commands is byte-identical, clippy clean, tests unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The two start-gate waits in GatewaySessionTests ignored the bool from
WaitAsync(timeout), so a wait that timed out let the task run on
un-synchronised and the race test could pass without racing. Both now
Assert.True the result. Every other timed semaphore wait in the Tests
project was already checked (the #205 lease helper, the new CountSignal)
or is token-only.
Mutation check: with Release(n-1) and a 200 ms gate wait, the original
code passes (vacuous) while the fixed code fails with "Timed out at the
start gate."
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DashboardSnapshotPublisher takes an optional TimeProvider (default
TimeProvider.System) driving the idle poll, the reconnect delay and, via
BoundedHubSend, the send timeout. Its tests now run on a fake clock with
exact assertions (reconnect gap == delay, 25 idle polls pull nothing, a
wedged send is abandoned at the fake send timeout).
DashboardSnapshotFeed has no time dependence, so no seam was added; its
tests lose the 100 ms sleep and poll loops in favour of CountSignal
signals. The Task.Delay(25) poll copies in the alarm-monitor tests and
several 5-10 ms poll loops now share TestSupport/Eventually.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The detail-code-to-text table had no direct tests; only three codes were
covered incidentally through MxStatusProxyConverterTests. The new
MxStatusDetailTextTests pins every known code's text, checks codes
outside the table (including the gaps inside it and the int extremes)
map to empty, sweeps -16..20000 to assert the table holds exactly the
pinned codes, and checks every text is non-blank, trimmed and distinct.
(mxaccessgw#178)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-O3: WorkerApplication.Run's failure lines logged only the exception
type. Each now also carries exception_message, scrubbed by the new
WorkerLogRedactor.RedactMessage: the value of any name=value / name: value
pair whose name looks sensitive and any mxgw_ API key token are replaced,
line breaks are flattened and the text is capped at 512 characters. The
AppDomain unhandled-exception hook stays type-only, since an exception
from an arbitrary thread can carry command values.
R21-O7: StaCommandDispatcher.ExecuteQueuedCommandAsync logs
WorkerCommandStarted (correlation id, method, queued_ms) and
WorkerCommandCompleted (status, HRESULT, elapsed_ms) when a CommandLogger
is set. IWorkerLogger has no Debug level, so the lines are opt-in through
MXGATEWAY_WORKER_LOG_COMMANDS=1|true (WorkerCommandLogging), which the
worker inherits from the gateway process environment. No payloads or
diagnostic messages are logged. Docs updated. (mxaccessgw#184)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A frame read that hits end of stream means the gateway closed its end of
the pipe (usually because it died). WorkerApplication mapped every
WorkerFrameProtocolException to ProtocolViolation (6); EndOfStream now
exits as PipeConnectionFailed (5), alongside the IOException a broken
pipe raises.
Event-queue overflow exited as the generic UnexpectedFailure (1). The
event drain loop now unwinds with WorkerEventQueueFaultException carrying
the fault category, and QueueOverflow exits with the new
WorkerExitCode.EventQueueOverflow (8). Other categories keep exit 1.
The gateway does not decode worker exit codes (it only logs the number),
so no gateway change is needed. WorkerBootstrap.md and
MxAccessWorkerInstanceDesign.md updated. (mxaccessgw#186)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
TryStartCommandTask returned silently when the session had stopped
accepting commands, so a command racing shutdown got no reply and no log
and the gateway's correlation wait ran to its timeout. The message loop
now answers such a command with a WorkerUnavailable reply (the status the
STA dispatcher already uses during shutdown) and logs
WorkerCommandRejectedShuttingDown. New test
CommandAfterShutdown_RepliesWorkerUnavailable (mxaccessgw#182)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-O32: SessionLeaseMonitorHostedService gets FakeTimeProvider tests (no sweep
before the interval, one sweep per interval with the clock's now, a failing
sweep does not stop the loop, interval floor, stop ends the loop), and
OrphanWorkerCleanupHostedService gets startup tests (sweeps once, failure
swallowed) plus a registration-order test (orphan cleanup before the session
hosted services). TickObservingTimeProvider gains WaitForWatchedTimersAsync so
Advance never races a service that has not armed its timer yet.
R21-O29: the 250 ms negative-assertion sleep in the stopping-WorkerClient test
is replaced by disposing the client and awaiting the pipe read-ahead to EOF.
Remaining real-clock sleeps (dashboard feed/publisher, alarm monitor tests)
need TimeProvider seams in production code and are left open.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds WorkerExecutableHealthCheck (configured worker .exe exists) and
GalaxySqlHealthCheck (IGalaxyRepository.TestConnectionAsync bounded by a 5 s
TimeProvider-driven timeout). Both are tagged ready only, never live, so an
outage takes the node out of rotation without triggering a restart; sessions
stay on the active tier so zero sessions is still healthy for Overview.
Registrations move into AddGatewayHealthChecks() so the tier tagging is
unit-tested. LDAP is not added (sign-in already surfaces bind failures).
docs/Diagnostics.md lists the two new checks.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HeartbeatLoopAsync waited with a plain Task.Delay, so heartbeat grace/stuck
detection could not be driven with a fake clock. It now uses
Task.Delay(interval, _timeProvider, ct).
Adds fake-time tests (grace boundary then fault; in-flight command within the
stuck ceiling) built on a new TickObservingTimeProvider, and replaces the
150 ms real-clock sleep in the within-ceiling negative test (mxaccessgw#205).
Both new tests fail without the fix.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The replay ring was bounded only by ReplayBufferCapacity (1024 events) and
ReplayRetentionSeconds. Event size is unbounded up to the worker frame size,
so a burst of large-array events could pin gigabytes per session while the
count looked small.
New MxGateway:Events:ReplayBufferMaxBytes (default 64 MiB, 0 disables) caps
the summed serialized size (MxEvent.CalculateSize, measured only when the cap
is on). Over the cap the oldest entries are evicted, exactly like the count
cap, so a resuming subscriber sees the existing ReplayGap path; an event bigger
than the whole cap is not retained. Every eviction (count, age or byte) now
clears its slot, so an evicted event is released immediately instead of
staying referenced until the slot is reused.
The 64 MiB default is a judgement call: it never binds for scalar traffic
(1024 x a few hundred bytes) and only trims large-array bursts.
Tests: SessionEventDistributorTests byte-cap eviction, single oversized event,
cap 0 = count only (the first two fail with the cap enforcement removed);
validator rejects a negative value. Docs: GatewayConfiguration.md, Sessions.md,
DesignDecisions.md; appsettings.json carries the key.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-O14: MxGateway:Worker:HeartbeatIntervalSeconds was documented as the
worker heartbeat send interval, but it never reached the worker (which sends
on its own fixed 5 s) and only set the gateway's HeartbeatCheckInterval. It is
renamed HeartbeatCheckIntervalSeconds (default 5, unchanged). The old key stays
as a deprecated alias with the same meaning so existing appsettings keep
working; setting both is a validation error. The grace-vs-interval check now
compares against the resolved check interval.
R21-O15: WorkerClientOptions.EventChannelFullModeTimeout and
HeartbeatStuckCeiling were always their compile-time defaults. They are now
MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds (default 5000, > 0)
and HeartbeatStuckCeilingSeconds (default 75, >= HeartbeatGraceSeconds). The
factory's mapping moved into SessionWorkerClientFactory.CreateWorkerClientOptions
so every field is set from configuration and is testable. The effective-config
record shows the resolved check interval and both new values.
Docs: GatewayConfiguration.md, GatewayProcessDesign.md sample,
MxAccessWorkerInstanceDesign.md heartbeat section; appsettings.json uses the
new keys.
Tests: CreateWorkerClientOptions mapping + defaults (fail under a mutated
mapping), alias honoured, both-keys conflict, and bounds for the new options.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
GetReadyWorkerClientAsync re-read the session and worker states every 25 ms
until the WorkerReadyWaitTimeoutMs deadline. It now waits on two
TaskCompletionSource signals - one completed by every GatewaySession _state /
_workerClient write, one by every WorkerClient _state write - and re-evaluates
only when one fires or the remaining budget elapses. Both signals are captured
under the lock before the state is evaluated, so a transition between the
evaluation and the await is not lost.
IWorkerClient gains GetStateChangedSignal(), a default interface member that
returns null; a client without a signal (the test fakes) is re-read on the old
25 ms interval, so the fakes need no change. The production WorkerClient always
publishes one. Default behaviour (WorkerReadyWaitTimeoutMs = 0, fail fast) is
unchanged.
Regression tests (SessionManagerTests): ...WakesWithoutPollTimer and
...FailsFastWithoutPollTimer never advance the FakeTimeProvider, so they hang
and fail against the poll loop; WorkerClientTests.GetStateChangedSignal_
CompletesOnEachTransition pins the worker-side signal.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-O12: GatewaySession already owned an async DisposeAsync teardown but did not
declare IAsyncDisposable, so anything holding it as IAsyncDisposable (or a DI
container) could not find the teardown. Declaring it changes no call site;
the existing await-using sites bound to the same method by pattern.
R21-O13: KillWorker(string) had no callers and was an ungated kill that
bypassed _closeLock. Deleted it; docs/Sessions.md and the test comments now
name KillWorkerWithCloseGateAsync, the real kill path.
Regression test: GatewaySessionTests.GatewaySession_ImplementsIAsyncDisposable
(fails before the interface is declared).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The launcher's Polly startup-probe pipeline was its only recorder and was
removed in mxaccessgw#188, so the counter, the RetryAttempted method and the
snapshot's RetryAttempts / RetryAttemptsByArea fields carried a permanent
zero. No dashboard page or in-repo alert rule references them. Metrics.md
drops the row and the snapshot shape (which also gains the missing
AlarmProviderSwitchCount field).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Add Dashboard DisableLogin/AutoLoginUser/CookieName, Alarms:Fallback and Tls,
matching the option classes and the per-option tables below the block.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
D11: BrowsePage search CTS is cancelled+disposed exactly once under a lock.
D12: the hub refreshes the cached health status before seeding a new connection.
D13: DashboardEventBroadcaster gains DisposeAsync that awaits the pumps (bounded).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
O19: reject a __Host- CookieName with RequireHttpsCookie=false at startup.
O20: project DisableLogin, AutoLoginUser, RequireHttpsCookie and CookieName into
EffectiveDashboardConfiguration and show them on the Settings page.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
--pepper on the command line lands in shell history and the process list.
Add --pepper-stdin (first stdin line), warn on stderr when --pepper is used,
and reject both together. Documented in GatewayProcessDesign.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Set retainedFileCountLimit=30, fileSizeLimitBytes=50 MiB and rollOnFileSizeLimit=true on the
file sink in appsettings.json and appsettings.Development.json; document under a new Logging
section in docs/GatewayConfiguration.md; add a guard test over the shipped config.
Adds SparseArrayException so the Sessions layer no longer throws Grpc.Core.RpcException;
MxAccessGatewayService.MapException maps it to StatusCode.InvalidArgument (client-visible
behaviour unchanged). Adds a service-level mapping test.
Auth.ApiKeys 0.3.0 throws a plain InvalidOperationException("pepper unavailable")
with no typed exception to catch, so take the issue's second option: check
IApiKeyPepperProvider before Create/Rotate and drop the message-matching catch.
Adds create/rotate missing-pepper tests.
The generation counter and key-id eviction index used ordinal comparison, so an
Invalidate spelled with different case than the presented token missed the guard.
Both now use OrdinalIgnoreCase; two tests added (fail without the fix).
WorkerProcessStartedProbe only checks HasExited, so a retry could never change
the outcome. Remove StartupProbeRetryAttempts/StartupProbeRetryDelayMilliseconds,
their validation, the launcher's Polly retry pipeline and the docs; the probe now
runs once under StartupTimeoutSeconds. Stale config keys are ignored by the binder.
ExecuteSafeUnaryAsync capped the whole retry pipeline at one DefaultCallTimeout
while each attempt carried the same deadline, so a DeadlineExceeded first
attempt consumed the whole budget and no retry ever ran. The outer cap is now
MaxAttempts x DefaultCallTimeout + (MaxAttempts-1) x MaxDelay in both
MxGatewayClient and GalaxyRepositoryClient. Adds a slow-first-attempt test
(fails without the fix) and a budget unit test.
0.2.1 on the feed predates the 488649c AcknowledgeAlarmRequest
operator_password/operator_domain fields; 0.3.0 ships them so OtOpcUa can
adopt operator-authenticated alarm ack (lmxopcua#749).
NT AUTHORITY\NETWORK marks a network logon, not a remote caller. On windev
a gateway run from an SSH session carries it in its own token, so the deny
entry locked the gateway's same-machine workers out (UnauthorizedAccess on
connect; 15 fake-worker tests failed). The ACL now holds exactly one entry:
FullControl for the gateway's user.
On Windows each worker pipe is created via NamedPipeServerStreamAcl with a
protected DACL granting FullControl only to the gateway process user (the
worker is launched under the same account) and denying NETWORK logons. On
non-Windows hosts (test matrix) the pipe uses PipeOptions.CurrentUserOnly.
Tests assert the factory ACL and the ACL read back from the created pipe
(Windows), and that a same-user client still connects.
Adds both secured bulk kinds to GatewayLogRedactor's credential-bearing set,
plus a contract cross-check test that walks every MxCommand payload descriptor
and fails if a secured or password-bearing command kind is not redacted.
Stream E made the worker wait once for all uncached tags. Fix the Go, Python, Rust, Java
and .NET library docs and CLI help text, the proto comment (default is the worker's
1000 ms), and the worker design doc. Regenerated Contracts Generated/ and the Java
generated protos; both diffs are comment-only.
- Sessions.md / GatewayConfiguration.md: the dashboard Browse live-data session is
unary-only, is reaped by UnattachedSessionGraceSeconds when idle, and reopens via
EnsureReadyAsync; only the alarm monitor is exempt.
- GatewayConfiguration.md: a refused second subscriber gets ResourceExhausted, not AlreadyExists.
- Grpc.md: rewrite the event streaming model around the SessionEventDistributor pump and
replay ring; add EventSubscriberLimitReached to the mapping snippet; describe what
clients see while the gateway is stopping.