Readiness covered only the auth store, so a gateway whose worker exe was
missing or of the wrong architecture reported ready while every OpenSession
failed. The new ready-tier check worker-executable applies the launcher's own
rule (path resolution, .exe, presence, PE architecture), now shared as
WorkerExecutableValidator.ResolveAndValidate, and reports only the launch
error code (the endpoint is anonymous). The session check stays on the
active tier, so zero sessions keeps readiness green.
The bounded Galaxy SQL check is not in this change: its tier, failure status
and probe cadence need a decision (see the issue).
Tests: WorkerExecutableHealthCheckTests (healthy, missing, wrong architecture,
and the ready-tier registration, which fails without the wiring).
Over TLS without -ca-cert the CLI was always skip-verify, with no way to ask
for verification against the system roots even though the library supports
it. The new common flag is wired to Options.RequireCertificateValidation in
both the gateway and Galaxy dialers and echoed in the redacted JSON options.
Test: tls_flag_test.go starts a self-signed TLS gRPC gateway; the default
open-session succeeds (positive control) and the same call with the flag
fails to dial (fails on the old code with 'flag provided but not defined').
Every RPC failure surfaced as a GatewayError with no way to tell a rejected
key from a missing scope, though docs/ClientLibrariesDesign.md requires the
two be treated distinctly. GatewayError now matches the new sentinels
ErrUnauthenticated and ErrPermissionDenied via errors.Is (by gRPC status code
only, so it works through the secret-redaction wrapper) and exposes Code().
Tests: errors_auth_test.go — sentinel matching per code, Code(), and a bufconn
AcknowledgeAlarm round trip proving the match survives the wire and the
password stays redacted (both behavioural tests fail without GatewayError.Is).
ExecuteSafeUnaryAsync capped the whole retry pipeline at one
DefaultCallTimeout while each attempt carried the same deadline, so a first
attempt ending in DeadlineExceeded spent the entire budget and the retry
never ran. The outer cap is now MaxAttempts x DefaultCallTimeout plus MaxDelay
per backoff (MxGatewayClientRetryPolicy.SafeUnaryBudget), in both
MxGatewayClient and GalaxyRepositoryClient; a budget too large for a
cancellation timer leaves only the per-attempt deadlines.
Tests: MxGatewayClientWireTests.SafeUnaryRetry_AfterDeadlineExceededFirstAttempt_RunsTheRetry
(real gRPC loopback server stalls the first Invoke; fails with Cancelled on
the old code), plus two SafeUnaryBudget unit tests.
WorkerProcessStartedProbe only checks HasExited and throws a non-retryable
WorkerProcessLaunchException, so StartupProbeRetryAttempts and
StartupProbeRetryDelayMilliseconds configured a Polly retry that no shipped
probe could trigger. Real readiness is WorkerClient's WorkerReady handshake
under the same StartupTimeoutSeconds budget, so the probe now runs once and
the two options, their validation, the Polly pipeline (and the server's now
unused Polly.Core reference) and their docs are removed. A stale value in
configuration is ignored by the binder.
Tests: WorkerProcessLauncherTests.LaunchAsync_WhenStartupProbeFails_DoesNotRetryTheProbe,
WorkerOptions_HasNoStartupProbeRetryOptions (both fail on the old code).
TryStartCommandTask returned silently when command intake had stopped, so a
command refused during shutdown got no reply and no log and the gateway's
correlation wait ran out its own timeout. The refusal now writes a
WorkerUnavailable reply for the correlation and logs
WorkerCommandRejectedAfterShutdown. StopAcceptingCommands is internal so the
test can reach the guard (the serial read loop cannot race it today).
Test: WorkerPipeSessionTests.CommandAfterShutdown_RepliesWorkerUnavailable.
0.2.1 on the feed predates the 488649c AcknowledgeAlarmRequest
operator_password/operator_domain fields; 0.3.0 ships them so OtOpcUa can
adopt operator-authenticated alarm ack (lmxopcua#749).
NT AUTHORITY\NETWORK marks a network logon, not a remote caller. On windev
a gateway run from an SSH session carries it in its own token, so the deny
entry locked the gateway's same-machine workers out (UnauthorizedAccess on
connect; 15 fake-worker tests failed). The ACL now holds exactly one entry:
FullControl for the gateway's user.
On Windows each worker pipe is created via NamedPipeServerStreamAcl with a
protected DACL granting FullControl only to the gateway process user (the
worker is launched under the same account) and denying NETWORK logons. On
non-Windows hosts (test matrix) the pipe uses PipeOptions.CurrentUserOnly.
Tests assert the factory ACL and the ACL read back from the created pipe
(Windows), and that a same-user client still connects.
Adds both secured bulk kinds to GatewayLogRedactor's credential-bearing set,
plus a contract cross-check test that walks every MxCommand payload descriptor
and fails if a secured or password-bearing command kind is not redacted.
Stream E made the worker wait once for all uncached tags. Fix the Go, Python, Rust, Java
and .NET library docs and CLI help text, the proto comment (default is the worker's
1000 ms), and the worker design doc. Regenerated Contracts Generated/ and the Java
generated protos; both diffs are comment-only.
- Sessions.md / GatewayConfiguration.md: the dashboard Browse live-data session is
unary-only, is reaped by UnattachedSessionGraceSeconds when idle, and reopens via
EnsureReadyAsync; only the alarm monitor is exempt.
- GatewayConfiguration.md: a refused second subscriber gets ResourceExhausted, not AlreadyExists.
- Grpc.md: rewrite the event streaming model around the SessionEventDistributor pump and
replay ring; add EventSubscriberLimitReached to the mapping snippet; describe what
clients see while the gateway is stopping.
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
- TryAttachWorkerClientAndMarkReadyAsync returns SessionAttachOutcome and refuses a
worker client that is already Faulted/Closed; the open discards it and fails OpenFailed.
- MapWorkerEventsAsync calls ObserveWorkerTerminalState when its first readiness check
fails, so a worker that dies between attach and the first pump read faults the session
and the faulted sweep reaps it.
- A caller-cancelled open rolls back, records no fault and rethrows the cancellation
(gRPC Cancelled); an attach refused because the drain closed the session reports
GatewayStopping.
- docs/Sessions.md updated.
Add archreview/2026-10-21 (overall summary + tracker with R21-* rows).
Re-triage the 68 cycle-1 Not-started rows in place: 21 Done, 13 Won't fix,
34 Open (promoted as R21-O1..O34). Correct the 2026-08-19 tracker header
so CLI-47..50, IPC-36/37, TST-35 read Done like their rows.
- AddItem/AddItem2/AddBufferedItem drop stale cache entries for a reused handle id
- ReadBulk: dedupe handles within a batch (advise/teardown once, fan out result);
pump every 16 items in add/advise and teardown phases
- alarm poll stops without recording a fault once StaRuntime.LoopFault is set
- StaRuntime skips the loop-fault handler during a requested shutdown
- StaHung loop fault diagnostic carries type + HRESULT only
- gateway.md documents ReadBulk's shared deadline
Live windev finding: after a client sent the ShutdownWorker control command
the worker exited 0 and WorkerClient moved to Closed on the shutdown ack,
but GatewaySession.State stayed Ready (even with an event stream attached)
until its lease expired.
IWorkerClient has no state-change callback, so the session observes the
worker from its own worker-event pump source: WorkerClient enters
Closed/Faulted before completing the event channel, so when the enumeration
ends ObserveWorkerTerminalState sees the terminal state and faults a Ready
session. Faulted rather than Closed: Closed means the gateway closed it and
would linger in the registry; a faulted session stays observable with the
reason and the faulted-session sweep reaps it, accounting SessionClosed
exactly once. Only a Ready session follows its worker, so a gateway-
initiated close is unaffected.
FakeWorkerHarness-based tests cover the ShutdownWorker exit and a broken
worker pipe.
- The metrics scheme skips its dashboard-cookie fallback while
Dashboard:DisableLogin is on, so the dev auto-login handler cannot
satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
vs gRPC per connection, so only the per-key aggregate is shared.
- Open path: the attach gate now waits on CancellationToken.None and the
manager discards (kills + disposes) the late worker client on ANY
exception out of the attach step, including the caller's own cancellation
(checked first, so an abandoned open never yields a session). Previously
an OperationCanceledException from the gate wait escaped before the
discard and orphaned the x86 worker. The gate Release is guarded against
ObjectDisposedException.
- Unattached grace: AttachInternalEventSubscriber sets a separate latch so
the gateway's own alarm-monitor session (internal subscriber only) is
never reaped; the idle clock starts at Ready (MarkReady stamps
_readyAtUtc) instead of at open, so a slow worker start does not eat the
window.
- KillWorkerAsync / CloseSessionCoreAsync: an ObjectDisposedException from
the close gate on a session that is no longer registered means a
concurrent close already removed, accounted and disposed it; both now
return AlreadyClosed with no fault, no second gauge decrement and no sweep
failure. The general kill catch only decrements the gauge when the
session is still registered. The sweep no longer counts AlreadyClosed
results among the sessions it closed.
New MxGateway:Metrics:RequireAuthentication (default false). When true,
/metrics requires an API key with the admin scope (verified through the
cached verifier and the shared API-key failure limiter) or a dashboard
login with a real dashboard role; anonymous localhost and
Authentication:Mode=Disabled do not qualify. An anonymous scrape gets a
plain 401 Bearer challenge, a throttled peer 429. /health stays anonymous
because the off-repo Overview dashboard polls /health/active.
MapZbMetrics returns IEndpointConventionBuilder, so the policy is attached
with RequireAuthorization.
CachingApiKeyVerifier's key-id index was only pruned by Invalidate, so it
grew with every token ever verified. A post-eviction callback now drops the
slot (under the same lock as Set + index, so a late callback cannot
unindex a re-cached entry). Concurrent misses for the same token and key
generation now share one inner verification, so a TTL expiry under load no
longer stampedes SQLite; keying the flight on the generation keeps the
revoke guarantee for joiners.
A client that opens a session and vanishes before it ever streams (e.g.
OtOpcUa's GalaxyDriver on a failed start) pinned a session slot and a live
x86 worker for the full DefaultLeaseSeconds (1800 s), because detach-grace
only applies to sessions that once had a subscriber.
MxGateway:Sessions:UnattachedSessionGraceSeconds (int, default 0 = off,
validated >= 0) makes the existing lease sweep reap a Ready session that
has never attached an event subscriber once it has been idle (no unary
activity) for the window, with the close reason unattached-grace-expired.
Ready-only, idle-from-last-activity and first-subscriber-cancels keep it
from catching slow starts or unary-only clients.
GatewaySession.ExtendLease had no production callers and did not fit the
rule, so it is removed; tests move leases through TouchClientActivity.
WorkerClient.StageWorkerEvent and StreamEvents called Enum.ToString per
event (per subscriber on the stream path). MxEventFamilyNames serves the
contract-fixed names from a frozen lookup, falling back to ToString for
undefined values so exported tag values are unchanged.
StreamDisconnected always recorded "Detached". The reason tag is now a
bounded EventStreamDisconnectReasons constant for how the stream ended:
SessionClosed, Overflow, WorkerFault, ClientCancelled, Error, or Detached.
- StreamAlarms: slow subscribers are completed with
AlarmFeedSubscriberOverflowException and surface as ResourceExhausted
(like a StreamEvents overflow) instead of warning-with-stack + Unavailable.
- GatewayAlarmMonitor.StreamAsync reuses the memoized alarm-set projection
instead of cloning the whole set under the broadcast lock per attach; the
prefix filter runs outside the lock.
- StreamEvents/StreamAlarms pass the call token to response writes; a write
failure after client cancel maps to Cancelled without a warning.
- Every MxCommandReply.correlation_id now echoes the request's
client_correlation_id (previously only constraint-denied replies did;
normal replies leaked the internal envelope id).
Rewrite the Pages 'Show:' lists to what each page actually renders (no
per-session subscriber/pending-command/queue counters, no worker memory/CPU,
no per-family command counters, counters not rates). Document the per-session
EventsHub pump, alarm-hub send bound and AlarmFeedReset marker, configuration
sent once per snapshot-hub connection, health-derived gateway status, Browse
search debounce/index, bounded disposal, 30s circuit retention with
connection-gated polling, per-viewer advise-set share, and AlarmsPage memo.
CloseAsync rethrew the graceful-shutdown exception after a successful
Kill, so the manager faulted the session, recorded a CloseFailed fault and
surfaced CloseFailed to the caller for the ordinary hung-worker case.
docs/Sessions.md reserves CloseFailed for both shutdown and kill failing.
A successful kill now completes the close as Closed; the shutdown failure
rides back on SessionCloseResult.WorkerShutdownFailure and the manager
logs it at warning level. Both-fail still raises CloseFailed.
- Search: the oninput setter scanned the whole Galaxy synchronously per
keystroke. It now debounces 150ms, runs on the thread pool with
cancellation (newer keystroke supersedes), and reads a shared singleton
DashboardBrowseSearchIndex of pre-lowercased references keyed on the
Galaxy cache Sequence.
- Disposal: DisposeAsync awaited both loops unbounded and the loops caught
only OCE. Loops now handle faults per iteration (poll faults surface in
_readError, deploy-notifier faults in a retrying warning), mutate the tree
on the dispatcher, drain concurrently with a 5s bound, and the CTSs are
disposed in finally.
- Circuits: DisconnectedCircuitRetentionPeriod is 30s (was the 3 min
default) and a scoped DashboardCircuitConnectionTracker (CircuitHandler)
lets the 2s ReadBulk poll skip ticks while the browser is disconnected.
- Reads pass a per-page viewer id to the live-data per-viewer share.
ShutdownAsync iterated a single registry snapshot while Kestrel still
accepted OpenSession calls (ApplicationStopping fires before hosted
services stop), so a session opened behind the snapshot had nobody left to
close it and its x86 worker outlived the gateway as an orphan.
SessionManager now observes IHostApplicationLifetime.ApplicationStopping
(and sets the same flag at the start of ShutdownAsync) and rejects
OpenSessionAsync with the new SessionManagerErrorCode.GatewayStopping,
re-checked after registration so the flag write before the drain's
snapshot closes the race. GatewayStopping maps to gRPC Unavailable and is
not a fault. ShutdownAsync re-snapshots until the registry is empty,
bounded at three passes.
The 256-tag LRU was global, so two Browse circuits on different areas evicted
each other's whole sets every poll. Tags now record the viewer that read them
last; each active viewer (read within 30s) owns MaxSubscribedTags/active
(floor 32). Eviction takes other viewers' excess over their share first
(inactive viewers' share is zero), then the reader's own untouched tags, and
never strips a neighbour within its share (soft cap). Single-viewer behaviour
is unchanged. IDashboardLiveDataService gains a viewerId ReadAsync overload.
Joseph Doherty