WorkerProcessStartedProbe only checks HasExited and throws a non-retryable
WorkerProcessLaunchException, so StartupProbeRetryAttempts and
StartupProbeRetryDelayMilliseconds configured a Polly retry that no shipped
probe could trigger. Real readiness is WorkerClient's WorkerReady handshake
under the same StartupTimeoutSeconds budget, so the probe now runs once and
the two options, their validation, the Polly pipeline (and the server's now
unused Polly.Core reference) and their docs are removed. A stale value in
configuration is ignored by the binder.
Tests: WorkerProcessLauncherTests.LaunchAsync_WhenStartupProbeFails_DoesNotRetryTheProbe,
WorkerOptions_HasNoStartupProbeRetryOptions (both fail on the old code).
NT AUTHORITY\NETWORK marks a network logon, not a remote caller. On windev
a gateway run from an SSH session carries it in its own token, so the deny
entry locked the gateway's same-machine workers out (UnauthorizedAccess on
connect; 15 fake-worker tests failed). The ACL now holds exactly one entry:
FullControl for the gateway's user.
On Windows each worker pipe is created via NamedPipeServerStreamAcl with a
protected DACL granting FullControl only to the gateway process user (the
worker is launched under the same account) and denying NETWORK logons. On
non-Windows hosts (test matrix) the pipe uses PipeOptions.CurrentUserOnly.
Tests assert the factory ACL and the ACL read back from the created pipe
(Windows), and that a same-user client still connects.
Adds both secured bulk kinds to GatewayLogRedactor's credential-bearing set,
plus a contract cross-check test that walks every MxCommand payload descriptor
and fails if a secured or password-bearing command kind is not redacted.
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
- TryAttachWorkerClientAndMarkReadyAsync returns SessionAttachOutcome and refuses a
worker client that is already Faulted/Closed; the open discards it and fails OpenFailed.
- MapWorkerEventsAsync calls ObserveWorkerTerminalState when its first readiness check
fails, so a worker that dies between attach and the first pump read faults the session
and the faulted sweep reaps it.
- A caller-cancelled open rolls back, records no fault and rethrows the cancellation
(gRPC Cancelled); an attach refused because the drain closed the session reports
GatewayStopping.
- docs/Sessions.md updated.
Live windev finding: after a client sent the ShutdownWorker control command
the worker exited 0 and WorkerClient moved to Closed on the shutdown ack,
but GatewaySession.State stayed Ready (even with an event stream attached)
until its lease expired.
IWorkerClient has no state-change callback, so the session observes the
worker from its own worker-event pump source: WorkerClient enters
Closed/Faulted before completing the event channel, so when the enumeration
ends ObserveWorkerTerminalState sees the terminal state and faults a Ready
session. Faulted rather than Closed: Closed means the gateway closed it and
would linger in the registry; a faulted session stays observable with the
reason and the faulted-session sweep reaps it, accounting SessionClosed
exactly once. Only a Ready session follows its worker, so a gateway-
initiated close is unaffected.
FakeWorkerHarness-based tests cover the ShutdownWorker exit and a broken
worker pipe.
- The metrics scheme skips its dashboard-cookie fallback while
Dashboard:DisableLogin is on, so the dev auto-login handler cannot
satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
vs gRPC per connection, so only the per-key aggregate is shared.
- Open path: the attach gate now waits on CancellationToken.None and the
manager discards (kills + disposes) the late worker client on ANY
exception out of the attach step, including the caller's own cancellation
(checked first, so an abandoned open never yields a session). Previously
an OperationCanceledException from the gate wait escaped before the
discard and orphaned the x86 worker. The gate Release is guarded against
ObjectDisposedException.
- Unattached grace: AttachInternalEventSubscriber sets a separate latch so
the gateway's own alarm-monitor session (internal subscriber only) is
never reaped; the idle clock starts at Ready (MarkReady stamps
_readyAtUtc) instead of at open, so a slow worker start does not eat the
window.
- KillWorkerAsync / CloseSessionCoreAsync: an ObjectDisposedException from
the close gate on a session that is no longer registered means a
concurrent close already removed, accounted and disposed it; both now
return AlreadyClosed with no fault, no second gauge decrement and no sweep
failure. The general kill catch only decrements the gauge when the
session is still registered. The sweep no longer counts AlreadyClosed
results among the sessions it closed.
New MxGateway:Metrics:RequireAuthentication (default false). When true,
/metrics requires an API key with the admin scope (verified through the
cached verifier and the shared API-key failure limiter) or a dashboard
login with a real dashboard role; anonymous localhost and
Authentication:Mode=Disabled do not qualify. An anonymous scrape gets a
plain 401 Bearer challenge, a throttled peer 429. /health stays anonymous
because the off-repo Overview dashboard polls /health/active.
MapZbMetrics returns IEndpointConventionBuilder, so the policy is attached
with RequireAuthorization.
CachingApiKeyVerifier's key-id index was only pruned by Invalidate, so it
grew with every token ever verified. A post-eviction callback now drops the
slot (under the same lock as Set + index, so a late callback cannot
unindex a re-cached entry). Concurrent misses for the same token and key
generation now share one inner verification, so a TTL expiry under load no
longer stampedes SQLite; keying the flight on the generation keeps the
revoke guarantee for joiners.
A client that opens a session and vanishes before it ever streams (e.g.
OtOpcUa's GalaxyDriver on a failed start) pinned a session slot and a live
x86 worker for the full DefaultLeaseSeconds (1800 s), because detach-grace
only applies to sessions that once had a subscriber.
MxGateway:Sessions:UnattachedSessionGraceSeconds (int, default 0 = off,
validated >= 0) makes the existing lease sweep reap a Ready session that
has never attached an event subscriber once it has been idle (no unary
activity) for the window, with the close reason unattached-grace-expired.
Ready-only, idle-from-last-activity and first-subscriber-cancels keep it
from catching slow starts or unary-only clients.
GatewaySession.ExtendLease had no production callers and did not fit the
rule, so it is removed; tests move leases through TouchClientActivity.
WorkerClient.StageWorkerEvent and StreamEvents called Enum.ToString per
event (per subscriber on the stream path). MxEventFamilyNames serves the
contract-fixed names from a frozen lookup, falling back to ToString for
undefined values so exported tag values are unchanged.
StreamDisconnected always recorded "Detached". The reason tag is now a
bounded EventStreamDisconnectReasons constant for how the stream ended:
SessionClosed, Overflow, WorkerFault, ClientCancelled, Error, or Detached.
- StreamAlarms: slow subscribers are completed with
AlarmFeedSubscriberOverflowException and surface as ResourceExhausted
(like a StreamEvents overflow) instead of warning-with-stack + Unavailable.
- GatewayAlarmMonitor.StreamAsync reuses the memoized alarm-set projection
instead of cloning the whole set under the broadcast lock per attach; the
prefix filter runs outside the lock.
- StreamEvents/StreamAlarms pass the call token to response writes; a write
failure after client cancel maps to Cancelled without a warning.
- Every MxCommandReply.correlation_id now echoes the request's
client_correlation_id (previously only constraint-denied replies did;
normal replies leaked the internal envelope id).
CloseAsync rethrew the graceful-shutdown exception after a successful
Kill, so the manager faulted the session, recorded a CloseFailed fault and
surfaced CloseFailed to the caller for the ordinary hung-worker case.
docs/Sessions.md reserves CloseFailed for both shutdown and kill failing.
A successful kill now completes the close as Closed; the shutdown failure
rides back on SessionCloseResult.WorkerShutdownFailure and the manager
logs it at warning level. Both-fail still raises CloseFailed.
- Search: the oninput setter scanned the whole Galaxy synchronously per
keystroke. It now debounces 150ms, runs on the thread pool with
cancellation (newer keystroke supersedes), and reads a shared singleton
DashboardBrowseSearchIndex of pre-lowercased references keyed on the
Galaxy cache Sequence.
- Disposal: DisposeAsync awaited both loops unbounded and the loops caught
only OCE. Loops now handle faults per iteration (poll faults surface in
_readError, deploy-notifier faults in a retrying warning), mutate the tree
on the dispatcher, drain concurrently with a 5s bound, and the CTSs are
disposed in finally.
- Circuits: DisconnectedCircuitRetentionPeriod is 30s (was the 3 min
default) and a scoped DashboardCircuitConnectionTracker (CircuitHandler)
lets the 2s ReadBulk poll skip ticks while the browser is disconnected.
- Reads pass a per-page viewer id to the live-data per-viewer share.
ShutdownAsync iterated a single registry snapshot while Kestrel still
accepted OpenSession calls (ApplicationStopping fires before hosted
services stop), so a session opened behind the snapshot had nobody left to
close it and its x86 worker outlived the gateway as an orphan.
SessionManager now observes IHostApplicationLifetime.ApplicationStopping
(and sets the same flag at the start of ShutdownAsync) and rejects
OpenSessionAsync with the new SessionManagerErrorCode.GatewayStopping,
re-checked after registration so the flag write before the drain's
snapshot closes the race. GatewayStopping maps to gRPC Unavailable and is
not a fault. ShutdownAsync re-snapshots until the registry is empty,
bounded at three passes.
The 256-tag LRU was global, so two Browse circuits on different areas evicted
each other's whole sets every poll. Tags now record the viewer that read them
last; each active viewer (read within 30s) owns MaxSubscribedTags/active
(floor 32). Eviction takes other viewers' excess over their share first
(inactive viewers' share is zero), then the reader's own untouched tags, and
never strips a neighbour within its share (soft cap). Single-viewer behaviour
is unchanged. IDashboardLiveDataService gains a viewerId ReadAsync overload.
FilteredAlarms() ran the full filter+sort twice per render (Showing card and
table) and Areas() once more, including every search keystroke render. Both
now go through DashboardAlarmView, recomputed only when the alarm generation
(bumped per poll) or the filter criteria change.
TryBeginCloseIfExpired set _closeStarted when it reserved a session, so the
sweep's own CloseAsync reported AlreadyClosed: true and CloseSessionCoreAsync
skipped SessionClosed: lease-expired, faulted-reaped and detach-grace-expired
closes never left the sessions.open gauge and under-counted sessions.closed.
The reservation now writes Closing only; the first CloseAsync to take the
close lock does the accounting exactly once.
SEC-27: DashboardSnapshot.GatewayStatus was hardcoded Healthy. It is now the
aggregate HealthCheckService status, evaluated once per snapshot tick (2s
bound, zero-wait gate) and cached for GetSnapshot; Unknown before the first
evaluation, on failure, or without a health service.
The startup-static EffectiveGatewayConfiguration is JsonIgnore'd on the
snapshot, so /hubs/snapshot ticks carry only volatile state. The hub sends it
once per connection (ConfigurationUpdated) and serves GetConfiguration().
Snapshot broadcasts are time-bounded (1s) so one slow client cannot delay
ticks. StatusBadge maps Unhealthy to Bad.
A session is registered before its worker is created, so an admin Kill or
the host drain could close it during the startup window and the worker
client that arrived afterwards was stored on the dead session, MarkReady
silently refused, SessionOpened recorded and the Closed session returned
to the caller with a live x86 worker nothing would ever stop.
AttachWorkerClient now returns false under _syncRoot when the session is
Closing/Closed/Faulted or disposed; the manager runs attach + MarkReady +
SessionOpened under the per-session close gate, discards (kills and
disposes) a refused worker client and fails the open with OpenFailed.
OpenRecorded gates the closed-side accounting so a never-opened session is
not counted as closed.
The open rollback goes through RemoveSessionAsync so per-session event
metrics are dropped like every other teardown.
AlarmsHubPublisher awaited every group send unbounded inside the StreamAsync
loop, so one slow client stalled the drain until the monitor dropped the
subscriber and the resubscribe replayed the full snapshot to everyone. Each
broadcast is now bounded (2s) and abandoned on timeout, and every
(re)subscription is preceded by an AlarmFeedReset frame so clients discard
state before the replayed snapshot.
GatewayGrpcAuthorizationInterceptor only overrode the unary and
server-streaming handlers; the base Interceptor passes client- and
duplex-streaming calls through unauthenticated. Both now run the same
API-key pipeline and fail closed to the admin scope, since there is no
single request message to resolve a scope from.
Publish fired a group SendAsync per event and returned, so a slow remote
viewer accumulated unbounded pending sends plus cloned MxEvents. Hub delivery
now goes through a 256-deep DropOldest channel per hub-watched session with a
single drain task and time-bounded sends. The registry counts hub viewers
separately and raises HubViewersReleased so the pump is torn down with the
last remote viewer; in-process-only sessions start no pump.
A bulk command with N denied items wrote N audit events into the bounded,
drop-on-full audit channel, so a low-privilege key could evict every other
pending audit record. Bulk filters now accumulate a ConstraintDenialSummary
(count, constraint names, <=20 sample targets) and write one aggregated
constraint-denied event per command.
The per-item constraint checks are synchronous, so IConstraintEnforcer's
Check* methods now return ValueTask instead of allocating a Task per item.
WorkerClient.ShutdownAsync marked the client Closing and then queued the
WorkerShutdown envelope on the caller's token alone; the shutdown-timeout CTS
was only created afterwards. The outbound channel is bounded, and a worker that
has stopped reading the pipe parks the write loop in a pipe write it can only
leave through _stopCts while the channel behind it fills with commands and
their cancels. Once Closing, the heartbeat watchdog skips every tick, so nothing
cancelled _stopCts and the enqueue waited for the caller's token - the lease
sweep's stopping token, which fires at host stop and never before - with
GatewaySession.CloseAsync holding its close lock throughout.
Create the linked timeout CTS before the enqueue and run the enqueue, the loop
waits and the process-exit wait on it, so the whole call is bounded by the
shutdown timeout. A stalled queue now surfaces as the existing ShutdownTimeout
fault, which cancels the loops, kills the worker process tree and hands
GatewaySession its kill fallback.
Test: WorkerClientTests.ShutdownAsync_WhenOutboundQueueIsFullBehindStalledPipeWrite_TimesOutAndKillsWorker
stalls the gateway-side stream (a worker that stopped reading), fills the
outbound channel with abandoned commands and their cancels, and asserts that
ShutdownAsync throws ShutdownTimeout inside the test bound and kills the fake
worker. It hung past the bound before the fix.
Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
Bumps ZB.MOM.WW.Auth.Abstractions, .ApiKeys, .AspNetCore, and .Ldap in
ZB.MOM.WW.MxGateway.Server.csproj and ZB.MOM.WW.MxGateway.IntegrationTests.csproj.
0.3.0 is additive (new IApiKeyStoreMigrator, SqliteAuthStoreMigrator implements it,
old ctors retained) — no source changes needed. Restore, Release build (0 warnings/
0 errors), and dotnet test on the NonWindows solution are unchanged versus the
pre-bump baseline: 1198 passed, 0 failed, 19 skipped (live-only tests, opt-in via
env vars).
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
The subtag fallback's coverage was fixed at SubscribeAlarms time and never
re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository
outage and returns whatever configuration contributed, so a discovery failure at
startup left the monitor subscribed with an empty watch-list for the rest of its
life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong
until the failover that needs the standby finds it covering zero alarms — with a
single LogWarning emitted possibly weeks earlier as the only evidence.
ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery
was attempted and whether it failed, with the failure message), which is what the
caller needed to tell "discovery produced nothing" from "nothing was configured".
GatewayAlarmMonitor grades that resolution against the effective mode and folds
the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge
and every late-joining subscriber:
- zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with
SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed
and the message. ForceAlarmManager builds no subtag consumer and is not graded.
- degraded is now the OR of "subtag is serving" and "the watch-list covers
nothing", so the Auto case where alarmmgr is healthy and the standby is empty
is visible. DashboardAlarmProviderStatus renders that combination as the amber
"Alarm Manager (no subtag standby)" badge rather than the failover label, which
would have named the wrong live provider.
- on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the
monitor re-resolves BEFORE publishing the status and reports any difference as
SubtagWatchListStale with both counts.
The re-resolution is deliberately NOT re-sent. The worker rejects a second
SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only
expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds
FailoverAlarmConsumer, which restarts on the primary that just failed and
discards the warm standby state, so it would flap the feed rather than repair it.
The finding's other alternative (treat empty coverage as a lifecycle failure) was
rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage
and take the working alarmmgr feed down with it. A monitor restart re-resolves and
re-sends, and the status now says so.
Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale
claims (the resolver method name, and "re-runs discovery on its reconcile cadence
and pushes an updated watch-list", which never happened); gateway.md gains the
standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the
consequence on the UseGalaxyRepository and Mode rows.
Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith-
Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift
drive the fake-worker monitor; two resolver cases pin the new provenance fields.
The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
The worker reports startup failures as structured WorkerFault frames --
MxAccessCreationException carries the COM HRESULT/ProgID context and a
WorkerFaultCategory -- so the gateway can tell "MXAccess is not installed / not
registered / access denied" apart from a pipe-level protocol problem.
ReadHandshakeEnvelopeAsync did not look inside: any body other than the expected
one became a ProtocolViolation naming only the body type, and that text is what
SessionManager puts in the operator log and the client's OpenFailed detail. The
information was lost exactly in the window where it matters most -- first
session open on a freshly deployed host, where the answer is almost always an
MXAccess installation or permission problem.
Special-case BodyCase == WorkerFault and throw WorkerClientException(
WorkerFaulted, CreateWorkerFaultMessage(...)) -- the same formatter the
steady-state fault path already uses, so both windows read alike. Every other
unexpected body still gets ProtocolViolation.
Test: StartAsync_WhenWorkerFaultsDuringHandshake_SurfacesFaultCategoryAndDiagnostic
has the fake worker answer the hello and then write a WorkerFault
{MxaccessCreationFailed} in place of WorkerReady; the existing
FakeWorkerHarness.EmitFaultAsync covered it, so the harness needed no change.
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only.
Its rationale -- the gateway has stopped waiting but the worker has not stopped
working, so a queued-but-not-started command should be dropped instead of
occupying the single STA behind a caller that is already gone -- applies exactly
as well to caller cancellation, which is the common abandonment path: gRPC
clients routinely set per-call deadlines shorter than the gateway's
CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in
arms that removed the pending entry and returned silently, leaving the
correlation to execute in its turn. Under a retry storm against a slow provider
the STA works through dead commands while live callers queue behind them --
precisely the backlog the cancel exists to shorten.
- TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and
takes the reason string, since it serves both arms.
- Both caller-cancellation arms route through AbandonCommandOnCallerCancellation,
which sends the cancel under the same `removed` ownership gate as the timeout
arm (so a reply that won the race is never chased) and skips it once _stopCts
has fired, where the outbound channel is being completed and every pending
command is failed anyway.
- New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label
those arms used, so metrics and logs stop recording a shutdown that did not
happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps
to, so nothing changes for clients.
Docs: docs/Grpc.md mapping line, gateway.md cancellation rules,
docs/GatewayProcessDesign.md pending-command section.
Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin
of the timeout test, driven by the caller token; verified failing without the
send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
Joseph Doherty