Live windev finding: after a client sent the ShutdownWorker control command
the worker exited 0 and WorkerClient moved to Closed on the shutdown ack,
but GatewaySession.State stayed Ready (even with an event stream attached)
until its lease expired.
IWorkerClient has no state-change callback, so the session observes the
worker from its own worker-event pump source: WorkerClient enters
Closed/Faulted before completing the event channel, so when the enumeration
ends ObserveWorkerTerminalState sees the terminal state and faults a Ready
session. Faulted rather than Closed: Closed means the gateway closed it and
would linger in the registry; a faulted session stays observable with the
reason and the faulted-session sweep reaps it, accounting SessionClosed
exactly once. Only a Ready session follows its worker, so a gateway-
initiated close is unaffected.
FakeWorkerHarness-based tests cover the ShutdownWorker exit and a broken
worker pipe.
- Open path: the attach gate now waits on CancellationToken.None and the
manager discards (kills + disposes) the late worker client on ANY
exception out of the attach step, including the caller's own cancellation
(checked first, so an abandoned open never yields a session). Previously
an OperationCanceledException from the gate wait escaped before the
discard and orphaned the x86 worker. The gate Release is guarded against
ObjectDisposedException.
- Unattached grace: AttachInternalEventSubscriber sets a separate latch so
the gateway's own alarm-monitor session (internal subscriber only) is
never reaped; the idle clock starts at Ready (MarkReady stamps
_readyAtUtc) instead of at open, so a slow worker start does not eat the
window.
- KillWorkerAsync / CloseSessionCoreAsync: an ObjectDisposedException from
the close gate on a session that is no longer registered means a
concurrent close already removed, accounted and disposed it; both now
return AlreadyClosed with no fault, no second gauge decrement and no sweep
failure. The general kill catch only decrements the gauge when the
session is still registered. The sweep no longer counts AlreadyClosed
results among the sessions it closed.
A client that opens a session and vanishes before it ever streams (e.g.
OtOpcUa's GalaxyDriver on a failed start) pinned a session slot and a live
x86 worker for the full DefaultLeaseSeconds (1800 s), because detach-grace
only applies to sessions that once had a subscriber.
MxGateway:Sessions:UnattachedSessionGraceSeconds (int, default 0 = off,
validated >= 0) makes the existing lease sweep reap a Ready session that
has never attached an event subscriber once it has been idle (no unary
activity) for the window, with the close reason unattached-grace-expired.
Ready-only, idle-from-last-activity and first-subscriber-cancels keep it
from catching slow starts or unary-only clients.
GatewaySession.ExtendLease had no production callers and did not fit the
rule, so it is removed; tests move leases through TouchClientActivity.
CloseAsync rethrew the graceful-shutdown exception after a successful
Kill, so the manager faulted the session, recorded a CloseFailed fault and
surfaced CloseFailed to the caller for the ordinary hung-worker case.
docs/Sessions.md reserves CloseFailed for both shutdown and kill failing.
A successful kill now completes the close as Closed; the shutdown failure
rides back on SessionCloseResult.WorkerShutdownFailure and the manager
logs it at warning level. Both-fail still raises CloseFailed.
ShutdownAsync iterated a single registry snapshot while Kestrel still
accepted OpenSession calls (ApplicationStopping fires before hosted
services stop), so a session opened behind the snapshot had nobody left to
close it and its x86 worker outlived the gateway as an orphan.
SessionManager now observes IHostApplicationLifetime.ApplicationStopping
(and sets the same flag at the start of ShutdownAsync) and rejects
OpenSessionAsync with the new SessionManagerErrorCode.GatewayStopping,
re-checked after registration so the flag write before the drain's
snapshot closes the race. GatewayStopping maps to gRPC Unavailable and is
not a fault. ShutdownAsync re-snapshots until the registry is empty,
bounded at three passes.
TryBeginCloseIfExpired set _closeStarted when it reserved a session, so the
sweep's own CloseAsync reported AlreadyClosed: true and CloseSessionCoreAsync
skipped SessionClosed: lease-expired, faulted-reaped and detach-grace-expired
closes never left the sessions.open gauge and under-counted sessions.closed.
The reservation now writes Closing only; the first CloseAsync to take the
close lock does the accounting exactly once.
A session is registered before its worker is created, so an admin Kill or
the host drain could close it during the startup window and the worker
client that arrived afterwards was stored on the dead session, MarkReady
silently refused, SessionOpened recorded and the Closed session returned
to the caller with a live x86 worker nothing would ever stop.
AttachWorkerClient now returns false under _syncRoot when the session is
Closing/Closed/Faulted or disposed; the manager runs attach + MarkReady +
SessionOpened under the per-session close gate, discards (kills and
disposes) a refused worker client and fails the open with OpenFailed.
OpenRecorded gates the closed-side accounting so a never-opened session is
not counted as closed.
The open rollback goes through RemoveSessionAsync so per-session event
metrics are dropped like every other teardown.
Address review of the #164-#166 fixes:
- The HRESULT check could not tell interop from programming errors
(a managed ArgumentException already defaults to 0x80070057). Wrap each
MXAccess call (AuthenticateUser, AddItem, Advise, Write) in a narrow
filter that converts an interop failure to a private marker exception,
and catch only that marker, so exceptions from the writer's own code
(argument checks, caching, the pump) propagate.
- A late OnWriteComplete for a dropped handle recreated the cache entry
that DropItem removed. The writer now records completions only for item
handles it holds on its live registration.
- The item-handle-reuse probe now asserts that handles within one
registration strictly increase, so a rerun is a regression check.
Refs #164, #165, #166
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
Add the Skip-gated ItemHandleReuseProbeTests probe and its windev result
(2026-10-20): within one MXAccess registration, item handles increase
monotonically and are never reused across remove/re-add, so the ack
writer's Unconfirmed drop-and-re-add mitigation holds and no per-handle
generation counter is needed. Numbering restarts only on a new
registration, which teardown already isolates. Document the guarantee at
the Dirty re-add and in AlarmProbeFindings.md.
Fixes#166
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
Add MxAccessWriteCompletionCache.Remove and RemoveServer and call them
from AlarmAckWriter when it drops an item (dirty re-add, LRU eviction,
failed AddItem/Advise/Write) and when it tears down its registration, so
the cache no longer keeps one entry per dropped handle for the life of
the worker.
Fixes#165
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
Resolve the operator user, password and comment before the interop try
blocks, and match ArgumentException as an MXAccess failure only when it
carries E_INVALIDARG and is not ArgumentNullException or
ArgumentOutOfRangeException. A programming error inside the try blocks
now propagates instead of being reported as an authentication or write
failure.
Fixes#164
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
- AlarmAckWriter: classify AuthenticateUser and AddItem/Advise/Write
failures by HRESULT for any interop exception (ArgumentException for
E_INVALIDARG, UnauthorizedAccessException, COMException); server-gone
HRESULTs at AuthenticateUser tear down the registration and report
WriteRejected.
- Bound the ack item cache with a 256-entry LRU (eviction best-effort
UnAdvise+RemoveItem).
- After an Unconfirmed ack, drop and re-add the item before the next
write so a late OnWriteComplete on the old handle is not credited.
- Pass StaRuntime.PumpPendingMessages to the ack writer through a
pump-aware alarm handler factory so long completion waits refresh the
STA watchdog heartbeat.
- Rust CLI: --operator is required for acknowledge-alarm.
- Docs: full ack status table, credential-oracle trade-off, upgrade note.
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
AlarmCommandHandler -> AlarmDispatcher -> IAlarmAckWriter: the dispatcher resolves the ack item through the consumer and the writer authenticates the operator and writes the comment. The executor builds AlarmAckOperator from operator_user/domain/password, maps AlarmAckOutcome to the reply (Applied/Unconfirmed Ok, AlarmNotFound/NotAckable InvalidRequest, AuthenticationFailed/WriteRejected MxaccessFailure) and reports escaping exceptions by type and HRESULT only, so operator_password cannot leak. Removes the inert managed/WNAL alarm-client ack path.
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
IMxAccessAlarmConsumer.ResolveAckTarget/ResolveAckTargetByName replace AcknowledgeByGuid/AcknowledgeByName. wnwrap maps Galaxy-provider alarms to <tag>.AckMsg and drops its inert parallel ack consumer; subtag resolves to the target's AckCommentSubtag and no longer advises or writes it; ISubtagAlarmSource.Write is removed. Does not build alone: AlarmDispatcher moves to the new API in the next commit.
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
The 2026-10-18 live test on the dev rig showed the opt-in managed ack path
cannot work as built: AlarmClient.RegisterConsumer returns -65 alongside the
worker's wnwrap reader (the worker then silently falls back to wnwrap), the
managed client does not recognise the wnwrap GUID even standalone, and the
gateway's GUID arm sends empty identity fields. Replace the docs' claim that
the path works with the measured results and tell operators to leave
MXGATEWAY_USE_MANAGED_ALARM_ACK unset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes for the opt-in managed GUID ack:
- Subtag-mode GUID acks (forced subtag, or failover on the standby) stay on
the consumer; their GUIDs are synthetic and only the subtag consumer can
resolve them. Forced-subtag sessions no longer build a managed client.
- A failed managed Initialize/Register/Subscribe drops back to the wnwrap
ack path instead of failing the alarm subscription.
- Drop stale crefs to a nonexistent Unverified member in
WnalNativeAckConsumer.
- Unit tests for routing, fallback, and disposal; document
MXGATEWAY_USE_MANAGED_ALARM_ACK.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a managed acknowledge path over AVEVA's modern alarm client so GUID
acks carry full operator identity (domain + full name) and work on builds
where the wnwrap COM AlarmAckByGUID is E_NOTIMPL. Default off; opt in with
MXGATEWAY_USE_MANAGED_ALARM_ACK=1 (wnwrap behavior unchanged otherwise).
- AaManagedAlarmAckConsumer: reflection wrapper over
aaAlarmManagedClient.AlarmClient (InitializeConsumer/RegisterConsumer/
Subscribe/AlarmAckByGUID); EnvGatedFactory degrades to null (wnwrap) if the
DLL can't load, so alarm reads never break.
- IManagedAlarmAckClient seam. AlarmDispatcher takes an optional client and
routes GUID acks to it (subscribing + disposing it alongside the reader).
- AlarmCommandHandler gains an optional managedAckFactory; WorkerPipeSession
wires AaManagedAlarmAckConsumer.EnvGatedFactory at the production sites.
- WnalNativeAckConsumer: confirmed native alarmlst.dll WNAL submit layer
(WNALalarmReqSetText / WNALUserRequest / WNALUserRequestEx + lpAlmAck
layout). List acquisition is intentionally not hand-rolled: the alarm-record
layout and selection semantics are provider-defined and not recoverable from
alarmlst.dll, so inferring them risks acking the wrong alarm.
- AlarmManagedGuidAckLiveSmokeTests: skip-gated live smoke that pulls a real
(non-degraded) alarm GUID and asserts the ack APPLIES (Acknowledge
transition / acked snapshot), not merely rc=0.
By-name acks stay on wnwrap (AlarmClient has no AlarmAckByName). Grounded in
static RE of alarmlst.dll, aaAlarmManagedClient.dll and alarmmgr.exe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
Bumps ZB.MOM.WW.Auth.Abstractions, .ApiKeys, .AspNetCore, and .Ldap in
ZB.MOM.WW.MxGateway.Server.csproj and ZB.MOM.WW.MxGateway.IntegrationTests.csproj.
0.3.0 is additive (new IApiKeyStoreMigrator, SqliteAuthStoreMigrator implements it,
old ctors retained) — no source changes needed. Restore, Release build (0 warnings/
0 errors), and dotnet test on the NonWindows solution are unchanged versus the
pre-bump baseline: 1198 passed, 0 failed, 19 skipped (live-only tests, opt-in via
env vars).
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
Joseph Doherty
dohertj2