Commit Graph

637 Commits

Author SHA1 Message Date
Joseph Doherty 2a7d6facce perf(dashboard): memoize AlarmsPage filtered rows and area list
FilteredAlarms() ran the full filter+sort twice per render (Showing card and
table) and Areas() once more, including every search keystroke render. Both
now go through DashboardAlarmView, recomputed only when the alarm generation
(bumped per poll) or the filter criteria change.
2026-10-22 06:19:34 -04:00
Joseph Doherty 7618cafb6c fix(worker): review 2026-10-21 stream E — cache eviction, phased ReadBulk, poll/STA fault handling
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m3s
ci / java (push) Successful in 2m17s
ci / portable (push) Failing after 4m42s
- RemoveItem/Unregister evict value + write-completion caches (leak under churn)
- ReadBulk snapshot path: add/advise all, one shared deadline, then tear down
- alarm poll faults only after 3 consecutive failures; transient ones logged
- AlarmCommandHandler disposes primary when standby build throws
- StaRuntime pumps between queued work items
- WRK-02: post-start STA loop death surfaces as StaHung WorkerFault
- WRK-09: AppDomain unhandled-exception hook, exit code 7
- RETRIGGER documented as reserved, never synthesized
- TST-05: live facts for Ping/GetSessionState/GetWorkerInfo/DrainEvents/ShutdownWorker
2026-10-22 06:19:32 -04:00
Joseph Doherty 09666cc719 fix(sessions): account SessionClosed for sweep-driven closes
TryBeginCloseIfExpired set _closeStarted when it reserved a session, so the
sweep's own CloseAsync reported AlreadyClosed: true and CloseSessionCoreAsync
skipped SessionClosed: lease-expired, faulted-reaped and detach-grace-expired
closes never left the sessions.open gauge and under-counted sessions.closed.
The reservation now writes Closing only; the first CloseAsync to take the
close lock does the accounting exactly once.
2026-10-22 06:19:20 -04:00
Joseph Doherty 6733519ae5 fix(dashboard): derive gateway status from health checks; send config once per hub connection
SEC-27: DashboardSnapshot.GatewayStatus was hardcoded Healthy. It is now the
aggregate HealthCheckService status, evaluated once per snapshot tick (2s
bound, zero-wait gate) and cached for GetSnapshot; Unknown before the first
evaluation, on failure, or without a health service.

The startup-static EffectiveGatewayConfiguration is JsonIgnore'd on the
snapshot, so /hubs/snapshot ticks carry only volatile state. The hub sends it
once per connection (ConfigurationUpdated) and serves GetConfiguration().
Snapshot broadcasts are time-bounded (1s) so one slow client cannot delay
ticks. StatusBadge maps Unhealthy to Bad.
2026-10-22 06:18:02 -04:00
Joseph Doherty 87f23d9712 fix(sessions): refuse late worker attach after close during startup; route open rollback through removal path
A session is registered before its worker is created, so an admin Kill or
the host drain could close it during the startup window and the worker
client that arrived afterwards was stored on the dead session, MarkReady
silently refused, SessionOpened recorded and the Closed session returned
to the caller with a live x86 worker nothing would ever stop.

AttachWorkerClient now returns false under _syncRoot when the session is
Closing/Closed/Faulted or disposed; the manager runs attach + MarkReady +
SessionOpened under the per-session close gate, discards (kills and
disposes) a refused worker client and fails the open with OpenFailed.
OpenRecorded gates the closed-side accounting so a never-opened session is
not counted as closed.

The open rollback goes through RemoveSessionAsync so per-session event
metrics are dropped like every other teardown.
2026-10-22 06:16:53 -04:00
Joseph Doherty 3688fa19e6 fix(dashboard): time-bound alarm hub sends and mark each resubscribe
AlarmsHubPublisher awaited every group send unbounded inside the StreamAsync
loop, so one slow client stalled the drain until the monitor dropped the
subscriber and the resubscribe replayed the full snapshot to everyone. Each
broadcast is now bounded (2s) and abandoned on timeout, and every
(re)subscription is preceded by an AlarmFeedReset frame so clients discard
state before the replayed snapshot.
2026-10-22 06:14:53 -04:00
Joseph Doherty 19cc74f21a fix(auth): authenticate client- and duplex-streaming gRPC calls
GatewayGrpcAuthorizationInterceptor only overrode the unary and
server-streaming handlers; the base Interceptor passes client- and
duplex-streaming calls through unauthenticated. Both now run the same
API-key pipeline and fail closed to the admin scope, since there is no
single request message to resolve a scope from.
2026-10-22 06:14:33 -04:00
Joseph Doherty f1db002b87 fix(dashboard): bound EventsHub delivery with one drop-oldest pump per session
Publish fired a group SendAsync per event and returned, so a slow remote
viewer accumulated unbounded pending sends plus cloned MxEvents. Hub delivery
now goes through a 256-deep DropOldest channel per hub-watched session with a
single drain task and time-bounded sends. The registry counts hub viewers
separately and raises HubViewersReleased so the pump is torn down with the
last remote viewer; in-process-only sessions start no pump.
2026-10-22 06:13:05 -04:00
Joseph Doherty ca0cf19a1f fix(auth): aggregate bulk constraint denials into one audit event; ValueTask constraint checks
A bulk command with N denied items wrote N audit events into the bounded,
drop-on-full audit channel, so a low-privilege key could evict every other
pending audit record. Bulk filters now accumulate a ConstraintDenialSummary
(count, constraint names, <=20 sample targets) and write one aggregated
constraint-denied event per command.

The per-item constraint checks are synchronous, so IConstraintEnforcer's
Check* methods now return ValueTask instead of allocating a Task per item.
2026-10-22 06:12:02 -04:00
Joseph Doherty b0100e738a fix(worker): bound ShutdownAsync's envelope enqueue by the shutdown timeout
WorkerClient.ShutdownAsync marked the client Closing and then queued the
WorkerShutdown envelope on the caller's token alone; the shutdown-timeout CTS
was only created afterwards. The outbound channel is bounded, and a worker that
has stopped reading the pipe parks the write loop in a pipe write it can only
leave through _stopCts while the channel behind it fills with commands and
their cancels. Once Closing, the heartbeat watchdog skips every tick, so nothing
cancelled _stopCts and the enqueue waited for the caller's token - the lease
sweep's stopping token, which fires at host stop and never before - with
GatewaySession.CloseAsync holding its close lock throughout.

Create the linked timeout CTS before the enqueue and run the enqueue, the loop
waits and the process-exit wait on it, so the whole call is bounded by the
shutdown timeout. A stalled queue now surfaces as the existing ShutdownTimeout
fault, which cancels the loops, kills the worker process tree and hands
GatewaySession its kill fallback.

Test: WorkerClientTests.ShutdownAsync_WhenOutboundQueueIsFullBehindStalledPipeWrite_TimesOutAndKillsWorker
stalls the gateway-side stream (a worker that stopped reading), fills the
outbound channel with abandoned commands and their cancels, and asserts that
ShutdownAsync throws ShutdownTimeout inside the test bound and kills the fake
worker. It hung past the bound before the fix.
2026-10-22 06:10:52 -04:00
Joseph Doherty 02228b2717 Merge branch 'main' into feat/ackmsg-as-operator
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m7s
ci / java (pull_request) Successful in 2m18s
ci / nightly-windev (pull_request) Has been skipped
ci / portable (push) Failing after 4m47s
ci / windows-x86 (pull_request) Successful in 2m40s
ci / portable (pull_request) Failing after 4m43s
ci / windows-x86 (push) Successful in 5m38s
Picks up the System.Security.Cryptography.Pkcs 10.0.13 pin (NU1903).

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:28:45 -04:00
Joseph Doherty 463748e395 fix(worker): classify ack interop failures at the COM call; ignore late completions
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m24s
ci / java (push) Successful in 2m32s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m38s
ci / portable (push) Failing after 2m52s
ci / portable (pull_request) Failing after 2m53s
ci / windows-x86 (pull_request) Successful in 8m8s
Address review of the #164-#166 fixes:

- The HRESULT check could not tell interop from programming errors
  (a managed ArgumentException already defaults to 0x80070057). Wrap each
  MXAccess call (AuthenticateUser, AddItem, Advise, Write) in a narrow
  filter that converts an interop failure to a private marker exception,
  and catch only that marker, so exceptions from the writer's own code
  (argument checks, caching, the pump) propagate.
- A late OnWriteComplete for a dropped handle recreated the cache entry
  that DropItem removed. The writer now records completions only for item
  handles it holds on its live registration.
- The item-handle-reuse probe now asserts that handles within one
  registration strictly increase, so a rerun is a regression check.

Refs #164, #165, #166

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:18:02 -04:00
Joseph Doherty bd82036d5f docs(worker): record live item-handle-reuse probe for ack writer
ci / windows-x86 (push) Successful in 1m29s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m47s
ci / nightly-windev (pull_request) Has been skipped
ci / portable (push) Failing after 3m1s
ci / portable (pull_request) Failing after 1m46s
ci / windows-x86 (pull_request) Successful in 1m24s
ci / java (pull_request) Successful in 2m36s
Add the Skip-gated ItemHandleReuseProbeTests probe and its windev result
(2026-10-20): within one MXAccess registration, item handles increase
monotonically and are never reused across remove/re-add, so the ack
writer's Unconfirmed drop-and-re-add mitigation holds and no per-handle
generation counter is needed. Numbering restarts only on a new
registration, which teardown already isolates. Document the guarantee at
the Dirty re-add and in AlarmProbeFindings.md.

Fixes #166

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:13:31 -04:00
Joseph Doherty f13950e453 fix(worker): forget write-completion entries for dropped ack item handles
Add MxAccessWriteCompletionCache.Remove and RemoveServer and call them
from AlarmAckWriter when it drops an item (dirty re-add, LRU eviction,
failed AddItem/Advise/Write) and when it tears down its registration, so
the cache no longer keeps one entry per dropped handle for the life of
the worker.

Fixes #165

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:11:37 -04:00
Joseph Doherty 219611c821 fix(worker): narrow AlarmAckWriter ArgumentException interop filter
Resolve the operator user, password and comment before the interop try
blocks, and match ArgumentException as an MXAccess failure only when it
carries E_INVALIDARG and is not ArgumentNullException or
ArgumentOutOfRangeException. A programming error inside the try blocks
now propagates instead of being reported as an authentication or write
failure.

Fixes #164

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:11:37 -04:00
Joseph Doherty a09fa0a4d6 fix: pin System.Security.Cryptography.Pkcs 10.0.13 (NU1903)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 10m44s
Microsoft.Data.SqlClient 6.0.2 pulls System.Security.Cryptography.Pkcs 9.0.4,
which carries GHSA-846f-f764-76mx (high) and GHSA-2fmv-rv2c-59x7 (moderate).
With TreatWarningsAsErrors the NU1902/NU1903 audit warnings failed restore.
10.0.13 patches both.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-20 07:07:59 -04:00
Joseph Doherty e7f56622c1 fix: address final ackmsg-as-operator review findings
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m0s
ci / windows-x86 (push) Successful in 2m11s
ci / portable (push) Failing after 2m21s
- AlarmAckWriter: classify AuthenticateUser and AddItem/Advise/Write
  failures by HRESULT for any interop exception (ArgumentException for
  E_INVALIDARG, UnauthorizedAccessException, COMException); server-gone
  HRESULTs at AuthenticateUser tear down the registration and report
  WriteRejected.
- Bound the ack item cache with a 256-entry LRU (eviction best-effort
  UnAdvise+RemoveItem).
- After an Unconfirmed ack, drop and re-add the item before the next
  write so a late OnWriteComplete on the old handle is not credited.
- Pass StaRuntime.PumpPendingMessages to the ack writer through a
  pump-aware alarm handler factory so long completion waits refresh the
  STA watchdog heartbeat.
- Rust CLI: --operator is required for acknowledge-alarm.
- Docs: full ack status table, credential-oracle trade-off, upgrade note.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:58:24 -04:00
Joseph Doherty 9409488008 docs: alarm acknowledgement writes AckMsg as the authenticated operator
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:44:21 -04:00
Joseph Doherty 2858e42e80 feat(gateway): forward operator credentials on alarm acknowledgement
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:40:23 -04:00
Joseph Doherty d6db0ff40d feat(worker): route alarm acks through the operator-authenticated AckMsg writer
ci / nightly-windev (push) Has been skipped
ci / java (push) Failing after 2m5s
ci / windows-x86 (push) Successful in 2m10s
ci / portable (push) Failing after 2m17s
AlarmCommandHandler -> AlarmDispatcher -> IAlarmAckWriter: the dispatcher resolves the ack item through the consumer and the writer authenticates the operator and writes the comment. The executor builds AlarmAckOperator from operator_user/domain/password, maps AlarmAckOutcome to the reply (Applied/Unconfirmed Ok, AlarmNotFound/NotAckable InvalidRequest, AuthenticationFailed/WriteRejected MxaccessFailure) and reports escaping exceptions by type and HRESULT only, so operator_password cannot leak. Removes the inert managed/WNAL alarm-client ack path.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:35:55 -04:00
Joseph Doherty 0d4b2aea2d refactor(worker): alarm consumers resolve the AckMsg item instead of acking
IMxAccessAlarmConsumer.ResolveAckTarget/ResolveAckTargetByName replace AcknowledgeByGuid/AcknowledgeByName. wnwrap maps Galaxy-provider alarms to <tag>.AckMsg and drops its inert parallel ack consumer; subtag resolves to the target's AckCommentSubtag and no longer advises or writes it; ISubtagAlarmSource.Write is removed. Does not build alone: AlarmDispatcher moves to the new API in the next commit.

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:35:49 -04:00
Joseph Doherty 8199553734 fix(worker): handle AckMsg add/advise/write COM failures and server death
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m55s
ci / java (push) Failing after 2m2s
ci / portable (push) Failing after 2m17s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:25:43 -04:00
Joseph Doherty 7900df3dce fix(worker): disambiguate MxStatusCategory in AlarmAckWriter
ci / nightly-windev (push) Has been skipped
ci / java (push) Failing after 2m25s
ci / portable (push) Failing after 2m26s
ci / windows-x86 (push) Successful in 2m27s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:22:27 -04:00
Joseph Doherty 76db822df3 feat(worker): acknowledge alarms via authenticated AckMsg write
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 56s
ci / java (push) Failing after 2m17s
ci / portable (push) Failing after 2m23s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:21:55 -04:00
Joseph Doherty 64dc1215fd feat(worker): attribute alarm ack transitions from wnwrap ACK fields
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m7s
ci / java (push) Failing after 2m17s
ci / portable (push) Failing after 2m23s
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:18:24 -04:00
Joseph Doherty a94b502ce6 feat(contracts): carry operator credentials on alarm acknowledgement
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
2026-10-19 19:16:39 -04:00
Joseph Doherty 6f4c7c3bf1 Keep subtag acks and alarm reads safe on the managed ack path
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (pull_request) Successful in 2m0s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m16s
ci / java (push) Successful in 2m19s
ci / portable (push) Failing after 2m31s
ci / portable (pull_request) Failing after 2m36s
ci / windows-x86 (push) Successful in 3m11s
Review fixes for the opt-in managed GUID ack:

- Subtag-mode GUID acks (forced subtag, or failover on the standby) stay on
  the consumer; their GUIDs are synthetic and only the subtag consumer can
  resolve them. Forced-subtag sessions no longer build a managed client.
- A failed managed Initialize/Register/Subscribe drops back to the wnwrap
  ack path instead of failing the alarm subscription.
- Drop stale crefs to a nonexistent Unverified member in
  WnalNativeAckConsumer.
- Unit tests for routing, fallback, and disposal; document
  MXGATEWAY_USE_MANAGED_ALARM_ACK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 13:11:01 -04:00
Joseph Doherty 6affbd57ea feat(worker): add opt-in managed GUID alarm-ack path
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m19s
ci / java (push) Successful in 2m8s
ci / portable (push) Failing after 2m23s
ci / nightly-windev (pull_request) Has been skipped
ci / windows-x86 (pull_request) Successful in 1m18s
ci / java (pull_request) Successful in 2m3s
ci / portable (pull_request) Failing after 2m17s
Adds a managed acknowledge path over AVEVA's modern alarm client so GUID
acks carry full operator identity (domain + full name) and work on builds
where the wnwrap COM AlarmAckByGUID is E_NOTIMPL. Default off; opt in with
MXGATEWAY_USE_MANAGED_ALARM_ACK=1 (wnwrap behavior unchanged otherwise).

- AaManagedAlarmAckConsumer: reflection wrapper over
  aaAlarmManagedClient.AlarmClient (InitializeConsumer/RegisterConsumer/
  Subscribe/AlarmAckByGUID); EnvGatedFactory degrades to null (wnwrap) if the
  DLL can't load, so alarm reads never break.
- IManagedAlarmAckClient seam. AlarmDispatcher takes an optional client and
  routes GUID acks to it (subscribing + disposing it alongside the reader).
- AlarmCommandHandler gains an optional managedAckFactory; WorkerPipeSession
  wires AaManagedAlarmAckConsumer.EnvGatedFactory at the production sites.
- WnalNativeAckConsumer: confirmed native alarmlst.dll WNAL submit layer
  (WNALalarmReqSetText / WNALUserRequest / WNALUserRequestEx + lpAlmAck
  layout). List acquisition is intentionally not hand-rolled: the alarm-record
  layout and selection semantics are provider-defined and not recoverable from
  alarmlst.dll, so inferring them risks acking the wrong alarm.
- AlarmManagedGuidAckLiveSmokeTests: skip-gated live smoke that pulls a real
  (non-degraded) alarm GUID and asserts the ack APPLIES (Acknowledge
  transition / acked snapshot), not merely rc=0.

By-name acks stay on wnwrap (AlarmClient has no AlarmAckByName). Grounded in
static RE of alarmlst.dll, aaAlarmManagedClient.dll and alarmmgr.exe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-18 12:42:07 -04:00
Joseph Doherty 83aa76c33d Retain fake worker fault diagnostics in test fixtures
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m20s
ci / java (push) Successful in 2m19s
ci / windows-x86 (pull_request) Successful in 1m20s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m23s
ci / portable (push) Successful in 9m10s
ci / portable (pull_request) Successful in 9m2s
2026-10-13 10:35:28 -04:00
Joseph Doherty 42c120aa83 Preserve framed reads across fake worker emission gates
ci / windows-x86 (push) Successful in 1m20s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m17s
ci / windows-x86 (pull_request) Successful in 1m25s
ci / nightly-windev (pull_request) Has been skipped
ci / java (pull_request) Successful in 2m27s
ci / portable (push) Successful in 9m43s
ci / portable (pull_request) Successful in 9m47s
2026-10-13 09:42:09 -04:00
Joseph Doherty f40409a688 docs: strip task-tracking IDs from code comments and fill missing XML docs
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m16s
ci / java (push) Successful in 2m5s
ci / portable (push) Failing after 5m9s
Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
2026-10-07 15:34:32 -04:00
Joseph Doherty 4f931e4b7e chore(deps): bump ZB.MOM.WW.Auth 0.2.1->0.3.0
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m20s
ci / java (push) Successful in 2m20s
ci / portable (push) Successful in 10m43s
Bumps ZB.MOM.WW.Auth.Abstractions, .ApiKeys, .AspNetCore, and .Ldap in
ZB.MOM.WW.MxGateway.Server.csproj and ZB.MOM.WW.MxGateway.IntegrationTests.csproj.
0.3.0 is additive (new IApiKeyStoreMigrator, SqliteAuthStoreMigrator implements it,
old ctors retained) — no source changes needed. Restore, Release build (0 warnings/
0 errors), and dotnet test on the NonWindows solution are unchanged versus the
pre-bump baseline: 1198 passed, 0 failed, 19 skipped (live-only tests, opt-in via
env vars).

Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
2026-09-03 10:57:10 -04:00
Joseph Doherty cc143b52cf ci(TST-35): run gateway suite on windev nightly
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m44s
ci / windows-x86 (push) Successful in 5m56s
ci / portable (push) Failing after 9m21s
2026-08-25 11:56:35 -04:00
Joseph Doherty 986c6c0ce4 fix(IPC-36): import obsolete attribute namespace
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m4s
ci / java (push) Successful in 2m22s
ci / portable (push) Failing after 9m44s
2026-08-25 11:49:02 -04:00
Joseph Doherty 484028e721 feat(GWC-37,WRK-32): refresh live alarm watch lists
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 51s
ci / java (push) Successful in 2m8s
ci / portable (push) Failing after 1h1m54s
2026-08-25 11:46:07 -04:00
Joseph Doherty d6c5a11ae6 fix(IPC-36,IPC-37): guard secondary contracts 2026-08-25 11:14:14 -04:00
Joseph Doherty 1875426e4b fix(SEC-38,SEC-39): preserve aggregate and fallback throttles 2026-08-25 11:00:29 -04:00
Joseph Doherty 780720165a fix(GWC-35): repair re-raised alarm snapshots 2026-08-25 10:58:21 -04:00
Joseph Doherty b0156c685e fix(GWC-33): unregister terminal worker queue gauge 2026-08-25 10:56:23 -04:00
Joseph Doherty 92aae28f07 chore(GWC-36,IPC-38,TST-33,TST-34,TST-36): close cycle-3 currency gaps 2026-08-25 10:52:39 -04:00
Joseph Doherty f2ab9cc879 Merge fix/c3-tracker-hygiene: GWC-31 perf-pass comments relabeled PERF-nn, archreview prefixes reserved in REVIEW-PROCESS.md, cycle-1/2 tracker housekeeping (GWC-16 Won't fix, IPC-08/IPC-31 Done, TST-06/17 refreshed, WRK-29..31 reserved)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m57s
ci / java (push) Successful in 2m11s
ci / portable (push) Successful in 9m23s
2026-08-20 09:40:05 -04:00
Joseph Doherty c6650c156b Merge fix/c3-session-lifecycle: GWC-32 WorkerClient dispose is terminal, GWC-34 attach-after-dispose throws SessionNotReady, GWC-37 subtag watch-list coverage graded on provider status + re-resolved at failover edge 2026-08-20 09:39:53 -04:00
Joseph Doherty 193221773d fix(GWC-37): grade the alarm subtag watch-list instead of latching it silently
The subtag fallback's coverage was fixed at SubscribeAlarms time and never
re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository
outage and returns whatever configuration contributed, so a discovery failure at
startup left the monitor subscribed with an empty watch-list for the rest of its
life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong
until the failover that needs the standby finds it covering zero alarms — with a
single LogWarning emitted possibly weeks earlier as the only evidence.

ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery
was attempted and whether it failed, with the failure message), which is what the
caller needed to tell "discovery produced nothing" from "nothing was configured".
GatewayAlarmMonitor grades that resolution against the effective mode and folds
the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge
and every late-joining subscriber:

- zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with
  SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed
  and the message. ForceAlarmManager builds no subtag consumer and is not graded.
- degraded is now the OR of "subtag is serving" and "the watch-list covers
  nothing", so the Auto case where alarmmgr is healthy and the standby is empty
  is visible. DashboardAlarmProviderStatus renders that combination as the amber
  "Alarm Manager (no subtag standby)" badge rather than the failover label, which
  would have named the wrong live provider.
- on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the
  monitor re-resolves BEFORE publishing the status and reports any difference as
  SubtagWatchListStale with both counts.

The re-resolution is deliberately NOT re-sent. The worker rejects a second
SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only
expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds
FailoverAlarmConsumer, which restarts on the primary that just failed and
discards the warm standby state, so it would flap the feed rather than repair it.
The finding's other alternative (treat empty coverage as a lifecycle failure) was
rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage
and take the working alarmmgr feed down with it. A monitor restart re-resolves and
re-sends, and the status now says so.

Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale
claims (the resolver method name, and "re-runs discovery on its reconcile cadence
and pushes an updated watch-list", which never happened); gateway.md gains the
standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the
consequence on the UseGalaxyRepository and Mode rows.

Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith-
Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift
drive the fake-worker monitor; two resolver cases pin the new provenance fields.
The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
2026-08-20 09:38:46 -04:00
Joseph Doherty cfbc99581b Merge fix/c3-ipc-boundaries: IPC-33 drain budget bounded under the gRPC send cap + validator upper coupling, IPC-39 pre-check sequence allowance, IPC-34 WorkerCancel on caller cancellation + CallerCancelled code, IPC-35 handshake WorkerFault surfaces category 2026-08-20 09:33:55 -04:00
Joseph Doherty ab41362a5c fix(IPC-35): surface a handshake-phase WorkerFault with its category and diagnostic
The worker reports startup failures as structured WorkerFault frames --
MxAccessCreationException carries the COM HRESULT/ProgID context and a
WorkerFaultCategory -- so the gateway can tell "MXAccess is not installed / not
registered / access denied" apart from a pipe-level protocol problem.
ReadHandshakeEnvelopeAsync did not look inside: any body other than the expected
one became a ProtocolViolation naming only the body type, and that text is what
SessionManager puts in the operator log and the client's OpenFailed detail. The
information was lost exactly in the window where it matters most -- first
session open on a freshly deployed host, where the answer is almost always an
MXAccess installation or permission problem.

Special-case BodyCase == WorkerFault and throw WorkerClientException(
WorkerFaulted, CreateWorkerFaultMessage(...)) -- the same formatter the
steady-state fault path already uses, so both windows read alike. Every other
unexpected body still gets ProtocolViolation.

Test: StartAsync_WhenWorkerFaultsDuringHandshake_SurfacesFaultCategoryAndDiagnostic
has the fake worker answer the hello and then write a WorkerFault
{MxaccessCreationFailed} in place of WorkerReady; the existing
FakeWorkerHarness.EmitFaultAsync covered it, so the harness needed no change.
2026-08-20 09:32:01 -04:00
Joseph Doherty b27035b3a6 fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only.
Its rationale -- the gateway has stopped waiting but the worker has not stopped
working, so a queued-but-not-started command should be dropped instead of
occupying the single STA behind a caller that is already gone -- applies exactly
as well to caller cancellation, which is the common abandonment path: gRPC
clients routinely set per-call deadlines shorter than the gateway's
CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in
arms that removed the pending entry and returned silently, leaving the
correlation to execute in its turn. Under a retry storm against a slow provider
the STA works through dead commands while live callers queue behind them --
precisely the backlog the cancel exists to shorten.

- TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and
  takes the reason string, since it serves both arms.
- Both caller-cancellation arms route through AbandonCommandOnCallerCancellation,
  which sends the cancel under the same `removed` ownership gate as the timeout
  arm (so a reply that won the race is never chased) and skips it once _stopCts
  has fired, where the outbound channel is being completed and every pending
  command is failed anyway.
- New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label
  those arms used, so metrics and logs stop recording a shutdown that did not
  happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps
  to, so nothing changes for clients.

Docs: docs/Grpc.md mapping line, gateway.md cancellation rules,
docs/GatewayProcessDesign.md pending-command section.

Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin
of the timeout test, driven by the caller token; verified failing without the
send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
2026-08-20 09:31:29 -04:00
Joseph Doherty 9f0c3fd9fa Merge fix/c3-dashboard-acl: SEC-41 ACL re-resolves tag grants from group claims, SEC-40 alarms hub/page behind non-bypassable authenticated-role policy, GWC-38 browse scope fails closed without identity under ApiKey mode 2026-08-20 09:31:21 -04:00
Joseph Doherty a71663a2a2 fix(IPC-39): make the CommandTooLarge pre-check allow for the sequence stamp
The pre-check measures the command envelope while Sequence is still 0, and an
unset protobuf field costs nothing to encode -- so the measured size is a lower
bound on the frame WriteLoopAsync emits after GWC-28 stamps the sequence. Being
a lower bound is the right kind of conservative for the knownSize question the
comment answers, and the wrong kind for the pre-check itself: it admits an
envelope in the last eleven bytes of the range that the writer then rejects with
a session-fatal MessageTooLarge.

Compare against MaxMessageBytes - MaxSequenceEncodingBytes (1 tag byte + a
10-byte varint, the widest 64-bit field encoding) and say in the comment that
the pre-check is a lower bound relying on that allowance.

Not reachable under the shipped validator (caller-controlled commands arrive
through gRPC capped at least 64 KiB below the pipe max, gateway-authored
commands are tiny), so this closes the argument rather than a live defect. The
existing CommandTooLarge test still passes unchanged.
2026-08-20 09:30:56 -04:00
Joseph Doherty 8257a0333e docs(SEC-40): correct the hub-mapping comment now that AlarmsHub uses its own policy 2026-08-20 09:30:45 -04:00
Joseph Doherty 577a0db862 fix(IPC-33): bound the DrainEvents byte budget against the gRPC send cap, not just the pipe frame max
The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.

Option (a) from the finding: close the coupling at both ends.

- GatewayOptionsValidator adds the upper half of the headroom rule
  (Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
  message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
  once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
  send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
  usable.

The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.

Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.

The worker half cannot be built on macOS; verified on windev separately.
2026-08-20 09:30:39 -04:00