Commit Graph

13 Commits

Author SHA1 Message Date
Joseph Doherty ff397b27be fix(gateway): remove the worker startup-probe retry options that could never retry (#188)
WorkerProcessStartedProbe only checks HasExited and throws a non-retryable
WorkerProcessLaunchException, so StartupProbeRetryAttempts and
StartupProbeRetryDelayMilliseconds configured a Polly retry that no shipped
probe could trigger. Real readiness is WorkerClient's WorkerReady handshake
under the same StartupTimeoutSeconds budget, so the probe now runs once and
the two options, their validation, the Polly pipeline (and the server's now
unused Polly.Core reference) and their docs are removed. A stale value in
configuration is ignored by the binder.

Tests: WorkerProcessLauncherTests.LaunchAsync_WhenStartupProbeFails_DoesNotRetryTheProbe,
WorkerOptions_HasNoStartupProbeRetryOptions (both fail on the old code).
2026-11-06 16:05:42 -05:00
Joseph Doherty a0b5e91227 fix(grpc): count a worker fault once, not once per StreamEvents subscriber
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
2026-10-22 08:42:35 -04:00
Joseph Doherty 6ae5ed905f fix(grpc): record the real StreamEvents disconnect reason
StreamDisconnected always recorded "Detached". The reason tag is now a
bounded EventStreamDisconnectReasons constant for how the stream ended:
SessionClosed, Overflow, WorkerFault, ClientCancelled, Error, or Detached.
2026-10-22 06:29:31 -04:00
Joseph Doherty 3619152b9f chore(GWC-31): relabel perf-pass GWC-3x comments to PERF-nn; tracker housekeeping
Sweep code/doc comments citing unregistered GWC-30/GWC-31 (2026-08-15 perf
pass) to PERF-02/PERF-07/PERF-22, keyed to task numbers in
docs/plans/2026-08-15-perf-review-remediation.md; add a PERF-ID register
table to that plan. Genuine cycle-2 GWC-30 citations (gateway-side frame-
reader 4-byte prefix fix in Server/Workers/WorkerFrameReader.cs and its
test) are left untouched.

Add a line to REVIEW-PROCESS.md reserving the GWC/WRK/IPC/SEC/CLI/TST
prefixes for the archreview domain registers.

Tracker housekeeping per archreview/2026-08-19/00-overall.md:
- GWC-16 -> Won't fix (rationale: PERF(followup) comment documents the
  double session lookup as measured-negligible, deliberately not collapsed)
- IPC-08 -> Done (closed by 7c1ea12/a1a38b5; residual tracked as IPC-34)
- IPC-31 -> Done (superseded by GWC-28's write-time sequence stamping)
- TST-06, TST-17 rows refreshed per cycle-3 60-testing-docs-gaps.md
- WRK-31 (and defensively WRK-29/30) registered as reserved/do-not-reuse
  interim IDs burned by e913dab, in both the cycle-2 tracker and the
  cycle-2 worker remediation doc

Flip Batch C to Done in archreview/2026-08-19/remediation/00-tracking.md.

Worker-side files (Worker/Ipc/WorkerFrameReader.cs, WorkerFrameWriter.cs)
are comment-only edits, unverified here (net48 x86, windev-only build).
2026-08-20 09:23:36 -04:00
Joseph Doherty 9735ac3b7c perf(metrics): pull-model worker queue gauge (fixes last-writer-wins), Interlocked command counters 2026-08-15 12:21:51 -04:00
Joseph Doherty d4154e340c fix(GWC-24): bound the worker event staging channel and unify the depth gauge
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.

Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.

Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.

Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
2026-08-07 05:35:07 -04:00
Joseph Doherty 5e2e40a927 perf(gateway): trim event/command hot-path allocations (GWC-06/07/15, IPC-05)
Behavior-preserving allocation cuts on the per-event/per-command path:
- GWC-06: StreamEvents send-timing uses Stopwatch.GetTimestamp() +
  GetElapsedTime() instead of a per-event Stopwatch allocation (same measured
  span).
- GWC-07/IPC-05 (event): MapEvent transfers ownership of the inner MxEvent
  instead of .Clone()-ing it. Safe: WorkerEvent is parsed fresh per pipe frame
  with the distributor pump as its single consumer (GWC-01), MapEvent runs once
  before fan-out, and every downstream consumer (subscribers, replay ring)
  only READS the event (WorkerSequence is stamped upstream; verified no
  post-mapping mutation). Comment documents the invariant + restore-clone
  caveat if a second consumer is added.
- IPC-05 (command): CreateCommandEnvelope no longer re-clones; MapCommand
  already isolated the graph from the caller-owned gRPC message.
- GWC-15: grpc_stream_queue.depth converts from a per-event push counter to an
  ObservableGauge summing registered channel sources at scrape time only
  (name/semantics unchanged); removes all per-event .Count/lock work.

Kept every load-bearing isolation clone (MapCommand, Invoke, bulk filters,
MapCommandReply). Server build clean (0 warnings); EventStream/Metrics/
Distributor/Mapper tests 62/62 (incl. formerly-flaky queue-depth tests, now
green under the lazy gauge, + 2 new MapEvent ownership tests). Docs: Metrics.md,
Grpc.md.

Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
2026-07-09 15:12:43 -04:00
Joseph Doherty 74e815c4d7 fix(archreview-p1): SEC-02/12/20 dashboard + observability hardening
- SEC-02: DashboardAuthorizationHandler restricts the loopback and
  Authentication:Mode=Disabled bypasses to read-only. They now satisfy only a
  Viewer-bearing requirement (AnyDashboardRole), never AdminOnly, so anonymous
  localhost can view the dashboard but cannot reach API-key CRUD or session
  Close/Kill at the policy layer (previously guarded only by service re-checks).
- SEC-12: DashboardSessionAdminService emits canonical AuditEvents through
  IAuditWriter (actions dashboard-close-session / dashboard-kill-worker, category
  SessionAdmin) on Success/Failure/Denied, mirroring the API-key audit path so
  destructive session actions leave durable, queryable rows.
- SEC-20: drop the unbounded session_id tag from the exported
  mxgateway.heartbeats.failed counter (per-session detail stays in the snapshot/log).

Docs updated same-change: CLAUDE.md (read-only loopback + 5-min bearer),
GatewayDashboardDesign.md (bypass scoping + session-admin audit), Metrics.md.
Server build clean; 30/30 targeted + 295/295 Dashboard/Security/App/Metrics sweep.
2026-07-09 07:47:51 -04:00
Joseph Doherty 17f16ea181 fix(archreview): security authz+hub cluster (SEC-07/05/08/11) + validation hardening
SEC-07: add QueryActiveAlarmsRequest -> events:read scope arm; fix two tests that
  constructed StreamAlarmsRequest instead of QueryActiveAlarmsRequest.
SEC-05: shorten hub-token lifetime 30m -> 5m; document that the ?access_token= query
  carriage must never be request-logged.
SEC-08: gateway-side CachingApiKeyVerifier (short TTL, keyed on a hash of the presented
  secret) skips the per-call store read+last_used write; CoalescingMarkApiKeyStore bounds
  last_used writes to <=1/key/min; identity constraints are cached. Invalidation is wired
  at the gateway admin sites (revoke/rotate/delete); short TTL backstops out-of-process CLI.
SEC-11: fixed-window rate limit on POST /auth/login + a per-peer (key-id) failure limiter
  checked before VerifyAsync; new MxGateway:Security options bound + validated.

Also fixes regressions from the SEC-01/04/06 commit (c185f62) that a narrow test filter
missed (all now covered by a full-suite checkpoint):
- Rooting check is cross-platform: accepts Windows C:\/UNC forms on Unix so the shipped
  appsettings path validates on the macOS dev box, still rejecting bare filenames.
- AddGatewayConfiguration TryAdds a non-production IHostEnvironment fallback so the validator
  resolves in minimal test/tooling containers; the real host + apikey CLI register the actual
  environment first (TryAdd no-op there).
- Test assembly defaults ASPNETCORE_ENVIRONMENT=Development (ModuleInitializer) so full-host
  tests exercise wiring instead of tripping the SEC-04/06 production guards.
- GatewayOptionsTests asserts the SEC-01 CommonApplicationData-derived default (platform-correct).

archreview: SEC-07/05/08/11 (P1). Verified: NonWindows build clean; full gateway suite
747 passed / 42 failed, where all 42 are the pre-existing macOS named-pipe-harness failures
(Unix-socket path limit) and 0 are validation/regression failures.
2026-07-09 07:31:35 -04:00
Joseph Doherty dbf550da8b docs(mxgateway): sync Metrics.md to renamed meter + seconds histogram units 2026-06-01 16:48:46 -04:00
Joseph Doherty dc9c0c950c rename: prefix gateway projects/namespaces with ZB.MOM.WW + sln→slnx
Apply the ZB.MOM.WW. prefix to all gateway-side projects, folders,
.csproj/.sln contents, C# namespaces, using directives, generated proto
C# (csharp_namespace + checked-in generated files), InternalsVisibleTo
attributes, project-name string literals (LoadProject, .sln lookups,
worker exe paths, staticwebassets manifest), and the install/script/doc
references that point at any of the above. Migrate the solution from
.sln to .slnx via `dotnet sln migrate` and delete the old file.

External-runtime identifiers are intentionally NOT prefixed so external
configuration keeps working:
- GatewayMetrics.cs MeterName ("MxGateway.Server")
- DashboardAuthenticationDefaults Scheme/Policy ("MxGateway.Dashboard")
- GatewayRequestLoggingMiddleware logger category ("MxGateway.Request")
- StaRuntime thread name ("MxGateway.Worker.STA")
- appsettings.json root section "MxGateway" + env-var prefix
  MxGateway__... and secret-name MxGateway:ApiKeyPepper
- C:\ProgramData\MxGateway\ data dir paths

Also fixes two tests that were not rename-related but became visible
while validating the rename:

- WorkerLiveMxAccessSmokeTests.ShutDownAsync: cancellation that the
  gateway service correctly maps to RpcException(Cancelled) per gRPC
  convention was being misclassified as a stream fault. Added a sibling
  catch on RpcException with StatusCode.Cancelled.

- IntegrationTestEnvironment.ResolveRepositoryRoot: extracted IsRepositoryRoot
  and made it accept either a .git marker OR a .sln/.slnx next to src/
  so the worker-exe walker works in non-git working copies.

clients/proto/proto-inputs.json's protoRoot updated to point at
src/ZB.MOM.WW.MxGateway.Contracts/Protos.

Verified by `dotnet build` and a full `dotnet test` of the .slnx with
MXGATEWAY_RUN_LIVE_{MXACCESS,LDAP,GALAXY}_TESTS=1:
  Tests: 472/472 pass
  Worker.Tests: 280/280 pass (4 dev-rig [Fact(Skip=...)] skipped)
  IntegrationTests: 18/18 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 16:22:23 -04:00
Joseph Doherty 51a9dadf62 Align docs with StyleGuide and add CLAUDE.md
- Rename 16 kebab-case docs to PascalCase per StyleGuide
- Move per-language client design docs from docs/ to clients/<lang>/
  alongside their READMEs
- Add ## Related Documentation sections to 15 docs that lacked one
- Fix sentence-case violations in H3 headings (StyleGuide rule)
- Update cross-references in gateway.md, client READMEs, scripts,
  and generate-proto.ps1 helpers to follow the new paths
- Add CLAUDE.md with build/test commands, the source-update
  verification matrix, the parity-first contract, and pointers
  to MXAccess and Galaxy Repository analysis sources

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 10:19:22 -04:00
Joseph Doherty 133c83029b Add Galaxy repository API and clients 2026-04-29 07:27:00 -04:00