Commit Graph

429 Commits

Author SHA1 Message Date
Joseph Doherty 26a90883fa Unify gateway styling and preserve identity through recovery
ci / java (push) Failing after 15m11s
ci / nightly-windev (push) Failing after 10m18s
ci / portable (push) Failing after 27m42s
ci / windows-x86 (push) Failing after 22m19s
2026-11-13 20:12:00 -05:00
Joseph Doherty b06921ce85 Lead gateway overview with readable fault context 2026-11-13 19:30:52 -05:00
Joseph Doherty c9a6c31cb8 feat(dashboard): make Galaxy browse and watch accessible 2026-11-13 18:19:08 -05:00
Joseph Doherty 5a1daab9ce test(dashboard): retain native pilot raw verification logs 2026-11-13 17:51:32 -05:00
Joseph Doherty 260d58afa1 fix(dashboard): pilot native dialog focus and safe dismissal 2026-11-13 17:51:02 -05:00
Joseph Doherty 3e004708d4 docs(ldap): windev runs Production on the shared GLAuth's LDAPS :3894 (#180)
ci / java (push) Failing after 15m34s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Failing after 21m32s
ci / windows-x86 (push) Failing after 21m47s
windev's MxAccessGw now runs DOTNET_/ASPNETCORE_ENVIRONMENT=Production, with
DisableLogin removed and MxGateway:Ldap on Transport=Ldaps, Port=3894,
AllowInsecure=false and the GLAuth cert pinned by SHA-256. Record the env
change and the rollback, and update the stale "GLAuth has no LDAPS" text.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 02:42:05 -05:00
Joseph Doherty 2cc0d57541 docs(deploy): QA vq03 + vq04 on 76a71bc8, #228 verified live on both (#229, #170)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m29s
ci / java (push) Failing after 9m20s
ci / portable (push) Failing after 16m20s
First real run of the #230-fixed Upgrade-MxGatewayService.ps1: rolling, ~8-9 s each,
env block untouched. #228 KeepsSentTimestamp passes on both (old build +5 h as the
negative control); #226 not observable on QA (empty alarm snapshot, no transitions).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:48:17 -05:00
Joseph Doherty 9e4be25807 test(live): capture OperationComplete trigger, subtag Clear and status rows on windev (#210, #211)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m23s
ci / java (push) Failing after 7m49s
ci / portable (push) Failing after 16m36s
Live MXAccess captures against the windev DEV Galaxy, recorded in
docs/LiveCaptures-2026-11-13.md:

- OperationComplete follows a Suspend/Activate whose synchronous MxStatus is
  Pending (advised, resolved item); one Ok row from RespondingLmx. Pinned by
  the new OperationCompleteLiveProbeTests live fact, with an
  unresolved-reference negative control. (#211 §3.1)
- No public COM event carries completion-only status bytes. All 71
  completions were typed MXSTATUS_PROXY[], so they stay raw permanently.
  (#211 §3.3)
- Subtag-mode Clear is drivable: the TestAlarm UDAs are MxSecurityOperate,
  so AuthenticateUser + WriteSecured toggles them. The new self-driven
  LiveSubtag_SelfDrivenRaiseThenClear_SynthesizesClear fact asserts the
  UnackAlm -> UnackRtn Clear (mutation-checked). (#210 §3.7)
- The B8 buffered "NoData bootstrap" is a ConfigurationError resolution
  failure (detail 6 / 1002). The DEV Galaxy has no buffered attribute, so a
  multi-sample capture needs IDE work. provider_switches was not driven. (#210)
- Status-row catalogue for lmxopcua#872: no Warning row seen, and Pending
  only on the synchronous Suspend/Activate return.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:34:46 -05:00
Joseph Doherty 4890a3d16e fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
ci / windows-x86 (push) Successful in 2m41s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Failing after 12m34s
ci / java (push) Failing after 14m50s
The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run
MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the
pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously.

- Image names come from the installed exes (NSSM Application, Worker env path) plus
  both known schemes; processes are also matched by exact path (Win32_Process).
- One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at
  the Application path, as the service host's child, is an abort.
- STOP_PENDING kill is scoped to this install's processes.
- Staged checks use the installed exe names; staged Worker must be x86; a staged
  self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade).
- Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the
  live copy; other live-only non-binary top-level files are carried; hash-verified.
- Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never
  values) after stop, swap, start and the stability window; any change aborts.
  Sensitive values are redacted from all output and notes.
- File-based master key (vd03) accepted and backed up; a missing env master key
  (windev) is a warning. Backup dir is ACL-restricted before anything is written.
- Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a
  mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA.
- -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under
  WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable".
- Install-MxGatewayService.ps1 process guard matches both schemes.
- Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions,
  passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed).
- Runbook updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:23:19 -05:00
Joseph Doherty 31f7f9fc6e docs(deploy): DEV vd03 on 76a71bc8, Worker time fixes #226 + #228 verified live (#229)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m19s
ci / portable (push) Failing after 24m22s
ci / java (push) Failing after 24m39s
Server and x86 Worker from 76a71bc8 on wonder-app-vd03, about 9 s downtime, env block and
pepper prefix unchanged. Write2 KeepsSentTimestamp passes; RuntimeAlert reads
2026-07-28T20:02:46.830Z. QA still pending (#170). Upgrade-script gaps filed as #230.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 21:23:40 -05:00
Joseph Doherty 76a71bc82d docs(deploy): pending DEV-then-QA deploy note for the Worker time fixes #226 + #228
ci / windows-x86 (push) Successful in 2m28s
ci / nightly-windev (push) Has been skipped
ci / java (push) Failing after 9m56s
ci / portable (push) Failing after 17m29s
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:59:45 -05:00
Joseph Doherty 9ef6fc8489 fix(worker): send the Write2/WriteSecured2 time argument as box-local time (#228)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m28s
ci / portable (push) Failing after 16m55s
ci / java (push) Failing after 17m47s
MXAccess reads the VT_DATE time argument of Write2, WriteSecured2 and their
bulk forms as box-local time and converts it to UTC itself. The worker passed
the proto Timestamp's UTC clock fields (the COM marshaler ignores
DateTime.Kind), so on vd03 (UTC-5) a Write2 sent at 01:33:00Z was stored at
06:33:00Z.

VariantConverter.ConvertToComTimestamp converts a timestamp_value instant with
TimeZoneInfo.ConvertTimeFromUtc into the worker's zone (TimeZoneInfo.Local by
default, DST included; an injectable zone for tests). The executor uses it for
the time argument only: value conversion is unchanged, and the read direction
(MXAccess reports UTC; ToTimestamp treats it as UTC) is unchanged. The one
residual loss is MXAccess-side: the repeated fall-back hour cannot be told
apart in an offset-less VT_DATE.

Tests: VariantConverterTimestampTests (UTC-5, US-Eastern DST both edges,
UTC+05:30, UTC, fall-back ambiguity, value path unchanged); the four executor
forwarding tests inject a fixed UTC-5 zone; the windev live Write2 smoke now
sends a past whole-second timestamp and asserts the stored one when applied.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:49:41 -05:00
Joseph Doherty 7481b95905 test(live): prove Write2 applies a value via supervisory advise on DEV (#212)
ci / nightly-windev (push) Has been skipped
ci / portable (push) Failing after 7m54s
ci / windows-x86 (push) Failing after 10m16s
ci / java (push) Failing after 10m17s
On the DEV gateway (wonder-app-vd03, 2b13564e), Write2 after
AdviseSupervisory applied the value to MESReceiver_025.MoveInBatchID
(0 -> 212212, OnWriteComplete Ok), and the value was restored to 0. After a
plain Advise, the same Write2 got SecurityError/1008, which is exactly what
the windev run saw. So that refusal came from the plain advise. It does not
show that windev has no writable attribute.

- Client.Tests: add the opt-in LiveGatewayWrite2Tests (gated by
  LiveGatewayWriteFact: endpoint, key file and write item env vars). It runs
  supervisory advise -> Write2 -> readback -> restore. The value test passed
  live on vd03. The timestamp-parity test needs a second opt-in; it fails on
  a non-UTC host until #228 is fixed.
- IntegrationTests: the windev Write2 smoke now uses AdviseSupervisory and
  restores the original value. It has not been re-run on windev.
- GatewayTesting.md: document the env vars. code-reviews IntegrationTests-033:
  correct the "no writable attribute" limit.

The run also found that the Write2 timestamp is read as host-local time
(stored +5 h on the UTC-5 box); filed as #228.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:40:46 -05:00
Joseph Doherty 04a40f2c1a docs(deploy): vd03 API-key pepper moved into the service environment (#171)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m50s
ci / java (push) Failing after 4m55s
ci / portable (push) Failing after 11m17s
Record of the 2026-11-12 config-only change on wonder-app-vd03: MxGateway__ApiKeyPepper
added to the MxAccessGw AppEnvironmentExtra block (same value, sha256 prefix match),
removed from Server\appsettings.json; authenticated round-trip with a pre-change key,
ScadaBridge MxGateway link unchanged. Runbook and configuration doc now say where the
pepper lives on each wonder box.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 19:49:42 -05:00
Joseph Doherty 447070a532 chore(clients/dotnet): bump Contracts + Client 0.3.0 -> 0.4.0 for publish (#227)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m34s
ci / java (push) Failing after 2m33s
ci / portable (push) Failing after 3m39s
Ships the opt-in stream-attached acknowledgement (afd504d5) so OtOpcUa
can adopt it (lmxopcua#877). Published to the Gitea feed 2026-11-12.
2026-11-12 07:32:01 -05:00
Joseph Doherty afd504d5ae feat(events): opt-in stream-attached acknowledgement on StreamEvents (#227)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m28s
ci / java (push) Failing after 2m21s
ci / portable (push) Failing after 3m33s
A fresh StreamEvents is never replayed, so an initial value published before
the gateway registers the stream's subscriber is lost (lmxopcua#865 —
slow-changing ScanState). Clients had nothing deterministic to wait on.

Contract (additive, proto3):
- StreamEventsRequest.emit_stream_attached = 3 (bool, default false)
- MxEvent.stream_attached = 15 (optional StreamAttached)
- message StreamAttached { uint64 live_after_sequence = 1; }
- OpenSessionReply capability "stream-events-attached-ack"
  (GatewayCapabilities.StreamEventsAttachedAck)

When requested, EventStreamService yields the sentinel as the FIRST message,
after AttachEventSubscriber/AttachEventSubscriberWithReplay has returned, so
receipt implies registration; refused attaches never see it. It precedes the
ReplayGap sentinel and replay on a resume. Shape mirrors ReplayGap: family
UNSPECIFIED, no body, worker_sequence 0.

Opt-in rather than always-on: ScadaBridge's MxGateway adapter stores every
streamed event's worker_sequence as its resume cursor, so an unrequested
sentinel would reset it to 0. Default wire behaviour is unchanged.

.NET client: StreamEventsAsync/StreamEventItemsAsync(after, emitStreamAttached),
MxEventStreamItem.IsStreamAttached, MxGatewaySession.SupportsStreamAttachedAck,
and OpenEventStreamAsync -> MxGatewayEventStream (awaits the sentinel, with
attach timeout; refuses older gateways).

Regenerated Contracts/Generated, the client protoset, Go and Python bindings
(also picks up comment-only drift already on main), synced the Rust vendored
proto and set emit_stream_attached in the Rust StreamEventsRequest literals.
No package versions bumped.

Tests: ordering (sentinel first, then events), race proof (event fanned right
after the sentinel is delivered — mutation-checked: fails when the sentinel is
yielded before registration), resume-with-gap order, refused attach emits no
sentinel, default emits none, contract field numbers, capability, and .NET
client wire tests (await, no-capability, older gateway, timeout).
2026-11-12 07:27:17 -05:00
Joseph Doherty 892bdc1a19 fix(worker): wnwrap alarm DATE+TIME are UTC; stop re-applying GMTOFFSET (#226)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m28s
ci / portable (push) Failing after 3m38s
QueryActiveAlarms / alarm transitions reported every wnwrap alarm late by
the record's GMTOFFSET-DSTADJUST: ParseTransitionTimestampUtc treated the
XML DATE+TIME as producer-local and added the offset, but the fields are
already UTC. DEV evidence: RuntimeAlert raise EventTimeUtc 2026-07-28
20:02:46.830 (EventTimeUTCOffsetMins -300) surfaced as 2026-07-29
01:02:46.830Z. The 2026-05-01 probe capture (TIME 13:26:14.709,
GMTOFFSET 240) was committed at 09:44 EDT, so its TIME is UTC too.

GMTOFFSET/DSTADJUST are no longer read. Tests updated + #226 regressions.
2026-11-12 07:20:44 -05:00
Joseph Doherty 2b13564e04 feat(ldap): pin the LDAP server certificate by SHA-256 thumbprint (#180)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m15s
ci / portable (push) Failing after 4m19s
New optional MxGateway:Ldap:ServerCertificateThumbprintSha256 (hex SHA-256
of the DER cert; case/colon/dash/space-insensitive; comma/semicolon-separated
string or array, so a rotation can carry old + new). When set, the dashboard's
shared LDAP provider gets a RemoteCertificateValidationCallback via
LdapOptions.ServerCertificateValidationCallback (Auth 0.4.1) that accepts the
server certificate iff its thumbprint matches a pin, ignoring chain/name
errors only then. Unset keeps default OS trust-store validation untouched.

The callback is an init-only, code-only property, so the ILdapAuthService
registration from AddZbLdapAuth is replaced with one built from the bound
options plus the callback. Pins are read straight from configuration because
the binder silently binds the scalar form onto a list as empty.

GatewayOptionsValidator rejects a pin that is not 64 hex chars after
normalization and any pin with Transport=None; the Production hard stops are
unchanged. Pins are shown on the Settings page. Docs: GatewayConfiguration.md,
glauth.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 11:52:14 -05:00
Joseph Doherty 53562ddc17 fix(worker): unregister-first shutdown cleanup order, guarded + logged (#175, R21-K10)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m32s
ci / java (push) Successful in 2m21s
ci / portable (push) Failing after 4m30s
ShutdownGracefully now Unregisters every server handle first (MXAccess drops its
items and advise subscriptions with it). Only for a server whose Unregister fails
does the UnAdvise -> RemoveItem fallback run. Each step is guarded, logged at Debug,
and the method is idempotent. #176 (AuthenticateUser cache) is unaffected.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 04:26:55 -05:00
Joseph Doherty 477fa55998 feat(deploy): on-box NSSM upgrade/rollback and new-box install scripts, runbook updated (mxaccessgw#220)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m12s
ci / portable (push) Failing after 4m11s
Add scripts/deploy/Upgrade-MxGatewayService.ps1 (preflight, stop with STOP_PENDING handling,
backup, Server+Worker swap, start, /health/ready + SHA verify, auto/manual rollback,
one-gateway guard, never touches the NSSM env block) and Install-MxGatewayService.ps1
(service install, restart/recovery settings, env block). Both support -DryRun/-WhatIf.
Dry-run validated on macOS pwsh against a fake layout; untested on a real box.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:10:26 -05:00
Joseph Doherty fccc21ba8e feat(events): distinguish ResourceExhausted causes with an mxgw-reason trailer (mxaccessgw#216)
Event-queue overflow, alarm-feed overflow, subscriber cap, session limit and
oversized commands all surfaced as ResourceExhausted. The status code is
unchanged; the gateway now attaches a machine-readable `mxgw-reason` trailer
(constants in Contracts ResourceExhaustedReasons). The .NET client maps
ResourceExhausted to MxGatewayResourceExhaustedException (an
MxGatewayException subclass; additive) exposing Reason/IsOverflow/
IsSubscriberLimit. Documented in docs/ClientBehaviorFixtures.md; the other
clients get a doc note only.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:02:24 -05:00
Joseph Doherty de86f8b80e ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m34s
ci / java (push) Successful in 2m37s
ci / portable (push) Failing after 5m0s
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):

- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
  New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
  succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
  2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
  second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
  Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
  handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
  harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
  removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
  Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
  Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
  until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
  csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
  window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
  memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
  in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
  classes migrating the one per-process secrets store. Tests now build the gateway through
  TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
  the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
  7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
  burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.

Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:36:34 -05:00
Joseph Doherty 22964e192b feat(gateway,worker): configurable worker watchdog grace; heartbeat settings in effective config and Settings page (mxaccessgw#224)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m24s
ci / java (push) Successful in 2m15s
ci / portable (push) Failing after 4m17s
- New MxGateway:Worker:WorkerHeartbeatGraceSeconds (default 15), exported as
  MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS and read by WorkerPipeSessionOptions.FromEnvironment
  (a grace the ceiling does not exceed falls back to the default so the worker
  still starts). Validation now checks WorkerHeartbeatStuckCeilingSeconds against
  the configured grace; the copied WorkerWatchdogGraceSeconds constant is gone.
- EffectiveWorkerConfiguration and the dashboard Settings page now show the worker
  heartbeat send interval, watchdog grace and stuck ceiling (render test added).
- Docs updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:29:10 -05:00
Joseph Doherty 2e52f9f5e8 feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m32s
ci / java (push) Successful in 2m57s
ci / portable (push) Failing after 5m1s
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.

- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
  and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
  WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
  MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
  worker settings use.
- The worker's production WorkerPipeSession reads them through
  WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
  ceiling not above the worker's 15 s watchdog grace, keeps the default rather
  than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
  send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
  the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:24:05 -05:00
Joseph Doherty 29d9510b4e perf(worker): batch frames skip per-frame TCS; stream the alarm-poll XML (mxaccessgw#177)
R21-K12: WorkerFrameWriter.WriteBatchAsync allocated a TaskCompletionSource
(plus its Task) per frame on the event drain hot path. PendingFrame now holds
its terminal state itself (Interlocked pending/succeeded/faulted/canceled) and
only single-frame WriteAsync frames, which race their completion against the
write lock, get a TCS. A batch caller reads each frame's outcome after its own
drain pass: every frame a pass claims is terminal before that pass releases the
lock, so nothing is left to await. First-failure-in-batch-order, tombstoning,
claim/cancel interlock and flush coalescing are unchanged.

R21-K13: WnWrapAlarmConsumer.ParseSnapshotXml streams the GetXmlCurrentAlarms2
payload with an XmlTextReader (same defaults XmlDocument.LoadXml uses:
Normalization off, DTDs parsed, no resolver; Significant whitespace only)
instead of building a full XmlDocument every 500 ms poll.

Behaviour is pinned by new tests: batch first-failure ordering (own drain and
drained by another lock holder), batch cancellation tombstoning, batch flush
failure, and a differential test running the replaced XmlDocument parser as an
oracle over a 48-payload corpus (CRLF, CDATA, entities, DTD, namespaces,
nesting, malformed). Mutation checks on windev: 8 of 8 injected regressions
(whitespace, normalization, DTD, namespace, CDATA, batch last-failure, batch
ignored failures, ignored tombstone) fail the new tests.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:24:05 -05:00
Joseph Doherty d9c60a78eb feat(worker): log exception messages and per-command start/end (mxaccessgw#184)
R21-O3: WorkerApplication.Run's failure lines logged only the exception
type. Each now also carries exception_message, scrubbed by the new
WorkerLogRedactor.RedactMessage: the value of any name=value / name: value
pair whose name looks sensitive and any mxgw_ API key token are replaced,
line breaks are flattened and the text is capped at 512 characters. The
AppDomain unhandled-exception hook stays type-only, since an exception
from an arbitrary thread can carry command values.

R21-O7: StaCommandDispatcher.ExecuteQueuedCommandAsync logs
WorkerCommandStarted (correlation id, method, queued_ms) and
WorkerCommandCompleted (status, HRESULT, elapsed_ms) when a CommandLogger
is set. IWorkerLogger has no Debug level, so the lines are opt-in through
MXGATEWAY_WORKER_LOG_COMMANDS=1|true (WorkerCommandLogging), which the
worker inherits from the gateway process environment. No payloads or
diagnostic messages are logged. Docs updated. (mxaccessgw#184)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:01:58 -05:00
Joseph Doherty f49b7a0cc1 fix(worker): exit 5 on gateway death, dedicated exit 8 on event-queue overflow (mxaccessgw#186)
A frame read that hits end of stream means the gateway closed its end of
the pipe (usually because it died). WorkerApplication mapped every
WorkerFrameProtocolException to ProtocolViolation (6); EndOfStream now
exits as PipeConnectionFailed (5), alongside the IOException a broken
pipe raises.

Event-queue overflow exited as the generic UnexpectedFailure (1). The
event drain loop now unwinds with WorkerEventQueueFaultException carrying
the fault category, and QueueOverflow exits with the new
WorkerExitCode.EventQueueOverflow (8). Other categories keep exit 1.

The gateway does not decode worker exit codes (it only logs the number),
so no gateway change is needed. WorkerBootstrap.md and
MxAccessWorkerInstanceDesign.md updated. (mxaccessgw#186)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:01:58 -05:00
Joseph Doherty 8c21678cb4 feat(health): worker-exe presence and bounded Galaxy SQL readiness checks (mxaccessgw#206)
Adds WorkerExecutableHealthCheck (configured worker .exe exists) and
GalaxySqlHealthCheck (IGalaxyRepository.TestConnectionAsync bounded by a 5 s
TimeProvider-driven timeout). Both are tagged ready only, never live, so an
outage takes the node out of rotation without triggering a restart; sessions
stay on the active tier so zero sessions is still healthy for Overview.

Registrations move into AddGatewayHealthChecks() so the tier tagging is
unit-tested. LDAP is not added (sign-in already surfaces bind failures).
docs/Diagnostics.md lists the two new checks.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:58:07 -05:00
Joseph Doherty 2f6e480a41 docs(runbooks): deploy/upgrade runbook for the NSSM-wrapped gateway service (mxaccessgw#207)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:43:52 -05:00
Joseph Doherty 07badd139e fix(gateway): bound the session replay ring by bytes as well as count (mxaccessgw#172)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m28s
ci / portable (push) Failing after 4m48s
The replay ring was bounded only by ReplayBufferCapacity (1024 events) and
ReplayRetentionSeconds. Event size is unbounded up to the worker frame size,
so a burst of large-array events could pin gigabytes per session while the
count looked small.

New MxGateway:Events:ReplayBufferMaxBytes (default 64 MiB, 0 disables) caps
the summed serialized size (MxEvent.CalculateSize, measured only when the cap
is on). Over the cap the oldest entries are evicted, exactly like the count
cap, so a resuming subscriber sees the existing ReplayGap path; an event bigger
than the whole cap is not retained. Every eviction (count, age or byte) now
clears its slot, so an evicted event is released immediately instead of
staying referenced until the slot is reused.

The 64 MiB default is a judgement call: it never binds for scalar traffic
(1024 x a few hundred bytes) and only trims large-array bursts.

Tests: SessionEventDistributorTests byte-cap eviction, single oversized event,
cap 0 = count only (the first two fail with the cap enforcement removed);
validator rejects a negative value. Docs: GatewayConfiguration.md, Sessions.md,
DesignDecisions.md; appsettings.json carries the key.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:39:49 -05:00
Joseph Doherty 7b21f304e9 fix(gateway): split the heartbeat check interval from the send interval; expose the two fixed worker-client timeouts (mxaccessgw#192)
R21-O14: MxGateway:Worker:HeartbeatIntervalSeconds was documented as the
worker heartbeat send interval, but it never reached the worker (which sends
on its own fixed 5 s) and only set the gateway's HeartbeatCheckInterval. It is
renamed HeartbeatCheckIntervalSeconds (default 5, unchanged). The old key stays
as a deprecated alias with the same meaning so existing appsettings keep
working; setting both is a validation error. The grace-vs-interval check now
compares against the resolved check interval.

R21-O15: WorkerClientOptions.EventChannelFullModeTimeout and
HeartbeatStuckCeiling were always their compile-time defaults. They are now
MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds (default 5000, > 0)
and HeartbeatStuckCeilingSeconds (default 75, >= HeartbeatGraceSeconds). The
factory's mapping moved into SessionWorkerClientFactory.CreateWorkerClientOptions
so every field is set from configuration and is testable. The effective-config
record shows the resolved check interval and both new values.

Docs: GatewayConfiguration.md, GatewayProcessDesign.md sample,
MxAccessWorkerInstanceDesign.md heartbeat section; appsettings.json uses the
new keys.

Tests: CreateWorkerClientOptions mapping + defaults (fail under a mutated
mapping), alias honoured, both-keys conflict, and bounds for the new options.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:39:49 -05:00
Joseph Doherty 11d8f76c5f fix(gateway): GatewaySession implements IAsyncDisposable; drop dead KillWorker(string) (mxaccessgw#191)
R21-O12: GatewaySession already owned an async DisposeAsync teardown but did not
declare IAsyncDisposable, so anything holding it as IAsyncDisposable (or a DI
container) could not find the teardown. Declaring it changes no call site;
the existing await-using sites bound to the same method by pattern.

R21-O13: KillWorker(string) had no callers and was an ungated kill that
bypassed _closeLock. Deleted it; docs/Sessions.md and the test comments now
name KillWorkerWithCloseGateAsync, the real kill path.

Regression test: GatewaySessionTests.GatewaySession_ImplementsIAsyncDisposable
(fails before the interface is declared).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:39:49 -05:00
Joseph Doherty d56353bf08 fix(gateway): remove the orphaned mxgateway.retries.attempted instrument (mxaccessgw#215)
The launcher's Polly startup-probe pipeline was its only recorder and was
removed in mxaccessgw#188, so the counter, the RetryAttempted method and the
snapshot's RetryAttempts / RetryAttemptsByArea fields carried a permanent
zero. No dashboard page or in-repo alert rule references them. Metrics.md
drops the row and the snapshot shape (which also gains the missing
AlarmProviderSwitchCount field).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:39:49 -05:00
Joseph Doherty 9906d7e57c docs(worker): state that residual events are discarded after WorkerShutdown (mxaccessgw#183)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:31:56 -05:00
Joseph Doherty ad7d2cbef0 docs(clients): add slow-consumer/backpressure comparison table (mxaccessgw#204)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:31:32 -05:00
Joseph Doherty 7c377cac7b docs(config): complete the Configuration Shape JSON block (mxaccessgw#209)
Add Dashboard DisableLogin/AutoLoginUser/CookieName, Alarms:Fallback and Tls,
matching the option classes and the per-option tables below the block.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:30:42 -05:00
Joseph Doherty 417b9e5a19 feat(apikey): add --pepper-stdin and warn on --pepper (mxaccessgw#194)
--pepper on the command line lands in shell history and the process list.
Add --pepper-stdin (first stdin line), warn on stderr when --pepper is used,
and reject both together. Documented in GatewayProcessDesign.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:27:37 -05:00
Joseph Doherty be0d585f8c fix(server): cap Serilog file-sink retention and size (mxaccessgw#208)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m31s
ci / java (push) Successful in 2m46s
ci / portable (push) Failing after 5m16s
Set retainedFileCountLimit=30, fileSizeLimitBytes=50 MiB and rollOnFileSizeLimit=true on the
file sink in appsettings.json and appsettings.Development.json; document under a new Logging
section in docs/GatewayConfiguration.md; add a guard test over the shipped config.
2026-11-09 12:10:20 -05:00
Joseph Doherty 42f69e4f00 fix(server): remove inert worker startup probe retry options (mxaccessgw#188)
WorkerProcessStartedProbe only checks HasExited, so a retry could never change
the outcome. Remove StartupProbeRetryAttempts/StartupProbeRetryDelayMilliseconds,
their validation, the launcher's Polly retry pipeline and the docs; the probe now
runs once under StartupTimeoutSeconds. Stale config keys are ignored by the binder.
2026-11-09 12:10:20 -05:00
Joseph Doherty e5e7f1854f chore(clients/dotnet): bump Contracts + Client 0.2.1 -> 0.3.0 for publish (#214)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m19s
ci / portable (push) Failing after 4m35s
0.2.1 on the feed predates the 488649c AcknowledgeAlarmRequest
operator_password/operator_domain fields; 0.3.0 ships them so OtOpcUa can
adopt operator-authenticated alarm ack (lmxopcua#749).
2026-11-06 16:00:21 -05:00
Joseph Doherty 777fef9cad fix(sessions): drop the deny-NETWORK pipe ACE (#187)
ci / windows-x86 (push) Successful in 1m26s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m30s
ci / portable (push) Failing after 4m47s
NT AUTHORITY\NETWORK marks a network logon, not a remote caller. On windev
a gateway run from an SSH session carries it in its own token, so the deny
entry locked the gateway's same-machine workers out (UnauthorizedAccess on
connect; 15 fake-worker tests failed). The ACL now holds exactly one entry:
FullControl for the gateway's user.
2026-11-06 15:52:46 -05:00
Joseph Doherty 34a7453ac3 fix(sessions): restrict the worker named pipe to the gateway's account (#187)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m20s
ci / portable (push) Failing after 4m46s
On Windows each worker pipe is created via NamedPipeServerStreamAcl with a
protected DACL granting FullControl only to the gateway process user (the
worker is launched under the same account) and denying NETWORK logons. On
non-Windows hosts (test matrix) the pipe uses PipeOptions.CurrentUserOnly.
Tests assert the factory ACL and the ACL read back from the created pipe
(Windows), and that a same-user client still connects.
2026-11-06 15:45:50 -05:00
Joseph Doherty 9354d69bf9 fix(diagnostics): redact WriteSecuredBulk/WriteSecured2Bulk payloads (#193)
Adds both secured bulk kinds to GatewayLogRedactor's credential-bearing set,
plus a contract cross-check test that walks every MxCommand payload descriptor
and fails if a secured or password-bearing command kind is not redacted.
2026-11-06 15:45:50 -05:00
Joseph Doherty 5a235776b6 docs: point markdown backlogs at Gitea issues (status lives in Gitea)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m28s
ci / java (push) Successful in 2m29s
ci / portable (push) Failing after 4m58s
2026-11-06 15:32:27 -05:00
Joseph Doherty dd623356f4 docs(clients): ReadBulk timeout is one shared deadline for the command, not per tag
Stream E made the worker wait once for all uncached tags. Fix the Go, Python, Rust, Java
and .NET library docs and CLI help text, the proto comment (default is the worker's
1000 ms), and the worker design doc. Regenerated Contracts Generated/ and the Java
generated protos; both diffs are comment-only.
2026-10-22 08:46:29 -04:00
Joseph Doherty 3ffba83723 docs: dashboard live-data session reaping, subscriber-refusal code, distributor-based StreamEvents model
- Sessions.md / GatewayConfiguration.md: the dashboard Browse live-data session is
  unary-only, is reaped by UnattachedSessionGraceSeconds when idle, and reopens via
  EnsureReadyAsync; only the alarm monitor is exempt.
- GatewayConfiguration.md: a refused second subscriber gets ResourceExhausted, not AlreadyExists.
- Grpc.md: rewrite the event streaming model around the SessionEventDistributor pump and
  replay ring; add EventSubscriberLimitReached to the mapping snippet; describe what
  clients see while the gateway is stopping.
2026-10-22 08:43:41 -04:00
Joseph Doherty a0b5e91227 fix(grpc): count a worker fault once, not once per StreamEvents subscriber
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
2026-10-22 08:42:35 -04:00
Joseph Doherty a178d68b60 fix(sessions): refuse a dead worker at attach, fault on pump readiness failure, keep cancel/stopping out of OpenFailed
- TryAttachWorkerClientAndMarkReadyAsync returns SessionAttachOutcome and refuses a
  worker client that is already Faulted/Closed; the open discards it and fails OpenFailed.
- MapWorkerEventsAsync calls ObserveWorkerTerminalState when its first readiness check
  fails, so a worker that dies between attach and the first pump read faults the session
  and the faulted sweep reaps it.
- A caller-cancelled open rolls back, records no fault and rethrows the cancellation
  (gRPC Cancelled); an attach refused because the drain closed the session reports
  GatewayStopping.
- docs/Sessions.md updated.
2026-10-22 08:41:02 -04:00
Joseph Doherty 20135734b3 Merge stream E (worker: cache eviction, ReadBulk shared deadline + dedupe, alarm poll threshold, STA loop fault, unhandled-exception hook, live control-command facts) 2026-10-22 08:28:55 -04:00
Joseph Doherty 2fad124bf8 Merge stream L (proto reserved/propagation comments + regen, stale docs, gateway test gaps) 2026-10-22 08:25:25 -04:00