The worker logged its MXSTATUS_PROXY record-info pin outcome only to its own
stderr, which the gateway's launcher does not capture, so operators never saw
whether the pin took.
- Contract (additive, internal IPC): WorkerReady.record_info_pin_status = 5.
No protocol version bump; an older peer ignores it or sends it empty.
Regenerated Contracts/Generated, the client protoset, the Rust vendored
proto, and the Go/Python/Java worker bindings (real delta only).
- Worker: the handshake copies MxStatusProxyRecordInfoPin.Status into
WorkerReady unless the runtime already set it.
- Gateway: WorkerClient records it (IWorkerClient.RecordInfoPinStatus, default
null) and logs it at Information when pinned, Warning otherwise; empty is
not logged. Shown as "Record-info pin" in the dashboard session details
Worker table via DashboardSessionSummary.WorkerRecordInfoPinStatus.
- Tests: drain-thread cancellation (task ends Canceled; RunAsync cancels,
runtime disposed, thread exits, no EventDrain stop-failure log) and fault
(exception on the thread faults RunAsync with the same instance) paths;
WorkerReady carries the pin status; gateway logging/recording and the
dashboard projection.
- Docs: gateway.md STA thread model (drain thread, pin step in COM lifetime),
MxAccessWorkerInstanceDesign, GatewayProcessDesign, GatewayDashboardDesign,
WorkerFrameProtocol.
At ~240 events/s the worker burned 8-12% system CPU on wonder-app-vd03. A WPR
trace put 72% of it in lmxproxy's per-event GetRecordInfoFromGuids (type
library reload through the registry, amplified by WOW64 and the EDR registry
filter) and 26% in the .NET Framework thread pool spinning after each
per-event work item.
- Worker: MxStatusProxyRecordInfoPin holds the LMXProxy ITypeLib and the
MXSTATUS_PROXY IRecordInfo for the process lifetime, so oleaut32 serves the
per-event lookup from cache (~1 ms -> <1 us per call). Best-effort, never
fatal; status logged at handshake.
- Worker: the event drain loop runs on a dedicated background thread with a
synchronous wait and a synchronous pipe flush, removing the per-event
thread-pool hops.
- Gateway: runtimeconfig sets ThreadPool.UnfairSemaphoreSpinLimit=10 to cut
PortableThreadPool spin-wait CPU.
windev bench at ~242 ev/s: worker 67% -> 6.2% of one core; gateway 8.1% -> 6.9%.
Worker suite 785 passed / 14 skipped (x86, windev).
The scheduled nightly-windev job failed most nights for host reasons and
filed a Gitea issue each time. Remove the cron trigger and the job, and add
a workflow_dispatch trigger whose windev_mode input (test/build/live) drives
the windows-x86 job, so the live tier runs on demand - from the Actions page
or via scripts/ci/run-windev-ci.sh live - the same way clients are packed and
published on demand. Docs updated to match.
Co-Authored-By: Claude <noreply@anthropic.com>
windev's MxAccessGw now runs DOTNET_/ASPNETCORE_ENVIRONMENT=Production, with
DisableLogin removed and MxGateway:Ldap on Transport=Ldaps, Port=3894,
AllowInsecure=false and the GLAuth cert pinned by SHA-256. Record the env
change and the rollback, and update the stale "GLAuth has no LDAPS" text.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
First real run of the #230-fixed Upgrade-MxGatewayService.ps1: rolling, ~8-9 s each,
env block untouched. #228 KeepsSentTimestamp passes on both (old build +5 h as the
negative control); #226 not observable on QA (empty alarm snapshot, no transitions).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Live MXAccess captures against the windev DEV Galaxy, recorded in
docs/LiveCaptures-2026-11-13.md:
- OperationComplete follows a Suspend/Activate whose synchronous MxStatus is
Pending (advised, resolved item); one Ok row from RespondingLmx. Pinned by
the new OperationCompleteLiveProbeTests live fact, with an
unresolved-reference negative control. (#211 §3.1)
- No public COM event carries completion-only status bytes. All 71
completions were typed MXSTATUS_PROXY[], so they stay raw permanently.
(#211 §3.3)
- Subtag-mode Clear is drivable: the TestAlarm UDAs are MxSecurityOperate,
so AuthenticateUser + WriteSecured toggles them. The new self-driven
LiveSubtag_SelfDrivenRaiseThenClear_SynthesizesClear fact asserts the
UnackAlm -> UnackRtn Clear (mutation-checked). (#210 §3.7)
- The B8 buffered "NoData bootstrap" is a ConfigurationError resolution
failure (detail 6 / 1002). The DEV Galaxy has no buffered attribute, so a
multi-sample capture needs IDE work. provider_switches was not driven. (#210)
- Status-row catalogue for lmxopcua#872: no Warning row seen, and Pending
only on the synchronous Suspend/Activate return.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run
MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the
pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously.
- Image names come from the installed exes (NSSM Application, Worker env path) plus
both known schemes; processes are also matched by exact path (Win32_Process).
- One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at
the Application path, as the service host's child, is an abort.
- STOP_PENDING kill is scoped to this install's processes.
- Staged checks use the installed exe names; staged Worker must be x86; a staged
self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade).
- Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the
live copy; other live-only non-binary top-level files are carried; hash-verified.
- Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never
values) after stop, swap, start and the stability window; any change aborts.
Sensitive values are redacted from all output and notes.
- File-based master key (vd03) accepted and backed up; a missing env master key
(windev) is a warning. Backup dir is ACL-restricted before anything is written.
- Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a
mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA.
- -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under
WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable".
- Install-MxGatewayService.ps1 process guard matches both schemes.
- Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions,
passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed).
- Runbook updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
0.3.1 stops the gRPC/HttpClient instrumentation tracing telemetry's own
OTLP export calls (the Serilog log sink's LogsService/Export spans).
The app's own outgoing gRPC/HTTP spans are unchanged.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.Telemetry(.Serilog) 0.1.0 -> 0.3.0. AddZbTelemetry now reads the whole
MxGateway:Telemetry section through the shared ApplyConfiguration: node.role defaults
to "standalone" (one gateway per box), MxGateway:Telemetry:SiteId / :NodeRole /
:ServiceVersion / :ResourceAttributes:* are operator keys, and service.version
defaults to the entry assembly's informational version (the git SHA). With
MxGateway:Telemetry:Exporter=Otlp the Serilog logs go to the same collector too.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Server and x86 Worker from 76a71bc8 on wonder-app-vd03, about 9 s downtime, env block and
pepper prefix unchanged. Write2 KeepsSentTimestamp passes; RuntimeAlert reads
2026-07-28T20:02:46.830Z. QA still pending (#170). Upgrade-script gaps filed as #230.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
MXAccess reads the VT_DATE time argument of Write2, WriteSecured2 and their
bulk forms as box-local time and converts it to UTC itself. The worker passed
the proto Timestamp's UTC clock fields (the COM marshaler ignores
DateTime.Kind), so on vd03 (UTC-5) a Write2 sent at 01:33:00Z was stored at
06:33:00Z.
VariantConverter.ConvertToComTimestamp converts a timestamp_value instant with
TimeZoneInfo.ConvertTimeFromUtc into the worker's zone (TimeZoneInfo.Local by
default, DST included; an injectable zone for tests). The executor uses it for
the time argument only: value conversion is unchanged, and the read direction
(MXAccess reports UTC; ToTimestamp treats it as UTC) is unchanged. The one
residual loss is MXAccess-side: the repeated fall-back hour cannot be told
apart in an offset-less VT_DATE.
Tests: VariantConverterTimestampTests (UTC-5, US-Eastern DST both edges,
UTC+05:30, UTC, fall-back ambiguity, value path unchanged); the four executor
forwarding tests inject a fixed UTC-5 zone; the windev live Write2 smoke now
sends a past whole-second timestamp and asserts the stored one when applied.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
On the DEV gateway (wonder-app-vd03, 2b13564e), Write2 after
AdviseSupervisory applied the value to MESReceiver_025.MoveInBatchID
(0 -> 212212, OnWriteComplete Ok), and the value was restored to 0. After a
plain Advise, the same Write2 got SecurityError/1008, which is exactly what
the windev run saw. So that refusal came from the plain advise. It does not
show that windev has no writable attribute.
- Client.Tests: add the opt-in LiveGatewayWrite2Tests (gated by
LiveGatewayWriteFact: endpoint, key file and write item env vars). It runs
supervisory advise -> Write2 -> readback -> restore. The value test passed
live on vd03. The timestamp-parity test needs a second opt-in; it fails on
a non-UTC host until #228 is fixed.
- IntegrationTests: the windev Write2 smoke now uses AdviseSupervisory and
restores the original value. It has not been re-run on windev.
- GatewayTesting.md: document the env vars. code-reviews IntegrationTests-033:
correct the "no writable attribute" limit.
The run also found that the Write2 timestamp is read as host-local time
(stored +5 h on the UTC-5 box); filed as #228.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Ran the Write2 live smoke on windev against origin/main 04a40f2c (x86 Release
worker, item TestChildObject.TestInt). It passed: Ok, hresult 0, and a matching
OnWriteComplete. MXAccess reported SecurityError/1008 on the completion, which
is the Galaxy refusing an unauthenticated write; plain Write got the same
status and the value did not change. Corrected the test comment that said the
reply meant the write succeeded.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Record of the 2026-11-12 config-only change on wonder-app-vd03: MxGateway__ApiKeyPepper
added to the MxAccessGw AppEnvironmentExtra block (same value, sha256 prefix match),
removed from Server\appsettings.json; authenticated round-trip with a pre-change key,
ScadaBridge MxGateway link unchanged. Runbook and configuration doc now say where the
pepper lives on each wonder box.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A fresh StreamEvents is never replayed, so an initial value published before
the gateway registers the stream's subscriber is lost (lmxopcua#865 —
slow-changing ScanState). Clients had nothing deterministic to wait on.
Contract (additive, proto3):
- StreamEventsRequest.emit_stream_attached = 3 (bool, default false)
- MxEvent.stream_attached = 15 (optional StreamAttached)
- message StreamAttached { uint64 live_after_sequence = 1; }
- OpenSessionReply capability "stream-events-attached-ack"
(GatewayCapabilities.StreamEventsAttachedAck)
When requested, EventStreamService yields the sentinel as the FIRST message,
after AttachEventSubscriber/AttachEventSubscriberWithReplay has returned, so
receipt implies registration; refused attaches never see it. It precedes the
ReplayGap sentinel and replay on a resume. Shape mirrors ReplayGap: family
UNSPECIFIED, no body, worker_sequence 0.
Opt-in rather than always-on: ScadaBridge's MxGateway adapter stores every
streamed event's worker_sequence as its resume cursor, so an unrequested
sentinel would reset it to 0. Default wire behaviour is unchanged.
.NET client: StreamEventsAsync/StreamEventItemsAsync(after, emitStreamAttached),
MxEventStreamItem.IsStreamAttached, MxGatewaySession.SupportsStreamAttachedAck,
and OpenEventStreamAsync -> MxGatewayEventStream (awaits the sentinel, with
attach timeout; refuses older gateways).
Regenerated Contracts/Generated, the client protoset, Go and Python bindings
(also picks up comment-only drift already on main), synced the Rust vendored
proto and set emit_stream_attached in the Rust StreamEventsRequest literals.
No package versions bumped.
Tests: ordering (sentinel first, then events), race proof (event fanned right
after the sentinel is delivered — mutation-checked: fails when the sentinel is
yielded before registration), resume-with-gap order, refused attach emits no
sentinel, default emits none, contract field numbers, capability, and .NET
client wire tests (await, no-capability, older gateway, timeout).
QueryActiveAlarms / alarm transitions reported every wnwrap alarm late by
the record's GMTOFFSET-DSTADJUST: ParseTransitionTimestampUtc treated the
XML DATE+TIME as producer-local and added the offset, but the fields are
already UTC. DEV evidence: RuntimeAlert raise EventTimeUtc 2026-07-28
20:02:46.830 (EventTimeUTCOffsetMins -300) surfaced as 2026-07-29
01:02:46.830Z. The 2026-05-01 probe capture (TIME 13:26:14.709,
GMTOFFSET 240) was committed at 09:44 EDT, so its TIME is UTC too.
GMTOFFSET/DSTADJUST are no longer read. Tests updated + #226 regressions.
New optional MxGateway:Ldap:ServerCertificateThumbprintSha256 (hex SHA-256
of the DER cert; case/colon/dash/space-insensitive; comma/semicolon-separated
string or array, so a rotation can carry old + new). When set, the dashboard's
shared LDAP provider gets a RemoteCertificateValidationCallback via
LdapOptions.ServerCertificateValidationCallback (Auth 0.4.1) that accepts the
server certificate iff its thumbprint matches a pin, ignoring chain/name
errors only then. Unset keeps default OS trust-store validation untouched.
The callback is an init-only, code-only property, so the ILdapAuthService
registration from AddZbLdapAuth is replaced with one built from the bound
options plus the callback. Pins are read straight from configuration because
the binder silently binds the scalar form onto a list as empty.
GatewayOptionsValidator rejects a pin that is not 64 hex chars after
normalization and any pin with Transport=None; the Production hard stops are
unchanged. Pins are shown on the Settings page. Docs: GatewayConfiguration.md,
glauth.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Opt-in [LiveMxAccessFact] mirroring the Write round-trip: Register/AddItem/
Advise, then Write2 with a VT_DATE timestamp value; asserts Ok + Write2 kind
and a matching OnWriteComplete. Skips without the live MXAccess env.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ShutdownGracefully now Unregisters every server handle first (MXAccess drops its
items and advise subscriptions with it). Only for a server whose Unregister fails
does the UnAdvise -> RemoveItem fallback run. Each step is guarded, logged at Debug,
and the method is idempotent. #176 (AuthenticateUser cache) is unaffected.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Auth 0.4.1 (BindMode stays ServiceAccount; empty password now refused in the library too; Username canonicalised from UserNameAttribute). No API breaks, no test changes needed.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Add scripts/deploy/Upgrade-MxGatewayService.ps1 (preflight, stop with STOP_PENDING handling,
backup, Server+Worker swap, start, /health/ready + SHA verify, auto/manual rollback,
one-gateway guard, never touches the NSSM env block) and Install-MxGatewayService.ps1
(service install, restart/recovery settings, env block). Both support -DryRun/-WhatIf.
Dry-run validated on macOS pwsh against a fake layout; untested on a real box.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
CanonicalForwardingApiKeyAuditStore now derives CorrelationId from the ambient
request: Activity.Current trace id, else the HTTP request TraceIdentifier
(GUID used verbatim, otherwise hashed to a stable GUID) via an optional
IHttpContextAccessor. The CLI path has neither and stays null. Auth library
unchanged (option b).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Event-queue overflow, alarm-feed overflow, subscriber cap, session limit and
oversized commands all surfaced as ResourceExhausted. The status code is
unchanged; the gateway now attaches a machine-readable `mxgw-reason` trailer
(constants in Contracts ResourceExhaustedReasons). The .NET client maps
ResourceExhausted to MxGatewayResourceExhaustedException (an
MxGatewayException subclass; additive) exposing Reason/IsOverflow/
IsSubscriberLimit. Documented in docs/ClientBehaviorFixtures.md; the other
clients get a doc note only.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):
- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
classes migrating the one per-process secrets store. Tests now build the gateway through
TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.
Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The MxGateway.Tests part of #223 (the Worker.Tests items are handled separately).
- DashboardSnapshotFeed gains two internal test seams: WhenIdleGateClosedAsync, a
TCS-backed signal that completes once no pump generation is accepting subscribers,
and IsPumpIdle (no live generation and the latest pump has exited). The TCS is created
lazily by a waiter, so production never allocates it.
- WatchAsync_WhenAJoinerLeavesWhileAnOldGenerationLingers_LeavesNoPumpRunning drops both
150 ms real sleeps: it waits on the idle-gate signal for the joiner's unsubscribe, and
asserts IsPumpIdle before the orphan-push check, which is then immediate. Reverting the
per-generation idle check to the old global "list is empty" check now fails the test
(5 s TimeoutException on the gate), where before the sleep simply elapsed.
- Eventually.WaitUntilAsync/WaitForAsync take an optional description (defaulting to the
condition's source text via CallerArgumentExpression) for the timeout message, and gain an
async-condition overload. The hand-rolled polling loops in EventStreamServiceTests,
GatewaySessionDashboardMirrorTests, DashboardEventBroadcasterTests and
AlarmsPageTruncationBannerTests now delegate to it, keeping their timeouts and messages.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- New MxGateway:Worker:WorkerHeartbeatGraceSeconds (default 15), exported as
MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS and read by WorkerPipeSessionOptions.FromEnvironment
(a grace the ceiling does not exceed falls back to the default so the worker
still starts). Validation now checks WorkerHeartbeatStuckCeilingSeconds against
the configured grace; the copied WorkerWatchdogGraceSeconds constant is gone.
- EffectiveWorkerConfiguration and the dashboard Settings page now show the worker
heartbeat send interval, watchdog grace and stuck ceiling (render test added).
- Docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV