Commit Graph

1146 Commits

Author SHA1 Message Date
Joseph Doherty 1a08b4f746 fix(galaxy): #872 detail-0 failure rows route on category, not detected_by
Every raw detected_by value names an MXAccess layer (requesting/responding
LMX, NMX, automation object); the gateway's own failures arrive as
ProtocolStatus. The 'non-zero source = transport failure' rule made a failure
the responding object detected a comms error. Item 1 of #872 (Warning/Pending
categories) stays open pending a captured row.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:15:05 -05:00
Joseph Doherty 970f2159ad docs(galaxy): IsSuccess checks the status category only
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:14:22 -05:00
Joseph Doherty 875b82175d fix(galaxy): #866 grade OK-status data changes by the item's OPC DA quality
The gateway fills Statuses on every data change (live OnDataChange, value-cache
replay and session snapshot all carry Quality too), and EventPump used the status
row whenever one was present, so Quality was never read: an off-scan, Bad or
Uncertain attribute with an OK status row was published Good. A failure status
row still wins; otherwise the quality byte grades the value (legacy parity:
quality >= 192 Good).

Live, read-only on VD03 (DEV): healthy ScanState and _TestArea attributes stay
Good (0x00000000) via quality 192; the stopped JoePlatform/JoeAppEngine ScanState
now publish BadCommunicationError (0x80050000) instead of generic Bad, with no
"unrecognised quality byte" warning.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:14:11 -05:00
Joseph Doherty 71b3b7f35f fix(galaxy): #866 map MxStatus failure details by the MxStatusDetail vocabulary, not as OPC DA quality
The gateway copies MXSTATUS_PROXY.detail verbatim, so a failure row's detail is
always an MxStatusDetail code. FromMxStatus routed any detail <= 0xFF through the
OPC DA quality table: MX_E_PlatformCommunicationError (2) on a stopped platform's
ScanState logged "Unrecognised MXAccess quality byte 0x02" and published generic
Bad (VD03 live capture), MX_E_InvalidObjectId (8) became BadNotConnected,
MX_E_Overflow (20) BadCommunicationError, and a detail in the Uncertain band
published Uncertain for a failure.

Every non-zero detail now maps per the gateway.md vocabulary (timeout,
communication, configuration, type, range, access, initialising), then by
category; the category-fallback warning logs once per (detail, category).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:06:26 -05:00
Joseph Doherty eba016096c Merge branch 'master' into worktree-agent-a720278a57ae41792
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:26:05 -05:00
Joseph Doherty b5d4e25905 Merge #864/#865 Galaxy host status: per-template platform seed, event stream started before probe registration; registry and reconnect races fixed
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:22:33 -05:00
Joseph Doherty 59d33ab938 test(galaxy): #865 review round 7 - pin the superseded-pump fault filter
Adds an escalated-stream-fault test seam and a test: the live pump's fault
escalates (positive controls before and after), a superseded pump's late
fault after a reopen does not; the absence is asserted only once both
pumps' callbacks are known to have run (HasEnded). Verified to fail with
the filter disabled. The filter's comment now says the identity check does
the work and warns about a future session bump while Healthy.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:20:49 -05:00
Joseph Doherty 13cc7b8353 fix(galaxy): #865 review round 6 - ignore stream faults from a superseded pump
Each EventPump's fault callback now carries the pump's identity; a fault
from a pump that is no longer the live one (or belongs to an older registry
session) is logged at Debug and not escalated, so a stale pump's late fault
cannot fail a good replay now that a failure during Replaying fails the
attempt. The replay-retry warning includes the fault.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:16:01 -05:00
Joseph Doherty d80bdf6369 fix(galaxy): #865 review round 5 - a failure during the replay fails the attempt
ReconnectSupervisor swallowed a transport failure reported while Replaying,
so a restarted event stream refused/ended a round trip after the replay
began still let the replay go Healthy over a dead stream (and the next
advise's re-report restarted at attempt 1 with no backoff). A failure
reported during Replaying is now remembered and turns that attempt's
success into a retry with backoff. ReplayAsync's synchronous dead-pump
throw yields to a teardown's cancellation.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:12:33 -05:00
Joseph Doherty 8653960ea3 fix(galaxy): #865 review round 4 - replay fails on a dead stream, backoff-aware wait
- ReplayAsync fails when its restarted pump's stream has already ended, so
  the supervisor backs off and stays Degraded instead of going Healthy over
  a dead stream (whose next advise re-report restarted at attempt 1 with no
  backoff - a tight reconnect loop).
- ReconnectSupervisor.AttemptInProgress (reopen through replay); advises
  wait for a stale pump's replacement only while an attempt is running, not
  for a whole backoff (the state still reads Reopening/Replaying there).
- ReconnectSupervisor: _disposed is set under the state lock and re-checked
  inside ReportTransportFailure, so a late report cannot start a recovery
  loop that StopAsync already missed.
- EventPump.HasEnded is a flag set before StreamRequested is released, and
  a stream that ends/faults on its first MoveNext is marked ended first.
- SubscriptionRegistry fast path: a miss now falls through to the locked
  path (lookup + session + window check atomically) instead of a lock-free
  re-lookup.
- Tests: replay over an immediately-ending stream fails; AttemptInProgress
  across attempt/backoff; ReportTransportFailure after stop; the
  reopen-window test uses a generous wait bound.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:07:26 -05:00
Joseph Doherty 4e492e6550 fix(galaxy): confirm unconfirmed native alarm acks against Galaxy's acknowledged transition (#869)
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.

- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
  and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
  and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
  (default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
  transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
  after AcknowledgeAsync, off its mailbox and outside the resilience
  pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:03:04 -05:00
Joseph Doherty 1c1705fa77 fix(galaxy): #865 review round 3 - budget-safe stream wait, dead pump, fast-path drop
- StreamRequestWait 1 s (under Tier-A Read 2 s / Subscribe 5 s budgets) and
  a stale pump is only waited on while the supervisor is reopening or
  replaying (or a restart is underway); in a reconnect backoff the advise
  proceeds at once. The no-stream log is a rate-limited Warning while
  Healthy, Debug otherwise.
- A current-session pump whose loop has ended is not a stream: if the
  supervisor already reports Healthy (its fault was swallowed while it was
  recovering) the failure is re-reported to start a new cycle.
- Subscribe/Read no longer start the pump directly (only
  AwaitCurrentEventStreamAsync does, honouring the replay restart flag);
  the teardown refusal is an explicit ThrowIfRuntimeStopping.
- SubscriptionRegistry.ResolveOrPark fast path: the handle lookup and the
  open-window check were not atomic, so a subscribe that registered the
  handle and closed its window in between got its initial value dropped as
  stale (found by a stress run: the first event of a bulk advise lost). It
  re-looks-up before declaring stale. Deterministic test via a seam.
- Tests: first Read on a fresh driver, replay of a data subscription.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:00:19 -05:00
Joseph Doherty 9060ff5561 fix(galaxy): #865 review round 2 - wait for the replay's pump, never race a second stream
Round 1 replaced a stale-session pump from any advise path and disposed the
old one in the background; that could start a second stream while the old
one still held the gateway's single-subscriber slot, and a kept
current-session pump could be dead with no fault re-reported.

- EnsureEventPumpStarted reuses again; only ReplayAsync restarts the pump.
  RestartEventPumpForReplay keeps a current-session pump only while its loop
  is running (EventPump.HasEnded), and flags the restart so no advise path
  starts a competing pump in the dispose gap.
- AwaitCurrentEventStreamAsync (bounded by StreamRequestWait) starts a pump
  only when none exists, WAITS for the replay while the pump is stale or
  restarting, waits for StreamRequested, and returns the session it verified.
  SubscribeAsync (per #861 attempt), read-via-subscribe and the probe hook
  issue and fence the advise under that session, so a reconnect after the
  check is caught as a straddle and retried.
- Read: a deadline (not caller) cancel during the wait falls through to
  per-ref results.
- Tests: reopen-window probe test now asserts the advise waits for the
  replay's single new stream; first Subscribe/Read wait for the stream; the
  fencing tests that advise in the window use a short wait bound; the
  production-path seed test uses an empty galaxy (a real probe advise now
  starts a stream against the fake endpoint).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:46:15 -05:00
Joseph Doherty cc1a7ec687 fix(galaxy): #865 review round 1 - stale-session pump, data-path stream wait
- EnsureEventPumpStarted replaces a pump created for an older registry
  session (the reopen -> replay window still held the dead session's pump,
  so an advise there lost its initial value and the replay skipped it);
  RestartEventPumpForReplay keeps a pump already on the current session.
- SubscribeAsync, read-via-subscribe and ReplayAsync also await the
  bounded StreamRequested wait before advising, not just the probes.
- TryEnsureEventPumpStarted tolerates a teardown flipping _runtimeStopping
  mid-check; the probe hook honours an already-cancelled seed token.
- StreamRequested is documented as narrowing, not closing, the attach race.
- Tests: reopen-window probe test, EventPump.StreamRequested tests, and the
  FixRound1 fresh-pump assertion now waits on a positive control.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:31:51 -05:00
Joseph Doherty cbd24d7a01 Merge #867 Heidenhain integer leaves keep their declared type; #868 ConfigPublishCoordinator survives ConfigDb outage at start
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:29:00 -05:00
Joseph Doherty e35f2a4e93 fix(galaxy): probe advises start the event stream first so initial ScanState arrives (#865)
The gateway's StreamEvents with after_worker_sequence = 0 is a fresh stream
and never replays, and the event pump started only on the first
Read/Subscribe, so platform ScanState probes advised before then lost their
initial value (and a driver with no authored tags never got any host status).

PerPlatformProbeWatcher now awaits an ensure-event-stream hook before every
probe SubscribeBulk (seed, discover sync, reconnect ResubscribeStaleAsync).
The driver's hook starts the pump (refused while a teardown is in progress)
and waits, bounded, on the new EventPump.StreamRequested signal - completed
once the stream loop's first MoveNext has issued the StreamEvents RPC, or
when the loop ends - so the advise never precedes the stream request.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:24:08 -05:00
Joseph Doherty c4bc1ef309 fix(galaxy): seed platforms with one template filter per request (#864)
The gateway's shared GalaxyHierarchyProjector ANDs template_chain_contains
entries, so the single request carrying both $WinPlatform and $AppEngine
matched no object and the probe watcher always seeded 0 platforms.
GatewayGalaxyPlatformSource now runs one paged sweep per template and
unions/dedupes by tag name. The tests use a fake gateway that applies the
projector's AND semantics (they fail on the old single request).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:24:03 -05:00
Joseph Doherty 8e97fc94fd fix(heidenhain): Tool/Number, Tool/Index, Tool/AxisCode publish their declared integer types (#867)
HeidenhainSession.Leaf's `case Lsv2Tool t` switch expression mixed long, int and
double? arms, so C# unified the natural type to double? and Tool/Number (Int64),
Tool/Index and Tool/AxisCode (Int32) were published as Double. Every arm in that
switch (and in the alarm-list, axes and feature-acquire switches in the same file)
is now boxed to object explicitly so no numeric arm can widen another.

New HeidenhainLeafRuntimeTypeTests asserts value.GetType() for every catalog value
leaf against its declared DataType (scalar + array), plus PLC/DATA binding coercion
for Int16/Int32/Int64/Boolean. Verified red on the old code (exactly the three Tool
leaves) and green with the fix.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:20:33 -05:00
Joseph Doherty 233e4bab25 Merge #863 TwinCAT Test Connect: drop the AmsNetId-as-IP TCP pre-check; probe via the local AMS router ADS handshake
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:14:58 -05:00
Joseph Doherty fdc9473819 fix(twincat): Test Connect probe no longer dials the AmsNetId as an IP (#863)
TwinCATDriverProbe ran a first-phase TCP check against the AmsNetId's
first four octets on the AMS port (801/851). An AmsNetId is not an IP and
the AMS port is not a TCP port, so Test Connect timed out against every
real runtime (Z35474: dialled 5.118.56.93, an unrelated public host).

The probe now opens no socket of its own: it takes the driver's path,
AdsClient.Connect + ReadStateAsync through the local AMS router, bounded
by the probe budget (Task.Run + WaitAsync), and classifies failures as
no local AMS router / no route or target unreachable / timeout / other.
The degrade-to-green 'TCP reachability only' outcome is gone - with no
TCP check nothing was contacted, so a missing router is reported red.

Tests: the regression test listens on the AmsNetId-derived endpoint and
asserts no connection arrives (fails on the old probe); classification
and budget tests drive the handshake through an internal seam.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:13:55 -05:00
Joseph Doherty bdbc47f5fc fix(mtconnect): serve an empty string-typed value as Good "" (#824)
Owner decision 2026-11-11 (Java parity): a genuinely empty observation on a
String/Reference or unauthored item is Good "" instead of BadNoCommunication.
Non-string items (numeric, bool, DateTime, arrays) keep BadNoCommunication -
the Agent supplied no value; BadTypeMismatch would wrongly blame the authored
type. The UNAVAILABLE sentinel is unchanged, and a structured observation with
no text stays BadNotSupported (structured check now precedes the empty check).
Parity matrix, quality-mapping section and drivers rule updated.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:56:30 -05:00
Joseph Doherty 33e10c2266 fix(galaxy): stranded subscriptions after a no-replay reconnect go Bad + Degraded (#862)
With Reconnect.ReplayOnSessionLost=false a recreated gateway session carries none of
the old advises, so every pre-reconnect subscription went silently deaf: nodes kept
their last value as Good and health returned to Healthy (only a Warning log).

ReplayAsync now records the stranded subscriptions and publishes BadNoCommunication
(StatusCodeMap.BadNoCommunication, 0x80310000; value null — no driver value cache)
for every tag of each, straight to OnDataChange so LastSuccessfulRead is not stamped.
GetHealth reports Degraded naming the setting while any stranded subscription is
still registered; it clears on UnsubscribeAsync (the host's re-subscribe path) or
runtime teardown, and pruned ids never pin it. A subscribe after the reconnect is an
ordinary new-session subscription. With nothing else flowing, the comms-loss
watchdog re-initializes the driver after 90 s (a delayed replay).

Tests: Runtime/StrandedSubscriptionTests (6). Docs: drivers/Galaxy.md, which also
no longer claims the replay uses the gateway's ReplaySubscriptions RPC.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:13:19 -05:00
Joseph Doherty a8d3790b09 fix(galaxy): #859 final review — probe replay per platform session, teardown session bump
BLOCKER: the reconnect replay chose work from the registry entries' sessions,
but ONE registry entry holds every probe handle and any probe publish on the
new session (a seed in the reopen→replay gap, or the straddle retry) re-stamps
it as current — the replay then skipped the probe watcher and the old
platforms stayed bound to handle 0 (host status frozen); a publish after the
snapshot made it re-subscribe everything (double advise, leaked handle).
ReplayAsync now excludes the probe entry from its snapshot and calls the new
PerPlatformProbeWatcher.ResubscribeStaleAsync on every reconnect, which
re-subscribes exactly the platforms whose ProbeHandle.Session predates the
current session. The replay-gap path keeps ResubscribeAsync (all platforms).

Also: ClearRegistry (teardown) starts a new registry session, so a late
PurgeParked from a teardown-orphaned call cannot reach the next runtime's
parked values; the read path's unused pump local is gone (call kept).

Tests go through the driver's reopen/replay: the probe old-session-number test
is rewritten that way, plus the seed-in-the-gap scenario (only P and Q are
re-subscribed, R keeps its new-session handle, all three resolve). Both checked
red with the probe replay removed / re-subscribing all platforms.

Refs #859, #861.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:07:59 -05:00
Joseph Doherty 133bc07da5 fix(galaxy): #859/#861 third review — read rebind release, probe session fence, reopen fence, pump restart
B-1: the subscribe-once read's cleanup unsubscribes what the registry holds at
removal (the replay may have rebound it to new-session handles), not its
original bindings; nothing when the entry still holds dead old-session
handles. Found while testing it: reads were attached to the pump INSTANCE, so
a read across a reconnect's pump restart never saw the replayed value — reads
now listen on a driver-level ReadDataChange raised for whichever pump is live.

I-A: PerPlatformProbeWatcher records the registry session with each probe
handle. Publishing goes through the new fenced TryUpsert and binds only
current-session handles; removal/stop unsubscribe and purge only those; a
seed reply that straddled a reconnect records nothing and is re-issued (max 3).

I-B: PurgeParked takes the session captured BEFORE the unsubscribe RPC and is
a no-op under the park gate if the session changed.

I-C: ReopenAsync starts a new registry session before AND after the gateway
session swap; ReplayAsync does so only when no reopen ran (injected subscriber
/ tests) and replays only subscriptions still holding older-session handles,
so a subscribe issued after the swap is live and not double-advised.

P-1: ReplayAsync restarts the pump before any early return (no entries used to
leave the dead pump in place). P-2: ReplayOnSessionLost=false logs a Warning;
misleading comment about gateway-side replay fixed.

M-a: TryRebind returns BindOutcome; UnknownId releases the handles just
advised for a subscription removed mid-replay. M-b: the replay/re-snapshot
pass reads the session once and stops if it changes. M-c: parked entries drop
their MxEvent when they leave the parked set. M-d (retry a straddled read)
not done.

Tests (TestWait) for each, each checked red against a mutation of its fix.

Refs #859, #861.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:02:55 -05:00
Joseph Doherty d08e900203 fix(galaxy): #859 second review + #861 — window-sequence trim, per-session handles, straddling subscribes
I-2: unclaimed parked events are trimmed by window sequence. Each window gets a
monotonically increasing sequence at Begin, each parked event is stamped with
the newest one, and on every End events stamped before the oldest open
window's sequence are dropped (amortised O(1) via a park-order queue). Under
concurrent reads the open count rarely hit 0, so trailing events used to pile
up to the bound and push genuine early values into overflow.

I-1 / #861: SubscriptionRegistry.StartNewSession() — called FIRST in
ReplayAsync on every reconnect — clears held events, invalidates the reverse
map (subscriptions kept) and bumps a session number. New-session events whose
number an old binding still held now park until the replay's per-subscription
Rebind promotes them, instead of resolving to the wrong tag. TryRegister /
TryRebind refuse a stale session atomically under the park gate: a
SubscribeAsync whose reply straddled a reconnect is re-issued on the new
session (max 3 attempts), a subscribe-once read returns BadCommunicationError,
ResubscribeAllAsync stops if the session changes again. Old-session handle
numbers are never unsubscribed (Remove reports whether the entry belongs to
the current session; the read's late-reply path checks too), and an old
session's pump can no longer resolve through the map (fast path checks the
session).

I-3 (interim): a completed UnsubscribeBulk purges events parked for its
handles (driver + probe watcher). Full fix needs a worker sequence on the
SubscribeBulk reply (mxaccessgw, optional) — documented.

M-1: a resolved handle with a stranded parked list promotes it and resolves
the live event (never parks it). M-2: ResolveOrPark returns the ready events
to replay first, one gate acquisition per event while anything is held.
M-3: SubscribeAsync/read dispose the window before the orphan-release RPC.
M-5: ResolveOrPark remarks document the single-dispatching-pump invariant.

Tests (TestWait): overlapping windows don't accumulate, purge, stranded list,
session fence + map invalidation, stale-session registration refused,
end-to-end replay with swapped handle numbers and event-before-reply,
subscribe and read straddling a reconnect, read deadline closes its window.
Each was checked red against a mutation of the fix it pins.

Refs #859, #861.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 10:41:16 -05:00
Joseph Doherty f89e0cc04c fix(galaxy): #859 review — read path window, atomic promote, session epoch, per-call windows
1. ReadAsync (subscribe-once) now opens the early-event window around its
   SubscribeBulk + registration; an initial value that overtook the reply was
   dropped and the read returned BadTimeout.
2. Register/Rebind write the reverse map and promote parked events in ONE
   critical section under the park gate, and ResolveOrPark resolves under that
   gate whenever anything is held (lock-free fast path only when nothing is),
   so a later live event can no longer overtake the parked initial value. A
   resolved handle that still holds a parked list queues behind it.
3. Held events are fenced by a session epoch: each new EventPump calls
   StartNewEpoch(), discarding parked AND promoted-but-unreplayed events; an
   older pump can neither park nor replay. Item-handle reuse across a reconnect
   can no longer put an old-session value on a new tag.
4. ResubscribeAllAsync opens one window per subscription, not across the whole
   replay.
5. Promoted-but-unreplayed (ready) events count toward MaxParkedEvents.
6. Begin/End of a window run under the park gate, so N->0 discard and 0->1
   open are atomic (no inherited leftovers).
Documented that ParkedEventsReady may run under _registryGate/_syncLock and
relies on the channel's AllowSynchronousContinuations=false.

Tests use TestWait (#858); each new test was checked red against a mutation
of the fix it pins.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 10:26:28 -05:00
Joseph Doherty 1a31e3f584 fix(galaxy): park data-change events that overtake their SubscribeBulk reply (#859)
mxaccessgw gives no reply-before-event ordering: the worker completes the STA
command via a RunContinuationsAsynchronously TCS, so the reply is framed on a
thread-pool continuation while the STA already pumps the OnDataChange Advise
queued (drained/framed by a separate loop), and the reply (unary Invoke) and
the event (StreamEvents) are separate HTTP/2 streams. The driver then also
registers handles concurrently with the pump's dispatch loop. MXAccess's
initial value could therefore be dropped as "stale after unsubscribe",
leaving the node at its pre-subscribe value until the next change.

SubscriptionRegistry gains an early-event window: every SubscribeBulk call
site (SubscribeAsync, ResubscribeAllAsync, PerPlatformProbeWatcher) opens
BeginPendingSubscribe() before the RPC and closes it after registering.
Inside the window unknown-handle events are parked (bounded, 100k); the
Register/Rebind that binds a handle promotes them, and the EventPump replays
them on its single dispatch thread before any later event for that handle
(woken by a sentinel when no further event arrives). Closing the last window
discards unclaimed events, so the stale drop is unchanged outside it.
New counters galaxy.events.early_{parked,replayed,overflow}.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 10:19:24 -05:00
Joseph Doherty 988046cbe4 fix(galaxy): WatchedPlatforms cannot show a platform before its probe binding is routable (#856)
PerPlatformProbeWatcher records a platform's handle and publishes the probe
binding into the driver registry as one step under _syncLock, but
WatchedPlatforms read the handle map lock-free - so it could report a
platform one step ahead of the registry Upsert. A ScanState event raised in
that window was dispatched by the EventPump, missed ResolveSubscribers and
was dropped as "stale after unsubscribe", leaving the host status empty.

That is the #856 flake: A_fetch_completing_after_init_populates_host_statuses
waited for WatchedPlatforms, emitted ScanState, and timed out at 10 s on the
final Running wait. Reproduced 11/700 (28 parallel runners x 25 + 14 CPU
burners), every failure at that step with the binding present by the time of
diagnosis and hostStatuses=[] - i.e. the event was dropped. 0/1400 after.

WatchedPlatforms now reads under _syncLock (test seam only; the dispatch path
already checks the map after the registry, the safe order). New
PerPlatformProbeWatcherVisibilityTests spins a reader on another thread and
asserts the registry entry exists the moment the platform is visible
(old code: torn in 1691/2000 iterations). WaitUntilAsync gains a describe
overload; the #856 test's waits now report elapsed time and the observed
seed/binding/pump/host-status chain on timeout.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 10:00:44 -05:00
Joseph Doherty 340d9d26b7 feat(mtconnect): CommsLossConfirmMs AdminUI field, refused-connection fast path reaches native conditions, dedicated live-gate Agent (#852)
- AdminUI MTConnect driver form: "Comms-loss confirm (ms)" in the Reconnect section, always written
  (0 = first refusal, negative = off is never dropped), case-insensitive load, unknown keys preserved,
  advisory at >= 90000 ms (the watchdog fires first). Form output byte-pinned and parsed through
  MTConnectDriver.ParseOptions on the driver side.
- Driver: a confirmed refused-connection outage now annotates every told CONDITION quality-only
  BadNoCommunication (state/ack kept, once per outage, UNAVAILABLE-Bad skipped); the first delivery
  afterwards re-diffs all subscribed conditions so unchanged ones get a quality-only Good.
- docker-dev/docker-compose.mtconnect-gate.yml: throwaway Agent in its adapter's network namespace so a
  stop is a genuine ConnectionRefused (a plain stopped container is a DNS failure = ambiguous).
- Docs: MTConnect.md comms-loss section + live-gate recipe, docker-dev README, drivers.md rule.

Refs #852
2026-11-11 06:23:18 -05:00
Joseph Doherty fc3a39b1ca fix(mtconnect): name the comms-loss status constant so the status-code parity guard can see it (#846) 2026-11-11 04:13:55 -05:00
Joseph Doherty 271b96a644 fix(security): #835/#836 review — unsaved endpoints never borrow the owner's secrets; audit credential use; path rule follows AuthType
I-1: Test Connect / browse configs may carry unsaved edits (the device modal merges
the stored DriverConfig with the typed device config). The server
(DriverInstanceIdentities.EffectiveOwnerAsync, used by AdminOperationsActor and
BrowserSessionService) honours the named owner only when every endpoint value
(new DriverCredentialRules.EndpointFields: Galaxy Endpoint, OpcUaClient
EndpointUrl/EndpointUrls/DiscoveryUrl, Heidenhain Host/Port) is one that driver
stores; otherwise the owner is null and its instance-named secrets are refused.
Every Test Connect / browse whose config carries an in-scope secret:/env: reference
is recorded (CredentialUseAudit: IAuditWriter event category CredentialUse + a
structured Information line: user, driver, endpoint hosts, outcome).

L-1: the UserCertificatePath rule is skipped only in an object that explicitly sets
AuthType to non-Certificate (string or ordinal); the deploy gate also checks the
runtime-merged driver+device config so the switch cannot be split from the path; the
form no longer persists the hidden field.
L-2: a ConfigDb fault in the browse check is a refusal; RawBrowseModal catches open
failures into its error message.
L-4: a relative UserCertificateDirectory resolves against AppContext.BaseDirectory.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 04:12:40 -05:00
Joseph Doherty 87416a230f Merge #835 #836 per-instance credential references, confined user-certificate path 2026-11-11 03:53:12 -05:00
Joseph Doherty 63331b0416 fix(security): per-instance credential references (#835) and confined OpcUaClient user-certificate path (#836)
#835: inside one driver namespace, a secret:<ns>/<X>/... reference whose first
segment names ANOTHER existing same-type driver instance (id or name,
case-insensitive; another-instance match beats an own-name match, so renaming
one's driver after a victim's unlocks nothing) is refused at save
(DriverConfigSaveGate.ValidateAsync), deploy (CredentialReferenceOtherInstance,
Heidenhain included), Test Connect (TestDriverConnect.DriverInstanceId, ConfigDb
read off the mailbox) and live browse (BrowserSessionService, fail-closed without
a ConfigDb). One rule: DriverCredentialPolicy.DescribeOtherInstanceProblems +
DriverInstanceIdentities. Shared names and the driver's own id stay valid; env:
is not instance-scoped. Running drivers are unchanged (warn-only window: the
deploy gate and the upgrade preflight / /raw banner list them).

#836: OpcUaClient UserCertificatePath is a relative path inside
Drivers:OpcUaClient:UserCertificateDirectory (default
<LocalAppData>/OtOpcUa/pki/user-certificates). Syntactic rule (CredentialFilePaths,
a DriverCredentialPolicy path field) at every #815 seam; at load time
ResolveSecretRefsAsync joins, canonicalises and requires lexical AND
link-resolved containment before the PFX password is resolved, and
LoadCertificate refuses an unresolved path. A running driver's pre-#836 absolute
path keeps loading with a rate-limited warning (LegacyCredentialReferenceTracker.ReportPath).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 03:52:48 -05:00
Joseph Doherty 19ffb3c902 feat(mtconnect): refused Agent connections mark values BadNoCommunication after a short confirmation; clamp the raised sample count to the Agent's bufferSize (#846)
- A /sample or re-baseline failure whose chain holds SocketError.ConnectionRefused, unbroken for
  CommsLossConfirmMs (new, default 2000; 0 = first refusal; negative = off), publishes every subscribed
  value BadNoCommunication once per outage; the first delivery afterwards republishes all of them from
  the index (unchanged values included). Timeouts/resets/DNS/pause keep the #527 watchdog contract.
- SampleCountCeiling clamps the adaptive raise to the /probe bufferSize (never below SampleCount).
- Docs: MTConnect.md comms-loss section + config key; drivers.md rule.

Refs #846
2026-11-11 03:48:05 -05:00
Joseph Doherty ddc1f29e3a feat(heidenhain): Part 9 condition class from alarm class/group, ALARM: selectors in the browse picker, unclaimed alarms counted + logged (#839)
- Core: optional AlarmEventArgs.ConditionClass (AlarmConditionClass Base/Process/Maintenance/System/Safety)
  carried by NativeAlarmProjector on AlarmConditionSnapshot and written to ConditionClassId/Name by
  OtOpcUaNodeManager (kept across class-less events, carried across a #641 rebuild).
- Heidenhain: HeidenhainAlarmClasses — HEIDENHAIN DNC-interface class/group names (corroborated by the
  captured iTNC 530 record); class 5 => Safety, group 1/2 => Process, 4/5 => System, else Base.
- DiscoverAsync offers AlarmConditions/ selector leaves (IsAlarm Boolean); browse-commit seeds the alarm object.
- Alarms no selector claims: otopcua.driver.alarms.unmatched counter, /hosts diagnostics, rate-limited Warning.

Refs #839
2026-11-11 03:41:14 -05:00
Joseph Doherty 61b9adeb9c Merge branch 'master' into fix/832-docker-dev-batch 2026-11-11 02:32:42 -05:00
Joseph Doherty e9064dbdf4 Merge fix/815-credential-ref-scoping
# Conflicts:
#	.claude/rules/drivers.md
#	docs/drivers/Heidenhain.md
2026-11-11 02:08:24 -05:00
Joseph Doherty 4563940792 fix(security): warn-only compatibility for deployed drivers + upgrade preflight (#815)
User decision: save, Test Connect, live browse and the deploy gate keep
refusing out-of-namespace credential references, but an ALREADY-DEPLOYED
driver must not fault at the upgrade restart or (Galaxy) at the next
gateway reconnect, since ReopenAsync re-resolves Gateway.ApiKeySecretRef.

- LegacyCredentialReferenceTracker (Core.Abstractions): the one, deprecated,
  call-site-chosen compatibility path. Running Galaxy (InitializeAsync,
  ReopenAsync, ack fallback) and OpcUaClient (session open) pass it and keep
  the pre-#815 semantics (case-insensitive prefixes, Galaxy file:, OpcUaClient
  literal pass-through) with a rate-limited Warning (driver/field/allowed
  form, never the value) and a credential-compat:<field> diagnostics entry
  (both drivers now implement IDriverDiagnostics). Browsers and probes pass
  none and stay hard-fail; Heidenhain has none.
- Deploy-gate message names the migration.
- Upgrade preflight: CredentialReferencePreflightService (admin nodes, once
  after ApplicationStarted, ConfigDb only, resolves nothing) logs a Warning
  per Draft row / Deployed-artifact finding; the /raw banner shows them plus
  Running drivers reporting warn-only use (driver-health diagnostics).
- docs/security.md: exact replacement table (incl. the old form default
  env:MX_API_KEY, file:, un-namespaced secret:, casing changes), the
  ReopenAsync reconnect behaviour, the warn-only window and its removal.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 02:07:33 -05:00
Joseph Doherty d3285af6b3 Merge feat/830-heidenhain-followups 2026-11-11 01:55:10 -05:00
Joseph Doherty 598c668108 Merge fix/816-opcuaclient-cert-host 2026-11-11 01:53:35 -05:00
Joseph Doherty 9aff90f884 fix(opcuaclient): never pass checkDomain for a None endpoint; document both SDK check sites (#816 review)
SDK 2.0.1 checks the server host name in TWO places: Session.OpenAsync (after
the chain is trusted) and DefaultSessionFactory.CreateChannelAsync, against the
certificate in the GetEndpoints reply and before any security-mode test. opc-plc
sends a certificate on its None endpoint, so `connect -u
opc.tcp://10.100.0.35:50000 --validate-host-name` was refused on an
unauthenticated channel, where a host check proves nothing.

All three call sites now pass checkDomain only when the selected endpoint's
SecurityMode != None: OpcUaEndpointSelection.CheckDomainFor for the driver
(OpenSessionOnEndpointAsync, so the failover sweep too) and the AdminUI browser;
the same expression in Client.Shared ConnectToEndpointAsync (shared by
failover). New live tests: driver None + flag on vs :50000 reads; browser None +
flag on vs the NAT fixture browses; the secure + flag refusals still hold.

Docs: name both check sites; a zone-qualified IPv6 host never matches (FindDomain
compares the zone too), so link-local needs validateServerHostName=false; the
match is loose (first label alone, subject DC= parts count, no wildcards), which
caveats the CA recommendation.
2026-11-11 01:53:01 -05:00
Joseph Doherty 8a6f0e3c84 fix(mtconnect): the Agent refuses count ABOVE bufferSize, not at it (#825)
Live gate against the cppagent 2.7 fixture: count=16384 with bufferSize
16384 streams; 16385 is OUT_OF_RANGE ('count' must be less than 16385).
Warn only when SampleCount > bufferSize, pin both sides of the edge
against the live Agent, and correct the #804 wording that said 'at or
above'.
2026-11-11 01:38:48 -05:00
Joseph Doherty 2b0eed1a8f feat(heidenhain): controller alarms as native Part 9 conditions; rediscovery on identity change (#831)
- ALARM: selectors (HeidenhainAlarmSelector, Contracts): ALARM:*, ALARM:OTHER,
  ALARM:CLASS=<r>[/GROUP=<r>][/NUMBER=<r>] (<r> = n | lo..hi; negative numbers allowed).
  An ALARM: address and the cross-driver alarm object only work together — the factory
  refuses either alone, so editor, save path, CSV and deploy inspector agree. Data type
  fixed Boolean; never a value (ReadAsync BadNotSupported), never in a data subscription.
- HeidenhainDriver implements IAlarmSource: one condition per ALARM: tag at its RawPath,
  driven by the SAME allowlisted R_RI 27/28 walk on the control's single loop, on the new
  Polling.AlarmMs cadence (default 2000 ms, only for controls carrying an ALARM: tag).
  FOCAS #647 transition rules (Raise / Retrigger / Clear), identity = class+group+number+text,
  deterministic display order, told-record on the driver instance, recorded only while a
  handler listens. A failed walk (refusal, overflow, comms loss) is quality-only, never a clear.
- Acknowledge is LOCAL-ONLY: AcknowledgeAsync is a no-op success (FOCAS/MTConnect posture);
  Lsv2ReadOnlyPolicy is unchanged — no ack or write request exists.
- Opt-in per tag, no driver flag (FOCAS #647 / MTConnect #626 precedent); Alarms/* arrays unchanged.
- IRediscoverable: a reconnect whose R_VR identity (model / NC / PLC / options) differs from
  the previous session raises OnRediscoveryNeeded once (advisory re-browse chip). No extra traffic.
- AdminUI: Heidenhain joins NativeAlarmModel's alarm drivers; the typed editor fixes ALARM:
  tags at Boolean and seeds/drops the alarm object with the selector; driver form gains
  "Alarm conditions (ms)". (HeidenhainDriverFormModel also carries the #830 credential-notice change.)

Tests: HeidenhainAlarmTests (grammar, pairing, rules, simulator e2e with the captured iTNC 530
alarm record, local-only ack = zero bytes sent, quality-only outage/refusal, rediscovery),
Runtime.Tests HeidenhainNativeAlarmEndToEndTests (real DriverHostActor condition), a
DraftValidator case, and the NativeAlarmDriverSetGuard now scans the Heidenhain driver.
2026-11-11 01:37:41 -05:00
Joseph Doherty 9742a66a9e docs(security): credential-reference namespaces + migration note (#815)
docs/security.md gains the canonical table of per-driver namespaces, the five
enforcement seams, the out-of-scope cases (appsettings ${secret:}, Sql
connectionStringRef) and the migration steps for existing configs. Galaxy,
OpcUaClient, Heidenhain, Configuration and driver-specs docs, the Galaxy
options XML docs, the drivers rule file and the docker-dev compose comment
point at it. docker-dev's env:GALAXY_MXGW_API_KEY stays valid (allowed legacy
prefix), so no rig change is needed.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 01:37:25 -05:00
Joseph Doherty ae91e6b7f7 feat(opcuaclient): opt-in server-certificate host-name check; NAT fixture keeps its PKI (#816, #818)
#816 finding: the SDK checks that the server certificate names the host only
when a session is created with checkDomain:true. The driver, the AdminUI
browser and Client.Shared all called the 8-argument
DefaultSessionFactory.CreateAsync, which forwards checkDomain:false — so no
host-name check ever ran (why a trusted NAT cert for *.invalid browsed fine
over an IP in #813). When enabled the SDK (Session.OpenAsync ->
ValidateDomains -> FindDomain) compares ConfiguredEndpoint.EndpointUrl.IdnHost
— the CONFIGURED host, because OpcUaEndpointSelection/EndpointSelector
rewrote the advertised one — against the cert's DNS/IP SANs; secured
sessions only, and the auto-accept handler accepts a mismatch.

Policy: certificateValidation.validateServerHostName, tri-state. Unset =
OpcUaCertificateValidationOptions.DefaultValidateServerHostName = false, the
pre-#816 behaviour, so nothing changes on upgrade; true = validate the
configured host; false = explicit NAT/port-publish opt-out. Threaded into the
driver, the AdminUI browser (same JSON) and the AdminUI form (a select that
keeps "unset" unset), Client.Shared ConnectionSettings.ValidateServerHostName
and the CLI --validate-host-name. A driver refusal names the opt-out.

#818: the NAT fixture's /app/pki is on the opc-plc-nat-pki volume (verified:
same thumbprint across --force-recreate), its PUBLIC cert is committed as
Docker/nat/opc-plc-nat-server.der, and NatPublishedEndpointBrowseTests run
every secure browse on a throwaway PKI (internal browser seam): refusal with
nothing trusted, positive with the cert trusted (unset/false host check), and
a BadCertificateHostNameInvalid refusal with the check on. The trusted cert is
OPCUA_NAT_SERVER_CERT or the committed one (skip on thumbprint mismatch);
OPCUA_NAT_SERVER_CERT_TRUSTED is retired. CI exports its throwaway NAT
container's public cert. ServerHostNameValidationTests pin the driver against
the plain opc-plc. MIN_EXECUTED not raised (needs a hosted run): +6 in
v2-integration (440 -> 446), documented in scripts/ci/README.md.
2026-11-11 01:33:00 -05:00
Joseph Doherty 8e8408a9ca fix(opcuaclient): keep the configured IPv6 zone index through the endpoint rewrite (#819)
.NET's Uri parses a zone ([fe80::1%eth0]) but drops it from Host, Authority,
AbsoluteUri and ToString(); only DnsSafeHost/IdnHost carry it. The shared
OpcUaEndpointUrlRewriter rebuilt the URL through UriBuilder, so the zone was
lost and link-local endpoints were unroutable. The SDK dials
new Uri(EndpointUrl).IdnHost (TcpByteTransport -> DnsEndPoint), which keeps
it, so splicing the zone back into the string is enough. Kept verbatim
(%eth0, %4, or RFC 6874 %25eth0) so the session dials exactly what
GetEndpoints reached. Applies to the driver, the AdminUI browser and the
Client CLI/UI (all share the rewriter).
2026-11-11 01:32:38 -05:00
Joseph Doherty a224fd6391 fix(security): scope operator-authored credential references per driver (#815)
A Designer could point a Galaxy / OpcUaClient endpoint at a host they control,
set a credential field to any reference the node can resolve
(env:ConnectionStrings__ConfigDb, env:ServerHistorian__ApiKey, secret:<any>,
Galaxy file:<any path>) and click Test Connect / live browse; the node resolved
the value and sent it to that host.

One shared rule in Core.Abstractions (CredentialReferences +
DriverCredentialPolicy) generalises the #803 Heidenhain namespace:
- Galaxy: secret:galaxy/... | secret:otopcua/galaxy/... | env:OTOPCUA_GALAXY_... |
  legacy env:GALAXY_MXGW_...; file: refused; dev:/literal API key unchanged.
- OpcUaClient: secret:opcuaclient/... | env:OTOPCUA_OPCUACLIENT_...; literals unchanged.
- Heidenhain adopts the shared parser/resolver (namespace unchanged).
Enforced at the AdminUI save path (DriverConfigSaveGate), the deploy gate
(CredentialReferenceDeployChecks, code CredentialReferenceOutOfScope), Test
Connect (AdminOperationsActor, before any probe), live browse
(BrowserSessionService, before any browser) and again at resolve time in every
driver/browser (fail closed, before any store/env read).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 01:28:58 -05:00
Joseph Doherty e7b485c0a2 feat(historian): serve DurationGood/DurationBad on the SQL backends; pin the #821 late-start facts (#820, #821)
#820: SqlServer + QuestDB now serve Part 13 DurationGood / DurationBad from the
same Good / Bad region totals TimeWeightedAggregator already computes for
PercentGood / PercentBad — reported in milliseconds (OPC UA Duration, the SDK's
unit), Uncertain time in neither (Mantis 0006166), same BadNoData / status /
Partial rules. AggregateFamilies.Served grows 17 -> 19, so i=11201 and the
up-front refusal follow automatically; Gateway still refuses both (explicit
served set, unchanged). Client.Shared AggregateType / mapper / CLI gain both.
The #796 oracle answers DurationGood through the Good-only subclass as well.
QualityMix conformance asserts both directly (Q and QP, hand values, ms, and
= PercentGood x interval), replaces the "refused up front" test, and adds
Duration* to the forward/reverse/truncated oracle comparisons.

#821 (facts only, no decision): SdkAggregateParityTests pins an Uncertain-first
late-start interval (QP over [0,20), both directions) against ours (Part 13
Total 163.96, TimeAverage 10.93), the frozen 1.5.378.182 calculators (188.96,
9.448), SDK 2.0.1's stock calculators (188.96, 9.448) and a Good-first control
(all 163.96); LiveAggregateLateStartFactsTests pins the same through the booted
server's runtime AggregateManager (live AggregateFilter) incl. DurationGood
15000 / PercentGood 75 there vs 8000 / 40 in HistoryRead (#809). QualityMix
adds the reverse late-start variant (reverse TW pin + reverse oracle windows).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 01:28:54 -05:00
Joseph Doherty f8f6781a24 feat(mtconnect): warn at Test Connect and deploy when SampleCount >= the Agent bufferSize (#825)
Parse the /probe Header bufferSize; MTConnectDriverProbe reads sampleCount
(default 1000) and answers Ok with a WARNING message (amber chip in the
Test Connect button); InitializeAsync logs the same Warning and exposes
agentBufferSize on the diagnostics bag. Live-checked against the shared
cppagent fixture (bufferSize 16384).
2026-11-11 01:23:33 -05:00
Joseph Doherty 5bbe1e3c7a fix(heidenhain): resolve the probe's secret resolver lazily — eager resolution crash-looped central nodes at boot (#803)
v2-ci / unit-tests (push) Successful in 1h9m7s
The Test-Connect probe set is resolved inside the AdminOperations Akka starter; resolving
ISecretResolver there walks AddZbSecrets -> IAuditWriter -> ActorAuditWriter, whose ctor needs
AuditWriterActorKey (registered later in the same builder). Found by the docker-dev live gate.
Regression test: resolving the probe set never resolves ISecretResolver (fails before the fix).
Also: TagRowProblem doc comment re-attached; Heidenhain credential help text names the
secret:heidenhain/ and env:OTOPCUA_HEIDENHAIN_ namespaces.
2026-11-11 00:27:37 -05:00