9aaab9ee421504b2cfec84e4cb03df82b6272888
78 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
71b3b7f35f |
fix(galaxy): #866 map MxStatus failure details by the MxStatusDetail vocabulary, not as OPC DA quality
The gateway copies MXSTATUS_PROXY.detail verbatim, so a failure row's detail is always an MxStatusDetail code. FromMxStatus routed any detail <= 0xFF through the OPC DA quality table: MX_E_PlatformCommunicationError (2) on a stopped platform's ScanState logged "Unrecognised MXAccess quality byte 0x02" and published generic Bad (VD03 live capture), MX_E_InvalidObjectId (8) became BadNotConnected, MX_E_Overflow (20) BadCommunicationError, and a detail in the Uncertain band published Uncertain for a failure. Every non-zero detail now maps per the gateway.md vocabulary (timeout, communication, configuration, type, range, access, initialising), then by category; the category-fallback warning logs once per (detail, category). Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
eba016096c |
Merge branch 'master' into worktree-agent-a720278a57ae41792
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
59d33ab938 |
test(galaxy): #865 review round 7 - pin the superseded-pump fault filter
Adds an escalated-stream-fault test seam and a test: the live pump's fault escalates (positive controls before and after), a superseded pump's late fault after a reopen does not; the absence is asserted only once both pumps' callbacks are known to have run (HasEnded). Verified to fail with the filter disabled. The filter's comment now says the identity check does the work and warns about a future session bump while Healthy. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
d80bdf6369 |
fix(galaxy): #865 review round 5 - a failure during the replay fails the attempt
ReconnectSupervisor swallowed a transport failure reported while Replaying, so a restarted event stream refused/ended a round trip after the replay began still let the replay go Healthy over a dead stream (and the next advise's re-report restarted at attempt 1 with no backoff). A failure reported during Replaying is now remembered and turns that attempt's success into a retry with backoff. ReplayAsync's synchronous dead-pump throw yields to a teardown's cancellation. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
77872b1d1c |
fix(runtime,galaxy): #869 review round 1 — confirmation off the actor scheduler, stop/async-fault tests
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop abandon-and-revert report must run after the actor stopped). - DriverInstanceActorAckConfirmationTests: a confirmation that faults after the handler returned is reported; one pending at actor stop is reported; a confirmed ack reports nothing (positive control). - Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not confirm; case-insensitive match does). - AdminUI min/max on the confirm-timeout field; Galaxy.md documents the reconnect, UNACK_RTN-as-Clear and RPC-budget limits. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
8653960ea3 |
fix(galaxy): #865 review round 4 - replay fails on a dead stream, backoff-aware wait
- ReplayAsync fails when its restarted pump's stream has already ended, so the supervisor backs off and stays Degraded instead of going Healthy over a dead stream (whose next advise re-report restarted at attempt 1 with no backoff - a tight reconnect loop). - ReconnectSupervisor.AttemptInProgress (reopen through replay); advises wait for a stale pump's replacement only while an attempt is running, not for a whole backoff (the state still reads Reopening/Replaying there). - ReconnectSupervisor: _disposed is set under the state lock and re-checked inside ReportTransportFailure, so a late report cannot start a recovery loop that StopAsync already missed. - EventPump.HasEnded is a flag set before StreamRequested is released, and a stream that ends/faults on its first MoveNext is marked ended first. - SubscriptionRegistry fast path: a miss now falls through to the locked path (lookup + session + window check atomically) instead of a lock-free re-lookup. - Tests: replay over an immediately-ending stream fails; AttemptInProgress across attempt/backoff; ReportTransportFailure after stop; the reopen-window test uses a generous wait bound. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
4e492e6550 |
fix(galaxy): confirm unconfirmed native alarm acks against Galaxy's acknowledged transition (#869)
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was applied and when it was only sent (no OnWriteComplete; the reply then carries a diagnostic). EvaluateReply treated both as success, so the H6 live gate saw a condition go Acknowledged while Galaxy never applied it. - EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic) and logs the gateway diagnostic on every branch that carries one. - GalaxyAckConfirmationTracker watches the alarm feed from before the RPC and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs (default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged transition; on timeout it refuses (BadTimeout + diagnostic). - New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it after AcknowledgeAsync, off its mailbox and outside the resilience pipeline; a failure is the existing #760 AlarmAckFailed revert. - AdminUI Galaxy form carries the new knob; docs updated. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
1c1705fa77 |
fix(galaxy): #865 review round 3 - budget-safe stream wait, dead pump, fast-path drop
- StreamRequestWait 1 s (under Tier-A Read 2 s / Subscribe 5 s budgets) and a stale pump is only waited on while the supervisor is reopening or replaying (or a restart is underway); in a reconnect backoff the advise proceeds at once. The no-stream log is a rate-limited Warning while Healthy, Debug otherwise. - A current-session pump whose loop has ended is not a stream: if the supervisor already reports Healthy (its fault was swallowed while it was recovering) the failure is re-reported to start a new cycle. - Subscribe/Read no longer start the pump directly (only AwaitCurrentEventStreamAsync does, honouring the replay restart flag); the teardown refusal is an explicit ThrowIfRuntimeStopping. - SubscriptionRegistry.ResolveOrPark fast path: the handle lookup and the open-window check were not atomic, so a subscribe that registered the handle and closed its window in between got its initial value dropped as stale (found by a stress run: the first event of a bulk advise lost). It re-looks-up before declaring stale. Deterministic test via a seam. - Tests: first Read on a fresh driver, replay of a data subscription. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
9060ff5561 |
fix(galaxy): #865 review round 2 - wait for the replay's pump, never race a second stream
Round 1 replaced a stale-session pump from any advise path and disposed the old one in the background; that could start a second stream while the old one still held the gateway's single-subscriber slot, and a kept current-session pump could be dead with no fault re-reported. - EnsureEventPumpStarted reuses again; only ReplayAsync restarts the pump. RestartEventPumpForReplay keeps a current-session pump only while its loop is running (EventPump.HasEnded), and flags the restart so no advise path starts a competing pump in the dispose gap. - AwaitCurrentEventStreamAsync (bounded by StreamRequestWait) starts a pump only when none exists, WAITS for the replay while the pump is stale or restarting, waits for StreamRequested, and returns the session it verified. SubscribeAsync (per #861 attempt), read-via-subscribe and the probe hook issue and fence the advise under that session, so a reconnect after the check is caught as a straddle and retried. - Read: a deadline (not caller) cancel during the wait falls through to per-ref results. - Tests: reopen-window probe test now asserts the advise waits for the replay's single new stream; first Subscribe/Read wait for the stream; the fencing tests that advise in the window use a short wait bound; the production-path seed test uses an empty galaxy (a real probe advise now starts a stream against the fake endpoint). Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
cc1a7ec687 |
fix(galaxy): #865 review round 1 - stale-session pump, data-path stream wait
- EnsureEventPumpStarted replaces a pump created for an older registry session (the reopen -> replay window still held the dead session's pump, so an advise there lost its initial value and the replay skipped it); RestartEventPumpForReplay keeps a pump already on the current session. - SubscribeAsync, read-via-subscribe and ReplayAsync also await the bounded StreamRequested wait before advising, not just the probes. - TryEnsureEventPumpStarted tolerates a teardown flipping _runtimeStopping mid-check; the probe hook honours an already-cancelled seed token. - StreamRequested is documented as narrowing, not closing, the attach race. - Tests: reopen-window probe test, EventPump.StreamRequested tests, and the FixRound1 fresh-pump assertion now waits on a positive control. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
e35f2a4e93 |
fix(galaxy): probe advises start the event stream first so initial ScanState arrives (#865)
The gateway's StreamEvents with after_worker_sequence = 0 is a fresh stream and never replays, and the event pump started only on the first Read/Subscribe, so platform ScanState probes advised before then lost their initial value (and a driver with no authored tags never got any host status). PerPlatformProbeWatcher now awaits an ensure-event-stream hook before every probe SubscribeBulk (seed, discover sync, reconnect ResubscribeStaleAsync). The driver's hook starts the pump (refused while a teardown is in progress) and waits, bounded, on the new EventPump.StreamRequested signal - completed once the stream loop's first MoveNext has issued the StreamEvents RPC, or when the loop ends - so the advise never precedes the stream request. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
33e10c2266 |
fix(galaxy): stranded subscriptions after a no-replay reconnect go Bad + Degraded (#862)
With Reconnect.ReplayOnSessionLost=false a recreated gateway session carries none of the old advises, so every pre-reconnect subscription went silently deaf: nodes kept their last value as Good and health returned to Healthy (only a Warning log). ReplayAsync now records the stranded subscriptions and publishes BadNoCommunication (StatusCodeMap.BadNoCommunication, 0x80310000; value null — no driver value cache) for every tag of each, straight to OnDataChange so LastSuccessfulRead is not stamped. GetHealth reports Degraded naming the setting while any stranded subscription is still registered; it clears on UnsubscribeAsync (the host's re-subscribe path) or runtime teardown, and pruned ids never pin it. A subscribe after the reconnect is an ordinary new-session subscription. With nothing else flowing, the comms-loss watchdog re-initializes the driver after 90 s (a delayed replay). Tests: Runtime/StrandedSubscriptionTests (6). Docs: drivers/Galaxy.md, which also no longer claims the replay uses the gateway's ReplaySubscriptions RPC. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
a8d3790b09 |
fix(galaxy): #859 final review — probe replay per platform session, teardown session bump
BLOCKER: the reconnect replay chose work from the registry entries' sessions, but ONE registry entry holds every probe handle and any probe publish on the new session (a seed in the reopen→replay gap, or the straddle retry) re-stamps it as current — the replay then skipped the probe watcher and the old platforms stayed bound to handle 0 (host status frozen); a publish after the snapshot made it re-subscribe everything (double advise, leaked handle). ReplayAsync now excludes the probe entry from its snapshot and calls the new PerPlatformProbeWatcher.ResubscribeStaleAsync on every reconnect, which re-subscribes exactly the platforms whose ProbeHandle.Session predates the current session. The replay-gap path keeps ResubscribeAsync (all platforms). Also: ClearRegistry (teardown) starts a new registry session, so a late PurgeParked from a teardown-orphaned call cannot reach the next runtime's parked values; the read path's unused pump local is gone (call kept). Tests go through the driver's reopen/replay: the probe old-session-number test is rewritten that way, plus the seed-in-the-gap scenario (only P and Q are re-subscribed, R keeps its new-session handle, all three resolve). Both checked red with the probe replay removed / re-subscribing all platforms. Refs #859, #861. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
133bc07da5 |
fix(galaxy): #859/#861 third review — read rebind release, probe session fence, reopen fence, pump restart
B-1: the subscribe-once read's cleanup unsubscribes what the registry holds at removal (the replay may have rebound it to new-session handles), not its original bindings; nothing when the entry still holds dead old-session handles. Found while testing it: reads were attached to the pump INSTANCE, so a read across a reconnect's pump restart never saw the replayed value — reads now listen on a driver-level ReadDataChange raised for whichever pump is live. I-A: PerPlatformProbeWatcher records the registry session with each probe handle. Publishing goes through the new fenced TryUpsert and binds only current-session handles; removal/stop unsubscribe and purge only those; a seed reply that straddled a reconnect records nothing and is re-issued (max 3). I-B: PurgeParked takes the session captured BEFORE the unsubscribe RPC and is a no-op under the park gate if the session changed. I-C: ReopenAsync starts a new registry session before AND after the gateway session swap; ReplayAsync does so only when no reopen ran (injected subscriber / tests) and replays only subscriptions still holding older-session handles, so a subscribe issued after the swap is live and not double-advised. P-1: ReplayAsync restarts the pump before any early return (no entries used to leave the dead pump in place). P-2: ReplayOnSessionLost=false logs a Warning; misleading comment about gateway-side replay fixed. M-a: TryRebind returns BindOutcome; UnknownId releases the handles just advised for a subscription removed mid-replay. M-b: the replay/re-snapshot pass reads the session once and stops if it changes. M-c: parked entries drop their MxEvent when they leave the parked set. M-d (retry a straddled read) not done. Tests (TestWait) for each, each checked red against a mutation of its fix. Refs #859, #861. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
d08e900203 |
fix(galaxy): #859 second review + #861 — window-sequence trim, per-session handles, straddling subscribes
I-2: unclaimed parked events are trimmed by window sequence. Each window gets a monotonically increasing sequence at Begin, each parked event is stamped with the newest one, and on every End events stamped before the oldest open window's sequence are dropped (amortised O(1) via a park-order queue). Under concurrent reads the open count rarely hit 0, so trailing events used to pile up to the bound and push genuine early values into overflow. I-1 / #861: SubscriptionRegistry.StartNewSession() — called FIRST in ReplayAsync on every reconnect — clears held events, invalidates the reverse map (subscriptions kept) and bumps a session number. New-session events whose number an old binding still held now park until the replay's per-subscription Rebind promotes them, instead of resolving to the wrong tag. TryRegister / TryRebind refuse a stale session atomically under the park gate: a SubscribeAsync whose reply straddled a reconnect is re-issued on the new session (max 3 attempts), a subscribe-once read returns BadCommunicationError, ResubscribeAllAsync stops if the session changes again. Old-session handle numbers are never unsubscribed (Remove reports whether the entry belongs to the current session; the read's late-reply path checks too), and an old session's pump can no longer resolve through the map (fast path checks the session). I-3 (interim): a completed UnsubscribeBulk purges events parked for its handles (driver + probe watcher). Full fix needs a worker sequence on the SubscribeBulk reply (mxaccessgw, optional) — documented. M-1: a resolved handle with a stranded parked list promotes it and resolves the live event (never parks it). M-2: ResolveOrPark returns the ready events to replay first, one gate acquisition per event while anything is held. M-3: SubscribeAsync/read dispose the window before the orphan-release RPC. M-5: ResolveOrPark remarks document the single-dispatching-pump invariant. Tests (TestWait): overlapping windows don't accumulate, purge, stranded list, session fence + map invalidation, stale-session registration refused, end-to-end replay with swapped handle numbers and event-before-reply, subscribe and read straddling a reconnect, read deadline closes its window. Each was checked red against a mutation of the fix it pins. Refs #859, #861. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
f89e0cc04c |
fix(galaxy): #859 review — read path window, atomic promote, session epoch, per-call windows
1. ReadAsync (subscribe-once) now opens the early-event window around its SubscribeBulk + registration; an initial value that overtook the reply was dropped and the read returned BadTimeout. 2. Register/Rebind write the reverse map and promote parked events in ONE critical section under the park gate, and ResolveOrPark resolves under that gate whenever anything is held (lock-free fast path only when nothing is), so a later live event can no longer overtake the parked initial value. A resolved handle that still holds a parked list queues behind it. 3. Held events are fenced by a session epoch: each new EventPump calls StartNewEpoch(), discarding parked AND promoted-but-unreplayed events; an older pump can neither park nor replay. Item-handle reuse across a reconnect can no longer put an old-session value on a new tag. 4. ResubscribeAllAsync opens one window per subscription, not across the whole replay. 5. Promoted-but-unreplayed (ready) events count toward MaxParkedEvents. 6. Begin/End of a window run under the park gate, so N->0 discard and 0->1 open are atomic (no inherited leftovers). Documented that ParkedEventsReady may run under _registryGate/_syncLock and relies on the channel's AllowSynchronousContinuations=false. Tests use TestWait (#858); each new test was checked red against a mutation of the fix it pins. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
1a31e3f584 |
fix(galaxy): park data-change events that overtake their SubscribeBulk reply (#859)
mxaccessgw gives no reply-before-event ordering: the worker completes the STA
command via a RunContinuationsAsynchronously TCS, so the reply is framed on a
thread-pool continuation while the STA already pumps the OnDataChange Advise
queued (drained/framed by a separate loop), and the reply (unary Invoke) and
the event (StreamEvents) are separate HTTP/2 streams. The driver then also
registers handles concurrently with the pump's dispatch loop. MXAccess's
initial value could therefore be dropped as "stale after unsubscribe",
leaving the node at its pre-subscribe value until the next change.
SubscriptionRegistry gains an early-event window: every SubscribeBulk call
site (SubscribeAsync, ResubscribeAllAsync, PerPlatformProbeWatcher) opens
BeginPendingSubscribe() before the RPC and closes it after registering.
Inside the window unknown-handle events are parked (bounded, 100k); the
Register/Rebind that binds a handle promotes them, and the EventPump replays
them on its single dispatch thread before any later event for that handle
(woken by a sentinel when no further event arrives). Closing the last window
discards unclaimed events, so the stale drop is unchanged outside it.
New counters galaxy.events.early_{parked,replayed,overflow}.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
|
||
|
|
4fdc3708f4 |
test(galaxy): presence waits survive a starved/loaded runner; absence asserted behind positive controls (#858)
Under heavy parallel load (14 concurrent test processes + 14 CPU burners, with DOTNET_ThreadPool_ForceMinWorkerThreads=1 as the reproducer) the Galaxy tests with 1-3 s fixed waits failed 16/70 runs. Every failure was a background loop started with Task.Run (alarm feed, DeployWatcher, EventPump, ReconnectSupervisor) not being SCHEDULED within the ceiling - e.g. DeployWatcher's stream had not even been opened after 2 s. No production race: the code under test was checked for each (alarm feed dispatch is synchronous in-order, DeployWatcher/ReconnectSupervisor state is lock- or loop-local, subscribe registers before returning). - New shared TestWait (UntilAsync / ForAsync / HealthyAsync): 30 s presence ceiling, failure reports elapsed ms + an observed-state snapshot (#856 shape; GalaxyDriverFixRound1Tests.WaitUntilAsync now delegates to it). - Converted the short waits in GatewayGalaxyAlarmFeedTests, DeployWatcherTests, GalaxyDriverSubscribeTests, GalaxyDriverRawPathKeyingTests (its WaitForAsync never threw on timeout), EventPumpStreamFaultTests, EventPumpReplayGapTests, ReconnectSupervisorTests; FixRound2 InitBudget 2 s -> 10 s (the parked fetch never completes, so any finite bound keeps the verdict). - Absence by short sleep replaced with positive controls: DeployWatcher bootstrap and same-deploy-time (follow with a real change, assert exactly that one raise - verified red when suppression is removed), subscribe failed-tag / unsubscribed handle (follow with a live handle's event), faulted pump (wait for its onStreamFault instead of 100 ms). - Fixed sleeps that raced scheduling: subscribe-once read tests wait for the read's registration (TrackedSubscriptionCountForTests) instead of 50 ms; supervisor Dispose test waits for reopen entry instead of 50 ms; the already-Healthy WaitForHealthy test asserts synchronous completion instead of a wall-clock bound. FakeSubscriber.Map is now a ConcurrentDictionary. Loaded run (14 parallel x 5, 14 burners, ForceMinWorkerThreads=1): before 16/70 failed runs; after 0/350 (70 + 210 + 70). Without the thread-pool override: 0/70 before and after on this host. Serial: 472 passed, 8 skipped. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
90de38910d |
test(galaxy): shutdown stream-cancel signal from the iterator's exit, not a token registration (#856)
Found while load-verifying #856: ShutdownAsync_cancels_the_event_stream_and_releases_the_pump timed out 7/700 under parallel load. The fake's StreamCancelled was set by a CancellationToken registration scoped by `using` inside the async iterator. On Cancel() the newer ReadAllAsync registration runs first and resumes the iterator on a pool thread, whose `using` can dispose our registration while Cancel() is still walking the callback list - so it never runs. Signal from the iterator's finally (when the token is cancelled) instead, which also proves the stream actually ended. 0/700 after. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
988046cbe4 |
fix(galaxy): WatchedPlatforms cannot show a platform before its probe binding is routable (#856)
PerPlatformProbeWatcher records a platform's handle and publishes the probe binding into the driver registry as one step under _syncLock, but WatchedPlatforms read the handle map lock-free - so it could report a platform one step ahead of the registry Upsert. A ScanState event raised in that window was dispatched by the EventPump, missed ResolveSubscribers and was dropped as "stale after unsubscribe", leaving the host status empty. That is the #856 flake: A_fetch_completing_after_init_populates_host_statuses waited for WatchedPlatforms, emitted ScanState, and timed out at 10 s on the final Running wait. Reproduced 11/700 (28 parallel runners x 25 + 14 CPU burners), every failure at that step with the binding present by the time of diagnosis and hostStatuses=[] - i.e. the event was dropped. 0/1400 after. WatchedPlatforms now reads under _syncLock (test seam only; the dispatch path already checks the map after the registry, the safe order). New PerPlatformProbeWatcherVisibilityTests spins a reader on another thread and asserts the registry entry exists the moment the platform is visible (old code: torn in 1691/2000 iterations). WaitUntilAsync gains a describe overload; the #856 test's waits now report elapsed time and the observed seed/binding/pump/host-status chain on timeout. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk |
||
|
|
a224fd6391 |
fix(security): scope operator-authored credential references per driver (#815)
A Designer could point a Galaxy / OpcUaClient endpoint at a host they control, set a credential field to any reference the node can resolve (env:ConnectionStrings__ConfigDb, env:ServerHistorian__ApiKey, secret:<any>, Galaxy file:<any path>) and click Test Connect / live browse; the node resolved the value and sent it to that host. One shared rule in Core.Abstractions (CredentialReferences + DriverCredentialPolicy) generalises the #803 Heidenhain namespace: - Galaxy: secret:galaxy/... | secret:otopcua/galaxy/... | env:OTOPCUA_GALAXY_... | legacy env:GALAXY_MXGW_...; file: refused; dev:/literal API key unchanged. - OpcUaClient: secret:opcuaclient/... | env:OTOPCUA_OPCUACLIENT_...; literals unchanged. - Heidenhain adopts the shared parser/resolver (namespace unchanged). Enforced at the AdminUI save path (DriverConfigSaveGate), the deploy gate (CredentialReferenceDeployChecks, code CredentialReferenceOutOfScope), Test Connect (AdminOperationsActor, before any probe), live browse (BrowserSessionService, before any browser) and again at resolve time in every driver/browser (fail closed, before any store/env read). Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
|
0702b31029 |
fix(alarms): refuse a keyed native ack with no Galaxy identity synchronously, from a cached key set (lmxopcua#765)
v2-ci / unit-tests (push) Waiting to run
Item 3 of #765. A keyed (API-key) session whose key had no Galaxy ack identity got Good and then a revert, because whether the key has an identity lives in the Secrets store, which must not be read under the SDK lock. GalaxyAckIdentityDirectory (Driver.Galaxy) keeps a node-wide set of the key ids that have a galaxy-ack-user secret. It lists this node's Secrets store (metadata only, nothing decrypted) and is read lock-free. GalaxyDriver.GetAcknowledgeRefusal (IAlarmAcknowledgePreflight, published through NativeAlarmAckPreflightView as in #760) now refuses a keyed session whose key the directory reports Absent, when the driver has no fallback ack account. A refusal also asks for a refresh. Fails open: before the first refresh, once the snapshot is older than StaleAfter, and from a signalled identity-secret change until the next refresh, the lookup answers Unknown and the ack takes the old Good-then-revert path. Refresh: GalaxyAckIdentityRefreshService runs on driver nodes, every Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds (default 30; StaleAfterSeconds 0 = three intervals; validated at start). It also refreshes early on secret changes. The Secrets library has no change event, so SecretsRegistration decorates ISecretCacheInvalidator, which is called by AdminUI /api-keys, /admin/secrets and the replication appliers. API-key changes need no event: a revoked key is already denied before the pre-check. Tests: Galaxy.Tests GalaxyAckIdentityDirectoryTests (16) and Host.Tests GalaxyAckIdentityCacheWiringTests (9). Seven separate mutations each turn at least one test red: pre-check ignores the directory, factory drops it, no invalidator decoration, no staleness check, no change-pending check, no refresh on refusal, loop never wakes early. Docs: Galaxy.md, AlarmTracking.md, Configuration.md, .claude/rules/alarms.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
|
b8202e1147 |
test: remaining xUnit1051/1031/1030, CS8602/8604, CS0618, CS0219, CS0649, NU1510 fixes (lmxopcua#725)
v2-ci / unit-tests (push) Has been cancelled
Awaited deadline oracles get the runner token; sync-over-async tests made async; nullable child models asserted via ShouldNotBeNull() return values; obsolete SDK ctors replaced with the telemetry overloads; dead locals/fixture and the redundant DI PackageReference removed. No suppressions. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
|
1d7a4b7a53 |
test: pass TestContext.Current.CancellationToken at xUnit1051 call sites (analyzer code fix, 451 sites) (lmxopcua#725)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
|
8e8d4cdf13 |
fix(alarms): surface a refused native alarm ack synchronously, revert one that fails later (lmxopcua#760)
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess, left the client with Good and the condition showing Acknowledged while Galaxy was never acked. Both options from the issue, layered: 1. Synchronous refusal for what the driver can decide without I/O. New optional IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an anonymous session when the driver has no AlarmAck fallback account. DriverHostActor publishes a condition NodeId -> owning-driver snapshot into NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new NativeAlarmAckPreflight seam, consulted after the authz and Primary gates, returns BadUserAccessDenied with the reason - nothing routed, applied or reported. A keyed session is not pre-checked (its identity lives in the Secrets store, which must not be read under the SDK lock). 2. Revert for everything else. RouteAlarmAck carries the condition NodeId; DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure, a driver without an alarm source, or an ack dropped while connecting / reconnecting. DriverHostActor re-projects the condition unacknowledged via NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is still pending - any driver transition projected since then wins. A disabled or unspawned driver is not reverted (unchanged). GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a non-zero native hresult (wrong password, no ack permission) instead of only warning, so that case reverts too; as a refusal it neither retries nor trips the breaker. Tests (each fails with its fix mutated out): projector revert/supersede, DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful ack control), preflight view snapshot, node-manager preflight unit test, over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no event; positive control), Galaxy preflight + hresult refusal. Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
|
dc0a24f224 |
feat(galaxy): operator-authenticated alarm ack — send Galaxy credential to mxaccessgw (#749)
v2-ci / unit-tests (push) Failing after 1h8m43s
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:
1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
held ONLY in the encrypted Secrets store
(otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
operator_user. Refusals are excluded from the resilience retry/breaker.
- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
acknowledger scrubs it from every gateway message it echoes and withholds
any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
path is fire-and-forget after the Part 9 method returns).
Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.
Closes #749
|
||
|
|
0d367590db |
test(galaxy): isolate process-wide telemetry tests; bounded stream-cancel wait
GalaxyTelemetryTests (ActivityListener) and GatewayGalaxyDataWriterFailClosedTests (MeterListener, filtered only by meter and instrument name) observe every Galaxy span or measurement in the process. They now run in a collection with parallelization disabled (F-FU6), because a parallel driver test, or its background probe seed, polluted their counts. The stream-cancel assertion in GalaxyDriverShutdownTests goes back to a bounded wait: the instantaneous IsCompleted check introduced in round 1 was racy. Refs #735 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT |
||
|
|
3e60a925dc |
fix(galaxy): background probe seed; generation-checked registrations
Review fix round 2 for lane F: - N1: the init platform seed no longer runs inline under the init token. Galaxy's Tier-A Initialize budget is 5 s, and a cold gateway DiscoverHierarchy can spend all of it, so the OperationCanceledException rethrow failed init, opened the Initialize breaker and cost a full rebuild per retry. InitializeAsync now only resolves the platform source (no network) and queues a background seed tied to a per-runtime CTS that the teardown cancels. A result that lands after its runtime was torn down is discarded. - N2: the seed worker serves the runtime that is live when each pass STARTS, so a request made under a new runtime while a stale pass is in flight always gets a pass of its own. - N3: SubscribeAsync and the subscribe-once read capture the runtime generation before SubscribeBulk and register only if it is unchanged, atomically with the teardown's registry Clear (_registryGate). On a mismatch they release the handles and fail the call. Re-subscribe Rebind uses the same gate, and the production rebuild clears the registry unconditionally. - PerPlatformProbeWatcher re-checks its stop after the SubscribeBulk await, atomically with the record and publish. A pass that straddles StopAsync releases the handles it acquired instead of writing the probe entry back into a cleared registry. Refs #735 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT |
||
|
|
77ca10aa37 |
test(galaxy): de-flake writer-lane and supervisor tests; align EffectiveTimeout doc
The writer concurrency fake parks on its release signal (the 300 ms delay could release the gate early on a starved pool), the supervisor test's cross-thread hold flag is Volatile, and the EffectiveTimeout doc now states the 1 s floor the code and tests implement. Refs #735 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT |
||
|
|
3456756cd1 |
fix(galaxy): seed the probe set at init, refuse pumps mid-teardown, bound the close
Review fix round 1 for lane F: - The ScanState probe watcher is seeded at InitializeAsync from a platform-only DiscoverHierarchy query (IGalaxyPlatformSource / GatewayGalaxyPlatformSource over the owned repository client) and re-seeded on every deploy-watcher rediscovery. The runtime never calls DiscoverAsync on a live driver, so per-platform host status was unreachable in production. A failed fetch logs a Warning and never fails init; re-seeds coalesce so a deploy during an in-flight pass still gets a pass that starts after it. - EnsureEventPumpStarted refuses while the runtime is stopping, BEFORE reusing the live pump (that pump belongs to the dying subscriber), and the production rebuild disposes any stale pump before wiring the new subscriber. - TeardownRuntimeAsync closes the MX session under the caller's token linked with a 5 s SessionCloseDeadline, so a blackholed gateway cannot hold DriverInstanceActor.PostStop for 30 s. - A runtime generation counter, bumped by every teardown, is re-checked before each Rebind in ResubscribeAllAsync, so a replay-gap re-snapshot that straddles a teardown cannot resurrect handles into the cleared registry. Refs #735 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT |
||
|
|
05a0e51fc1 |
fix(galaxy): GalaxyMxSession.CloseAsync(ct) bounds the gateway CloseSession
The SDK's DisposeAsync closes with no token, i.e. under the 30 s DefaultCallTimeout, so a blackholed gateway parked the driver's teardown for the whole of it. CloseAsync(ct) runs the close under the caller's token, drops the channel on a cancelled/failed close (the gateway reaps the session on its idle lease) and leaves the session re-openable. A close followed by the final dispose closes the gateway session once. Refs #735 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT |
||
|
|
6dedf5158f |
feat(galaxy): wire DiscoverPageSize and ProbeTimeoutSeconds; concurrent first-touch write lanes
Review 2026-10-21 G-5, G-6, G-8 ("wire them all").
- G-5: GatewayGalaxyHierarchySource now pages the DiscoverHierarchy RPC
itself (DiscoverHierarchyRawAsync) at Repository.DiscoverPageSize. The
mxaccessgw client's typed DiscoverHierarchyAsync overloads hard-code a
5000-row page and DiscoverHierarchyOptions carries no page size, so the
authored knob reached nothing; the raw request takes page_size on the
wire, so it is honoured here (non-positive → 5000). The loop keeps the
client's repeated-page-token guard.
- G-6: GalaxyDriverProbe reads ProbeTimeoutSeconds from the config and
runs under min(caller budget, configured cap) — the knob can shorten a
probe, never extend it past the host's budget. The TCP preflight now
shares that deadline instead of hanging on the caller's token alone.
- G-8: GatewayGalaxyDataWriter runs a batch as one lane per distinct tag,
concurrently and bounded at 8 (MaxConcurrentWriteLanes); same-tag
writes keep their authored order inside one lane and results land at
their request's index. Acks stay serial by design.
- GalaxyDriverForm: help text now says what the two knobs actually do.
Refs #735
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
|
||
|
|
1b3169273d |
fix(galaxy): wire ScanState probing, single-sweep discovery, liveness stamp, dispatch-loop fault
Review 2026-10-21 G-4, G-7, G-11, G-10. - G-4: PerPlatformProbeWatcher publishes its probe handles into the driver's SubscriptionRegistry under the reserved ProbeSubscriptionId (-1), so the EventPump's fan-out finally reaches the watcher — every ScanState event used to be dropped as "stale after unsubscribe" and per-platform host status never updated. ResubscribeAllAsync delegates the reserved entry to the watcher's new ResubscribeAsync, so the probe survives a reconnect; StopAsync removes the entry. Probe traffic is routed by the reserved handle and never reaches the public OnDataChange. The watcher is also built on the seam path (subscriber only), which is what makes the wiring testable without a gateway. - G-7: GalaxyDiscoverer.DiscoverAsync returns the hierarchy it fetched and the driver selects its $WinPlatform/$AppEngine probe targets from that same sweep instead of asking the gateway for the whole galaxy twice. - G-11: OnPumpDataChange stamps LastSuccessfulRead from inbound stream data, throttled (1 s default), so the comms-loss watchdog has honest liveness evidence under a Degraded supervisor. - G-10: an EventPump dispatch-loop exit that is not a cancellation now reports a stream fault so the supervisor recreates the pump, instead of leaving a deaf pump reporting Healthy. Refs #735 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT |
||
|
|
2d25c916b6 |
fix(galaxy): make ShutdownAsync a real runtime teardown so re-init rebuilds
GalaxyDriver.ShutdownAsync was a no-op: the MX session, both owned gRPC channels, the EventPump, the alarm feed, the DeployWatcher, the probe watcher and the reconnect supervisor were released only by DisposeAsync, which the host never calls — one leaked gateway session per respawn, removal and /raw browse, and a watchdog Shutdown→Initialize that neither tore down nor rebuilt anything (review 2026-10-21 G-1, G-2, G-9, G-3). - ShutdownAsync and DisposeAsync share one ordered TeardownRuntimeAsync (supervisor first, then deploy/probe watchers, pump, alarm feed, session, channels); owned seams are nulled so the next InitializeAsync rebuilds; injected seams survive a shutdown and drop only on the final dispose. Lifecycle calls are serialised behind a gate. - A failed production connect releases its partial runtime (no orphaned client per watchdog retry) and lands Faulted before and after. - StartDeployWatcherAsync stops the previous watcher before starting a replacement, and now starts on a production instance's FIRST InitializeAsync (the old guard returned whenever discovery had not run here, which on a runtime instance is always). - ReconnectSupervisor releases each cycle's CancellationTokenSource when its loop exits and gains an awaitable StopAsync; PerPlatformProbeWatcher gains an awaitable StopAsync. - EventPump treats any exception under cancellation as a clean stop, so a teardown never wakes the supervisor. - Internal seams for tests: an IGalaxyDeployWatchSource ctor parameter and a session-open override forwarded to the owned GalaxyMxSession. Refs #735 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT |
||
|
|
b39bb6794d |
fix(alarms): native-alarm re-statements no longer duplicate history or hide acked alarms (#656)
- AlarmTransitionKind.ActiveAcknowledged (appended): active AND acked applied as ONE projection. Galaxy ActiveAcked snapshot entries and OpcUaClient inactive->(active,acked) use it; the projector writes 'Activated' only if not already active, 'Acknowledged' only if not acked. - AlarmEventArgs.IsRestatement (optional, default false): set only by Galaxy snapshot entries and OpcUaClient first sight. A flagged Retrigger on an Active+unacked condition with the same message and severity, or a flagged Clear on an inactive condition, writes no /alerts row. - NativeAlarmProjector reports the history rows per transition; ForwardNativeAlarm writes one row per entry. Unflagged sources (FOCAS/MTConnect/AbCip, live Galaxy) are unchanged. - Tests: projector rules each with a must-write control; OpcUaClient transition table; Galaxy feed/driver mapping; Galaxy + OpcUaClient end-to-end through DriverHostActor. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
e7dd742541 |
Merge b647-focas-alarms: FOCAS native alarms reach Part 9 conditions (ALARM: address) + respawn keeps acks host-side (#647)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
89961af6e6 |
fix(alarms): #647 review — Raise on an active condition is a re-statement; FOCAS outage quality + health
MAJOR-1: a deploy that changes a driver's config respawns it, and the fresh instance's empty told record re-sent Raise for every still-active alarm, un-acking acknowledged conditions and writing a duplicate "Activated" alerts row. NativeAlarmProjector now treats Raise on an already-active condition as a re-statement (keeps Acked; ForwardNativeAlarm skips the alerts row); only Retrigger (or inactive->active) un-acks. Sources whose upstream ack state is authoritative send Retrigger for an unacked re-statement: OpcUaClient on first sight (no prior told), Galaxy for an unacked active snapshot entry. Runtime E2E drives a FRESH FOCAS driver after a redeploy and asserts the ack survives. MAJOR-2: a failed FOCAS alarm read annotates each subscribed condition BadNoCommunication once per outage (#626 quality-only path) with one Warning (MINOR-6), lifted to Good on recovery; failed alarm reads degrade driver health with LastSuccessfulRead frozen (successful ones advance it), so an alarm-only driver trips the #527 comms-loss watchdog. MINOR-3 CLAUDE.md #630 bullet; MINOR-4 split line; MINOR-5 a non-empty cnc_rdalmmsg2 payload that fits neither stride throws; MINOR-7 value-tag + parsed-address tables are built and swapped, never cleared in place under the retained poll engine. Docs corrected (FOCAS.md, AlarmTracking.md, CLAUDE.md). Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
55ae230fa4 |
ci(#606): Gitea-compatible trx upload + deterministic fixes for the CI-load test flakes
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3 on the test step. - Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR (TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget. - Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process (abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s. - Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick). - Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps (wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log. - Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped. - infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
dabd8eb955 |
fix(address-space): exact element-type check for array writes + array-capability shape edges (#610 review R4)
MAJOR-1: the SDK skips TypeInfo.IsInstanceOfDataType on any node with an OnWriteValue handler (every writable equipment node), and the node manager's fail-fast treated numeric<->numeric as compatible, so a UInt32[] to a Galaxy Int32 array matched MxValueEncoder's int[] arm by CLR covariance and was written bit-reinterpreted with a Good result. EvaluateEquipmentWriteStructure now matches an ARRAY write's element type exactly (Byte[]/ByteString the only equivalence; Variant[]/object[] refused) -> BadTypeMismatch, nothing dispatched. MxValueEncoder's int[]/long[] arms are exact-type guarded too, and the Galaxy parity test's false "the SDK refuses" premise is corrected (the UInt32/UInt64 exclusion is now the encoder's own answer). Scalar writes keep numeric widening by decision: consumers write CLR-typed values (Double to a Float node) and every driver's scalar encoder range-checks. MINOR: Modbus row honours maxRegistersPerWrite (smallest of DriverConfig / the tag's DeviceConfig, default 123) and FC15's 1968-coil ceiling, and requires region + numeric address; Modbus/S7/Galaxy require isArray + integer arrayLength >= 1 (Galaxy: an unbounded node would let a short write resize the attribute); an array value to a scalar node is refused inline. Docs: Uns.md array-write rules + upgrade note (ReadWrite array tags become client-writable on first boot), Modbus.md, Galaxy.md. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
b7a48e2b93 |
Merge #610: server-side array writes gated per driver capability (DriverArrayWriteSupport); ArrayWritable on the surgical whitelist + UNS Backing; shape/IndexRange refusal; ValueRank>1 out of scope
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe # Conflicts: # docs/drivers/S7.md |
||
|
|
18f8eb5d2f |
feat(address-space): server-side array writes gated per driver capability (#610)
Array nodes were forced read-only for every driver. A new single authority, Core.Abstractions DriverArrayWriteSupport, says which (driver, element type) pairs encode a whole-array write: Modbus (Coils / numeric HoldingRegisters), S7 (Byte/Int16/UInt16/Int32/UInt32/Float32), OpcUaClient (pass-through) and Galaxy (Boolean/Int32/Int64/Float/Double/String/DateTime). Everything else stays read-only, so the server never advertises a write the driver refuses. - Composer derives RawTagPlan.ArrayWritable + UnsReferenceVariable .BackingArrayWritable at compose time (both seams, byte-parity); the applier's one writability rule is Writable && (!IsArray || ArrayWritable) for materialise and surgical paths, raw and UNS alike. - ArrayWritable is on the classifier's surgical raw-tag whitelist and mirrored on the UNS record (UnsReferenceBackingParityTests updated). - Node manager refuses a wrong-length / non-array value on an array node (BadTypeMismatch) and any IndexRange write (BadWriteNotSupported) inline, before dispatch; WriteOperate/ACL gates and the revert are unchanged. - Per-driver parity tests pin the table against the real Modbus write path, S7 EncodeArrayBlock and Galaxy MxValueEncoder (UInt32/UInt64 excluded: array covariance reinterprets their bits). - Over-the-wire proof in OpcUaServer.IntegrationTests (ArrayWriteWireTests). - ValueRank > 1 documented as out of scope by decision (docs/Uns.md). Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
e77bca385a |
fix(galaxy,s7): R2 review minors — subscribe-once deadline, probe vs backoff, write-fault ownership, null array elements, write opt-in
- Galaxy: deadline bounds the wait, not the subscribe RPC; BadTimeout for all tags and late handles unsubscribed; doc comment relocated - S7: probe connect runs on the connect deadline so a timeout opens the backoff window - S7: a successful write only retires a Faulted the write side owns - S7: null array elements rejected (BadTypeMismatch); multi-PDU array writes documented non-atomic - S7 integration: array write test gated on S7_SIM_ALLOW_WRITES=1 Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
7c4c7f9653 |
fix(galaxy): seam hardening - advise unmark on throw, dup attributeRef surfaced, bounded subscribe-once read; delete dormant historian connection probe (#605)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
|
c51c59987b |
fix(galaxy): publish an unrepresentable value as BadTypeMismatch, not a good null
Reverses the conservative call from the previous commit at the user's direction, and they were right: "surface it on a counter first" left the wire lying. An MXAccess event can carry Good quality while its payload is a kind MxValueDecoder has no mapping for, so the old behaviour published null-under-Good — which a client cannot tell apart from a genuine null reading. A value the gateway DID send would be consumed as a value it did not, with the only evidence in a log nobody is tailing. Bad data is the honest wire representation of lost data. BadTypeMismatch is an existing StatusCodeMap constant, deliberately reused rather than written inline: StatusCodeParityTests finds status constants by reflecting over Good/Uncertain/Bad-prefixed uint fields and checking them against the pinned SDK, so an inline literal would be invisible to that guard (Gitea #497, where 16 wrong constants had accumulated across 6 drivers exactly this way). Logging is now once per DISTINCT value kind rather than once per pump. The set is bounded by the MxValue oneof enum so a broken producer still cannot flood, but a second unrepresentable kind appearing later gets its own line — warn-once-per-pump would have silently swallowed it, which is the same class of bug as the null this commit fixes. The test's fixture matters more than its assertion: the undecodable event carries Quality = 192 (OPC DA Good), so a decoder that gave up silently would publish Good and fail. Paired with a positive control — a decodable value at the same quality must still publish Good — so it cannot be satisfied by a pump that marks everything Bad. Falsified by reverting the status to Good: the test reds, and only that test. Verified: 331 Galaxy tests green, Core.Abstractions 298 (the parity guard), solution builds 0 errors. Docs corrected in the two places that recorded the old decision as deliberate. |
||
|
|
4769044f88 |
feat(galaxy): handle the replay-gap sentinel and sparse-array values
Closes the two follow-ups left open by the MxGateway 0.2.0 bump. Reading the
gateway's own contract first changed what each of them is, and one of my earlier
claims was simply wrong -- both corrections are recorded below.
Replay gap
----------
A sentinel MxEvent carries family = UNSPECIFIED, so it fell through EventPump's
family switch into the default drop: we silently ignored a documented MUST ("you
missed events -- discard local state and re-snapshot"). It is now intercepted and
answered by re-issuing SubscribeBulk for every tracked subscription, since
MXAccess delivers an initial value per item on subscribe -- re-subscribing IS the
re-snapshot. It does not restart the event pump (the stream carrying the sentinel
is healthy, and the handler runs on that stream's own read loop, so disposing the
pump from there would deadlock on itself), it does not block the read loop, and
re-entry is gated so a burst collapses to one pass instead of a subscribe storm
per sentinel.
Handled on the READ loop, not in the dispatch loop, because everything past that
point is subject to newest-dropped channel pressure -- and "you missed events" is
the one signal whose loss is unrecoverable. Pinned by a test that fills the
channel and still expects the sentinel.
⚠️ This cannot fire in the shipped configuration, and I did not pretend otherwise
or paper over it by enabling resume. The gateway emits the sentinel only for a
stream opened with after_worker_sequence > 0; GatewayGalaxySubscriber passes 0,
which the gateway treats as "fresh stream, not a resume -- plain attach, no
replay, no sentinel". Enabling resume is a larger and genuinely delicate change:
the sequence is only meaningful within ONE gateway session, our reconnect
recreates the session, and replaying a remembered sequence against a fresh
session sets the server's live watermark ABOVE every event it is about to send --
silencing the stream instead of recovering it. That deserves its own change with
a live gate, which I cannot run from here.
Sparse arrays -- correcting the last commit message
---------------------------------------------------
I wrote that the bump "makes handling it possible" on the decode path. That is
wrong: MxSparseArray is documented write-only, the gateway expands it before
forwarding, and "the worker never receives or produces it". So the decode path
is unreachable by contract and the write path has no use for it either -- it is a
wire compaction for a WHOLE-array write, not a partial update (unmentioned
indices are RESET, deliberately, because the gateway cannot read-modify-write
without racing the provider). It is therefore the wrong primitive for an OPC UA
IndexRange write, which expects untouched elements preserved -- and WriteRequest
carries no IndexRange in the first place.
It is decoded anyway, as contract defence: the alternative if a future gateway
ever emits one is a silent null published under Good quality. Fill semantics
mirror the gateway's own SparseArrayExpander exactly (epoch for Time, empty
string, Integer widening to long[] when any element is int64) so a value that
round-trips through a write reads back as the same CLR array.
Also: the decoder is now total by contract, since it runs inside the dispatch
loop where an escaping exception kills dispatch for the life of the pump. A
malformed payload degrades instead of throwing.
A real defect found in the same code
------------------------------------
A stream the GATEWAY completes -- a closed or evicted session -- ends the RPC
without an error, so nothing threw, the supervisor was never notified, and the
pump exited silently. GetHealth() kept reporting Healthy, which also means the
#527 comms-loss watchdog could never trip, because it requires a non-Healthy
state. The driver went permanently deaf with no signal anywhere. This one is
live, unlike the two above. Now reported as a fault; the falsification control is
the existing CleanShutdown_DoesNotInvokeOnStreamFault, since our own disposal
also ends the stream and must NOT be reported.
Value kinds we cannot represent are now counted and warned once rather than
published as an indistinguishable null. The status is deliberately NOT downgraded
to Bad: nothing can reach that branch today, and changing what an OPC UA client
sees for a live shape is not a change to make without a gate. The counter exists
so the next person has evidence instead of a guess.
Verified: 330 Galaxy tests green (12 new), plus Galaxy.Browser 21, Runtime 558,
Core 351; full solution builds 0 errors with the warning census unchanged. Every
new test was falsified first -- disabling the gap interception reds 3 of 4 (the
throwing-callback test passes vacuously when the callback never runs, by
construction), disabling sparse decode plus the representability signal reds all
7 decoder tests, and both were restored and re-verified green.
Docs: the Galaxy.Performance invariant was stated as received = dispatched +
dropped + in-flight, which this breaks -- a sentinel is received but never enters
the channel -- so it now reads + replay_gaps. Galaxy.md gains the three
operational notes; DriverLifecycle.md records that "reaches a non-Healthy state"
is an obligation a driver can silently fail to meet, which is the general shape
of the clean-end bug.
|
||
|
|
205cc4a4a3 |
feat(galaxy): per-code OPC UA mapping for the MxStatusDetail vocabulary
Follow-up to
|
||
|
|
20a816dbf4 |
feat(galaxy): consume the gateway's correlated write statuses; close archreview 06/S-1
The mxaccessgw OnWriteComplete correlation is live (server b948e69, worker 53f69cd — all four unary write kinds), so every write reply's statuses[0] now carries the real COM commit outcome. Live-gating the cutover on windev found and fixed three defects: - StatusCodeMap.FromMxStatus truncated the correlated failure row's MxStatus detail code to a byte (1007 & 0xFF = 239 -> OPC DA Good band), mapping a REFUSED write to Good. New invariant: a row that is not IsSuccess() never maps Good — details > 0xFF (and contradictory Good-band substatuses) route by category (SECURITY_ERROR -> BadUserAccessDenied, named const covered by StatusCodeParityTests; COMMUNICATION/CONFIGURATION keep their codes). - The borrow-from-subscription-registry write optimization could never commit: a subscription handle is plain-advised, and MXAccess refuses both the no-login write through it (SECURITY_ERROR detail 1008, live-captured) and the supervisory advise upgrade (E_INVALIDARG) — one advise per HANDLE, not per attribute. Removed: writer subscribedHandleSource ctor param + borrow branch, GalaxyDriver wiring, SubscriptionRegistry.TryResolveItemHandle + its forward index and test file. The writer always AddItems its own supervisory-advised handle, which commits fine while the same attribute is subscribed (live-proven; pinned by the new write-while-subscribed smoke). - (gateway-side, fixed there) the correlation initially covered WriteSecured only, while the dominant no-login path emits plain Write. Live gate, all legs PASSED against windev: happy write Good off a confirmed status row with galaxy.writes.unconfirmed + advise_failed both FLAT; read-only refusal surfaces Bad-family (the #5 node-revert input Galaxy could never produce before); WriteCompletionWaitMilliseconds=0 control -> provisional Good + metered unconfirmed (override removed after, restoration re-verified). New env-gated smokes: Live_correlated_write_statuses_confirm_commit_and_surface_refusal, Live_write_to_subscribed_tag_commits_via_own_handle (replaces the borrow smoke), GatewayGalaxyWriteMeterLiveTests (run filtered — meters are process-wide). |
||
|
|
95295210b0 |
fix(test): scope the EventPump counter capture to its own pump (#503)
Root cause was cross-test measurement leakage, not a timing budget. EventPump's `Meter` is STATIC — one instance per process — and a `MeterListener` can only select by *meter* in `InstrumentPublished`. `StartMeterCapture` therefore received measurements from every `EventPump` alive anywhere in the assembly. xUnit runs test classes in parallel and several build pumps (`EventPumpStreamFaultTests`, the `Driver-X` test in this same file), so a sibling's events landed in these counters. That is why the flake needed a busy box: it needed two tests to overlap. Measured directly during the gate — `Received = 11` in a run that emitted 10. The old `Received == Dispatched + Dropped + InFlight` assertion is what turned the leak into a failure. `InFlight` is DERIVED as `Received - Dispatched - Dropped`, so the identity is a tautology that cannot fail on a real defect — only on a foreign increment landing between its four separate `Interlocked.Read` calls. Hence the reported subject, `counters.Received`. It is deleted, along with the `InFlight` member, and replaced by direct assertions on the three MEASURED counters. Three further defects in the same test, all found on the way: - **The "held" dispatch loop was never held.** `OnDataChange` is `EventHandler<T>` — void-returning — so `async (_, _) => await gate` is async VOID: it returned to the dispatch loop at the first await and blocked nothing. Measured `Dispatched = 2..3`, never 0. Drops happened only when the producer happened to outrun the consumer, so the arrangement the doc comment describes was fiction and the assertions rode on scheduling luck. Now a synchronous `ManualResetEventSlim.Wait()` on the loop's own thread, pinned by `Dispatched.ShouldBe(0)`. - **Fixed `Task.Delay(150)`** replaced by a presence poll on the drop count (#500's rule). Dropped starts at 0, so it cannot be satisfied by the initial state. - **Emission is staged** — one event, wait for the dispatcher to enter the handler, then the other nine — so the arithmetic is exact (`10 - 1 held - 2 buffered = 7`) rather than scheduler-dependent. The gate release moved into `finally`, ahead of disposal: `DisposeAsync` awaits the dispatch loop, which is now genuinely parked on that gate, so a failed assertion would otherwise hang instead of failing. Verified: 15/15 clean under the exact contention the issue measured (Runtime + AdminUI + Core concurrently) — where the half-fixed version still failed on run 4. Falsifiability: restoring the async-void handler turns the new assertions red. |
||
|
|
b95b99ba8e |
fix(drivers): correct 16 wrong OPC UA status-code constants + guard them by reflection (#497)
Every value verified against Opc.Ua.StatusCodes in the pinned SDK (1.5.378.106) by reflection, not by copying siblings. All are client-visible: OPC UA clients branch on status. The three named in #497: - Historian.Gateway SampleMapper "BadNoData" 0x800E0000 -> 0x809B0000 (0x800E0000 is BadServerHalted) - Galaxy "BadTimeout" 0x800B0000 -> 0x800A0000 (0x800B0000 is BadServiceUnsupported) - TwinCAT BadTypeMismatch 0x80730000 -> 0x80740000 (0x80730000 is BadWriteNotSupported) BadTypeMismatch was wrong in three MORE drivers the issue did not name — FOCAS, AbLegacy and AbCip carried the same 0x80730000. Every call site is a FormatException/InvalidCastException conversion failure, so the name was right and the value was wrong in all four. Adding the reflection guard then surfaced nine further defects offline tests had never checked: - Galaxy GoodLocalOverride 0x00D80000 -> 0x00960000 (not a UA code at all) - Galaxy UncertainLastUsableValue 0x40A40000 -> 0x40900000 - Galaxy UncertainSensorNotAccurate 0x408D0000 -> 0x40930000 - Galaxy UncertainEngineeringUnitsExceeded 0x408E0000 -> 0x40940000 - Galaxy UncertainSubNormal 0x408F0000 -> 0x40950000 - TwinCAT BadOutOfService 0x80BE0000 -> 0x808D0000 (was BadProtocolVersionUnsupported) - TwinCAT BadInvalidState 0x80350000 -> 0x80AF0000 (was BadAttributeIdInvalid) - AbCip/AbLegacy GoodMoreData 0x00A70000 -> 0x00A60000 (was GoodCommunicationEvent) - Historian.Gateway GatewayQualityMapper Good_LocalOverride 0x00D80000 -> 0x00960000 Galaxy's whole Uncertain block read as though the OPC DA quality byte could be shifted into the UA substatus position; it cannot — the substatuses are an unrelated enumeration. GatewayQualityMapper already held the correct table, which is what the Galaxy values are now reconciled against. Guard: StatusCodeParityTests (Core.Abstractions.Tests, which already project- references every driver) reflects over every `const uint` in the deployed ZB.MOM.WW.OtOpcUa.Driver.*.dll whose name reads as a status code, and asserts it equals Opc.Ua.StatusCodes.<name>. Discovery is by convention, so a new driver assembly referenced by that project is covered with no edit here. It went red on all nine defects above before the fixes and now checks 109 constants green. Because reflection can only see a NAMED constant, two inline literals were hoisted so the guard can reach them: Galaxy's BadTimeout/Bad into StatusCodeMap, and GatewayQualityMapper's 15-entry table into a new HistorianStatusCodes. An inline `0x800B0000u, // BadTimeout` at a call site is exactly how the Galaxy defect survived. The drivers stay SDK-free by design; only the test project references Opc.Ua.Core, as the oracle. Also corrected: tests that pinned the wrong values (Galaxy StatusCodeMapTests, SampleMapperTests, GatewayQualityMapperTests — all written from the same bad source), a mislabelled comment in DriverInstanceActorTests, and stale constants in two design docs, one of them the unexecuted MTConnect driver design that would have propagated BadNoData's wrong value into a new driver. The CLI's SnapshotFormatter value->name table was checked against the SDK and is correct; no change needed. |
||
|
|
1424a21419 |
feat(secrets): G-2a secret: arm on GalaxySecretRef via ISecretResolver (Task 7)
Add a secret:NAME arm to GalaxySecretRef.ResolveApiKey that resolves the Galaxy gateway API key through the shared ISecretResolver — fail-closed if the secret is absent (never falls through to the cleartext literal arm), retiring the dev:/literal in-DB path for production. Because GetAsync is async the method becomes ResolveApiKeyAsync; the await cascade threads ISecretResolver by ctor injection into GalaxyDriver + GalaxyDriverBrowser and (since GalaxyDriver is built by a static factory closure, not DI) through GalaxyDriverFactoryExtensions + DriverFactoryBootstrap (which pulls the real resolver from the service provider — registered unconditionally in Slice 1). A NullSecretResolver null-object backs the parse-only/test paths only; the runtime path always gets the real resolver (verified end-to-end). TDD: 3 new secret:-arm tests (resolve / fail-closed-on-absent / no-literal-warning) RED without the arm, GREEN with it; 338 Galaxy tests pass; no sync-over-async. |