Commit Graph

4883 Commits

Author SHA1 Message Date
Joseph Doherty 730bb19e6b Merge #869 Galaxy native alarm ack: unconfirmed gateway OK waits for Galaxy's acknowledged transition, reverts on timeout
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:30:40 -05:00
Joseph Doherty eba016096c Merge branch 'master' into worktree-agent-a720278a57ae41792
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:26:05 -05:00
Joseph Doherty b421c7ca91 fix(runtime): #869 review round 3 — key a native-ack revert to the ack that failed
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:25:58 -05:00
Joseph Doherty b5d4e25905 Merge #864/#865 Galaxy host status: per-template platform seed, event stream started before probe registration; registry and reconnect races fixed
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:22:33 -05:00
Joseph Doherty 50b7b030a9 Merge #855 folder event delivery perf harness (Category=Perf, opt-in)
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:21:27 -05:00
Joseph Doherty a8f4be616b test(opcua): opt-in perf harness for #855 folder event per-item delivery
FolderEventDeliveryPerfTests (Category=Perf, skipped unless OTOPCUA_EVENT_PERF=1)
reports per-alarm-report node-manager Lock hold and end-to-end delivery latency
for one equipment folder with N = 1/10/50/200 event items, ACL on/off, folder
subscribers publishing or stalled at a full queue; the latency of an i=2253
subscriber while folder subscribers stall; and a QueueSize sensitivity sweep.
Compiles against the pre-#855 tree for a before/after comparison.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:20:58 -05:00
Joseph Doherty 59d33ab938 test(galaxy): #865 review round 7 - pin the superseded-pump fault filter
Adds an escalated-stream-fault test seam and a test: the live pump's fault
escalates (positive controls before and after), a superseded pump's late
fault after a reopen does not; the absence is asserted only once both
pumps' callbacks are known to have run (HasEnded). Verified to fail with
the filter disabled. The filter's comment now says the identity check does
the work and warns about a future session bump while Healthy.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:20:49 -05:00
Joseph Doherty 5001e40690 fix(runtime): #869 review round 2 — a carry-through transition no longer settles a pending local ack
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.

Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:19:52 -05:00
Joseph Doherty 13cc7b8353 fix(galaxy): #865 review round 6 - ignore stream faults from a superseded pump
Each EventPump's fault callback now carries the pump's identity; a fault
from a pump that is no longer the live one (or belongs to an older registry
session) is logged at Debug and not escalated, so a stale pump's late fault
cannot fail a good replay now that a failure during Replaying fails the
attempt. The replay-retry warning includes the fault.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:16:01 -05:00
Joseph Doherty ca6d131078 Merge #868 third review fixes: truncated failure reasons, poison ack writes counted, bounded deadline deferral, cache-booted revision unconfirmed
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:14:17 -05:00
Joseph Doherty 780c87eba3 fix(controlplane,runtime): #868 third review — bounded failure reasons, poison ack writes, cache-booted revisions unconfirmed
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
  nvarchar(2048), so on SQL Server the write failed forever, the ack retried
  forever, the deadline deferred forever and every later deploy was refused.
  - One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
    used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
    (exception messages and degraded reasons).
  - A non-transient DbUpdateException (not concurrency, no transient
    DbException / timeout / socket / IO underneath) is POISON: the ack is
    counted from memory with an Error, never retried.
  - Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
    past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
  written before the outcome) and never acks. Cache-booted revisions
  (BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
  re-dispatch re-applies and acks the real outcome instead of a clean
  short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
  path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
  needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
  remainder timed out nodes that applied). No dispatched-at column: a second
  restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
  rolling-upgrade across this change.

Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:13:36 -05:00
Joseph Doherty d80bdf6369 fix(galaxy): #865 review round 5 - a failure during the replay fails the attempt
ReconnectSupervisor swallowed a transport failure reported while Replaying,
so a restarted event stream refused/ended a round trip after the replay
began still let the replay go Healthy over a dead stream (and the next
advise's re-report restarted at attempt 1 with no backoff). A failure
reported during Replaying is now remembered and turns that attempt's
success into a retry with backoff. ReplayAsync's synchronous dead-pump
throw yields to a teardown's cancellation.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:12:33 -05:00
Joseph Doherty 77872b1d1c fix(runtime,galaxy): #869 review round 1 — confirmation off the actor scheduler, stop/async-fault tests
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
  depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
  abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
  handler returned is reported; one pending at actor stop is reported; a
  confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
  confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
  reconnect, UNACK_RTN-as-Clear and RPC-budget limits.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:11:49 -05:00
Joseph Doherty 8653960ea3 fix(galaxy): #865 review round 4 - replay fails on a dead stream, backoff-aware wait
- ReplayAsync fails when its restarted pump's stream has already ended, so
  the supervisor backs off and stays Degraded instead of going Healthy over
  a dead stream (whose next advise re-report restarted at attempt 1 with no
  backoff - a tight reconnect loop).
- ReconnectSupervisor.AttemptInProgress (reopen through replay); advises
  wait for a stale pump's replacement only while an attempt is running, not
  for a whole backoff (the state still reads Reopening/Replaying there).
- ReconnectSupervisor: _disposed is set under the state lock and re-checked
  inside ReportTransportFailure, so a late report cannot start a recovery
  loop that StopAsync already missed.
- EventPump.HasEnded is a flag set before StreamRequested is released, and
  a stream that ends/faults on its first MoveNext is marked ended first.
- SubscriptionRegistry fast path: a miss now falls through to the locked
  path (lookup + session + window check atomically) instead of a lock-free
  re-lookup.
- Tests: replay over an immediately-ending stream fails; AttemptInProgress
  across attempt/backoff; ReportTransportFailure after stop; the
  reopen-window test uses a generous wait bound.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:07:26 -05:00
Joseph Doherty c9e5c8dcf7 Merge #855 folder event items: per-item delivery under the node manager, ConditionRefresh cut matches the Server object
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:07:21 -05:00
Joseph Doherty ec17a8797b fix(opcua): #855 review 4 — per-item "known" in the folder cut; pin the post-stall liveness check
- RefreshDirectItemsWithCut keys its "decided off Lock" set per (item,
  condition), so a condition one item's walk saw is still decided for another.
- The deferral warning no longer blames slow folder subscribers (no folder of
  ours defers since #855).
- FolderEventFloodWireTests' removal test now asserts the folder item got
  nothing and the Server object exactly the pre-stall event (it passed with
  the post-stall liveness re-check deleted).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:05:06 -05:00
Joseph Doherty 4e492e6550 fix(galaxy): confirm unconfirmed native alarm acks against Galaxy's acknowledged transition (#869)
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.

- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
  and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
  and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
  (default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
  transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
  after AcknowledgeAsync, off its mailbox and outside the resilience
  pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:03:04 -05:00
Joseph Doherty 1c1705fa77 fix(galaxy): #865 review round 3 - budget-safe stream wait, dead pump, fast-path drop
- StreamRequestWait 1 s (under Tier-A Read 2 s / Subscribe 5 s budgets) and
  a stale pump is only waited on while the supervisor is reopening or
  replaying (or a restart is underway); in a reconnect backoff the advise
  proceeds at once. The no-stream log is a rate-limited Warning while
  Healthy, Debug otherwise.
- A current-session pump whose loop has ended is not a stream: if the
  supervisor already reports Healthy (its fault was swallowed while it was
  recovering) the failure is re-reported to start a new cycle.
- Subscribe/Read no longer start the pump directly (only
  AwaitCurrentEventStreamAsync does, honouring the replay restart flag);
  the teardown refusal is an explicit ThrowIfRuntimeStopping.
- SubscriptionRegistry.ResolveOrPark fast path: the handle lookup and the
  open-window check were not atomic, so a subscribe that registered the
  handle and closed its window in between got its initial value dropped as
  stale (found by a stress run: the first event of a bulk advise lost). It
  re-looks-up before declaring stale. Deterministic test via a seam.
- Tests: first Read on a fresh driver, replay of a data subscription.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:00:19 -05:00
Joseph Doherty 74496b4fec Merge #868 second review fixes: node row stays Applying until outcome, pending ack persists survive deadline/restart, write-once terminal states
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:54:47 -05:00
Joseph Doherty 494dcb5b87 fix(controlplane,runtime): #868 second review — final-outcome node rows, unpersisted acks survive deadline and restart
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
   as the revision advanced, up to ~105 s before the address-space outcome.
   The coordinator's recovery counts any non-Applying row as an ack, so a
   coordinator restart in that window sealed CLEAN while the node was about to
   report degraded (#601). The row now stays Applying until ReleasePendingAck
   (or the no-publish-actor path) writes the final Applied, clean or with the
   degraded reason. A node restart in the window replays the orphan Applying
   row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
   HandleDeadline persists that backlog first and defers its decision while
   any remain, so a node that acked is never timed out as silent; PreRestart
   re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
   pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
   the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
   a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
   closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
   health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
   failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
   apply to a row still Dispatching/AwaitingApplyAcks.

Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:54:30 -05:00
Joseph Doherty 7783fadae1 fix(opcua): #855 review 3 — plain refresh skips removed conditions; per-hop failure isolation
- Both plain ConditionRefresh paths (Server object and folder) skip a
  condition a deploy removed or replaced between the off-Lock walk and the
  queueing (the SDK base did not either; only the cut path checked).
- A failure delivering at one notifier hop is logged and the walk continues
  to the remaining hops; a failure deciding the Server-object items withholds
  only them.
- Test: a condition removed during a plain refresh (folder + Server object).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:51:03 -05:00
Joseph Doherty 9060ff5561 fix(galaxy): #865 review round 2 - wait for the replay's pump, never race a second stream
Round 1 replaced a stale-session pump from any advise path and disposed the
old one in the background; that could start a second stream while the old
one still held the gateway's single-subscriber slot, and a kept
current-session pump could be dead with no fault re-reported.

- EnsureEventPumpStarted reuses again; only ReplayAsync restarts the pump.
  RestartEventPumpForReplay keeps a current-session pump only while its loop
  is running (EventPump.HasEnded), and flags the restart so no advise path
  starts a competing pump in the dispose gap.
- AwaitCurrentEventStreamAsync (bounded by StreamRequestWait) starts a pump
  only when none exists, WAITS for the replay while the pump is stale or
  restarting, waits for StreamRequested, and returns the session it verified.
  SubscribeAsync (per #861 attempt), read-via-subscribe and the probe hook
  issue and fence the advise under that session, so a reconnect after the
  check is caught as a straddle and retried.
- Read: a deadline (not caller) cancel during the wait falls through to
  per-ref results.
- Tests: reopen-window probe test now asserts the advise waits for the
  replay's single new stream; first Subscribe/Read wait for the stream; the
  fencing tests that advise in the window use a short wait bound; the
  production-path seed test uses an empty galaxy (a real probe advise now
  starts a stream against the fake endpoint).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:46:15 -05:00
Joseph Doherty f775625805 fix(opcua): #855 review 2 — frozen replay skips non-live conditions; per-hop ACL resolution; drain never inline
- QueueFrozenReplay's verdict refuses a condition a deploy removed or replaced
  since the cut (Server-object and folder), so the cut path cannot replay a
  ghost or judge a replaced scripted alarm on its old folder.
- DeliverToNotifierItems resolves the #826 decision's item-independent part
  (live condition, notifier check, scopes) once per hop; only the grant and
  SDK role checks stay per item.
- A deferred delivery's drain starts on the pool, never inline under Lock.
- Test: a condition removed between the cut and its replay (folder + Server
  object).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:40:06 -05:00
Joseph Doherty ff85b8511a Merge #868 review fixes: recovered fully-acked deployments seal, stashed acks beat a past deadline, failed ack writes retry
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:38:57 -05:00
Joseph Doherty 19c7b5d7e4 fix(controlplane): #868 review — complete fully-acked recoveries, replay queued acks before a deadline, retry failed ack persists
Review findings on 02503a4a (ConfigPublishCoordinator):

1 (HIGH) A recovered deployment whose every expected ack was already in
  NodeDeploymentState (a seal whose SaveChanges failed, a fused node's own row)
  was never completed: inside the deadline it waited for acks that had landed,
  past it HandleDeadline's "race" guard returned without sealing — stuck in
  AwaitingApplyAcks, refusing every later deploy. ApplyRecovery now completes
  it (Seal / PartiallyFailed), and HandleDeadline completes a full ack set
  instead of returning.
2 (MED-HIGH) The past-deadline recovery of a populated expected set ran
  HandleDeadline inline before the stash replay, discarding on-time acks queued
  during the outage. It is now self-sent, behind the replay. The zero-rows
  past-deadline branch stays inline on purpose: no ack can count there, and a
  self-sent deadline would time out a queued dispatch of that same row right
  after publishing it.
3 (MED) A failed PersistNodeAck threw, restarted the actor and dropped the ack
  (driver-only nodes never write their own row and do not resend). Acks are now
  persisted before being counted, and a failed persist is re-queued on a timer.
  The class remarks document that the Loading stash is node-local (lost on a
  hand-over; queued dispatches are recovered from their rows).
Minor: MarkReady/Become(Ready) only after the recovery applied; a throw while
applying is a failed load (backoff) rather than a tight restart loop; a queued
dispatch matching the recovered Dispatching row logs Info, not the "lost
dispatch" Warning.

Tests (ConfigPublishCoordinatorDbOutageTests): fully-acked recovery seals /
partially fails inside and past the deadline (4 cases); queued ack applied
before a past-deadline recovery; failed ack persist retried. All 6 fail on
02503a4a.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:38:45 -05:00
Joseph Doherty 2bdff975fa Merge #842 ScriptWorker Windows suspend path verified on windev; #789 RSS budget 200 MB + windev re-measure
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:34:42 -05:00
Joseph Doherty 8f30c1e521 docs(script-runner): windev re-measure (#789) and Windows soft-clock test run (#842)
§12.1 gains the 2026-11-11 windev (VMware VM) perf table: idle p50 581 us
(MISS), kill latency after T+G 8-16 ms (PASS), kill both shards 2.12-2.14 s
(MISS by ~6 %), worker RSS 160 MB compiled / 36 MB warm-cache (PASS under the
revised 200 MB budget), host CPU +131 % vs InProcess (MISS, untuned).

It also records the first Windows run of Category=ScriptWorker: 107/0/2 from
C:\build, all 13 WorkerSoftClockTests cases (5 freezing a real worker with
NtSuspendProcess) green with no code change. From a user-profile directory
the restricted-identity isolation test fails because the restricted token
cannot read the worker dll there (profile ACL), which is environment, not a
defect.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:34:07 -05:00
Joseph Doherty cc1a7ec687 fix(galaxy): #865 review round 1 - stale-session pump, data-path stream wait
- EnsureEventPumpStarted replaces a pump created for an older registry
  session (the reopen -> replay window still held the dead session's pump,
  so an advise there lost its initial value and the replay skipped it);
  RestartEventPumpForReplay keeps a pump already on the current session.
- SubscribeAsync, read-via-subscribe and ReplayAsync also await the
  bounded StreamRequested wait before advising, not just the probes.
- TryEnsureEventPumpStarted tolerates a teardown flipping _runtimeStopping
  mid-check; the probe hook honours an already-cancelled seed token.
- StreamRequested is documented as narrowing, not closing, the attach race.
- Tests: reopen-window probe test, EventPump.StreamRequested tests, and the
  FixRound1 fresh-pump assertion now waits on a positive control.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:31:51 -05:00
Joseph Doherty d2b1545659 fix(opcua): #855 review 1 — re-decide frozen replay entries; a failed delivery is not a deferral
- QueueFrozenReplay re-asks each item's verdict right before each entry's gate
  hold (Server-object and folder cut paths): a grant revoked between the
  pre-Lock verdicts and the replay no longer lets the replay through. A denied
  entry is also removed from the cut's re-sent set.
- ReportEventWithoutBlocking logs and returns on a delivery that faulted
  inline instead of booking it as a deferral episode (which ran the drain and
  ClearStaleRefreshCuts under the event-report gate while Lock was held).
- OnConditionReportEvent isolates failures per Server-object item.
- Tests: revocation between cut and replay (folder + Server object), throwing
  hop not booked as a deferral; production-overload sampling inside Lock.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:31:02 -05:00
Joseph Doherty fb44ec2a9c Merge #821 live TimeAverage/Total late-start with Uncertain first sample matches HistoryRead (Part 13 partial interval)
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:30:27 -05:00
Joseph Doherty 8a0032bfdf fix(aggregates): live TimeAverage/Total leave the span before an Uncertain first value out (#821)
When the data starts inside an interval (no start bound) and its first value is
Uncertain, the frozen 1.5.378.182 AverageAggregateCalculator integrated the
uncovered span as a ramp from 0 and divided TimeAverage by the whole interval:
GetRegionsInValueSet (ignoreBadData, sloped) overwrites the BadNoData start
region's status with UncertainDataSubNormal in its one branch without a not-Bad
guard. Live AggregateFilter therefore disagreed with HistoryRead and Part 13
§5.4.3.6/§5.4.3.8 ("No Start Bound: Calculate Partial Interval"); QP: Total
188.96 vs 163.96, TimeAverage 9.448 vs 10.931. SDK 2.0.1 stock does the same.
Owner-approved 2026-11-11.

- New app-owned CoveredTimeAverageAggregateCalculator subclasses the baseline
  and recomputes only TimeAverage/Total, restoring the Bad status of a region
  that starts at a Bad point (only the start bound can). Only such an interval's
  value changes; status, timestamps, type and bits are the baseline's. Stepped
  and TreatUncertainAsBad=true never reached the defect. CompatibilityAggregate-
  Factory routes exactly those two IDs; copied SDK files untouched.
- Golden test: 0 fixture vectors change (every data set has a usable value at
  -10 s), asserted. New sweep over late-start variants of every data set (cut
  +5 s / +25 s, first non-Bad sample Uncertain, all directions/stepping/configs/
  incremental): 150 / 142 of 384 change, each pinned to a baseline-derived
  oracle (untouched baseline's value with that sample Good, its status/bits on
  the variant itself) - never our calculator. Plus a hand-computed case.
- Live-vs-HistoryRead parity (TimeWeightedAggregator) forward + reverse, no
  masks, with discrimination against the pre-#821 baseline; QualityMix now
  compares QP's TimeAverage/Total with the live oracle on both SQL backends;
  booted-runtime facts test expects 163.96 / 10.93.
- README recorded decision, ServerAggregateConfiguration remarks, oracle
  remarks, docs/Historian.md and .claude/rules/historian.md updated.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:30:01 -05:00
Joseph Doherty cbd24d7a01 Merge #867 Heidenhain integer leaves keep their declared type; #868 ConfigPublishCoordinator survives ConfigDb outage at start
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:29:00 -05:00
Joseph Doherty 02503a4a9f fix(controlplane): ConfigPublishCoordinator survives an unreachable ConfigDb at singleton start (#868)
PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.

- PreStart now only subscribes and kicks off an async recovery load piped back
  to the actor. A failure retries with exponential backoff (1 s doubling to
  30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
  stashes every message (DispatchDeployment logged as queued, ApplyAck too);
  they are replayed in order after the recovered state is applied. Recovery
  decisions (re-dispatch / time out) now run inline before the replay instead
  of via self-tells, and a duplicate dispatch of the in-flight deployment with
  a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
  "config-publish" ready-tier health check: Degraded while loading (with the
  attempt count and last error), Healthy when ready or not hosted here.

Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:28:39 -05:00
Joseph Doherty e35f2a4e93 fix(galaxy): probe advises start the event stream first so initial ScanState arrives (#865)
The gateway's StreamEvents with after_worker_sequence = 0 is a fresh stream
and never replays, and the event pump started only on the first
Read/Subscribe, so platform ScanState probes advised before then lost their
initial value (and a driver with no authored tags never got any host status).

PerPlatformProbeWatcher now awaits an ensure-event-stream hook before every
probe SubscribeBulk (seed, discover sync, reconnect ResubscribeStaleAsync).
The driver's hook starts the pump (refused while a teardown is in progress)
and waits, bounded, on the new EventPump.StreamRequested signal - completed
once the stream loop's first MoveNext has issued the StreamEvents RPC, or
when the loop ends - so the advise never precedes the stream request.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:24:08 -05:00
Joseph Doherty c4bc1ef309 fix(galaxy): seed platforms with one template filter per request (#864)
The gateway's shared GalaxyHierarchyProjector ANDs template_chain_contains
entries, so the single request carrying both $WinPlatform and $AppEngine
matched no object and the probe watcher always seeded 0 platforms.
GatewayGalaxyPlatformSource now runs one paged sweep per template and
unions/dedupes by tag name. The tests use a fake gateway that applies the
projector's AND semantics (they fail on the old single request).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:24:03 -05:00
Joseph Doherty 8e97fc94fd fix(heidenhain): Tool/Number, Tool/Index, Tool/AxisCode publish their declared integer types (#867)
HeidenhainSession.Leaf's `case Lsv2Tool t` switch expression mixed long, int and
double? arms, so C# unified the natural type to double? and Tool/Number (Int64),
Tool/Index and Tool/AxisCode (Int32) were published as Double. Every arm in that
switch (and in the alarm-list, axes and feature-acquire switches in the same file)
is now boxed to object explicitly so no numeric arm can widen another.

New HeidenhainLeafRuntimeTypeTests asserts value.GetType() for every catalog value
leaf against its declared DataType (scalar + array), plus PLC/DATA binding coercion
for Int16/Int32/Int64/Boolean. Verified red on the old code (exactly the three Tool
leaves) and green with the fix.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:20:33 -05:00
Joseph Doherty d9b23e0b95 feat(opcua): per-item delivery for folder event items; folder ConditionRefresh gets the #854 cut (#855)
Condition events reaching one of our notifiers no longer go through the SDK
MonitoredNode2 channel: the walk serves the attached items itself, per item and
per event (audit guard, uncached INodeManager3 verdict, item gate/cut,
QueueEvent), as the Server-object path does. No notifier of ours defers any
more; the #838 deferral queue and the #854 cut stay as the safety net and
folder items share the cut (RefreshDirectItems, flag on and off). An event
whose walk suspends is marked deferred before it suspends.

Tests produce deferral with a notifier-stall probe instead of holding a folder
item's SDK lock (which would now block the reporter). New
FolderConditionRefreshWireTests; design note in docs/plans.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:18:11 -05:00
Joseph Doherty 233e4bab25 Merge #863 TwinCAT Test Connect: drop the AmsNetId-as-IP TCP pre-check; probe via the local AMS router ADS handshake
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:14:58 -05:00
Joseph Doherty fdc9473819 fix(twincat): Test Connect probe no longer dials the AmsNetId as an IP (#863)
TwinCATDriverProbe ran a first-phase TCP check against the AmsNetId's
first four octets on the AMS port (801/851). An AmsNetId is not an IP and
the AMS port is not a TCP port, so Test Connect timed out against every
real runtime (Z35474: dialled 5.118.56.93, an unrelated public host).

The probe now opens no socket of its own: it takes the driver's path,
AdsClient.Connect + ReadStateAsync through the local AMS router, bounded
by the probe budget (Task.Run + WaitAsync), and classifies failures as
no local AMS router / no route or target unreachable / timeout / other.
The degrade-to-green 'TCP reachability only' outcome is gone - with no
TCP check nothing was contacted, so a missing router is reported red.

Tests: the regression test listens on the AmsNetId-derived endpoint and
asserts no connection arrives (fails on the old probe); classification
and budget tests drive the handshake through an internal seam.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:13:55 -05:00
Joseph Doherty 93d1809484 docs(script-runner): §8.2 worker idle RSS budget revised to 200 MB + scripts (#789)
The owner revised the per-shard worker idle RSS budget from 80 MB to 200 MB
+ scripts on 2026-11-11: compiles run in the worker, so Roslyn's code and
metadata stay resident (measured 159-187 MB) and no GC setting moves it.
The perf harness now reports both RSS rows against the 200 MB constant with
a PASS/MISS mark; A-3 records what the measurement showed.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:10:59 -05:00
Joseph Doherty 70bc960111 Merge #809 live PercentGood/DurationGood count Good only (Part 13), matching HistoryRead
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:05:51 -05:00
Joseph Doherty ed05d50795 fix(aggregates): live PercentGood/DurationGood count Good time only (#809)
The frozen 1.5.378.182 StatusAggregateCalculator counts Uncertain regions as
Good under TreatUncertainAsBad=false, so live AggregateFilter PercentGood /
DurationGood disagreed with HistoryRead (Part 13 §5.4.3.31/§5.4.3.33, Mantis
0006166) whenever Uncertain data was present. Owner-approved 2026-11-11.

- New app-owned GoodOnlyStatusAggregateCalculator subclasses the baseline and
  replaces only the PercentGood/DurationGood computation (Good regions only);
  CompatibilityAggregateFactory routes exactly those two IDs to it. The copied
  SDK files are untouched; the other 35 aggregates are byte-identical.
- Golden test: the 64 changed vectors (38 PercentGood + 26 DurationGood) are
  exempted exactly — those whose baseline output differs from the same case's
  baseline TreatUncertainAsBad=true twin — and pinned to that twin (baseline
  evidence, not regenerated from the implementation); plus a full Good-only
  pin over all 352 PercentGood/DurationGood vectors and a hand-computed case.
- Live-vs-history parity over Good/Uncertain/Bad mixes (no masks), with a
  discrimination check against the pre-#809 baseline; the booted-runtime facts
  test now expects 8000 ms / 40 %.
- SdkAggregateOracle: Run/RunLive = the registered factory; RunSdkAsIs = the
  untouched baseline; its Part13GoodCalculator duplicate is removed.
- README (recorded decision), ServerAggregateConfiguration remarks,
  docs/Historian.md and .claude/rules/historian.md updated.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:05:34 -05:00
Joseph Doherty c3b77a2e4f Merge #786 fleet diagnostics: Faulted devices counts distinct devices across the pair, Faulted only; Stopped column
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:58:51 -05:00
Joseph Doherty 43bdeff2b6 fix(adminui): Faulted-devices tile counts distinct Faulted devices across the cluster pair (#786)
Owner decision 2026-11-11: /fleet/diagnostics "Faulted devices" counts DISTINCT
devices across the nodes of the selected node's cluster, Faulted state only.

- DriverHostActor: FaultedDevices = HostState.Faulted only (Stopped no longer
  counted); new StoppedDevices + FaultedHostNames. Counting extracted to the
  testable DiagnosticsFromHealth.
- DriverInstanceDiagnostics: StoppedDevices (default 0) and FaultedHostNames
  (default null) are additive with defaults, so a pre-#786 sender's snapshot
  still deserializes over Akka's default JSON serializer (pinned by a test).
- FleetDiagnosticsAggregation (pure): identity key = (driver-instance string
  id, HostName), case-insensitive; a single-host driver / nameless pre-#786
  count is combined by MAX across nodes, never summed. Cluster peers come from
  enabled ClusterNode rows of the selected node's ClusterId.
- FleetDiagnostics.razor: tile fetches the cluster peers and shows the
  distinct count + "n of m nodes answered"; per-driver table gains a Stopped
  devices column.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:58:35 -05:00
Joseph Doherty 8c712c5ee0 Merge #824 MTConnect empty string Good (Java parity); #837 host-name check stays opt-in, toggle in Security panel
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:58:17 -05:00
Joseph Doherty e6ec5a0948 docs(opcuaclient): keep server host-name check opt-in; surface it in the Security panel (#837)
Owner decision 2026-11-11: certificateValidation.validateServerHostName stays
opt-in (unset = off). The AdminUI toggle moves from the "Subscriptions, limits &
discovery" panel to the Security panel beside mode/policy, with help text on the
NAT/IP caveat, IPv6 zone ids and the loose first-label match. Binding unchanged
(tri-state, "Default (off)" leaves the key absent). OpcUaClient.md states the
decision; drivers rule updated; source-scan test pins placement + caveats.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:57:59 -05:00
Joseph Doherty bdbc47f5fc fix(mtconnect): serve an empty string-typed value as Good "" (#824)
Owner decision 2026-11-11 (Java parity): a genuinely empty observation on a
String/Reference or unauthored item is Good "" instead of BadNoCommunication.
Non-string items (numeric, bool, DateTime, arrays) keep BadNoCommunication -
the Agent supplied no value; BadTypeMismatch would wrongly blame the authored
type. The UNAVAILABLE sentinel is unchanged, and a structured observation with
no text stays BadNotSupported (structured check now precedes the empty check).
Parity matrix, quality-mapping section and drivers rule updated.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:56:30 -05:00
Joseph Doherty e857135e8b Merge #862 Galaxy: reconnect with ReplayOnSessionLost=false publishes BadNoCommunication for stranded subscriptions and reports Degraded
v2-ci / unit-tests (push) Failing after 1h12m31s
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:13:43 -05:00
Joseph Doherty 33e10c2266 fix(galaxy): stranded subscriptions after a no-replay reconnect go Bad + Degraded (#862)
With Reconnect.ReplayOnSessionLost=false a recreated gateway session carries none of
the old advises, so every pre-reconnect subscription went silently deaf: nodes kept
their last value as Good and health returned to Healthy (only a Warning log).

ReplayAsync now records the stranded subscriptions and publishes BadNoCommunication
(StatusCodeMap.BadNoCommunication, 0x80310000; value null — no driver value cache)
for every tag of each, straight to OnDataChange so LastSuccessfulRead is not stamped.
GetHealth reports Degraded naming the setting while any stranded subscription is
still registered; it clears on UnsubscribeAsync (the host's re-subscribe path) or
runtime teardown, and pruned ids never pin it. A subscribe after the reconnect is an
ordinary new-session subscription. With nothing else flowing, the comms-loss
watchdog re-initializes the driver after 90 s (a delayed replay).

Tests: Runtime/StrandedSubscriptionTests (6). Docs: drivers/Galaxy.md, which also
no longer claims the replay uses the gateway's ReplaySubscriptions RPC.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:13:19 -05:00
Joseph Doherty 793935614f Merge #859/#861 Galaxy: early events parked across the subscribe-reply race; per-session fencing across reconnects
Four independent reviews (blockers fixed each round).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 11:08:14 -05:00