Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.
DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The host handed the SDK a provider-less telemetry context, so SDK-side
refusals — identity-token decryption (the SDK 2 RsaEncryptedSecret clock
window, #873), certificate validation, secure-channel faults — left no server
log line (found on windev during the #800 Windows validation). The hosted
service now creates one ServerSdkTelemetry context from its ILoggerFactory
(Warning and above, SDK categories, rate-limited 20/min per category) and
shares it between OtOpcUaSdkServer and the ApplicationInstance; released on
dispose. Test: an untrusted client's rejection reaches the captured log.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The fixed name lives on the SQL Server shared by every checkout and CI, so a
concurrent run's create or finally-drop could collide (failed once in the
local pre-push suite, 3/3 green on rerun; cause plausible, not proven).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Since #855 condition events are queued to folder event items under the node
manager Lock. At a full DiscardOldest queue the SDK's EventQueueHandler
dequeues the head of EventMonitoredItemQueue, a List.RemoveAt(0) shift of the
whole queue (~1 us/item at QueueSize 1000, ~10 us at 10000).
OtOpcUaSdkServer.CreateMonitoredItemQueueFactory now wraps the SDK factory in
RingEventQueueFactory: event queues are RingEventMonitoredItemQueue (O(1)
discard), data-change queues, durable requests and disposal go to the SDK
factory unchanged. A factory set through StandardServer.MonitoredItemQueueFactory
is used as-is.
The SDK queue is storage only (decompiled 2.0.1): DiscardOldest, overflow-event
positions and discarded/overflow counters live in EventQueueHandler, which is
unchanged. The ring reproduces QueueSize 0 initial + BadInternalError enqueue,
drop-oldest at Count == QueueSize, FIFO dequeue, SetQueueSize keep-newest /
keep-oldest, the newest-1000 duplicate window via IsEventRaisedFor, no-op
Dispose. RingEventMonitoredItemQueueParityTests drive both queues, and the SDK
EventQueueHandler over both factories, with the same random sequences.
FolderEventDeliveryPerfTests gains Ring_vs_sdk_event_queue (A/B in one process
via the internal OtOpcUaSdkServer.RingEventQueues switch).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Every raw detected_by value names an MXAccess layer (requesting/responding
LMX, NMX, automation object); the gateway's own failures arrive as
ProtocolStatus. The 'non-zero source = transport failure' rule made a failure
the responding object detected a comms error. Item 1 of #872 (Warning/Pending
categories) stays open pending a captured row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The gateway fills Statuses on every data change (live OnDataChange, value-cache
replay and session snapshot all carry Quality too), and EventPump used the status
row whenever one was present, so Quality was never read: an off-scan, Bad or
Uncertain attribute with an OK status row was published Good. A failure status
row still wins; otherwise the quality byte grades the value (legacy parity:
quality >= 192 Good).
Live, read-only on VD03 (DEV): healthy ScanState and _TestArea attributes stay
Good (0x00000000) via quality 192; the stopped JoePlatform/JoeAppEngine ScanState
now publish BadCommunicationError (0x80050000) instead of generic Bad, with no
"unrecognised quality byte" warning.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- MeshPskProviderRefreshTests: GatedResolver.Entered was set by the priming
refresh and never reset, so Wait() returned before the held refresh reached
the store and the mid-refresh call-count snapshot raced its increment
(expected 1, was 2). Hold() now resets it.
- FailedCredentialLockoutTests all-bad burst: the 15 waiters could queue
before a slot holder ran the increment after BeginAsync (authenticated 0
vs 5). The wait now covers both conditions; the security assertions are
unchanged.
Both looped 20x locally, 0 failures.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The gateway copies MXSTATUS_PROXY.detail verbatim, so a failure row's detail is
always an MxStatusDetail code. FromMxStatus routed any detail <= 0xFF through the
OPC DA quality table: MX_E_PlatformCommunicationError (2) on a stopped platform's
ScanState logged "Unrecognised MXAccess quality byte 0x02" and published generic
Bad (VD03 live capture), MX_E_InvalidObjectId (8) became BadNotConnected,
MX_E_Overflow (20) BadCommunicationError, and a detail in the Uncertain band
published Uncertain for a failure.
Every non-zero detail now maps per the gateway.md vocabulary (timeout,
communication, configuration, type, range, access, initialising), then by
category; the category-fallback warning logs once per (detail, category).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The filter-group registry holds groups weakly (the filter->group table is
the only strong reference), sweeps dead entries amortised and counts only
live shapes against the 4096 cap; filters past the cap are counted on the
new meter otopcua.opcua.condition_events.filter_groups_unshared.
- A group validates its where clause too (restored, never-validated filters
evaluate like validated ones).
- Test: value-equal filters cost one projection per locale set per report.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Filter-group buckets include the where clause's (value-based) hash, so
per-source filters sharing a select list are not all scanned under Lock.
- Locale keys are cached per session array.
- Tests: a notifier reached twice in one walk (Line above two wired
equipment) gets each event once on both paths (fails without the
per-report set); queued events re-projected by a modify are identical on
both paths; locale keys unambiguous and where clauses hash by value.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
FolderEventDeliveryPerfTests (Category=Perf, skipped unless OTOPCUA_EVENT_PERF=1)
reports per-alarm-report node-manager Lock hold and end-to-end delivery latency
for one equipment folder with N = 1/10/50/200 event items, ACL on/off, folder
subscribers publishing or stalled at a full queue; the latency of an i=2253
subscriber while folder subscribers stall; and a QueueSize sensitivity sweep.
Compiles against the pre-#855 tree for a before/after comparison.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Adds an escalated-stream-fault test seam and a test: the live pump's fault
escalates (positive controls before and after), a superseded pump's late
fault after a reopen does not; the absence is asserted only once both
pumps' callbacks are known to have run (HasEnded). Verified to fail with
the filter disabled. The filter's comment now says the identity check does
the work and warns about a future session bump while Healthy.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.
Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
ReconnectSupervisor swallowed a transport failure reported while Replaying,
so a restarted event stream refused/ended a round trip after the replay
began still let the replay go Healthy over a dead stream (and the next
advise's re-report restarted at attempt 1 with no backoff). A failure
reported during Replaying is now remembered and turns that attempt's
success into a retry with backoff. ReplayAsync's synchronous dead-pump
throw yields to a teardown's cancellation.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
handler returned is reported; one pending at actor stop is reported; a
confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
reconnect, UNACK_RTN-as-Clear and RPC-budget limits.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- ReplayAsync fails when its restarted pump's stream has already ended, so
the supervisor backs off and stays Degraded instead of going Healthy over
a dead stream (whose next advise re-report restarted at attempt 1 with no
backoff - a tight reconnect loop).
- ReconnectSupervisor.AttemptInProgress (reopen through replay); advises
wait for a stale pump's replacement only while an attempt is running, not
for a whole backoff (the state still reads Reopening/Replaying there).
- ReconnectSupervisor: _disposed is set under the state lock and re-checked
inside ReportTransportFailure, so a late report cannot start a recovery
loop that StopAsync already missed.
- EventPump.HasEnded is a flag set before StreamRequested is released, and
a stream that ends/faults on its first MoveNext is marked ended first.
- SubscriptionRegistry fast path: a miss now falls through to the locked
path (lookup + session + window check atomically) instead of a lock-free
re-lookup.
- Tests: replay over an immediately-ending stream fails; AttemptInProgress
across attempt/backoff; ReportTransportFailure after stop; the
reopen-window test uses a generous wait bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- RefreshDirectItemsWithCut keys its "decided off Lock" set per (item,
condition), so a condition one item's walk saw is still decided for another.
- The deferral warning no longer blames slow folder subscribers (no folder of
ours defers since #855).
- FolderEventFloodWireTests' removal test now asserts the folder item got
nothing and the Server object exactly the pre-stall event (it passed with
the post-stall liveness re-check deleted).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.
- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
(default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
after AcknowledgeAsync, off its mailbox and outside the resilience
pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- StreamRequestWait 1 s (under Tier-A Read 2 s / Subscribe 5 s budgets) and
a stale pump is only waited on while the supervisor is reopening or
replaying (or a restart is underway); in a reconnect backoff the advise
proceeds at once. The no-stream log is a rate-limited Warning while
Healthy, Debug otherwise.
- A current-session pump whose loop has ended is not a stream: if the
supervisor already reports Healthy (its fault was swallowed while it was
recovering) the failure is re-reported to start a new cycle.
- Subscribe/Read no longer start the pump directly (only
AwaitCurrentEventStreamAsync does, honouring the replay restart flag);
the teardown refusal is an explicit ThrowIfRuntimeStopping.
- SubscriptionRegistry.ResolveOrPark fast path: the handle lookup and the
open-window check were not atomic, so a subscribe that registered the
handle and closed its window in between got its initial value dropped as
stale (found by a stress run: the first event of a bulk advise lost). It
re-looks-up before declaring stale. Deterministic test via a seam.
- Tests: first Read on a fresh driver, replay of a data subscription.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Both plain ConditionRefresh paths (Server object and folder) skip a
condition a deploy removed or replaced between the off-Lock walk and the
queueing (the SDK base did not either; only the cut path checked).
- A failure delivering at one notifier hop is logged and the walk continues
to the remaining hops; a failure deciding the Server-object items withholds
only them.
- Test: a condition removed during a plain refresh (folder + Server object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Round 1 replaced a stale-session pump from any advise path and disposed the
old one in the background; that could start a second stream while the old
one still held the gateway's single-subscriber slot, and a kept
current-session pump could be dead with no fault re-reported.
- EnsureEventPumpStarted reuses again; only ReplayAsync restarts the pump.
RestartEventPumpForReplay keeps a current-session pump only while its loop
is running (EventPump.HasEnded), and flags the restart so no advise path
starts a competing pump in the dispose gap.
- AwaitCurrentEventStreamAsync (bounded by StreamRequestWait) starts a pump
only when none exists, WAITS for the replay while the pump is stale or
restarting, waits for StreamRequested, and returns the session it verified.
SubscribeAsync (per #861 attempt), read-via-subscribe and the probe hook
issue and fence the advise under that session, so a reconnect after the
check is caught as a straddle and retried.
- Read: a deadline (not caller) cancel during the wait falls through to
per-ref results.
- Tests: reopen-window probe test now asserts the advise waits for the
replay's single new stream; first Subscribe/Read wait for the stream; the
fencing tests that advise in the window use a short wait bound; the
production-path seed test uses an empty galaxy (a real probe advise now
starts a stream against the fake endpoint).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay's verdict refuses a condition a deploy removed or replaced
since the cut (Server-object and folder), so the cut path cannot replay a
ghost or judge a replaced scripted alarm on its old folder.
- DeliverToNotifierItems resolves the #826 decision's item-independent part
(live condition, notifier check, scopes) once per hop; only the grant and
SDK role checks stay per item.
- A deferred delivery's drain starts on the pool, never inline under Lock.
- Test: a condition removed between the cut and its replay (folder + Server
object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review findings on 02503a4a (ConfigPublishCoordinator):
1 (HIGH) A recovered deployment whose every expected ack was already in
NodeDeploymentState (a seal whose SaveChanges failed, a fused node's own row)
was never completed: inside the deadline it waited for acks that had landed,
past it HandleDeadline's "race" guard returned without sealing — stuck in
AwaitingApplyAcks, refusing every later deploy. ApplyRecovery now completes
it (Seal / PartiallyFailed), and HandleDeadline completes a full ack set
instead of returning.
2 (MED-HIGH) The past-deadline recovery of a populated expected set ran
HandleDeadline inline before the stash replay, discarding on-time acks queued
during the outage. It is now self-sent, behind the replay. The zero-rows
past-deadline branch stays inline on purpose: no ack can count there, and a
self-sent deadline would time out a queued dispatch of that same row right
after publishing it.
3 (MED) A failed PersistNodeAck threw, restarted the actor and dropped the ack
(driver-only nodes never write their own row and do not resend). Acks are now
persisted before being counted, and a failed persist is re-queued on a timer.
The class remarks document that the Loading stash is node-local (lost on a
hand-over; queued dispatches are recovered from their rows).
Minor: MarkReady/Become(Ready) only after the recovery applied; a throw while
applying is a failed load (backoff) rather than a tight restart loop; a queued
dispatch matching the recovered Dispatching row logs Info, not the "lost
dispatch" Warning.
Tests (ConfigPublishCoordinatorDbOutageTests): fully-acked recovery seals /
partially fails inside and past the deadline (4 cases); queued ack applied
before a past-deadline recovery; failed ack persist retried. All 6 fail on
02503a4a.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- EnsureEventPumpStarted replaces a pump created for an older registry
session (the reopen -> replay window still held the dead session's pump,
so an advise there lost its initial value and the replay skipped it);
RestartEventPumpForReplay keeps a pump already on the current session.
- SubscribeAsync, read-via-subscribe and ReplayAsync also await the
bounded StreamRequested wait before advising, not just the probes.
- TryEnsureEventPumpStarted tolerates a teardown flipping _runtimeStopping
mid-check; the probe hook honours an already-cancelled seed token.
- StreamRequested is documented as narrowing, not closing, the attach race.
- Tests: reopen-window probe test, EventPump.StreamRequested tests, and the
FixRound1 fresh-pump assertion now waits on a positive control.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay re-asks each item's verdict right before each entry's gate
hold (Server-object and folder cut paths): a grant revoked between the
pre-Lock verdicts and the replay no longer lets the replay through. A denied
entry is also removed from the cut's re-sent set.
- ReportEventWithoutBlocking logs and returns on a delivery that faulted
inline instead of booking it as a deferral episode (which ran the drain and
ClearStaleRefreshCuts under the event-report gate while Lock was held).
- OnConditionReportEvent isolates failures per Server-object item.
- Tests: revocation between cut and replay (folder + Server object), throwing
hop not booked as a deferral; production-overload sampling inside Lock.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
When the data starts inside an interval (no start bound) and its first value is
Uncertain, the frozen 1.5.378.182 AverageAggregateCalculator integrated the
uncovered span as a ramp from 0 and divided TimeAverage by the whole interval:
GetRegionsInValueSet (ignoreBadData, sloped) overwrites the BadNoData start
region's status with UncertainDataSubNormal in its one branch without a not-Bad
guard. Live AggregateFilter therefore disagreed with HistoryRead and Part 13
§5.4.3.6/§5.4.3.8 ("No Start Bound: Calculate Partial Interval"); QP: Total
188.96 vs 163.96, TimeAverage 9.448 vs 10.931. SDK 2.0.1 stock does the same.
Owner-approved 2026-11-11.
- New app-owned CoveredTimeAverageAggregateCalculator subclasses the baseline
and recomputes only TimeAverage/Total, restoring the Bad status of a region
that starts at a Bad point (only the start bound can). Only such an interval's
value changes; status, timestamps, type and bits are the baseline's. Stepped
and TreatUncertainAsBad=true never reached the defect. CompatibilityAggregate-
Factory routes exactly those two IDs; copied SDK files untouched.
- Golden test: 0 fixture vectors change (every data set has a usable value at
-10 s), asserted. New sweep over late-start variants of every data set (cut
+5 s / +25 s, first non-Bad sample Uncertain, all directions/stepping/configs/
incremental): 150 / 142 of 384 change, each pinned to a baseline-derived
oracle (untouched baseline's value with that sample Good, its status/bits on
the variant itself) - never our calculator. Plus a hand-computed case.
- Live-vs-HistoryRead parity (TimeWeightedAggregator) forward + reverse, no
masks, with discrimination against the pre-#821 baseline; QualityMix now
compares QP's TimeAverage/Total with the live oracle on both SQL backends;
booted-runtime facts test expects 163.96 / 10.93.
- README recorded decision, ServerAggregateConfiguration remarks, oracle
remarks, docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.
- PreStart now only subscribes and kicks off an async recovery load piped back
to the actor. A failure retries with exponential backoff (1 s doubling to
30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
stashes every message (DispatchDeployment logged as queued, ApplyAck too);
they are replayed in order after the recovered state is applied. Recovery
decisions (re-dispatch / time out) now run inline before the replay instead
of via self-tells, and a duplicate dispatch of the in-flight deployment with
a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
"config-publish" ready-tier health check: Degraded while loading (with the
attempt count and last error), Healthy when ready or not hosted here.
Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The gateway's StreamEvents with after_worker_sequence = 0 is a fresh stream
and never replays, and the event pump started only on the first
Read/Subscribe, so platform ScanState probes advised before then lost their
initial value (and a driver with no authored tags never got any host status).
PerPlatformProbeWatcher now awaits an ensure-event-stream hook before every
probe SubscribeBulk (seed, discover sync, reconnect ResubscribeStaleAsync).
The driver's hook starts the pump (refused while a teardown is in progress)
and waits, bounded, on the new EventPump.StreamRequested signal - completed
once the stream loop's first MoveNext has issued the StreamEvents RPC, or
when the loop ends - so the advise never precedes the stream request.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The gateway's shared GalaxyHierarchyProjector ANDs template_chain_contains
entries, so the single request carrying both $WinPlatform and $AppEngine
matched no object and the probe watcher always seeded 0 platforms.
GatewayGalaxyPlatformSource now runs one paged sweep per template and
unions/dedupes by tag name. The tests use a fake gateway that applies the
projector's AND semantics (they fail on the old single request).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
HeidenhainSession.Leaf's `case Lsv2Tool t` switch expression mixed long, int and
double? arms, so C# unified the natural type to double? and Tool/Number (Int64),
Tool/Index and Tool/AxisCode (Int32) were published as Double. Every arm in that
switch (and in the alarm-list, axes and feature-acquire switches in the same file)
is now boxed to object explicitly so no numeric arm can widen another.
New HeidenhainLeafRuntimeTypeTests asserts value.GetType() for every catalog value
leaf against its declared DataType (scalar + array), plus PLC/DATA binding coercion
for Int16/Int32/Int64/Boolean. Verified red on the old code (exactly the three Tool
leaves) and green with the fix.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Joseph Doherty