OpcUaApplicationHost left ServerConfiguration.MaxRegistrationInterval at the
SDK default (30000) with a null RegistrationEndpoint, so SDK 2.0.1's
StandardServer armed its registration timer against opc.tcp://localhost:4840,
which on an OtOpcUa node is the server itself (no LDS). It answered
BadServiceUnsupported and the SDK retried forever, visible since #874.
Registration is now opt-in: ApplyDiscoveryRegistration sets
MaxRegistrationInterval = 0 (the value that keeps StandardServer from arming
the timer) unless OpcUa:DiscoveryServerUrl is set, in which case the server
registers against that URL every OpcUa:DiscoveryRegistrationIntervalMs
(default 30000) over SignAndEncrypt/Basic256Sha256, like the SDK's own
default endpoint. The validator requires an absolute opc.tcp:// URL and a
positive interval when the URL is set. Documented in Configuration.md and
OpcUaServer.md.
Co-Authored-By: Claude <noreply@anthropic.com>
- Host reader: the frame read and its handling are caught separately. Only the
read's I/O exceptions mean the pipe ended; anything HandleFrame's inline
completion work throws now kills that worker as ProtocolFault (outstanding
runs -> worker restarted / BadResourceUnavailable) instead of escaping a
dedicated thread (process crash) or being swallowed as a clean end (an
InvalidOperationException used to be, leaving the worker unread until a Crash
2 s later).
- Worker reader: the same catch-all around HandleFrame (exit ProtocolFault).
- Worker: Cancel() runs on the pool, so a script's blocking
ctx.CancellationToken callback cannot stop the worker reading its pipe.
- Host write-stall check: ScriptWorkerFrameWriter.WriteInProgressFor; the
heartbeat monitor kills a worker whose pipe write has been blocked longer than
max(10 s, silence limit) as KilledUnresponsive, which fails the write and
releases the producer (seam submit, deploy load, Cancel). Fixes the wrong
'heartbeat monitor ends it' comment on the pipe buffer quota.
- StopAsync sends Shutdown off the caller so its 1 s budget holds on a full pipe.
- WindowsBlockingPipeStream refuses the async surface (Stream's base versions
serialise a write behind a waiting read).
- Tests: PipeFaultTests (throwing completion callback -> ProtocolFault and a
BadResourceUnavailable sibling, shard serves again; a worker that stops reading
but heartbeats is killed within the bound and the blocked producer returns,
via the Hello.test_stall_reader_on_diag seam); BlockingPipeStreamTests dispose
with a write blocked on a full pipe, and with a read and a write both pending;
Category=ScriptWorker on the protocol, handshake and pipe test classes.
Mutation checks: no stall kill -> stall test red; catch-all without the fault
-> callback test red (Crash).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A .NET pipe opened Asynchronous completes even its synchronous Read/Write
through the thread pool's completion port, so every frame on Windows cost a
completion packet plus a pool work item (and the pool's spin-wait). New
WindowsBlockingPipeStream issues its own overlapped ReadFile/WriteFile with the
OVERLAPPED.hEvent low bit set (no completion packet) and waits on that event on
the calling thread; reads and writes stay concurrent, unlike a synchronous
handle. ScriptWorkerFrameCodec.BlockingStream picks it on Windows and the pipe
itself on Unix, so both ends now use the dedicated blocking reader + synchronous
combining writer on every OS (UseBlockingIo and the async read loops are gone).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.
DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The host handed the SDK a provider-less telemetry context, so SDK-side
refusals — identity-token decryption (the SDK 2 RsaEncryptedSecret clock
window, #873), certificate validation, secure-channel faults — left no server
log line (found on windev during the #800 Windows validation). The hosted
service now creates one ServerSdkTelemetry context from its ILoggerFactory
(Warning and above, SDK categories, rate-limited 20/min per category) and
shares it between OtOpcUaSdkServer and the ApplicationInstance; released on
dispose. Test: an untrusted client's rejection reaches the captured log.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Since #855 condition events are queued to folder event items under the node
manager Lock. At a full DiscardOldest queue the SDK's EventQueueHandler
dequeues the head of EventMonitoredItemQueue, a List.RemoveAt(0) shift of the
whole queue (~1 us/item at QueueSize 1000, ~10 us at 10000).
OtOpcUaSdkServer.CreateMonitoredItemQueueFactory now wraps the SDK factory in
RingEventQueueFactory: event queues are RingEventMonitoredItemQueue (O(1)
discard), data-change queues, durable requests and disposal go to the SDK
factory unchanged. A factory set through StandardServer.MonitoredItemQueueFactory
is used as-is.
The SDK queue is storage only (decompiled 2.0.1): DiscardOldest, overflow-event
positions and discarded/overflow counters live in EventQueueHandler, which is
unchanged. The ring reproduces QueueSize 0 initial + BadInternalError enqueue,
drop-oldest at Count == QueueSize, FIFO dequeue, SetQueueSize keep-newest /
keep-oldest, the newest-1000 duplicate window via IsEventRaisedFor, no-op
Dispose. RingEventMonitoredItemQueueParityTests drive both queues, and the SDK
EventQueueHandler over both factories, with the same random sequences.
FolderEventDeliveryPerfTests gains Ring_vs_sdk_event_queue (A/B in one process
via the internal OtOpcUaSdkServer.RingEventQueues switch).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The sweep threshold never rises above the 4096 cap (a threshold the count
could not reach stopped sweeping, and sharing, for good); at the cap a
sweep for dead shapes runs after every 256 refusals.
- The per-report "queued" mark is taken after the filter group resolves,
so a failure there leaves the item unmarked.
- The bucket key also carries TypeDefinitionId and IndexRange.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The filter-group registry holds groups weakly (the filter->group table is
the only strong reference), sweeps dead entries amortised and counts only
live shapes against the 4096 cap; filters past the cap are counted on the
new meter otopcua.opcua.condition_events.filter_groups_unshared.
- A group validates its where clause too (restored, never-validated filters
evaluate like validated ones).
- Test: value-equal filters cost one projection per locale set per report.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Filter-group buckets include the where clause's (value-based) hash, so
per-source filters sharing a select list are not all scanned under Lock.
- Locale keys are cached per session array.
- Tests: a notifier reached twice in one walk (Line above two wired
equipment) gets each event once on both paths (fails without the
per-report set); queued events re-projected by a modify are identical on
both paths; locale keys unambiguous and where clauses hash by value.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.
Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
handler returned is reported; one pending at actor stop is reported; a
confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
reconnect, UNACK_RTN-as-Clear and RPC-budget limits.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- RefreshDirectItemsWithCut keys its "decided off Lock" set per (item,
condition), so a condition one item's walk saw is still decided for another.
- The deferral warning no longer blames slow folder subscribers (no folder of
ours defers since #855).
- FolderEventFloodWireTests' removal test now asserts the folder item got
nothing and the Server object exactly the pre-stall event (it passed with
the post-stall liveness re-check deleted).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.
- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
(default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
after AcknowledgeAsync, off its mailbox and outside the resilience
pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Both plain ConditionRefresh paths (Server object and folder) skip a
condition a deploy removed or replaced between the off-Lock walk and the
queueing (the SDK base did not either; only the cut path checked).
- A failure delivering at one notifier hop is logged and the walk continues
to the remaining hops; a failure deciding the Server-object items withholds
only them.
- Test: a condition removed during a plain refresh (folder + Server object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay's verdict refuses a condition a deploy removed or replaced
since the cut (Server-object and folder), so the cut path cannot replay a
ghost or judge a replaced scripted alarm on its old folder.
- DeliverToNotifierItems resolves the #826 decision's item-independent part
(live condition, notifier check, scopes) once per hop; only the grant and
SDK role checks stay per item.
- A deferred delivery's drain starts on the pool, never inline under Lock.
- Test: a condition removed between the cut and its replay (folder + Server
object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review findings on 02503a4a (ConfigPublishCoordinator):
1 (HIGH) A recovered deployment whose every expected ack was already in
NodeDeploymentState (a seal whose SaveChanges failed, a fused node's own row)
was never completed: inside the deadline it waited for acks that had landed,
past it HandleDeadline's "race" guard returned without sealing — stuck in
AwaitingApplyAcks, refusing every later deploy. ApplyRecovery now completes
it (Seal / PartiallyFailed), and HandleDeadline completes a full ack set
instead of returning.
2 (MED-HIGH) The past-deadline recovery of a populated expected set ran
HandleDeadline inline before the stash replay, discarding on-time acks queued
during the outage. It is now self-sent, behind the replay. The zero-rows
past-deadline branch stays inline on purpose: no ack can count there, and a
self-sent deadline would time out a queued dispatch of that same row right
after publishing it.
3 (MED) A failed PersistNodeAck threw, restarted the actor and dropped the ack
(driver-only nodes never write their own row and do not resend). Acks are now
persisted before being counted, and a failed persist is re-queued on a timer.
The class remarks document that the Loading stash is node-local (lost on a
hand-over; queued dispatches are recovered from their rows).
Minor: MarkReady/Become(Ready) only after the recovery applied; a throw while
applying is a failed load (backoff) rather than a tight restart loop; a queued
dispatch matching the recovered Dispatching row logs Info, not the "lost
dispatch" Warning.
Tests (ConfigPublishCoordinatorDbOutageTests): fully-acked recovery seals /
partially fails inside and past the deadline (4 cases); queued ack applied
before a past-deadline recovery; failed ack persist retried. All 6 fail on
02503a4a.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay re-asks each item's verdict right before each entry's gate
hold (Server-object and folder cut paths): a grant revoked between the
pre-Lock verdicts and the replay no longer lets the replay through. A denied
entry is also removed from the cut's re-sent set.
- ReportEventWithoutBlocking logs and returns on a delivery that faulted
inline instead of booking it as a deferral episode (which ran the drain and
ClearStaleRefreshCuts under the event-report gate while Lock was held).
- OnConditionReportEvent isolates failures per Server-object item.
- Tests: revocation between cut and replay (folder + Server object), throwing
hop not booked as a deferral; production-overload sampling inside Lock.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
When the data starts inside an interval (no start bound) and its first value is
Uncertain, the frozen 1.5.378.182 AverageAggregateCalculator integrated the
uncovered span as a ramp from 0 and divided TimeAverage by the whole interval:
GetRegionsInValueSet (ignoreBadData, sloped) overwrites the BadNoData start
region's status with UncertainDataSubNormal in its one branch without a not-Bad
guard. Live AggregateFilter therefore disagreed with HistoryRead and Part 13
§5.4.3.6/§5.4.3.8 ("No Start Bound: Calculate Partial Interval"); QP: Total
188.96 vs 163.96, TimeAverage 9.448 vs 10.931. SDK 2.0.1 stock does the same.
Owner-approved 2026-11-11.
- New app-owned CoveredTimeAverageAggregateCalculator subclasses the baseline
and recomputes only TimeAverage/Total, restoring the Bad status of a region
that starts at a Bad point (only the start bound can). Only such an interval's
value changes; status, timestamps, type and bits are the baseline's. Stepped
and TreatUncertainAsBad=true never reached the defect. CompatibilityAggregate-
Factory routes exactly those two IDs; copied SDK files untouched.
- Golden test: 0 fixture vectors change (every data set has a usable value at
-10 s), asserted. New sweep over late-start variants of every data set (cut
+5 s / +25 s, first non-Bad sample Uncertain, all directions/stepping/configs/
incremental): 150 / 142 of 384 change, each pinned to a baseline-derived
oracle (untouched baseline's value with that sample Good, its status/bits on
the variant itself) - never our calculator. Plus a hand-computed case.
- Live-vs-HistoryRead parity (TimeWeightedAggregator) forward + reverse, no
masks, with discrimination against the pre-#821 baseline; QualityMix now
compares QP's TimeAverage/Total with the live oracle on both SQL backends;
booted-runtime facts test expects 163.96 / 10.93.
- README recorded decision, ServerAggregateConfiguration remarks, oracle
remarks, docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.
- PreStart now only subscribes and kicks off an async recovery load piped back
to the actor. A failure retries with exponential backoff (1 s doubling to
30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
stashes every message (DispatchDeployment logged as queued, ApplyAck too);
they are replayed in order after the recovered state is applied. Recovery
decisions (re-dispatch / time out) now run inline before the replay instead
of via self-tells, and a duplicate dispatch of the in-flight deployment with
a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
"config-publish" ready-tier health check: Degraded while loading (with the
attempt count and last error), Healthy when ready or not hosted here.
Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events reaching one of our notifiers no longer go through the SDK
MonitoredNode2 channel: the walk serves the attached items itself, per item and
per event (audit guard, uncached INodeManager3 verdict, item gate/cut,
QueueEvent), as the Server-object path does. No notifier of ours defers any
more; the #838 deferral queue and the #854 cut stay as the safety net and
folder items share the cut (RefreshDirectItems, flag on and off). An event
whose walk suspends is marked deferred before it suspends.
Tests produce deferral with a notifier-stall probe instead of holding a folder
item's SDK lock (which would now block the reporter). New
FolderConditionRefreshWireTests; design note in docs/plans.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The frozen 1.5.378.182 StatusAggregateCalculator counts Uncertain regions as
Good under TreatUncertainAsBad=false, so live AggregateFilter PercentGood /
DurationGood disagreed with HistoryRead (Part 13 §5.4.3.31/§5.4.3.33, Mantis
0006166) whenever Uncertain data was present. Owner-approved 2026-11-11.
- New app-owned GoodOnlyStatusAggregateCalculator subclasses the baseline and
replaces only the PercentGood/DurationGood computation (Good regions only);
CompatibilityAggregateFactory routes exactly those two IDs to it. The copied
SDK files are untouched; the other 35 aggregates are byte-identical.
- Golden test: the 64 changed vectors (38 PercentGood + 26 DurationGood) are
exempted exactly — those whose baseline output differs from the same case's
baseline TreatUncertainAsBad=true twin — and pinned to that twin (baseline
evidence, not regenerated from the implementation); plus a full Good-only
pin over all 352 PercentGood/DurationGood vectors and a hand-computed case.
- Live-vs-history parity over Good/Uncertain/Bad mixes (no masks), with a
discrimination check against the pre-#809 baseline; the booted-runtime facts
test now expects 8000 ms / 40 %.
- SdkAggregateOracle: Run/RunLive = the registered factory; RunSdkAsIs = the
untouched baseline; its Part13GoodCalculator duplicate is removed.
- README (recorded decision), ServerAggregateConfiguration remarks,
docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11: /fleet/diagnostics "Faulted devices" counts DISTINCT
devices across the nodes of the selected node's cluster, Faulted state only.
- DriverHostActor: FaultedDevices = HostState.Faulted only (Stopped no longer
counted); new StoppedDevices + FaultedHostNames. Counting extracted to the
testable DiagnosticsFromHealth.
- DriverInstanceDiagnostics: StoppedDevices (default 0) and FaultedHostNames
(default null) are additive with defaults, so a pre-#786 sender's snapshot
still deserializes over Akka's default JSON serializer (pinned by a test).
- FleetDiagnosticsAggregation (pure): identity key = (driver-instance string
id, HostName), case-insensitive; a single-host driver / nameless pre-#786
count is combined by MAX across nodes, never summed. Cluster peers come from
enabled ClusterNode rows of the selected node's ClusterId.
- FleetDiagnostics.razor: tile fetches the cluster peers and shows the
distinct count + "n of m nodes answered"; per-driver table gains a Stopped
devices column.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11: certificateValidation.validateServerHostName stays
opt-in (unset = off). The AdminUI toggle moves from the "Subscriptions, limits &
discovery" panel to the Security panel beside mode/policy, with help text on the
NAT/IP caveat, IPv6 zone ids and the loose first-label match. Binding unchanged
(tri-state, "Default (off)" leaves the key absent). OpcUaClient.md states the
decision; drivers rule updated; source-scan test pins placement + caveats.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The Controls panel primed its per-node view from the snapshot store with
ReceivedUtc = now, so a node that stopped publishing before the dialog
opened looked live for the first 30 s stale window.
InMemoryDriverStatusSnapshotStore now records each snapshot's LOCAL
receive time (TimeProvider-injectable) and exposes it via
GetAllWithReceiveTime(); the panel primes through the pure seam
DriverStatusByNode.Prime/PrimedReceiveTime, which keeps the store's
receive time (clamped to now; still skew-proof). A store that does not
track one (default interface impl -> null) falls back to the publish
time only when that is already older than the stale window, else now.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review of 1622d07f:
1. Perf: every i=2253 refresh ran the whole notifier walk + permission checks +
QueueEvent under Lock. Now, with nothing deferred (the common case), the refresh is
the SDK-shaped walk off Lock (Lock only to copy the root notifiers), kept only if no
deferral episode existed at any moment of it (count 0 before and after, episode
unchanged); otherwise the cut path also runs and its later replay supersedes. The
cut path takes ACL/role verdicts off Lock, then one short Lock hold: watermark +
frozen snapshots of Retain conditions from _alarmConditions (no tree walk) + per-item
cut and "replay pending" marker; the replay is queued OFF Lock, one event per hold of
the item gate. While pending, a condition event newer than the cut marks its
condition so the older frozen entry is dropped rather than following it. A
concurrent fast-path refresh cancels a pending cut replay of the same item.
Measured, 3000 retained conditions, longest Lock wait seen by another thread during
one refresh: 1622d07f 95-106 ms idle / 102-127 ms deferring; now 0-3 ms idle /
23-32 ms deferring.
2. ACL: a retained condition omitted from the replay for want of a grant is never
withheld (RefreshCut.RetainedAtCut); only Retain=true events of a condition omitted
because it was NOT retained are.
3. One gate/cut state per item for its lifetime (ConditionalWeakTable keyed by the
item): an unsubscribe + resubscribe between refresh and drain keeps the cut.
4. Cuts carry their deferral episode; ClearStaleRefreshCuts runs when the count
returns to zero, and a stale-episode cut is inert.
5. Dispose guard for the custom refresh (ObjectDisposedException once disposing;
Dispose waits for refreshes in flight).
6. Docs: a refresh waits at most for one in-progress QueueEvent on the item.
Tests: resubscribe variants, ACL-grant-lands test, pending-replay probe test,
dispose test, 3000-condition Lock-hold measurement with a probe pinning that neither
path queues its replay under Lock; stress invariant tightened (never backwards outside a
refresh; each refresh ends on the newest delivered state) and tolerant of the SDK's own
BadRefreshInProgress overlap refusal. Each new test fails against its targeted mutation.
While a folder subscriber's SDK channel is full, a condition's later events wait in
its #838 deferral queue and reach i=2253 items only when the drain gets to them, off
Lock. A ConditionRefresh in that window replayed the current state, then the older
deferred events followed RefreshEnd (stale state, or a condition the refresh had just
dropped re-added until the newest drained).
- Condition events are reported as ConditionEventSnapshot (an InstanceStateSnapshot
carrying a report sequence, condition NodeId and Retain); events queued behind a
deferred report are marked Deferred.
- Server-object refresh (ConditionRefresh and ConditionRefresh2, flag on AND off) now
runs in RefreshServerObjectItems: watermark, "anything deferred?", the replay and its
queueing in ONE Lock hold, so the replay is exactly the state at the cut. Per item it
queues the replay and installs a RefreshCut under the item's gate.
- OnConditionReportEvent holds that gate across "superseded? -> QueueEvent" for
deferred events only. Withheld from that item alone: events at/below the cut whose
condition the replay re-sent, or Retain=true events of a condition it omitted; an
older Retain=false event of an omitted condition is still delivered. Queues, other
Server-object items and folder items are untouched; refresh never waits for a drain.
- Lock order Lock -> item gate -> SDK item lock, same as a synchronous report.
- Meter otopcua.opcua.condition_events.superseded.
- Folder items keep a documented residual (SDK channel backlog + memoised per-item
verdict leave no per-event hook).
Tests: ConditionRefreshDeferralWireTests (deterministic overload by holding the folder
item's SDK lock; 6 variants fail before the fix; probe-driven gate test fails with the
gate removed; concurrent reports/refreshes/stall stress test fails before the fix).
The deploy gate's merged-config path check is now one helper,
CredentialReferenceDeployChecks.MergedOnlyProblems (problems the runtime merge
reveals that the per-config checks do not), shared by the gate and
CredentialReferencePreflight.Scan - so an AuthType switch in one config and a
UserCertificatePath in the other is listed as a Draft finding '(merged with its
devices)' on the banner before a deploy refuses it.
Co-Authored-By: Claude <noreply@anthropic.com>
RawTreeService.CreateDriverAsync / RenameDriverAsync / UpdateDriverAsync run
DriverConfigSaveGate.ValidateNameAsync -> DriverInstanceIdentities.DescribeNameChangeProblemsAsync:
- claim: the new name may not be the instance part (secret: segment or env: part)
of a reference STORED on another same-type driver's DriverConfig or DeviceConfig -
naming a driver 'gw' used to fail every sharer's next deploy;
- pin: a driver may not be renamed away from a name its own references are named
after (unless they also name its id or the new name) - the rename would free that
name for every other driver's save / Test Connect / browse. Re-pointing to the id
in the same config-modal save is allowed; a case-only rename is never refused.
Messages name the other driver, id, device and field path, never a value. The
driver + device reads are pinned on SQL Server by a [SqlServerFact].
Co-Authored-By: Claude <noreply@anthropic.com>
env:<PREFIX><INSTANCE>_... is the per-instance convention for every prefix of a
driver's scope (OTOPCUA_GALAXY_, GALAXY_MXGW_, OTOPCUA_OPCUACLIENT_,
OTOPCUA_HEIDENHAIN_), INSTANCE = a driver id or name upper-cased with every other
character as '_'. A variable whose part after the prefix equals or starts with
(at a '_' boundary) ANOTHER same-type instance's id or name is refused with
CredentialReferenceOtherInstance at save, deploy, Test Connect and browse - the
same DriverCredentialPolicy.DescribeOtherInstanceProblems the secret: rule uses
(now over CollectInstanceReferences / ReferenceNames). Another-instance matches
win over the owner's, as for secrets. Shared variables that name no instance
(GALAXY_MXGW_API_KEY) keep working. HasInstanceSegments now also gates on env:,
so an env:-only browse needs the ConfigDb (fail-closed without it).
Co-Authored-By: Claude <noreply@anthropic.com>
- AdminUI MTConnect driver form: "Comms-loss confirm (ms)" in the Reconnect section, always written
(0 = first refusal, negative = off is never dropped), case-insensitive load, unknown keys preserved,
advisory at >= 90000 ms (the watchdog fires first). Form output byte-pinned and parsed through
MTConnectDriver.ParseOptions on the driver side.
- Driver: a confirmed refused-connection outage now annotates every told CONDITION quality-only
BadNoCommunication (state/ack kept, once per outage, UNAVAILABLE-Bad skipped); the first delivery
afterwards re-diffs all subscribed conditions so unchanged ones get a quality-only Good.
- docker-dev/docker-compose.mtconnect-gate.yml: throwaway Agent in its adapter's network namespace so a
stop is a genuine ConnectionRefused (a plain stopped container is a DNS failure = ambiguous).
- Docs: MTConnect.md comms-loss section + live-gate recipe, docker-dev README, drivers.md rule.
Refs #852
DriverStatusPanel (the /hosts Controls dialog) showed one snapshot per driver - the most
recently published across both nodes of the redundant pair - so a driver Faulted on one node
and Healthy on its peer flipped on every publish. It now primes from the store's GetAll() and
keeps one entry per ReportingNodeId (DriverStatusByNode): per-node state chip in the header,
per-node last-success / 5-min errors / last error, per-node stale dimming. A snapshot with no
node id (pre-#847 publisher) renders unlabelled, sorted last. A primed copy older than a racing
live publish for the same node is dropped.
Joseph Doherty