Both builds served HistoryRead from the real AVEVA Historian via the DEV
HistorianGateway (vd03, ea9f21c): 225 cases / 3,855 rows / events, 0 differences.
Raw also matches direct SQL on Runtime.dbo.History (6/6) and histgw-cli; simple
aggregates match a recomputation from raw (336/336). Modified, HistoryUpdate,
seven refused aggregates and HistoryServerCapabilities answer as before. No write
reached the historian. Findings (pre-existing, not SDK): all HistoryServerCapabilities
booleans false; AVEVA attribute-alarm events unreachable per source.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Profiled the remaining lmxopcua#789 misses on windev, a VMware VM, using per-thread CPU and context-switch accounting
plus ETW CPU sampling.
- One blocking hand-off costs ~26 us of latency and ~18 us of CPU there, about 10x macOS.
- An idle evaluation makes four hand-offs in a row: host writer -> worker reader -> evaluation thread -> host reader
-> dispatcher.
- 48% of host CPU at 10k/s is in the kernel: the per-RunAsync NtWriteFile, thread-pool wakes, reads and waits.
Spin-before-block was measured at every hand-off and is not kept. It meets 150 us idle p50 (118 us) only by keeping
about two host cores and four worker cores busy for one stream. At 10k/s it reads +250 points.
Harness: new "paced 10,000/s, batched" row. It uses the same rate and keys, with each 1 ms tick submitted as one
RunBatch call, which is the production seams' shape. It runs after the original four phases, so their rows stay
comparable. On windev it halves host CPU, to +36..+45 points against +71..+84 unbatched. That is still above 25.
Design doc:
- 12.8: the record.
- 14 #15: the owner's decision on the Windows budgets. Recommended: measure on bare-metal Windows, then state the two
budgets per platform, with the batched row as the gate. The alternative is a shared-memory transport.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
OpcUaApplicationHost left ServerConfiguration.MaxRegistrationInterval at the
SDK default (30000) with a null RegistrationEndpoint, so SDK 2.0.1's
StandardServer armed its registration timer against opc.tcp://localhost:4840,
which on an OtOpcUa node is the server itself (no LDS). It answered
BadServiceUnsupported and the SDK retried forever, visible since #874.
Registration is now opt-in: ApplyDiscoveryRegistration sets
MaxRegistrationInterval = 0 (the value that keeps StandardServer from arming
the timer) unless OpcUa:DiscoveryServerUrl is set, in which case the server
registers against that URL every OpcUa:DiscoveryRegistrationIntervalMs
(default 30000) over SignAndEncrypt/Basic256Sha256, like the SDK's own
default endpoint. The validator requires an absolute opc.tcp:// URL and a
positive interval when the URL is set. Documented in Configuration.md and
OpcUaServer.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Measured on windev (3 alternated sessions) and macOS: a pool resume in place
of the dispatcher, and an inline-on-reader upper bound, with the harness as
committed and with its paced continuation made synchronous.
- pool: 10k/s host CPU unchanged, idle p50 and idle CPU worse (macOS idle CPU
x3 from pool spin); only the closed loop gains. Not kept.
- inline (unsafe for every seam; the ceiling for a VirtualTag-only opt-in):
~8 points of a core at 10k/s raw and ~50 us idle p50 on windev with a cheap
continuation, nothing with the harness as committed. Both budgets still
missed by a wide margin, so the opt-in was not built.
No code change. Records the remaining attribution (harness pool hop ~15
points raw; reader wake + syscalls; worker side untouched).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.
DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PASS: gauge baseline 2, +2 with the rig's driver on both centrals, 4 after 20
config deploys (40 respawns; rig opened 39 / closed 37). Final drop after
removal not observed: the Mac's VPN to VD03 dropped right after the settle.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Read-only ConfigDb preflight (scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql)
covering every deploy gate added since the deployed build: VirtualTag/Calculation effective
triggers (#783), VT->VT reads and VirtualTagDependencyCycle (#791), newly forbidden script
APIs (#792, E2), credential-reference namespace / other-instance / reserved-name rules
(#815#835#836#848#849#850), MTConnectMultipleDevices, Sql query/write gates (#781),
Heidenhain, UNS effective-name collisions and the FOCAS renames/value changes. Verified on a
throwaway b8b4c18f-schema database (every check fires on seeded rows) and clean against the
docker-dev config.
Upgrade notes (docs/deploy/vd03-upgrade-b8b4c18f-to-master.md): the EnableSnapshotIsolation
migration and how to apply it by hand, the env-var audit (Security:ApiKeys:Pepper is now
required and absent on the box; JWT key validated at start), SDK 2.0.1 / CA KeyUsage, the
ScadaBridge-visible changes, swap/rollback and post-deploy checks.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.
Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
handler returned is reported; one pending at actor stop is reported; a
confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
reconnect, UNACK_RTN-as-Clear and RPC-budget limits.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.
- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
(default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
after AcknowledgeAsync, off its mailbox and outside the resilience
pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay's verdict refuses a condition a deploy removed or replaced
since the cut (Server-object and folder), so the cut path cannot replay a
ghost or judge a replaced scripted alarm on its old folder.
- DeliverToNotifierItems resolves the #826 decision's item-independent part
(live condition, notifier check, scopes) once per hop; only the grant and
SDK role checks stay per item.
- A deferred delivery's drain starts on the pool, never inline under Lock.
- Test: a condition removed between the cut and its replay (folder + Server
object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
§12.1 gains the 2026-11-11 windev (VMware VM) perf table: idle p50 581 us
(MISS), kill latency after T+G 8-16 ms (PASS), kill both shards 2.12-2.14 s
(MISS by ~6 %), worker RSS 160 MB compiled / 36 MB warm-cache (PASS under the
revised 200 MB budget), host CPU +131 % vs InProcess (MISS, untuned).
It also records the first Windows run of Category=ScriptWorker: 107/0/2 from
C:\build, all 13 WorkerSoftClockTests cases (5 freezing a real worker with
NtSuspendProcess) green with no code change. From a user-profile directory
the restricted-identity isolation test fails because the restricted token
cannot read the worker dll there (profile ACL), which is environment, not a
defect.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay re-asks each item's verdict right before each entry's gate
hold (Server-object and folder cut paths): a grant revoked between the
pre-Lock verdicts and the replay no longer lets the replay through. A denied
entry is also removed from the cut's re-sent set.
- ReportEventWithoutBlocking logs and returns on a delivery that faulted
inline instead of booking it as a deferral episode (which ran the drain and
ClearStaleRefreshCuts under the event-report gate while Lock was held).
- OnConditionReportEvent isolates failures per Server-object item.
- Tests: revocation between cut and replay (folder + Server object), throwing
hop not booked as a deferral; production-overload sampling inside Lock.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
When the data starts inside an interval (no start bound) and its first value is
Uncertain, the frozen 1.5.378.182 AverageAggregateCalculator integrated the
uncovered span as a ramp from 0 and divided TimeAverage by the whole interval:
GetRegionsInValueSet (ignoreBadData, sloped) overwrites the BadNoData start
region's status with UncertainDataSubNormal in its one branch without a not-Bad
guard. Live AggregateFilter therefore disagreed with HistoryRead and Part 13
§5.4.3.6/§5.4.3.8 ("No Start Bound: Calculate Partial Interval"); QP: Total
188.96 vs 163.96, TimeAverage 9.448 vs 10.931. SDK 2.0.1 stock does the same.
Owner-approved 2026-11-11.
- New app-owned CoveredTimeAverageAggregateCalculator subclasses the baseline
and recomputes only TimeAverage/Total, restoring the Bad status of a region
that starts at a Bad point (only the start bound can). Only such an interval's
value changes; status, timestamps, type and bits are the baseline's. Stepped
and TreatUncertainAsBad=true never reached the defect. CompatibilityAggregate-
Factory routes exactly those two IDs; copied SDK files untouched.
- Golden test: 0 fixture vectors change (every data set has a usable value at
-10 s), asserted. New sweep over late-start variants of every data set (cut
+5 s / +25 s, first non-Bad sample Uncertain, all directions/stepping/configs/
incremental): 150 / 142 of 384 change, each pinned to a baseline-derived
oracle (untouched baseline's value with that sample Good, its status/bits on
the variant itself) - never our calculator. Plus a hand-computed case.
- Live-vs-HistoryRead parity (TimeWeightedAggregator) forward + reverse, no
masks, with discrimination against the pre-#821 baseline; QualityMix now
compares QP's TimeAverage/Total with the live oracle on both SQL backends;
booted-runtime facts test expects 163.96 / 10.93.
- README recorded decision, ServerAggregateConfiguration remarks, oracle
remarks, docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events reaching one of our notifiers no longer go through the SDK
MonitoredNode2 channel: the walk serves the attached items itself, per item and
per event (audit guard, uncached INodeManager3 verdict, item gate/cut,
QueueEvent), as the Server-object path does. No notifier of ours defers any
more; the #838 deferral queue and the #854 cut stay as the safety net and
folder items share the cut (RefreshDirectItems, flag on and off). An event
whose walk suspends is marked deferred before it suspends.
Tests produce deferral with a notifier-stall probe instead of holding a folder
item's SDK lock (which would now block the reporter). New
FolderConditionRefreshWireTests; design note in docs/plans.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
TwinCATDriverProbe ran a first-phase TCP check against the AmsNetId's
first four octets on the AMS port (801/851). An AmsNetId is not an IP and
the AMS port is not a TCP port, so Test Connect timed out against every
real runtime (Z35474: dialled 5.118.56.93, an unrelated public host).
The probe now opens no socket of its own: it takes the driver's path,
AdsClient.Connect + ReadStateAsync through the local AMS router, bounded
by the probe budget (Task.Run + WaitAsync), and classifies failures as
no local AMS router / no route or target unreachable / timeout / other.
The degrade-to-green 'TCP reachability only' outcome is gone - with no
TCP check nothing was contacted, so a missing router is reported red.
Tests: the regression test listens on the AmsNetId-derived endpoint and
asserts no connection arrives (fails on the old probe); classification
and budget tests drive the handshake through an internal seam.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The owner revised the per-shard worker idle RSS budget from 80 MB to 200 MB
+ scripts on 2026-11-11: compiles run in the worker, so Roslyn's code and
metadata stay resident (measured 159-187 MB) and no GC setting moves it.
The perf harness now reports both RSS rows against the 200 MB constant with
a PASS/MISS mark; A-3 records what the measurement showed.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The frozen 1.5.378.182 StatusAggregateCalculator counts Uncertain regions as
Good under TreatUncertainAsBad=false, so live AggregateFilter PercentGood /
DurationGood disagreed with HistoryRead (Part 13 §5.4.3.31/§5.4.3.33, Mantis
0006166) whenever Uncertain data was present. Owner-approved 2026-11-11.
- New app-owned GoodOnlyStatusAggregateCalculator subclasses the baseline and
replaces only the PercentGood/DurationGood computation (Good regions only);
CompatibilityAggregateFactory routes exactly those two IDs to it. The copied
SDK files are untouched; the other 35 aggregates are byte-identical.
- Golden test: the 64 changed vectors (38 PercentGood + 26 DurationGood) are
exempted exactly — those whose baseline output differs from the same case's
baseline TreatUncertainAsBad=true twin — and pinned to that twin (baseline
evidence, not regenerated from the implementation); plus a full Good-only
pin over all 352 PercentGood/DurationGood vectors and a hand-computed case.
- Live-vs-history parity over Good/Uncertain/Bad mixes (no masks), with a
discrimination check against the pre-#809 baseline; the booted-runtime facts
test now expects 8000 ms / 40 %.
- SdkAggregateOracle: Run/RunLive = the registered factory; RunSdkAsIs = the
untouched baseline; its Part13GoodCalculator duplicate is removed.
- README (recorded decision), ServerAggregateConfiguration remarks,
docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11: certificateValidation.validateServerHostName stays
opt-in (unset = off). The AdminUI toggle moves from the "Subscriptions, limits &
discovery" panel to the Security panel beside mode/policy, with help text on the
NAT/IP caveat, IPv6 zone ids and the loose first-label match. Binding unchanged
(tri-state, "Default (off)" leaves the key absent). OpcUaClient.md states the
decision; drivers rule updated; source-scan test pins placement + caveats.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11 (Java parity): a genuinely empty observation on a
String/Reference or unauthored item is Good "" instead of BadNoCommunication.
Non-string items (numeric, bool, DateTime, arrays) keep BadNoCommunication -
the Agent supplied no value; BadTypeMismatch would wrongly blame the authored
type. The UNAVAILABLE sentinel is unchanged, and a structured observation with
no text stays BadNotSupported (structured check now precedes the empty check).
Parity matrix, quality-mapping section and drivers rule updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
With Reconnect.ReplayOnSessionLost=false a recreated gateway session carries none of
the old advises, so every pre-reconnect subscription went silently deaf: nodes kept
their last value as Good and health returned to Healthy (only a Warning log).
ReplayAsync now records the stranded subscriptions and publishes BadNoCommunication
(StatusCodeMap.BadNoCommunication, 0x80310000; value null — no driver value cache)
for every tag of each, straight to OnDataChange so LastSuccessfulRead is not stamped.
GetHealth reports Degraded naming the setting while any stranded subscription is
still registered; it clears on UnsubscribeAsync (the host's re-subscribe path) or
runtime teardown, and pruned ids never pin it. A subscribe after the reconnect is an
ordinary new-session subscription. With nothing else flowing, the comms-loss
watchdog re-initializes the driver after 90 s (a delayed replay).
Tests: Runtime/StrandedSubscriptionTests (6). Docs: drivers/Galaxy.md, which also
no longer claims the replay uses the gateway's ReplaySubscriptions RPC.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
B-1: the subscribe-once read's cleanup unsubscribes what the registry holds at
removal (the replay may have rebound it to new-session handles), not its
original bindings; nothing when the entry still holds dead old-session
handles. Found while testing it: reads were attached to the pump INSTANCE, so
a read across a reconnect's pump restart never saw the replayed value — reads
now listen on a driver-level ReadDataChange raised for whichever pump is live.
I-A: PerPlatformProbeWatcher records the registry session with each probe
handle. Publishing goes through the new fenced TryUpsert and binds only
current-session handles; removal/stop unsubscribe and purge only those; a
seed reply that straddled a reconnect records nothing and is re-issued (max 3).
I-B: PurgeParked takes the session captured BEFORE the unsubscribe RPC and is
a no-op under the park gate if the session changed.
I-C: ReopenAsync starts a new registry session before AND after the gateway
session swap; ReplayAsync does so only when no reopen ran (injected subscriber
/ tests) and replays only subscriptions still holding older-session handles,
so a subscribe issued after the swap is live and not double-advised.
P-1: ReplayAsync restarts the pump before any early return (no entries used to
leave the dead pump in place). P-2: ReplayOnSessionLost=false logs a Warning;
misleading comment about gateway-side replay fixed.
M-a: TryRebind returns BindOutcome; UnknownId releases the handles just
advised for a subscription removed mid-replay. M-b: the replay/re-snapshot
pass reads the session once and stops if it changes. M-c: parked entries drop
their MxEvent when they leave the parked set. M-d (retry a straddled read)
not done.
Tests (TestWait) for each, each checked red against a mutation of its fix.
Refs #859, #861.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
I-2: unclaimed parked events are trimmed by window sequence. Each window gets a
monotonically increasing sequence at Begin, each parked event is stamped with
the newest one, and on every End events stamped before the oldest open
window's sequence are dropped (amortised O(1) via a park-order queue). Under
concurrent reads the open count rarely hit 0, so trailing events used to pile
up to the bound and push genuine early values into overflow.
I-1 / #861: SubscriptionRegistry.StartNewSession() — called FIRST in
ReplayAsync on every reconnect — clears held events, invalidates the reverse
map (subscriptions kept) and bumps a session number. New-session events whose
number an old binding still held now park until the replay's per-subscription
Rebind promotes them, instead of resolving to the wrong tag. TryRegister /
TryRebind refuse a stale session atomically under the park gate: a
SubscribeAsync whose reply straddled a reconnect is re-issued on the new
session (max 3 attempts), a subscribe-once read returns BadCommunicationError,
ResubscribeAllAsync stops if the session changes again. Old-session handle
numbers are never unsubscribed (Remove reports whether the entry belongs to
the current session; the read's late-reply path checks too), and an old
session's pump can no longer resolve through the map (fast path checks the
session).
I-3 (interim): a completed UnsubscribeBulk purges events parked for its
handles (driver + probe watcher). Full fix needs a worker sequence on the
SubscribeBulk reply (mxaccessgw, optional) — documented.
M-1: a resolved handle with a stranded parked list promotes it and resolves
the live event (never parks it). M-2: ResolveOrPark returns the ready events
to replay first, one gate acquisition per event while anything is held.
M-3: SubscribeAsync/read dispose the window before the orphan-release RPC.
M-5: ResolveOrPark remarks document the single-dispatching-pump invariant.
Tests (TestWait): overlapping windows don't accumulate, purge, stranded list,
session fence + map invalidation, stale-session registration refused,
end-to-end replay with swapped handle numbers and event-before-reply,
subscribe and read straddling a reconnect, read deadline closes its window.
Each was checked red against a mutation of the fix it pins.
Refs #859, #861.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1. ReadAsync (subscribe-once) now opens the early-event window around its
SubscribeBulk + registration; an initial value that overtook the reply was
dropped and the read returned BadTimeout.
2. Register/Rebind write the reverse map and promote parked events in ONE
critical section under the park gate, and ResolveOrPark resolves under that
gate whenever anything is held (lock-free fast path only when nothing is),
so a later live event can no longer overtake the parked initial value. A
resolved handle that still holds a parked list queues behind it.
3. Held events are fenced by a session epoch: each new EventPump calls
StartNewEpoch(), discarding parked AND promoted-but-unreplayed events; an
older pump can neither park nor replay. Item-handle reuse across a reconnect
can no longer put an old-session value on a new tag.
4. ResubscribeAllAsync opens one window per subscription, not across the whole
replay.
5. Promoted-but-unreplayed (ready) events count toward MaxParkedEvents.
6. Begin/End of a window run under the park gate, so N->0 discard and 0->1
open are atomic (no inherited leftovers).
Documented that ParkedEventsReady may run under _registryGate/_syncLock and
relies on the channel's AllowSynchronousContinuations=false.
Tests use TestWait (#858); each new test was checked red against a mutation
of the fix it pins.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw gives no reply-before-event ordering: the worker completes the STA
command via a RunContinuationsAsynchronously TCS, so the reply is framed on a
thread-pool continuation while the STA already pumps the OnDataChange Advise
queued (drained/framed by a separate loop), and the reply (unary Invoke) and
the event (StreamEvents) are separate HTTP/2 streams. The driver then also
registers handles concurrently with the pump's dispatch loop. MXAccess's
initial value could therefore be dropped as "stale after unsubscribe",
leaving the node at its pre-subscribe value until the next change.
SubscriptionRegistry gains an early-event window: every SubscribeBulk call
site (SubscribeAsync, ResubscribeAllAsync, PerPlatformProbeWatcher) opens
BeginPendingSubscribe() before the RPC and closes it after registering.
Inside the window unknown-handle events are parked (bounded, 100k); the
Register/Rebind that binds a handle promotes them, and the EventPump replays
them on its single dispatch thread before any later event for that handle
(woken by a sentinel when no further event arrives). Closing the last window
discards unclaimed events, so the stale drop is unchanged outside it.
New counters galaxy.events.early_{parked,replayed,overflow}.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Joseph Doherty