ZbTelemetryOptions.Meters/ActivitySources are strict allowlists. The host named only
ZB.MOM.WW.OtOpcUa and the LocalDb meter, so the in-process Galaxy driver's galaxy.* spans
and its event pump's galaxy.events.* counters (both on "ZB.MOM.WW.OtOpcUa.Driver.Galaxy")
were silently dropped from /metrics and OTLP. Found exercising OTLP end to end against a
collector (scadaproj#13).
- Galaxy driver: public GalaxyDriverTelemetryNames (MeterName / ActivitySourceName);
GalaxyTelemetry and EventPump reference it.
- Host: ObservabilityExtensions.ObservedMeters / ObservedActivitySources include it.
- Host.IntegrationTests: ObservabilityAllowlistTests assert on the listener side (span
sampled, instrument enabled) with an unregistered-name negative control; dropping the
Galaxy entries fails 3 of 8.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.LocalDb and .Replication at 0.3.7 (published 2026-11-12).
Wire-compatible; no code change needed.
- 0.3.6 (scadaproj#36): the library sync service enforces
LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
its refuse-every-stream-when-blank rule; doc comment updated. docker-dev
site-a pair: both nodes now read one variable,
OTOPCUA_SITE_A_LOCALDB_SYNC_KEY (same dev default as before), so an
override can never split the pair.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
node offline past TombstoneRetention no longer brings a pruned
deployment back. The ledger grows while a peer is down; optional
TombstoneHardCap (documented in Configuration.md) bounds it.
Runbook, Configuration.md and the cluster-topology rule updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
GalaxyAlarmPriority, the one table live Galaxy alarms (MxAccessSeverityMapper)
and AVEVA A&E history (EventMapper, #886) map the AVEVA priority through, ranked
a higher number as more severe. AVEVA ranks priority 1 as the most urgent, so a
priority-1 alarm read 250 and a priority-999 system event 1000.
It now uses AVEVA's default priority ranges, most urgent first:
AVEVA priority AVEVA severity bucket OPC UA Severity
1-250 1 Critical Critical 1000
251-500 2 High High 750
501-750 3 Medium Medium 500
751-999 4 Low Low 250
(history only) 0 not an alarm - 1
Out-of-range priorities clamp (<= 0 Critical, > 999 Low). Evidence, vd03
Runtime.dbo.Events read-only: priority 1/100 alarms carry AVEVA severity 1,
400/500 severity 2, 600/700 severity 3, and every priority-999 System.* /
User.* row severity 0 with IsAlarm 0.
- Live and history now use the same number. GalaxyDriver passes the table's
exact value as AlarmEventArgs.OpcUaSeverity. Before, it passed only the bucket,
which NativeAlarmProjector re-mapped through its generic 200/500/700/900
ladder, so live and history disagreed even before the direction fix.
- History: a row whose AVEVA severity is 0 (System.Start/Stop/Deploy,
User.Write...) is not an alarm and reads Severity 1, below every alarm band.
For alarms AVEVA's 1-4 severity is still never the value: the live feed
carries only the priority, so it cannot use it, and history must agree.
- Docs: docs/Historian.md gains "Galaxy alarm priority -> Severity
(lmxopcua#887)"; docs/AlarmTracking.md updated.
- Live re-run of the #801/#886 harness against vd03 HistorianGateway
ea9f21c (read-only, network-only): 224 of 233 cases identical to the #886
run, the 9 others differ only in Severity (priority-700 alarm rows 750 ->
500, 19 system/user events 1000 -> 1). Record: aveva-history-801.md
"#887 re-run", report aveva-gate-sdk2-887.json.gz.
Client-visible: every OPC UA client that ranks or filters Galaxy alarms by
Severity sees the order reversed (live and HistoryReadEvents). ScadaBridge has
no code that depends on the old values; its own mixed severity directions are
ScadaBridge#129.
Tests updated because they pinned the inverted table:
- Driver.Galaxy.Tests MxAccessSeverityMapperTests (rewritten: edges 1/250/251/
500/501/750/751/999, 0/-100/int.MinValue/1000/int.MaxValue clamps,
monotonic-decreasing sweep, equals GalaxyAlarmPriority for -2..1001)
- Driver.Galaxy.Tests GatewayGalaxyAlarmFeedTests (750 and 600 -> Medium)
- Driver.Historian.Gateway.Tests EventMapperTests (band table, priority 999
-> 250, 1 -> 1000; system events -> 1)
- OpcUaServer.IntegrationTests GalaxyEquipmentEventHistoryWireTests (700 ->
500, 100 -> 1000, System.Stop -> 1; the fake now stores AVEVA's real 0-4
severity)
New tests:
- Runtime.Tests NativeAlarmRestatementEndToEndTests
.Galaxy_live_condition_severity_is_the_AVEVA_priority_band (8 rows: the real
feed -> real GalaxyDriver -> host -> projector; the same rows EventMapperTests
pins on the history path)
- EventMapperTests.Aveva_severity_zero_marks_an_event_that_is_not_an_alarm
(5 rows, incl. the priority-999 alarm positive control)
- GalaxyDriverAlarmSourceTests: OpcUaSeverity passes through
Mutation check, 8 mutants, all killed: the pre-#887 inverted table; the
driver dropping OpcUaSeverity; no not-an-alarm rule; each of the three band
edges off by one; history ignoring priority; a missing AVEVA severity read
as 0.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK memoizes INodeManager3.ValidateEventRolePermissions per (item, event type,
SourceNode). #826 kept it fresh by reflecting into the private
MonitoredNode2.InvalidatePermissionCaches (plus a detach/re-attach fallback). Since #855
condition events never take that channel, and the search for #878 found nothing
production-raised that does except a condition's SDK method audits, which reach it only
with auditing on, SignAndEncrypt and a SecurityAdmin session (nothing grants that role).
Under enforcement the channel verdict is now a flat deny for items on our nodes, so no
cached verdict depends on a grant, and InvalidateFolderEventVerdicts, the reflection,
the fallback, its counter and the guard test are removed. Method audits still reach
Server-object items, decided per event.
Tests: FolderEventVerdictInvalidationTests -> FolderEventVerdictFreshnessTests; new
revoked-grant test (positive control: a still-granted session) and method-audit test
(SecurityAdmin mapped for the test; positive controls: Server-object item, flag off);
the plain-event test runs each mode on a fresh server. Mutation-checked.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A notifier folder read one event source, its NodeId identifier (a RawPath or an EquipmentId). AVEVA stores a
Galaxy object's A&E history under the object's tag_name (Source_Object), so AVEVA's own alarm and event
history reached no production folder. Its lower-case property keys were also missed, so every row read
Severity 1 with no ConditionName.
Sources. A folder's event history now reads its own source plus the Galaxy object of every native condition
that fires on it live: those parented under it and those WireAlarmNotifiers wired to it. Live and history use
the same membership. The object comes from the condition's Galaxy alarm reference, its raw tag's attributeRef,
up to the first '.':
- The composer reads it with the driver's own parser (GalaxyAttributeRef, shared with GalaxyDriver), only for
a GalaxyMxGateway tag with an alarm (RawTagPlan.NativeAlarmReference, part of plan equality).
- The applier binds it after materialising the condition (IOpcUaAddressSpaceSink.BindNativeAlarmHistory,
forwarded by the deferred sink). A rebuild or the condition's removal forgets it.
Each object is read with the #882 rule: its own events and <Object>.<Attribute> rows, never a prefix sibling.
Fan-out. historiangw's ReadEvents filter is a single predicate, so there is one request per source; the wire
contract is unchanged. EventSourceFanOut merges the reads as one source:
- order is (time, source position, arrival), reversed for a reverse read; ties are never re-sorted by EventId,
so the boundary skip stays stable;
- duplicates by EventId are dropped and the merge is cut to the cap;
- a truncated source bounds the page at the time it stopped.
Paging, continuation points (bound to the whole source set), the walked-back reverse read and the tie-cluster
over-fetch are unchanged. A folder with one source is read exactly as before.
ACL. Every source is read inside the folder's own HistoryRead, after AuthorizeHistoryRead (#826). A denied
folder reads nothing and fails closed.
Mapping (EventMapper, property names now case-insensitive):
- Severity: an AVEVA row's priority goes through GalaxyAlarmPriority, the table live Galaxy alarms use
(MxAccessSeverityMapper now delegates to it), so history matches live. Otherwise severity (OtOpcUa's own
rows, which AVEVA stores lower-cased), clamped. AVEVA's own 1-4 severity is ignored. The table ranks a
higher priority as more severe, which inverts AVEVA's convention: filed as #887.
- An AVEVA row whose source_conditionvariable names a native condition that fires on the folder takes that
condition's SourceNode and live ConditionName. Otherwise ConditionName is alarmname or the condition
variable's attribute.
- alarm_id/alarmid map to AlarmId.
Tests:
- GalaxyEquipmentEventHistoryWireTests (4, real SDK session, production Gateway backend, AVEVA-shaped fake
gateway): equipment EqA backed by objects Oven_007 + OvenDoor_07. Covers object and attribute events from
both objects, the prefix sibling excluded, cross-object and in-object ties, forward paging at 1-4 a page,
reverse whole and paged, SourceNode/ConditionName/Severity, the raw device folder, ACL deny with no backend
call plus a granted positive control, and a rebuild forgetting the bindings.
- EventSourceFanOutTests (8), EventMapper AVEVA cases (11), GalaxyAttributeRefTests (16), a composer test and
an applier ordering test.
- Mutation-checked, 17 mutants, all killed. Covered: parent-only membership, no scope, a case-sensitive
condition match, no condition rewrite, no truncation bound, EventId tie order, no sort, no cap, no stop at
the bound, unreversed source order, no dedupe, no '.' boundary, priority ignored, case-sensitive keys, no
applier bind, no deferred forward, and a composer that ignores the driver type. The forget-on-rematerialise
guard survives as belt-and-braces: the rebuild clear covers it.
- Live, read-only, no WinRM, vd03 HistorianGateway ea9f21c: equipment folder EQ-886-ZipsenLeft (objects
DevWinPlatform + Left_001). It read 0 events before and 92 now, equal to the unfiltered read's rows for both
objects. 1/7-event pages and reverse whole/5-event pages agree, and 73 rows resolve to their live condition.
diff_runs vs the #882 run: 221/226 identical. The 5 that differ are expected: DevWinPlatform Severity
1 -> 750/1000, and #881 capabilities.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ConfigurationBinder appends configured collection items to an existing list, so
the pre-filled [None, Sign, SignAndEncrypt] default made "drop None" a no-op:
every deployment that listed only the secure profiles still advertised an
unsigned/unencrypted None endpoint.
EnabledSecurityProfiles is now null by default; ResolveEnabledSecurityProfiles()
applies the unchanged three-profile default only when nothing is configured.
A set-but-empty list fails startup validation and BuildSecurityPolicies no
longer falls back to a None endpoint on empty input (it throws). Sweep of all
options classes found no other non-empty pre-filled collection default.
Tests: options binding (indexed, JSON, env forms, default, empty, bad name), a
booted-server wire test (no None endpoint, None connection refused with
BadSecurityPolicyRejected, secure connection works; default case is the control).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
SDK 2.0.1's UserNameIdentityTokenHandler encrypts a password longer than 64 bytes
(a private RsaEncryptedSecretPasswordThreshold, no setting) as an RsaEncryptedSecret
with EncryptionAlgorithm=null. SDK 1.5.378 (.106 and .182) decrypts only the legacy
format under RSA policies; its SecurityPolicies.Decrypt answers a null algorithm with
BadIdentityTokenInvalid ("Unexpected encryption algorithm") before any key check.
Every production API key is 65 chars, so no SDK 2 OtOpcUa client could use one
against QA (b8b4c18f), a rollback target or an older third-party server.
LegacyRsaUserNameIdentityTokenHandler (Driver.OpcUaClient.Session, plugged in through
the public UserIdentity(IUserIdentityTokenHandler) ctor) always uses the legacy
format, RSA-OAEP(len|password|serverNonce) under the server certificate, under an RSA
policy. That is what SDK 2 sends for passwords of 64 bytes or less and what SDK 1.5
clients always send. The legacy format has no size limit (the plaintext is
length-framed over as many RSA blocks as it needs), and an SDK 2 server reads both
formats. Policy None, the ECC policies and a missing certificate go to the SDK
handler unchanged.
OpcUaUserIdentityFactory.UserName is now the one UserName identity the driver, the
AdminUI browse and Client.Shared (CLI/UI) present. Client.Shared references .Session
for it.
Tests: handler unit tests (all three RSA policies, the 1.5-required algorithm URI,
SDK 2 server-side decrypt, the session Copy() keeps the type, None unchanged, and an
SDK control pinning the 64/65-byte switch); a Client.Shared identity test; and an
SDK2<->SDK2 integration test where a 65-char key is admitted in both formats and a
wrong 65-char key reaches the key check (BadIdentityTokenRejected). The #800 Windows
gate's SDK 2 CLI 65-char case now gates against the 1.5 server too.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK starts Server.ServiceLevel at 255 (full service), and OpcUaPublishActor publishes nothing
until the node's first redundancy snapshot arrives, so a restarting pair node advertised 255 for the
~10-13 s it took to join, out-ranking the real Primary (250) for any client that picks the highest
ServiceLevel.
OtOpcUaSdkServer now writes StartupServiceLevel = 0 in OnNodeManagerStartedAsync: after the SDK's
CreateServerObjectAsync wrote 255 and before the server goes Running, so no session can ever read
255. 0 is the design's existing Down / Detached tier (ServiceLevelCalculator returns 0 for a member
that is not Up/Joining, and a node with no snapshot has no role yet). It is below the SDK 2
session-headroom floor (200), so it stands through session churn before the #884 claim, and the
claim's bind replays any level already requested. Single-node deployments are unaffected beyond
the window: the first snapshot publishes 250 as before.
New Node_with_no_requested_level_advertises_the_startup_level_until_one_is_published: reads 0
before the bind, 0 after a bind with nothing to replay, then 250 once published. Without the
startup write it reads 255 on the first session (mutation-checked). docs/Redundancy.md updated for
#884 and #885.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
SDK 2.0.1 recomputes Server.ServiceLevel from session headroom on every session create/close and
overwrites any value >= 200 unless a provider has claimed it through
IServerServiceLevelControl.ClaimServiceLevelControl. SdkServiceLevelPublisher never claimed it, so on
the docker-dev pair both nodes read 255 as soon as a client connected while their own metrics showed
250 (Primary) and 240 (Secondary); the Stale tier's 200 was lifted to 255 too. Levels below 200 (100,
0) were untouched, which is why the bootstrap 100 showed briefly.
The publisher now claims the value at construction and writes through the SDK's writer; a server
without the claim (test doubles) keeps the diagnostics-guarded write. New
Published_level_survives_later_sessions (250/240/200) fails without the claim (reads 255) and passes
with it.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
HistorianGateway answers a per-source ReadEvents (Source_Object = DevWinPlatform) with the
object's own events (SourceName DevWinPlatform) AND its attribute alarms, which it reports as
<Object>.<Attribute> (Source_ProcessVariable, e.g. DevWinPlatform.CPULoadAvg). The Gateway
backend's defensive client-side source filter compared SourceName for equality, so it dropped
the attribute alarms: 70 of the 72 DevWinPlatform events on vd03.
The filter (and the past-the-cap "more events" signal) now uses EventSourceMatch: the source
itself, or <source>.<anything> on the '.' boundary, ordinal. DevWinPlatform never matches
DevWinPlatform2 or DevWinPlatform2.X. This needs no wire change and still makes one request.
Each event keeps its own SourceName. SourceNode stays the condition its stored alarm id names,
which is null for AVEVA rows. The attribute events ride the notifier's own read, so the
per-folder HistoryRead verdict (#826) governs them: a denied folder answers
BadUserAccessDenied before any backend call. The SqlServer and QuestDB backends still match
exactly; they hold only OtOpcUa's own events.
Tests:
- GatewayAttributeEventSourceTests (16): the boundary theory, attribute-in/sibling-out through
the data source, the drain signal, and an exact attribute-named read.
- AttributeEventHistoryWireTests (2, over a real SDK session with the production Gateway
backend on an AVEVA-shaped deterministic gateway): object + attribute events with their own
SourceName, a prefix sibling excluded, paging, and the ACL-on deny with no backend call plus a
granted positive control.
- Mutation-checked: exact match fails both wire tests; dropping the '.' boundary fails the
wire sibling check and 6 unit tests.
- Live AVEVA harness re-run against vd03 HistorianGateway ea9f21c (read-only, no WinRM):
folder DevWinPlatform 2 -> 72 events (whole, 72 1-event pages, reverse); the new #882 check
passes; 221 of 225 shared cases are identical to the #801 SDK2 baseline, and the 4 that
differ are exactly the DevWinPlatform event cases. Recorded in aveva-history-801.md.
Follow-up #886: production folders name RawPaths/EquipmentIds, not Galaxy objects, plus the AVEVA
severity/condition property mapping.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Every Boolean under Server/ServerCapabilities/HistoryServerCapabilities read
false (the SDK default) even though data and event history were served, so a
spec-following client skipped HistoryRead entirely.
Each backend now declares what it serves through its own interface:
- IHistorianDataSource.ReadCapabilities (Data / Events / ServerTimestamps):
Gateway Data|Events; SqlServer + QuestDb add ServerTimestamps (they store
ServerTimeUtc, the Gateway stamps read time); Null declares None. A
reflection guard requires every production source to declare it itself.
- IHistorianDataUpdater.Capabilities (required member): SqlServer and QuestDb
declare Insert/Replace/Update/DeleteRaw/DeleteAtTime; the Gateway provider
has no updater.
The node manager writes the flags on every source/updater assignment
(AdvertisedHistoryCapabilities), and the same declarations drive dispatch:
an update the updater does not declare is BadHistoryOperationUnsupported
without reaching it, the HistoryWrite AccessLevel bit follows a non-empty
declaration, and event-history notifier promotion follows the Events flag.
Event and annotation update flags stay false (nothing dispatches them).
Tests: booted-server wire test per backend + Null + a SqlServer->Gateway
rewire pairing on one session; dispatch gating unit test; the #801 live
gate now asserts the Gateway flags (not run here).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
mxaccessgw#227 (Contracts/Client 0.4.0) adds an opt-in attach acknowledgement on
StreamEvents: a StreamAttached sentinel MxEvent sent first, only after the
stream's subscriber is registered on the session. This closes the #865 attach
race that waiting for "stream requested" only narrowed.
- Bump vendored MxGateway.Client/Contracts 0.3.0 -> 0.4.0 (nuget-packages/,
pulled from the Gitea feed; local-mxgw stays the mapped source).
- IGalaxySubscriber.SupportsStreamAttachedAck (default false, so fakes and older
seams keep the fallback); GatewayGalaxySubscriber reads it from the live
session's capabilities (stream-events-attached-ack) and passes
emitStreamAttached accordingly; TracedGalaxySubscriber forwards it.
- EventPump: records AttachAckExpected before the stream opens (published
before StreamRequested completes); intercepts ev.StreamAttached ahead of the
ReplayGap check - completes StreamAttached and continues: never dispatched,
never counted as received, never a resume cursor (worker_sequence 0). The
signal also completes when the loop ends so no waiter hangs. One pump = one
stream, so a reconnect's new pump has a fresh signal. New counter
galaxy.events.stream_attached.
- AwaitCurrentEventStreamAsync: with the capability, waits on StreamAttached
within the same StreamRequestWait budget (1 s); a missing ack logs a
rate-limited Warning and proceeds instead of hanging. Without the capability
the old StreamRequested wait is the fallback.
Tests (StreamAttachedAckTests, 6): ack-gated subscribe gets its initial value;
capability absent uses the fallback; missing ack times out with a warning; the
sentinel is never dispatched; a dead stream releases ack waiters; reconnect
gives each stream a fresh signal. With the ack wait skipped, 3 of them fail.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The rule fix itself landed in 1a08b4f7 (detail-0 failure rows route on
category); its test only exercised raw_detected_by = 3. Add per-source
coverage for every MxStatusSource value mxaccessgw's MapSource emits
(-1, 0..5, and an out-of-range value) across the failure categories, plus
the live-captured shape the old rule got wrong (OperationalError detected
by the responding automation object, mxaccessgw docs/AlarmProbeFindings.md)
and a RequestingLmx CommunicationError positive control. All 10 new cases
fail against the old 'detail 0 && raw_detected_by != 0 => comms' rule.
Comment in FromMxStatus now cites MapSource and says why NMX (2/3) is not
a transport signal. Item 1 (Warning/Pending) stays open pending a capture.
Co-Authored-By: Claude <noreply@anthropic.com>
OpcUaApplicationHost left ServerConfiguration.MaxRegistrationInterval at the
SDK default (30000) with a null RegistrationEndpoint, so SDK 2.0.1's
StandardServer armed its registration timer against opc.tcp://localhost:4840,
which on an OtOpcUa node is the server itself (no LDS). It answered
BadServiceUnsupported and the SDK retried forever, visible since #874.
Registration is now opt-in: ApplyDiscoveryRegistration sets
MaxRegistrationInterval = 0 (the value that keeps StandardServer from arming
the timer) unless OpcUa:DiscoveryServerUrl is set, in which case the server
registers against that URL every OpcUa:DiscoveryRegistrationIntervalMs
(default 30000) over SignAndEncrypt/Basic256Sha256, like the SDK's own
default endpoint. The validator requires an absolute opc.tcp:// URL and a
positive interval when the URL is set. Documented in Configuration.md and
OpcUaServer.md.
Co-Authored-By: Claude <noreply@anthropic.com>
- Host reader: the frame read and its handling are caught separately. Only the
read's I/O exceptions mean the pipe ended; anything HandleFrame's inline
completion work throws now kills that worker as ProtocolFault (outstanding
runs -> worker restarted / BadResourceUnavailable) instead of escaping a
dedicated thread (process crash) or being swallowed as a clean end (an
InvalidOperationException used to be, leaving the worker unread until a Crash
2 s later).
- Worker reader: the same catch-all around HandleFrame (exit ProtocolFault).
- Worker: Cancel() runs on the pool, so a script's blocking
ctx.CancellationToken callback cannot stop the worker reading its pipe.
- Host write-stall check: ScriptWorkerFrameWriter.WriteInProgressFor; the
heartbeat monitor kills a worker whose pipe write has been blocked longer than
max(10 s, silence limit) as KilledUnresponsive, which fails the write and
releases the producer (seam submit, deploy load, Cancel). Fixes the wrong
'heartbeat monitor ends it' comment on the pipe buffer quota.
- StopAsync sends Shutdown off the caller so its 1 s budget holds on a full pipe.
- WindowsBlockingPipeStream refuses the async surface (Stream's base versions
serialise a write behind a waiting read).
- Tests: PipeFaultTests (throwing completion callback -> ProtocolFault and a
BadResourceUnavailable sibling, shard serves again; a worker that stops reading
but heartbeats is killed within the bound and the blocked producer returns,
via the Hello.test_stall_reader_on_diag seam); BlockingPipeStreamTests dispose
with a write blocked on a full pipe, and with a read and a write both pending;
Category=ScriptWorker on the protocol, handshake and pipe test classes.
Mutation checks: no stall kill -> stall test red; catch-all without the fault
-> callback test red (Crash).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
With a zero quota an overlapped write stays pending until the peer has read the
bytes, so the new blocking writer waited out a peer wake-up per frame (windev
closed loop 54k -> 26k evals/s). With a quota the write completes once buffered.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A .NET pipe opened Asynchronous completes even its synchronous Read/Write
through the thread pool's completion port, so every frame on Windows cost a
completion packet plus a pool work item (and the pool's spin-wait). New
WindowsBlockingPipeStream issues its own overlapped ReadFile/WriteFile with the
OVERLAPPED.hEvent low bit set (no completion packet) and waits on that event on
the calling thread; reads and writes stay concurrent, unlike a synchronous
handle. ScriptWorkerFrameCodec.BlockingStream picks it on Windows and the pipe
itself on Unix, so both ends now use the dedicated blocking reader + synchronous
combining writer on every OS (UseBlockingIo and the async read loops are gone).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.
DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The host handed the SDK a provider-less telemetry context, so SDK-side
refusals — identity-token decryption (the SDK 2 RsaEncryptedSecret clock
window, #873), certificate validation, secure-channel faults — left no server
log line (found on windev during the #800 Windows validation). The hosted
service now creates one ServerSdkTelemetry context from its ILoggerFactory
(Warning and above, SDK categories, rate-limited 20/min per category) and
shares it between OtOpcUaSdkServer and the ApplicationInstance; released on
dispose. Test: an untrusted client's rejection reaches the captured log.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Since #855 condition events are queued to folder event items under the node
manager Lock. At a full DiscardOldest queue the SDK's EventQueueHandler
dequeues the head of EventMonitoredItemQueue, a List.RemoveAt(0) shift of the
whole queue (~1 us/item at QueueSize 1000, ~10 us at 10000).
OtOpcUaSdkServer.CreateMonitoredItemQueueFactory now wraps the SDK factory in
RingEventQueueFactory: event queues are RingEventMonitoredItemQueue (O(1)
discard), data-change queues, durable requests and disposal go to the SDK
factory unchanged. A factory set through StandardServer.MonitoredItemQueueFactory
is used as-is.
The SDK queue is storage only (decompiled 2.0.1): DiscardOldest, overflow-event
positions and discarded/overflow counters live in EventQueueHandler, which is
unchanged. The ring reproduces QueueSize 0 initial + BadInternalError enqueue,
drop-oldest at Count == QueueSize, FIFO dequeue, SetQueueSize keep-newest /
keep-oldest, the newest-1000 duplicate window via IsEventRaisedFor, no-op
Dispose. RingEventMonitoredItemQueueParityTests drive both queues, and the SDK
EventQueueHandler over both factories, with the same random sequences.
FolderEventDeliveryPerfTests gains Ring_vs_sdk_event_queue (A/B in one process
via the internal OtOpcUaSdkServer.RingEventQueues switch).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Every raw detected_by value names an MXAccess layer (requesting/responding
LMX, NMX, automation object); the gateway's own failures arrive as
ProtocolStatus. The 'non-zero source = transport failure' rule made a failure
the responding object detected a comms error. Item 1 of #872 (Warning/Pending
categories) stays open pending a captured row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The gateway fills Statuses on every data change (live OnDataChange, value-cache
replay and session snapshot all carry Quality too), and EventPump used the status
row whenever one was present, so Quality was never read: an off-scan, Bad or
Uncertain attribute with an OK status row was published Good. A failure status
row still wins; otherwise the quality byte grades the value (legacy parity:
quality >= 192 Good).
Live, read-only on VD03 (DEV): healthy ScanState and _TestArea attributes stay
Good (0x00000000) via quality 192; the stopped JoePlatform/JoeAppEngine ScanState
now publish BadCommunicationError (0x80050000) instead of generic Bad, with no
"unrecognised quality byte" warning.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The gateway copies MXSTATUS_PROXY.detail verbatim, so a failure row's detail is
always an MxStatusDetail code. FromMxStatus routed any detail <= 0xFF through the
OPC DA quality table: MX_E_PlatformCommunicationError (2) on a stopped platform's
ScanState logged "Unrecognised MXAccess quality byte 0x02" and published generic
Bad (VD03 live capture), MX_E_InvalidObjectId (8) became BadNotConnected,
MX_E_Overflow (20) BadCommunicationError, and a detail in the Uncertain band
published Uncertain for a failure.
Every non-zero detail now maps per the gateway.md vocabulary (timeout,
communication, configuration, type, range, access, initialising), then by
category; the category-fallback warning logs once per (detail, category).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The sweep threshold never rises above the 4096 cap (a threshold the count
could not reach stopped sweeping, and sharing, for good); at the cap a
sweep for dead shapes runs after every 256 refusals.
- The per-report "queued" mark is taken after the filter group resolves,
so a failure there leaves the item unmarked.
- The bucket key also carries TypeDefinitionId and IndexRange.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The filter-group registry holds groups weakly (the filter->group table is
the only strong reference), sweeps dead entries amortised and counts only
live shapes against the 4096 cap; filters past the cap are counted on the
new meter otopcua.opcua.condition_events.filter_groups_unshared.
- A group validates its where clause too (restored, never-validated filters
evaluate like validated ones).
- Test: value-equal filters cost one projection per locale set per report.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Filter-group buckets include the where clause's (value-based) hash, so
per-source filters sharing a select list are not all scanned under Lock.
- Locale keys are cached per session array.
- Tests: a notifier reached twice in one walk (Line above two wired
equipment) gets each event once on both paths (fails without the
per-report set); queued events re-projected by a modify are identical on
both paths; locale keys unambiguous and where clauses hash by value.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Adds an escalated-stream-fault test seam and a test: the live pump's fault
escalates (positive controls before and after), a superseded pump's late
fault after a reopen does not; the absence is asserted only once both
pumps' callbacks are known to have run (HasEnded). Verified to fail with
the filter disabled. The filter's comment now says the identity check does
the work and warns about a future session bump while Healthy.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.
Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Each EventPump's fault callback now carries the pump's identity; a fault
from a pump that is no longer the live one (or belongs to an older registry
session) is logged at Debug and not escalated, so a stale pump's late fault
cannot fail a good replay now that a failure during Replaying fails the
attempt. The replay-retry warning includes the fault.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
ReconnectSupervisor swallowed a transport failure reported while Replaying,
so a restarted event stream refused/ended a round trip after the replay
began still let the replay go Healthy over a dead stream (and the next
advise's re-report restarted at attempt 1 with no backoff). A failure
reported during Replaying is now remembered and turns that attempt's
success into a retry with backoff. ReplayAsync's synchronous dead-pump
throw yields to a teardown's cancellation.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
handler returned is reported; one pending at actor stop is reported; a
confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
reconnect, UNACK_RTN-as-Clear and RPC-budget limits.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- ReplayAsync fails when its restarted pump's stream has already ended, so
the supervisor backs off and stays Degraded instead of going Healthy over
a dead stream (whose next advise re-report restarted at attempt 1 with no
backoff - a tight reconnect loop).
- ReconnectSupervisor.AttemptInProgress (reopen through replay); advises
wait for a stale pump's replacement only while an attempt is running, not
for a whole backoff (the state still reads Reopening/Replaying there).
- ReconnectSupervisor: _disposed is set under the state lock and re-checked
inside ReportTransportFailure, so a late report cannot start a recovery
loop that StopAsync already missed.
- EventPump.HasEnded is a flag set before StreamRequested is released, and
a stream that ends/faults on its first MoveNext is marked ended first.
- SubscriptionRegistry fast path: a miss now falls through to the locked
path (lookup + session + window check atomically) instead of a lock-free
re-lookup.
- Tests: replay over an immediately-ending stream fails; AttemptInProgress
across attempt/backoff; ReportTransportFailure after stop; the
reopen-window test uses a generous wait bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- RefreshDirectItemsWithCut keys its "decided off Lock" set per (item,
condition), so a condition one item's walk saw is still decided for another.
- The deferral warning no longer blames slow folder subscribers (no folder of
ours defers since #855).
- FolderEventFloodWireTests' removal test now asserts the folder item got
nothing and the Server object exactly the pre-stall event (it passed with
the post-stall liveness re-check deleted).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.
- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
(default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
after AcknowledgeAsync, off its mailbox and outside the resilience
pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- StreamRequestWait 1 s (under Tier-A Read 2 s / Subscribe 5 s budgets) and
a stale pump is only waited on while the supervisor is reopening or
replaying (or a restart is underway); in a reconnect backoff the advise
proceeds at once. The no-stream log is a rate-limited Warning while
Healthy, Debug otherwise.
- A current-session pump whose loop has ended is not a stream: if the
supervisor already reports Healthy (its fault was swallowed while it was
recovering) the failure is re-reported to start a new cycle.
- Subscribe/Read no longer start the pump directly (only
AwaitCurrentEventStreamAsync does, honouring the replay restart flag);
the teardown refusal is an explicit ThrowIfRuntimeStopping.
- SubscriptionRegistry.ResolveOrPark fast path: the handle lookup and the
open-window check were not atomic, so a subscribe that registered the
handle and closed its window in between got its initial value dropped as
stale (found by a stress run: the first event of a bulk advise lost). It
re-looks-up before declaring stale. Deterministic test via a seam.
- Tests: first Read on a fresh driver, replay of a data subscription.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Both plain ConditionRefresh paths (Server object and folder) skip a
condition a deploy removed or replaced between the off-Lock walk and the
queueing (the SDK base did not either; only the cut path checked).
- A failure delivering at one notifier hop is logged and the walk continues
to the remaining hops; a failure deciding the Server-object items withholds
only them.
- Test: a condition removed during a plain refresh (folder + Server object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Joseph Doherty