#694 item 3 — historiangw#27 is fixed and now live-verified through OtOpcUa:
- ServerHistorian:Gateway:EnableNativeTimeWeighted defaults to true (false is
the kill switch for a gateway older than 9f711c1). TimeAverage / Total are
served and advertised again (12 aggregates).
- The gateway sends the interval as max(1, round(ms)) and AVEVA clips its last
interval at the request end (live). NativeAggregatePlan therefore sends a
multi-interval grid only for whole-ms intervals; a fractional interval is read
one interval per request (at most 1,000, else BadTooManyOperations), and every
one-interval request goes out rounded UP to whole ms so AVEVA answers one
clipped row. Live, a 3.0004 s clipped interval sent at its own width lost its
last 0.4 ms (Total 123 vs 123.0172).
#746 — HistorianGateway Contracts/Client 1.0.0 surface:
- Alarm history: GatewayAlarmHistorianWriter sends SendEvents batches of <=200.
A failed per-event result is re-sent alone via SendEvent (classified as
before); a refused batch (InvalidArgument etc.) goes event by event so a
poison event dead-letters alone; a systemic fault retries the rest without
sending more; Unimplemented (older gateway) falls back to SendEvent for good.
IHistorianGatewayClient.SendEventsAsync defaults to Unimplemented.
- Events reads: a capped page sends cap + 1 as ReadEventsRequest.max_events, so
the gateway completes the stream instead of the client cancelling it. An
ambiguous capped answer (foreign rows filtered out of the server's cut) or a
refused cap is re-read uncapped with the existing EventDrainBudget drain.
Tests: GatewayServerCappedEventsTests, GatewayAlarmWriterBatchTests,
GatewayNativeWholeMillisecondGridTests (gateway-faithful rounding fake), gate
tests flipped to the new default; every new test mutation-checked (each
mutation of the production change fails at least one). Live harness
AvevaGatewayHistorianFollowupsLiveGateTests (read-only) for #694/#700/#746.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A UnsTagReference to a raw tag that carries an "alarm" object produced a UNS
value Variable that read BadWaitingForInitialData forever (G6 live gate,
TestAlarm002Condition on testmachine-015). BuildUnsReferenceVariables now skips
those references; the equipment gets the alarm only as the raw condition wired
as an event notifier of its folder, which is unchanged (the reference still
feeds RawTagPlan.ReferencingEquipmentPaths).
Why condition-only and not "feed the mirror":
- The raw side has no value node for such a tag: AddressSpaceApplier
materialises an alarm tag as a condition OR a value variable, never both.
There is no raw value for the UNS mirror to mirror.
- DriverHostActor skips alarm tags when it builds the value fan-out and write
routing maps (`continue` before the UNS registration), and the driver leaves
them out of its value subscription ("subscribed to 5 refs" with 6 tags in
G6). Feeding the mirror would mean subscribing a value the design says the
tag does not have.
- Docs already state condition-only: ScriptedAlarms.md ("instead of a value
variable"), AlarmTracking.md ("only the condition, with no sibling value
variable"), the v3 design (native alarm = one condition at the raw tag,
surfaced at equipment via notifiers).
- The composer tests that asserted the mirror were incidental: f4f3e17e
(v3 B4-WP1) put the alarm on the fixture's Speed tag to test raw alarm
attachment, and later tests reused Speed for writable / historian / array
checks of the mirror. No commit or doc chose a value mirror for alarm tags.
- ScadaBridge does not read it: its OtOpcUa alarm path is event subscriptions
on the equipment/raw folders (v3 alarm B/C live gate), and a node that never
had a value cannot be a working binding. G6 itself authored the alarm's
Boolean as a separate value tag (TestAlarm002), which is the supported way to
read it.
Tests: the composer fixture moves the alarm onto its own HighTemp tag
(Speed is now a plain historized value tag), with a new test asserting no UNS
Variable for the alarm reference, with positive controls (the raw tag still
carries the equipment path; the value references still emit). New real-SDK
test Alarm_bearing_reference_surfaces_as_the_condition_only_against_real_SDK:
no Variable at the UNS path, the Speed mirror present, one condition, the
equipment folder wired as its notifier. Both new tests fail with the skip
removed. Docs: Uns.md, ScriptedAlarms.md, G6 evidence note.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The run covered browse (raw + UNS), read, subscribe, write and readback, and a native alarm
through Part 9 to an ack confirmed in Galaxy and a clear. The run found #895 (fixed in
c971b171) and #896 (open). DeployWatcher was not exercised: there is no headless Galaxy
redeploy.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
One owner-approved ack of CVDAisle_1.TableAlarms_008.RuntimeAlert at
2026-11-13T04:17:22Z through docker-dev site-b-1 -> vd03 mxaccessgw 76a71bc8.
.Acked True, .AckMsg = comment, gateway feed ACTIVE_ACKED, v_AlarmEventHistory2
ACK_ALM with the NAM operator and OperatorNode MxGateway. The 11-11 failure
was the vd03 gateway build (4f931e4, pre-AckMsg wnwrap AlarmAckByName no-op),
not #226/#228.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Read-only ADS gate from WW_DEV_VM: port 10000 Config, port 851 absent (error 6).
Boot project is the committed OtOpcUaTwinCatFixture without GVL_Perf/FB_PerfChurn.
Member writes, coalescing and #761 array writes remain blocked; no multi-hop hardware.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Re-ran every gate docker-dev and the fixtures can run, on master 412da27c plus the
deploy-API fix: A (now with dotnet-counters), B, C, D check 4 (after #785), E, H1
live end-to-end over OPC UA (N=0 year window: 6 pages <= 10k, continuation points,
all 52,560 values), the gate-F removal drop on the VD03 gateway (4 -> 2 within 15 s),
and all fixture lanes incl. the first G1 MTConnect run. All pass.
Recorded: the AbCip "writes" combination ran no writes (AB_DIFF_ENDPOINT gates them
off), a C-style pause past 15 s leaves a split SITE-B pair, and simulator evidence for
four of the five hardware userGates (TwinCAT has no simulator). The hardware
userGates stay open.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- #890: installer at f75ee7d0 re-applied on both nodes with -DenyPaths root, pki, data and
C:\ProgramData\ZB\kek (the Secrets KEK sits outside the install root); second apply 12/12 unchanged.
- Script-runner canary authored in the AdminUI and deployed (Calculation tag + VirtualTag, 1 s timers):
Good and advancing on both endpoints; kept on QA as a runner monitor.
- #794 step 5: the live worker token is restricted (SeChangeNotify only; identity/Everyone/Users/RESTRICTED)
and is denied read and write on pki, data, app, logs and the KEKs; network sockets none. Finding #891.
- #780: QA runaway drill passed (killed at the 4000 ms hard deadline twice, quarantined, siblings Good,
ServiceLevel 250/240 unchanged); drill removed, redeploy back to the canary revision 82dd0598.
- #780 P4 recorded: InProcess stays the default and the rollback; StuckOrphanQuarantineFactor documented
as in-process only.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The installer denied NT SERVICE\OtOpcUaScriptWorker only on -InstallRoot. On the wonder layout the node's keys and
secrets sit beside it (E:\ApiInstall\OtOpcUa\pki with the application certificate's private key, data\ with
secrets.db and LocalDb), inheriting BUILTIN\Users:RX. Users is a restricting SID of the worker token, so a worker
could read them.
- Install-ScriptWorkerIsolation.ps1 -DenyPaths <dir>,...: each folder gets the same inherited deny as InstallRoot
(new SensitiveDirDeny steps). Refuses drive/share roots, script-worker\ and the PE cache; folders must exist;
a comma/semicolon string works for powershell -File.
- The deny is read AND write (0xD0157: ReadData, WriteData, AppendData, WriteEA, DeleteChild, WriteAttributes,
Delete, WRITE_DAC, WRITE_OWNER). Traverse/Execute, ReadAttributes, ReadControl and Synchronize stay so the
loader still reaches script-worker\. The write half stops a worker planting a cert in pki\trusted\certs where
a data drive lets Users create files in subfolders.
- The list is recorded under the anchor service's key (Parameters\DenyPaths, step DenyPathRecord): a re-run
removes the deny from folders dropped from the list, -Uninstall removes every recorded one without repeating
-DenyPaths. -WhatIf/-PlanOnly print every step.
- After an apply, WARN for each protected-DACL item under a denied folder that the inherited deny cannot reach;
without -DenyPaths, WARN for pki/data/logs/backup/config/secrets siblings of InstallRoot.
Tests:
- Dry-run harness 46 -> 86 assertions (QA layout plan, mask bits, refusals, stale removal, record-driven
rollback, host account never denied). Mutations (no sibling deny, deny->allow, mask back to ReadData, no
stale removal, rollback ignoring the record, script-worker not refused) each turn it red.
- Test-ScriptWorkerIsolation.Windows.ps1 (Windows PowerShell 5.1, windev): 34/34, incl. -DenyPaths apply,
11/11 unchanged re-apply, shorter-list re-apply, uninstall from the record, second uninstall unchanged.
- New InstallerLayoutIsolationTests (OTOPCUA_ISOLATION_INSTALLER_E2E=1, elevated Windows): a scratch wonder
layout under the real restricted worker token. Before install and after uninstall the token reads pki/data/
logs/backup/appsettings and plants a trust-list file; installed it can do neither, still reads script-worker\,
a real worker launched from the layout runs a script, and the host still reads everything. Mutating the
-DenyPaths denies to a no-op mask turns it red on pki\own\private\otopcua.pfx (reproduces #890).
QA vq03/vq04 are NOT re-applied; the command is in docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Owner scope 2026-11-12 (Windows x64 + Linux, unchanged cycle-3 caps, >=7200 s, SDK 1.5 comparator).
- Linux ARM64 (late-slope, 7200.12 s, 1964 cycles / 491 restarts, shutdown verified, analyzer-validated):
first rejection RSS at cycle 247 / 900.73 s, 358,682,624 > 358,678,528 (+4 KiB); peak 382.98 MiB vs
limit 342.06 MiB; late plateau medians 344.4/341.1/341.9 MiB, OLS -1.90 MiB/h. FAIL.
- Windows x64 on windev (same workload, working set as RSS, handle table as descriptors): first rejection RSS
at cycle 285 / 1130.76 s, 303,505,408 > 302,174,208; peak 319.49 vs limit 288.18 MiB; plateau ~314-315 MiB,
+0.93 MiB/h. FAIL. Also incomplete: the launcher's wall-clock outer deadline fired early after a +552 s
w32time step and ended the run at test-phase 7014 s (rejection had already latched); launchers now use a
Stopwatch.
- SDK 1.5.378.182 comparator (efb9197a publication, Windows, 7201 s): RSS peak +65.8 MiB (never over its cap),
plateau ~+48 MiB, but the +32 handle cap tripped at cycle 17 (max 674 vs 642).
- Managed flat on both SDK2 runs; growth is the generated-factory JIT burst (+40k/+44k methods) - Linux JIT
doublemapper ~180 MiB; Windows private bytes grew ~25 MiB of a ~132 MiB working-set rise.
- Retained: b61f3e05 cycle-1 fixture failure (#808 unplaced UNS folder), Windows/SDK1 smoke diagnostics.
No cap raised, no warm baseline, no failed run counted as passing.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.LocalDb and .Replication at 0.3.7 (published 2026-11-12).
Wire-compatible; no code change needed.
- 0.3.6 (scadaproj#36): the library sync service enforces
LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
its refuse-every-stream-when-blank rule; doc comment updated. docker-dev
site-a pair: both nodes now read one variable,
OTOPCUA_SITE_A_LOCALDB_SYNC_KEY (same dev default as before), so an
override can never split the pair.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
node offline past TombstoneRetention no longer brings a pruned
deployment back. The ledger grows while a peer is down; optional
TombstoneHardCap (documented in Configuration.md) bounds it.
Runbook, Configuration.md and the cluster-topology rule updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
GalaxyAlarmPriority, the one table live Galaxy alarms (MxAccessSeverityMapper)
and AVEVA A&E history (EventMapper, #886) map the AVEVA priority through, ranked
a higher number as more severe. AVEVA ranks priority 1 as the most urgent, so a
priority-1 alarm read 250 and a priority-999 system event 1000.
It now uses AVEVA's default priority ranges, most urgent first:
AVEVA priority AVEVA severity bucket OPC UA Severity
1-250 1 Critical Critical 1000
251-500 2 High High 750
501-750 3 Medium Medium 500
751-999 4 Low Low 250
(history only) 0 not an alarm - 1
Out-of-range priorities clamp (<= 0 Critical, > 999 Low). Evidence, vd03
Runtime.dbo.Events read-only: priority 1/100 alarms carry AVEVA severity 1,
400/500 severity 2, 600/700 severity 3, and every priority-999 System.* /
User.* row severity 0 with IsAlarm 0.
- Live and history now use the same number. GalaxyDriver passes the table's
exact value as AlarmEventArgs.OpcUaSeverity. Before, it passed only the bucket,
which NativeAlarmProjector re-mapped through its generic 200/500/700/900
ladder, so live and history disagreed even before the direction fix.
- History: a row whose AVEVA severity is 0 (System.Start/Stop/Deploy,
User.Write...) is not an alarm and reads Severity 1, below every alarm band.
For alarms AVEVA's 1-4 severity is still never the value: the live feed
carries only the priority, so it cannot use it, and history must agree.
- Docs: docs/Historian.md gains "Galaxy alarm priority -> Severity
(lmxopcua#887)"; docs/AlarmTracking.md updated.
- Live re-run of the #801/#886 harness against vd03 HistorianGateway
ea9f21c (read-only, network-only): 224 of 233 cases identical to the #886
run, the 9 others differ only in Severity (priority-700 alarm rows 750 ->
500, 19 system/user events 1000 -> 1). Record: aveva-history-801.md
"#887 re-run", report aveva-gate-sdk2-887.json.gz.
Client-visible: every OPC UA client that ranks or filters Galaxy alarms by
Severity sees the order reversed (live and HistoryReadEvents). ScadaBridge has
no code that depends on the old values; its own mixed severity directions are
ScadaBridge#129.
Tests updated because they pinned the inverted table:
- Driver.Galaxy.Tests MxAccessSeverityMapperTests (rewritten: edges 1/250/251/
500/501/750/751/999, 0/-100/int.MinValue/1000/int.MaxValue clamps,
monotonic-decreasing sweep, equals GalaxyAlarmPriority for -2..1001)
- Driver.Galaxy.Tests GatewayGalaxyAlarmFeedTests (750 and 600 -> Medium)
- Driver.Historian.Gateway.Tests EventMapperTests (band table, priority 999
-> 250, 1 -> 1000; system events -> 1)
- OpcUaServer.IntegrationTests GalaxyEquipmentEventHistoryWireTests (700 ->
500, 100 -> 1000, System.Stop -> 1; the fake now stores AVEVA's real 0-4
severity)
New tests:
- Runtime.Tests NativeAlarmRestatementEndToEndTests
.Galaxy_live_condition_severity_is_the_AVEVA_priority_band (8 rows: the real
feed -> real GalaxyDriver -> host -> projector; the same rows EventMapperTests
pins on the history path)
- EventMapperTests.Aveva_severity_zero_marks_an_event_that_is_not_an_alarm
(5 rows, incl. the priority-999 alarm positive control)
- GalaxyDriverAlarmSourceTests: OpcUaSeverity passes through
Mutation check, 8 mutants, all killed: the pre-#887 inverted table; the
driver dropping OpcUaSeverity; no not-an-alarm rule; each of the three band
edges off by one; history ignoring priority; a missing AVEVA severity read
as 0.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK memoizes INodeManager3.ValidateEventRolePermissions per (item, event type,
SourceNode). #826 kept it fresh by reflecting into the private
MonitoredNode2.InvalidatePermissionCaches (plus a detach/re-attach fallback). Since #855
condition events never take that channel, and the search for #878 found nothing
production-raised that does except a condition's SDK method audits, which reach it only
with auditing on, SignAndEncrypt and a SecurityAdmin session (nothing grants that role).
Under enforcement the channel verdict is now a flat deny for items on our nodes, so no
cached verdict depends on a grant, and InvalidateFolderEventVerdicts, the reflection,
the fallback, its counter and the guard test are removed. Method audits still reach
Server-object items, decided per event.
Tests: FolderEventVerdictInvalidationTests -> FolderEventVerdictFreshnessTests; new
revoked-grant test (positive control: a still-granted session) and method-audit test
(SecurityAdmin mapped for the test; positive controls: Server-object item, flag off);
the plain-event test runs each mode on a fresh server. Mutation-checked.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A notifier folder read one event source, its NodeId identifier (a RawPath or an EquipmentId). AVEVA stores a
Galaxy object's A&E history under the object's tag_name (Source_Object), so AVEVA's own alarm and event
history reached no production folder. Its lower-case property keys were also missed, so every row read
Severity 1 with no ConditionName.
Sources. A folder's event history now reads its own source plus the Galaxy object of every native condition
that fires on it live: those parented under it and those WireAlarmNotifiers wired to it. Live and history use
the same membership. The object comes from the condition's Galaxy alarm reference, its raw tag's attributeRef,
up to the first '.':
- The composer reads it with the driver's own parser (GalaxyAttributeRef, shared with GalaxyDriver), only for
a GalaxyMxGateway tag with an alarm (RawTagPlan.NativeAlarmReference, part of plan equality).
- The applier binds it after materialising the condition (IOpcUaAddressSpaceSink.BindNativeAlarmHistory,
forwarded by the deferred sink). A rebuild or the condition's removal forgets it.
Each object is read with the #882 rule: its own events and <Object>.<Attribute> rows, never a prefix sibling.
Fan-out. historiangw's ReadEvents filter is a single predicate, so there is one request per source; the wire
contract is unchanged. EventSourceFanOut merges the reads as one source:
- order is (time, source position, arrival), reversed for a reverse read; ties are never re-sorted by EventId,
so the boundary skip stays stable;
- duplicates by EventId are dropped and the merge is cut to the cap;
- a truncated source bounds the page at the time it stopped.
Paging, continuation points (bound to the whole source set), the walked-back reverse read and the tie-cluster
over-fetch are unchanged. A folder with one source is read exactly as before.
ACL. Every source is read inside the folder's own HistoryRead, after AuthorizeHistoryRead (#826). A denied
folder reads nothing and fails closed.
Mapping (EventMapper, property names now case-insensitive):
- Severity: an AVEVA row's priority goes through GalaxyAlarmPriority, the table live Galaxy alarms use
(MxAccessSeverityMapper now delegates to it), so history matches live. Otherwise severity (OtOpcUa's own
rows, which AVEVA stores lower-cased), clamped. AVEVA's own 1-4 severity is ignored. The table ranks a
higher priority as more severe, which inverts AVEVA's convention: filed as #887.
- An AVEVA row whose source_conditionvariable names a native condition that fires on the folder takes that
condition's SourceNode and live ConditionName. Otherwise ConditionName is alarmname or the condition
variable's attribute.
- alarm_id/alarmid map to AlarmId.
Tests:
- GalaxyEquipmentEventHistoryWireTests (4, real SDK session, production Gateway backend, AVEVA-shaped fake
gateway): equipment EqA backed by objects Oven_007 + OvenDoor_07. Covers object and attribute events from
both objects, the prefix sibling excluded, cross-object and in-object ties, forward paging at 1-4 a page,
reverse whole and paged, SourceNode/ConditionName/Severity, the raw device folder, ACL deny with no backend
call plus a granted positive control, and a rebuild forgetting the bindings.
- EventSourceFanOutTests (8), EventMapper AVEVA cases (11), GalaxyAttributeRefTests (16), a composer test and
an applier ordering test.
- Mutation-checked, 17 mutants, all killed. Covered: parent-only membership, no scope, a case-sensitive
condition match, no condition rewrite, no truncation bound, EventId tie order, no sort, no cap, no stop at
the bound, unreversed source order, no dedupe, no '.' boundary, priority ignored, case-sensitive keys, no
applier bind, no deferred forward, and a composer that ignores the driver type. The forget-on-rematerialise
guard survives as belt-and-braces: the rebuild clear covers it.
- Live, read-only, no WinRM, vd03 HistorianGateway ea9f21c: equipment folder EQ-886-ZipsenLeft (objects
DevWinPlatform + Left_001). It read 0 events before and 92 now, equal to the unfiltered read's rows for both
objects. 1/7-event pages and reverse whole/5-event pages agree, and 73 rows resolve to their live condition.
diff_runs vs the #882 run: 221/226 identical. The 5 that differ are expected: DevWinPlatform Severity
1 -> 750/1000, and #881 capabilities.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ConfigurationBinder appends configured collection items to an existing list, so
the pre-filled [None, Sign, SignAndEncrypt] default made "drop None" a no-op:
every deployment that listed only the secure profiles still advertised an
unsigned/unencrypted None endpoint.
EnabledSecurityProfiles is now null by default; ResolveEnabledSecurityProfiles()
applies the unchanged three-profile default only when nothing is configured.
A set-but-empty list fails startup validation and BuildSecurityPolicies no
longer falls back to a None endpoint on empty input (it throws). Sweep of all
options classes found no other non-empty pre-filled collection default.
Tests: options binding (indexed, JSON, env forms, default, empty, bad name), a
booted-server wire test (no None endpoint, None connection refused with
BadSecurityPolicyRejected, secure connection works; default case is the control).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
SDK 2.0.1's UserNameIdentityTokenHandler encrypts a password longer than 64 bytes
(a private RsaEncryptedSecretPasswordThreshold, no setting) as an RsaEncryptedSecret
with EncryptionAlgorithm=null. SDK 1.5.378 (.106 and .182) decrypts only the legacy
format under RSA policies; its SecurityPolicies.Decrypt answers a null algorithm with
BadIdentityTokenInvalid ("Unexpected encryption algorithm") before any key check.
Every production API key is 65 chars, so no SDK 2 OtOpcUa client could use one
against QA (b8b4c18f), a rollback target or an older third-party server.
LegacyRsaUserNameIdentityTokenHandler (Driver.OpcUaClient.Session, plugged in through
the public UserIdentity(IUserIdentityTokenHandler) ctor) always uses the legacy
format, RSA-OAEP(len|password|serverNonce) under the server certificate, under an RSA
policy. That is what SDK 2 sends for passwords of 64 bytes or less and what SDK 1.5
clients always send. The legacy format has no size limit (the plaintext is
length-framed over as many RSA blocks as it needs), and an SDK 2 server reads both
formats. Policy None, the ECC policies and a missing certificate go to the SDK
handler unchanged.
OpcUaUserIdentityFactory.UserName is now the one UserName identity the driver, the
AdminUI browse and Client.Shared (CLI/UI) present. Client.Shared references .Session
for it.
Tests: handler unit tests (all three RSA policies, the 1.5-required algorithm URI,
SDK 2 server-side decrypt, the session Copy() keeps the type, None unchanged, and an
SDK control pinning the 64/65-byte switch); a Client.Shared identity test; and an
SDK2<->SDK2 integration test where a 65-char key is admitted in both formats and a
wrong 65-char key reaches the key check (BadIdentityTokenRejected). The #800 Windows
gate's SDK 2 CLI 65-char case now gates against the 1.5 server too.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK starts Server.ServiceLevel at 255 (full service), and OpcUaPublishActor publishes nothing
until the node's first redundancy snapshot arrives, so a restarting pair node advertised 255 for the
~10-13 s it took to join, out-ranking the real Primary (250) for any client that picks the highest
ServiceLevel.
OtOpcUaSdkServer now writes StartupServiceLevel = 0 in OnNodeManagerStartedAsync: after the SDK's
CreateServerObjectAsync wrote 255 and before the server goes Running, so no session can ever read
255. 0 is the design's existing Down / Detached tier (ServiceLevelCalculator returns 0 for a member
that is not Up/Joining, and a node with no snapshot has no role yet). It is below the SDK 2
session-headroom floor (200), so it stands through session churn before the #884 claim, and the
claim's bind replays any level already requested. Single-node deployments are unaffected beyond
the window: the first snapshot publishes 250 as before.
New Node_with_no_requested_level_advertises_the_startup_level_until_one_is_published: reads 0
before the bind, 0 after a bind with nothing to replay, then 250 once published. Without the
startup write it reads 255 on the first session (mutation-checked). docs/Redundancy.md updated for
#884 and #885.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
HistorianGateway answers a per-source ReadEvents (Source_Object = DevWinPlatform) with the
object's own events (SourceName DevWinPlatform) AND its attribute alarms, which it reports as
<Object>.<Attribute> (Source_ProcessVariable, e.g. DevWinPlatform.CPULoadAvg). The Gateway
backend's defensive client-side source filter compared SourceName for equality, so it dropped
the attribute alarms: 70 of the 72 DevWinPlatform events on vd03.
The filter (and the past-the-cap "more events" signal) now uses EventSourceMatch: the source
itself, or <source>.<anything> on the '.' boundary, ordinal. DevWinPlatform never matches
DevWinPlatform2 or DevWinPlatform2.X. This needs no wire change and still makes one request.
Each event keeps its own SourceName. SourceNode stays the condition its stored alarm id names,
which is null for AVEVA rows. The attribute events ride the notifier's own read, so the
per-folder HistoryRead verdict (#826) governs them: a denied folder answers
BadUserAccessDenied before any backend call. The SqlServer and QuestDB backends still match
exactly; they hold only OtOpcUa's own events.
Tests:
- GatewayAttributeEventSourceTests (16): the boundary theory, attribute-in/sibling-out through
the data source, the drain signal, and an exact attribute-named read.
- AttributeEventHistoryWireTests (2, over a real SDK session with the production Gateway
backend on an AVEVA-shaped deterministic gateway): object + attribute events with their own
SourceName, a prefix sibling excluded, paging, and the ACL-on deny with no backend call plus a
granted positive control.
- Mutation-checked: exact match fails both wire tests; dropping the '.' boundary fails the
wire sibling check and 6 unit tests.
- Live AVEVA harness re-run against vd03 HistorianGateway ea9f21c (read-only, no WinRM):
folder DevWinPlatform 2 -> 72 events (whole, 72 1-event pages, reverse); the new #882 check
passes; 221 of 225 shared cases are identical to the #801 SDK2 baseline, and the 4 that
differ are exactly the DevWinPlatform event cases. Recorded in aveva-history-801.md.
Follow-up #886: production folders name RawPaths/EquipmentIds, not Galaxy objects, plus the AVEVA
severity/condition property mapping.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Both builds served HistoryRead from the real AVEVA Historian via the DEV
HistorianGateway (vd03, ea9f21c): 225 cases / 3,855 rows / events, 0 differences.
Raw also matches direct SQL on Runtime.dbo.History (6/6) and histgw-cli; simple
aggregates match a recomputation from raw (336/336). Modified, HistoryUpdate,
seven refused aggregates and HistoryServerCapabilities answer as before. No write
reached the historian. Findings (pre-existing, not SDK): all HistoryServerCapabilities
booleans false; AVEVA attribute-alarm events unreachable per source.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Profiled the remaining lmxopcua#789 misses on windev, a VMware VM, using per-thread CPU and context-switch accounting
plus ETW CPU sampling.
- One blocking hand-off costs ~26 us of latency and ~18 us of CPU there, about 10x macOS.
- An idle evaluation makes four hand-offs in a row: host writer -> worker reader -> evaluation thread -> host reader
-> dispatcher.
- 48% of host CPU at 10k/s is in the kernel: the per-RunAsync NtWriteFile, thread-pool wakes, reads and waits.
Spin-before-block was measured at every hand-off and is not kept. It meets 150 us idle p50 (118 us) only by keeping
about two host cores and four worker cores busy for one stream. At 10k/s it reads +250 points.
Harness: new "paced 10,000/s, batched" row. It uses the same rate and keys, with each 1 ms tick submitted as one
RunBatch call, which is the production seams' shape. It runs after the original four phases, so their rows stay
comparable. On windev it halves host CPU, to +36..+45 points against +71..+84 unbatched. That is still above 25.
Design doc:
- 12.8: the record.
- 14 #15: the owner's decision on the Windows budgets. Recommended: measure on bare-metal Windows, then state the two
budgets per platform, with the batched row as the gate. The alternative is a shared-memory transport.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
OpcUaApplicationHost left ServerConfiguration.MaxRegistrationInterval at the
SDK default (30000) with a null RegistrationEndpoint, so SDK 2.0.1's
StandardServer armed its registration timer against opc.tcp://localhost:4840,
which on an OtOpcUa node is the server itself (no LDS). It answered
BadServiceUnsupported and the SDK retried forever, visible since #874.
Registration is now opt-in: ApplyDiscoveryRegistration sets
MaxRegistrationInterval = 0 (the value that keeps StandardServer from arming
the timer) unless OpcUa:DiscoveryServerUrl is set, in which case the server
registers against that URL every OpcUa:DiscoveryRegistrationIntervalMs
(default 30000) over SignAndEncrypt/Basic256Sha256, like the SDK's own
default endpoint. The validator requires an absolute opc.tcp:// URL and a
positive interval when the URL is set. Documented in Configuration.md and
OpcUaServer.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Measured on windev (3 alternated sessions) and macOS: a pool resume in place
of the dispatcher, and an inline-on-reader upper bound, with the harness as
committed and with its paced continuation made synchronous.
- pool: 10k/s host CPU unchanged, idle p50 and idle CPU worse (macOS idle CPU
x3 from pool spin); only the closed loop gains. Not kept.
- inline (unsafe for every seam; the ceiling for a VirtualTag-only opt-in):
~8 points of a core at 10k/s raw and ~50 us idle p50 on windev with a cheap
continuation, nothing with the harness as committed. Both budgets still
missed by a wide margin, so the opt-in was not built.
No code change. Records the remaining attribution (harness pool hop ~15
points raw; reader wake + syscalls; worker side untouched).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.
DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PASS: gauge baseline 2, +2 with the rig's driver on both centrals, 4 after 20
config deploys (40 respawns; rig opened 39 / closed 37). Final drop after
removal not observed: the Mac's VPN to VD03 dropped right after the settle.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Read-only ConfigDb preflight (scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql)
covering every deploy gate added since the deployed build: VirtualTag/Calculation effective
triggers (#783), VT->VT reads and VirtualTagDependencyCycle (#791), newly forbidden script
APIs (#792, E2), credential-reference namespace / other-instance / reserved-name rules
(#815#835#836#848#849#850), MTConnectMultipleDevices, Sql query/write gates (#781),
Heidenhain, UNS effective-name collisions and the FOCAS renames/value changes. Verified on a
throwaway b8b4c18f-schema database (every check fires on seeded rows) and clean against the
docker-dev config.
Upgrade notes (docs/deploy/vd03-upgrade-b8b4c18f-to-master.md): the EnableSnapshotIsolation
migration and how to apply it by hand, the env-var audit (Security:ApiKeys:Pepper is now
required and absent on the box; JWT key validated at start), SDK 2.0.1 / CA KeyUsage, the
ScadaBridge-visible changes, swap/rollback and post-deploy checks.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.
Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Joseph Doherty