Commit Graph

177 Commits

Author SHA1 Message Date
Joseph Doherty e39a125069 fix(acl): close the #808 review findings — reference variables, owner folder chain, NUL keys, unknown ids (#808)
Review of af5c9744:
1. UNS reference variables (s=Area/Line/Equip/Eff) of a moved or renamed
   equipment stayed readable/writable under the OLD names' grants: they now
   resolve as ReferenceVariable and are authorized at their split path only
   while it equals their live equipment folder's placement (live chain ==
   generation chain) + one name; otherwise denied (Read and Write).
2. TryPlaceScriptedCondition now also requires the owner folder's live chain
   to match the generation (AuthorizeScriptedCondition gains ownerFolderChain),
   so a moved equipment's alarm/children/commands follow its folder.
3. Identifiers containing NUL are denied before the decision cache, keeping
   client ids disjoint from the NUL-prefixed resolved-scope keys (a client
   could otherwise poison a real folder's cached verdict for its key shard).
4. An id no live node holds is Unknown: placed at [\0unknown] so only
   Cluster / Namespace-uns grants decide it (a removed opaque id can no longer
   split onto a same-named narrow grant; narrow grantees get no NodeDeleted for
   removed UNS folders/vtags/conditions — documented); 4-segment
   reference-shaped ids keep their split.
5. A wire test drives placements from DeploymentArtifact.ParseDataPlaneAuthority.

Baseline updates: #797 owner tests (unit + wire) and Sdk2RepeatedAlarmAuthority
now install folder scopes; Sdk2HistoryAuthority's UNS alias used the RAW path
as its UNS id (never minted in production) and now carries EqA/HistoryAlias.
Each new guard was falsified (reverting it fails its test).
2026-11-10 23:44:11 -05:00
Joseph Doherty af5c9744b7 fix(acl): place opaque-id UNS folders, virtual tags and scripted conditions for every operation (#808)
UNS Area/Line/Equipment folders are minted under their opaque entity ids
(s=UA-/UL-/EQ-...), virtual tags under s=EQ-.../Name and scripted conditions
(+ #629 children) under s=<ScriptedAlarmId>[/...], while every UNS grant is
planted at a NAME path. DataPlaneAuthorizer split those ids, so with
Security:Acl:Enabled a narrow (Area/Line/Equipment) grant could not Browse,
Read, subscribe or HistoryRead them, and Server-object delivery never reached
it — so a narrow key never held the EventId Ack/Confirm/AddComment need.

- The generation carries UnsFolderScopes (folderId -> path + id chain), built
  by DeploymentArtifact from the same AclScopePathResolver as the grants; ids
  two entity kinds share are dropped; plumbed through IDataPlaneAuthoritySink.
- OtOpcUaNodeManager.ResolveUnsAclSubject reports live facts (folder id
  ancestry / scripted owner binding); attached by the Authorizer setter.
- Evaluate places the subject against the CURRENT generation before the
  decision cache, which now caches the walk keyed on the resolved scope.
  Folders require live chain == generation chain (a moved equipment is denied
  under old and new line until the rebuild). Owned nodes (vtags, conditions,
  children) get owner path + a reserved segment, so Tag grants and their
  Browse transit never expose them. Conditions share TryPlaceScriptedCondition
  with the #797 command gate.
- Server-object visibility of a scripted condition is decided on the
  condition alone, so a moved alarm cannot reach its old owner there.

Tests: UnsFolderScopeWireTests (opaque ids, end-to-end browse -> EventId ->
Acknowledge on an Equipment grant; sibling denied; Tag-only grant; moved alarm
and moved equipment stale windows). Rig tests now install folder scopes
explicitly (ServerEventRig.FolderScopes); the #797 recreate test deliberately
asserts the moved alarm is invisible to the Cluster observer in the window.
2026-11-10 23:12:46 -05:00
Joseph Doherty 3d1b86b451 fix(acl): authorize scripted alarm commands at their owning equipment's scope (#797)
Scripted condition NodeIds are opaque ScriptedAlarmIds, so splitting them as a
topology path could never match an Equipment grant: a narrow equipment grant was
denied while a Cluster grant passed.

- OtOpcUaNodeManager binds each LIVE scripted condition instance to the
  EquipmentId it was materialised under (only when it hangs under that folder);
  dropped on remove / drop-recreate / equipment-subtree removal / rebuild; a
  same-id re-materialise under another equipment re-parents and re-binds.
- DataPlaneAuthoritySnapshot carries EquipmentScopePaths (EquipmentId ->
  [uns, Area, Line, Equipment]) decoded from the artifact by the same
  AclScopePathResolver call that plants Equipment grants.
- DataPlaneAuthorizer.AuthorizeScriptedCondition re-reads key, grant and
  generation per call (uncached) and evaluates at the owner path; unbound,
  replaced or unplaceable owners deny under every grant. No id-split or
  realm-root fallback.
- Native conditions (RawPath) and the ACL-off role gate are unchanged.
  NodeIds and AlarmCommand.AlarmId are unchanged.

Sdk2OpaqueScriptedScopeBaselineTests deliberately flipped: it pinned the denial
and now pins owner-grant allow / sibling deny / cluster allow. Follow-up #808
(folder ids + Server-object event visibility for narrow UNS grants).
2026-11-10 21:23:57 -05:00
Joseph Doherty 41e65f6a22 fix(opcua): restore alarm component reference after SDK creation 2026-11-10 12:14:35 -05:00
Joseph Doherty a21104d27c fix(opcua): scope application audit receive permissions to validation 2026-11-10 11:41:31 -05:00
Joseph Doherty 09ac0e58cb fix(opcua): preserve repeated alarm methods and owned write reverts 2026-11-10 11:04:53 -05:00
Joseph Doherty ac5e9d84f0 fix: propagate SDK request cancellation to historian calls 2026-11-10 10:39:39 -05:00
Joseph Doherty 9712e66ff4 fix: retain certificate ownership and optional alarm workflows 2026-11-10 10:37:28 -05:00
Joseph Doherty b71620145b fix: preserve SDK role identity and continuation authority lifecycle 2026-11-10 10:30:18 -05:00
Joseph Doherty 3dcb035f89 fix: preserve history sentinels and alarm SDK metadata 2026-11-10 10:21:41 -05:00
Joseph Doherty f9a006dab4 fix: initialize SDK telemetry and preserve ByteString boundaries 2026-11-10 10:11:32 -05:00
Joseph Doherty 9af1a824b3 refactor: migrate OPC UA server SDK 2.0 compilation and boundaries 2026-11-10 10:04:27 -05:00
Joseph Doherty e790938f94 fix(historian): QuestDB modified reads keep a landed change whose marker was lost; H2 review NITs (lmxopcua#778)
QuestDB modified-history reads dropped every still-pending trail row that was
not the latest at its source time, so V0->V1 (applied marker never written)
followed by V1->V2 hid the V0->V1 change for good. A pending row below the
latest is now served when the NEXT trail row at that time recorded a different
prior value (kind, value, status or the prior's server_ts) - the next change saw
what this one wrote; an orphan of a change that never landed leaves the same
prior for the next row and stays hidden. QuestDB window functions reject
VARCHAR, so the next row's prior comes from a LEFT JOIN on (ts, lead(id)).
The updater's comment and marker-failure log no longer claim modified reads
"still serve it" unconditionally.

Regression tests (QuestDB 10.0.1 throwaway container): marker lost then a
replace / delete at the same time; an orphan followed by a marker-lost retry
then a later change (orphan still dropped); the random marker-mix fuzz now
varies the recorded priors. All four fail on the old filter.

NITs:
- SqlServerHistorianValueWriter: a SqlException raised while the token is
  cancelled (SqlClient's "Operation cancelled by user") is rethrown as
  OperationCanceledException instead of booked as a failed write. Integration
  test holds a TABLOCKX on Sample and cancels mid-INSERT; it returned false before.
- SqlServerHistorianDataUpdater.DeleteRawBatches is an Interlocked counter.
- HistoryReadEvents: an event filter whose SDK validation throws (unknown
  operator) failed with the generic BadUnexpectedError; the node manager now
  re-throws any failed event-filter validation as BadEventFilterInvalid.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 16:17:24 -05:00
Joseph Doherty 29665c1872 feat(opcua): HistoryReadEvents projects EventType, ConditionName, SourceNode; honours the where-clause (lmxopcua#746)
HistoryReadEvents projected only EventId / SourceName / Time / ReceiveTime /
Message / Severity: a select clause asking for EventType, ConditionName or
SourceNode (what most generic alarm viewers ask for) got nulls, and the
event filter's where-clause (an OfType) was ignored.

- HistoricalEvent gains optional EventTypeName, ConditionName and AlarmId.
  SqlServer, QuestDB and Gateway event reads fill them from what their
  alarm writers already store (AlarmTypeName / AlarmName / AlarmId; the
  gateway event Type and its AlarmName / AlarmId properties).
- The node manager maps the type name to its Part 9 event type (unknown or
  non-event => AlarmConditionType), projects ConditionName, and resolves
  SourceNode to the materialised condition the alarm id names (the live
  events' SourceNode), else null.
- The where-clause is evaluated per event with the SDK content filter
  through an IFilterTarget (OfType via the type tree; other operators
  against the projected fields), after the page's cursor is taken, so
  paging is unchanged.

Tested in the node manager, over a real client session (the Client.CLI has
no `historyread --events` mode yet; filed as a follow-up), and by alarm
writer -> event read round trips on SqlServer and QuestDB.

Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:40:47 -05:00
Joseph Doherty 8e8d4cdf13 fix(alarms): surface a refused native alarm ack synchronously, revert one that fails later (lmxopcua#760)
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:

1. Synchronous refusal for what the driver can decide without I/O. New optional
   IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
   anonymous session when the driver has no AlarmAck fallback account.
   DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
   NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
   NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
   returns BadUserAccessDenied with the reason - nothing routed, applied or
   reported. A keyed session is not pre-checked (its identity lives in the
   Secrets store, which must not be read under the SDK lock).

2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
   DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
   a driver without an alarm source, or an ack dropped while connecting /
   reconnecting. DriverHostActor re-projects the condition unacknowledged via
   NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
   still pending - any driver transition projected since then wins. A disabled
   or unspawned driver is not reverted (unchanged).

GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.

Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:12:17 -05:00
Joseph Doherty dc0a24f224 feat(galaxy): operator-authenticated alarm ack — send Galaxy credential to mxaccessgw (#749)
v2-ci / unit-tests (push) Failing after 1h8m43s
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:

1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
   held ONLY in the encrypted Secrets store
   (otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
   no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
   FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
   env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
   operator_user. Refusals are excluded from the resilience retry/breaker.

- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
  nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
  RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
  is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
  ("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
  account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
  acknowledger scrubs it from every gateway message it echoes and withholds
  any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
  Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
  path is fire-and-forget after the Part 9 method returns).

Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.

Closes #749
2026-11-06 16:23:00 -05:00
Joseph Doherty f966efcfad fix(opcua): replay writes deferred during an off-thread plan; cache-ref and guard fixes (Lane B fix round 1)
C1 (critical): B6 computes the plan off the pinned thread while B1 made writes
non-creating, so a value/status/alarm write for a node the in-flight plan adds
was skipped, and change-driven drivers never republish it (BadWaitingForInitialData
indefinitely; on a full rebuild every re-subscribe value landed on discarded
nodes). OpcUaPublishActor now applies writes immediately AND remembers the
latest per (realm, id) while a plan is in flight or queued (value+status
compose; alarm state/quality coalesce to the latest, which the node's delta
gate makes inert on an existing condition), replays it after each apply before
answering, and drops it once the pipeline is idle. Bounded (250k keys by
default; beyond it new keys are not held, logged once).

I2: the data-plane-authority parse runs in its own try inside the plan task; a
throw is logged and never degrades the address-space apply.

I3: the SDK's DeleteMonitoredItems releases an item's component-cache
reference on a Good delete; a tombstoned item's reference was already
released, so the delete took one from the re-created node's entry. Released
items are remembered and the orphan delete takes one back first. Event items
never held a reference (EventManager + SubscribeToEvents), so the three
event-item releases are gone.

I4: direct tests for the HistoryUpdate override's per-request updater check
and the HistoryReadEvents override's source guard (both reached by restoring
the bit after a reset - the B7 race window); fixed the false remark.

Minors: the MaterialiseAlarmCondition kind-swap tombstones the dropped
instance; a bytes-less rebuild arriving mid-plan queues behind it (replies in
order); a PlanReady/PlanFailed from a previous actor incarnation is ignored and
its requester answered degraded; the tie-cluster end bound of an uncapped read
fits against the backend ceiling (defence in depth - guarded by a sweep test);
the lock-time benchmark stays in the default run, headroom noted.

Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:30:11 -04:00
Joseph Doherty be22b48b1d feat(opcua): advertise the historian raw cap as MaxReturnDataValues on source sync
Lane H1 follow-through: SyncAdvertisedAggregates now also calls
HistoryPaging.AdvertiseMaxReturnDataValues(Server, source
.MaxRawValuesPerRead) so HistoryServerCapabilities/MaxReturnDataValues
(i=11273) follows the historian source on every assignment (the Null
default advertises 0 = no limit); a false return logs a Warning (the cap
is enforced by HistoryRead regardless).

Test: NodeManagerMaxReturnDataValuesSyncTests reads i=11273 back through
the SDK master node manager after setting a capped source (7000, 250)
and after resetting to the Null source (0).

Refs #732 #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty 8a1116e658 fix(opcua): fit the raw end bound against the page's effective cap
H1 fuzz finding (routed to Lane B because AppendEndBoundAsync lives in
OtOpcUaNodeManager.cs): a bounded HistoryRead-Raw page sized BELOW
NumValuesPerNode by the backend's row ceiling (RawPageLimit.Plan) took
the end bound whenever its data fitted, because the fit was judged
against NumValuesPerNode instead of the page's effective cap — a full
ceiling-sized page came back with cap + 1 values. The bound is now
fitted against the page allowance (the ceiling-adjusted data cap plus
the start bound a first page already carries); when the page is full
the bound goes to the next page through the continuation point.

Test: NodeManagerHistoryReadBoundsTests
.A_full_ceiling_sized_page_carries_the_end_bound_on_the_next_page
(ceiling 4 ⇒ 3 data rows; first page = start bound + 3 with a CP, second
page = the end bound alone; control: no ceiling ⇒ one page of 5).

Refs #732 #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty d90f35b6be fix(opcua): refresh historian capability bits on late source/updater assignment (B7)
Review 01 U3. EnsureFolderIsEventNotifier granted the EventNotifiers
.HistoryRead bit (and registered the event-history source) only when a
real HistorianDataSource was wired at promotion time, and EnsureVariable
composed the AccessLevels.HistoryWrite bit from whether a
HistorianDataUpdater existed at materialise time — so an assignment
after the first apply left existing folders without event history and
existing nodes refused by the SDK's HistoryUpdate admission until a full
rebuild, and a source reset to Null never took the bit back.

The two setters now re-evaluate: HistorianDataSource walks every promoted
folder (bit + source registration, both directions), HistorianDataUpdater
recomposes the AccessLevel of every historized variable (Writable read
back off the node's own CurrentWrite bit). The Host's boot ordering is no
longer load-bearing.

Tests: NodeManagerCapabilityRefreshTests (late source lights the bit on
an already-promoted folder and a reset takes it back, live-event
subscription untouched; late updater grants HistoryWrite to historized
nodes only and a reset takes it back, authored Writable preserved).

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty 1578efd4a9 fix(opcua): write-path robustness, injected clock, configurable quotas (B4)
Review 01 L3/H5/H4/P6/P4.

- L3/H5 OnEquipmentTagWrite's dispatch is factored into DispatchDeviceWrite:
  a gateway that throws synchronously is refused inline with
  BadDeviceFailure (it used to escape the handler under the SDK Lock), and
  a faulted/cancelled dispatch task has its exception observed and logged
  and carries the real reason into the revert/audit event instead of
  surfacing as UnobservedTaskException at finalization.
- H4 every timestamp the manager stamps (condition quality, timed-shelve
  expiry, method-transition events, carry-over, revert, revalidation)
  reads the injected Clock; RemainingShelveMs takes the clock reading.
- P6 OpcUa:Quotas:* (OpcUaQuotaOptions on OpcUaApplicationHostOptions):
  Min/MaxRequestThreadCount, MaxQueuedRequestCount with the former
  hard-coded 5/100/200 as defaults, and the TransportQuotas members as
  nullable overrides over the SDK defaults.
- P4 the SDK line exposes IHistoryReadAsyncNodeManager but CustomNodeManager2
  offers no async virtual to override (adopting it means re-implementing
  the base dispatcher), so the lever is the admission cap:
  MaxConcurrentHistoryReadBatches defaults to clamp(2*ProcessorCount, 4, 16).
  The Host still overrides it from ServerHistorianOptions (default 16) —
  follow-up F3.

Tests: NodeManagerWriteDispatchTests (sync throw refused; faulted task
reverts; success leaves the value; clock-stamped revert),
OpcUaApplicationHostQuotaTests.

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty dca822dc59 perf(opcua): de-quadratic the lock-time teardown paths (B3)
Review 01 P1/P2/P3/P5 — work done under the manager-wide SDK Lock on the
pinned publish thread, where every Read/Browse/Publish waits.

- P1 RebuildAddressSpace detaches the shared root's top-level children
  once instead of every leaf from its parent (NodeState.RemoveChild is a
  linear scan + RemoveAt, ~k²/2 compares per k-child folder); root
  notifiers are cleared as a list instead of one O(F) RemoveRootNotifier
  each; ForgetPredefinedSubtree / TombstoneMonitoredSubtree are iterative
  with one child buffer.
- P2 RemoveEquipmentSubtree walks a parent→children index maintained by
  EnsureFolder/EnsureVariable/MaterialiseAlarmCondition (keyed by the
  parent id the caller named) instead of prefix-scanning six maps per
  removed equipment; a surviving condition's wiring into a removed folder
  is found through the folder's own notifier list; descendants are not
  detached from a folder that is itself being discarded.
- P3 LowerAllEventsMonitoring(condition) replaces the full O(C) reconcile
  on RemoveAlarmConditionNode / RemoveEquipmentSubtree / the kind-swap.
- P5 AlarmNotifierWiring.Folders is a reference-identity HashSet and an
  already-wired pair skips the SDK's linear AddNotifier dedup.

Benchmark (NodeManagerLockTimeBenchmarkTests, M-series Mac under load,
three runs each): rebuild of a 10k-tag folder 75–150 ms → 5–6 ms;
removing 200 equipment (250 signals each) from a 50k-node space
471–681 ms → 58–64 ms; all-events bookkeeping visits for 10 condition
removals over 50 conditions 900 → 10. The two timing legs assert < 1 s;
the P3 leg asserts the visit count (machine-independent).

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:31 -04:00
Joseph Doherty eaac65286d fix(opcua): tombstone MonitoredNode2 on every node removal (B2a)
Review 01 L2. The SDK keys its fan-out lists by NodeId and its component
cache hands back the CACHED NodeState for a NodeId it already holds, so a
rebuild or a surgical remove+re-add left a NEW item on the same NodeId —
from any session — bound to the DEAD node: created Good, fed its stale
initial value, never again, until every session had deleted its items.

Every removal path (RebuildAddressSpace, RemoveVariableNode,
RemoveAlarmConditionNode incl. the condition's instance-scoped children,
RemoveEquipmentSubtree) now queues BadNodeIdUnknown to the node's data
items, drops the MonitoredNodes entry, releases each item's component
cache reference and forgets the _liveItems/_eventItems entries. Items
the ACL pass stood down (on no fan-out list) are swept by NodeId. A later
client delete of an orphaned item is completed by the DeleteMonitoredItem
override / the SubscribeToEvents unsubscribe arm instead of answering
BadMonitoredItemIdInvalid or NRE-ing in MonitoredNode2.Remove. The
revalidation restore re-resolves the node by NodeId and no longer
re-attaches to Handle.Node blindly. Timestamps now come from the injected
Clock (UtcNow).

Tests: StaleMonitoredNodeWireTests (real SDK, three legs: rebuild,
surgical remove+re-add, second session; each keeps the old item alive,
proves the new item receives a value written after its creation, the old
one gets BadNodeIdUnknown, and its delete succeeds — all three fail with
the tombstone neutered), NodeManagerStaleMonitoredNodeTests (stood-down
item swept on surgical remove and rebuild; sibling removal leaves it
restorable).

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:31 -04:00
Joseph Doherty 00300d0f60 fix(opcua): no write ever creates a node; cross-kind id guard (B1)
Review 01 L1/L4/H2. WriteValue/WriteValues are non-creating (an unknown
id is skipped, counted and Debug-logged, as WriteStatuses always was), so
a driver publish racing a surgical remove can no longer resurrect the id
as a flat BaseDataType phantom under the shared root — nor hand a later
re-add of the same tag the phantom's parent/DataType/writability through
EnsureVariable's early return. WriteAlarmCondition for an unmaterialised
id is a logged no-op instead of minting the legacy bool[2] placeholder
(AddPredefinedNode is AddOrUpdate, so it replaced the SDK lookup entry of
any folder sharing the id). The applier no longer writes a condition
state against every removed equipment id (a folder, never a condition).
EnsureFolder/EnsureVariable/MaterialiseAlarmCondition refuse an id already
held by another node kind in the same realm. CreateVariable is deleted.

Tests: NodeManagerPhantomNodeTests (publish racing a remove leaves no
node; the re-add gets the authored shape and is live; unknown alarm id
is a no-op; the guard in all three directions, realm-qualified); the
sink / batch-write / write-statuses tests now declare their variables.

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:31 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
Joseph Doherty 97668d26d3 fix(historian): respect gateway ceilings and history value contracts
v2-ci / build (pull_request) Successful in 5m10s
v2-ci / unit-tests (pull_request) Failing after 28m34s
2026-10-13 03:51:04 -04:00
Joseph Doherty 3c3513ea3e fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
2026-10-13 01:39:32 -04:00
Joseph Doherty a5a0496176 Merge fix/693-gateway-attime: Gateway AtTime from raw reads + shared evaluator, node type hint (#693) 2026-10-13 01:18:09 -04:00
Joseph Doherty 0d249f3ee9 fix(historian): #693 AtTime type hint — Boolean nodes stepped, integer nodes rounded to their type
Review follow-up: on the Gateway every value is a Double (AVEVA stores a Boolean
as a 0/1 Int2), so the raw-evaluated AtTime interpolated a Boolean node to 0.5.

- IHistorianDataSource gains a default-interface overload
  ReadAtTimeAsync(fullReference, timestamps, TypeCode? valueType, ct) that
  forwards to the existing one (SqlServer/QuestDB/Null/fakes unchanged).
- OtOpcUaNodeManager.HistoryReadAtTime passes the node's scalar DataType
  (AtTimeValueType: built-in type -> TypeCode; arrays/other types => null);
  Raw and UNS NodeIds alike.
- GatewayAtTimeReader converts each raw row to the hinted kind BEFORE the shared
  AtTimeEvaluator runs (Boolean = value != 0 => stepped; integers => Int64 =>
  interpolated, rounded half away from zero, returned as the node's integer
  type; floats Double), so the evaluator's own type branch decides.
- AtTimeEvaluator boxed every interpolated Int64 as a Double (`cond ? (long)x :
  v` widens to double); its "keeping its type" test passed because Shouldly
  ShouldBe compares numbers across types. Fixed (affects SqlServer/QuestDB too)
  and pinned with ShouldBeOfType.
2026-10-13 00:52:03 -04:00
Joseph Doherty 8dfa165110 fix(historian): #692 every HistoryRead-Raw page is full — no empty page between values
A resumed raw page asked the backend for only N rows from the boundary timestamp
(inclusive), then trimmed the boundary ties already emitted: it netted N - skip values,
and at N = 1 nothing — the empty "full" page was taken for a stalled tie cluster and a
MaxTieClusterOverfetch + 1 (65,537-row) over-fetch stepped one tick, returning an EMPTY
page with a continuation point between every value (8 values = 17 pages / 25 reads),
forward and reverse, with or without ReturnBounds, on every backend.

Each page's read now asks for N + boundarySkip + 1 rows: the ties it trims, the page and
one lookahead row. The lookahead row is the 'more data' signal, so an exact multiple ends
on its last full page (no trailing empty page). The over-fetch fires only when every row
past the trimmed ties is still on the boundary (a genuine cluster larger than the page);
a cursor left inside a cluster is flagged WithinTieCluster and over-fetches directly.
A page that never leaves the boundary adds the prior skip to its cursor.

Low items: BoundedReadHasAnythingAsync carries its walk hint and, when there is no data,
the end bound it found (HistoryEndBound.Resolved + DataExhausted), so the next page reads
nothing; HistoryBounds.IsPlaceholder is an explicit marker (ConditionalWeakTable over the
NotFound instances) instead of status+null shape, so a stored Bad_BoundNotFound row is data.

Tests: NodeManagerHistoryReadPageFillTests (N=1 every shape/direction/bounds, tie clusters
straddling boundaries at N=1/2/3, exact-multiple, carried end bound + walk hint, stored
placeholder-shaped row). Existing paging tests updated where they encoded the short pages.
2026-10-13 00:42:13 -04:00
Joseph Doherty 934909b430 fix(historian): #691 review — clamp Gateway at-or-after read, GoodNoData when nothing exists, HonorReturnBounds kill switch
1. The at-or-after bound read on a forward-only backend (Gateway) ends at
   now + ReverseRawReader.LookAhead instead of DateTime.MaxValue (the gateway
   only checks start <= end, so a year-9999 end reached AVEVA); a bound beyond
   the horizon is not found without a read. SqlServer/QuestDB keep MaxValue:
   they bind it safely and may hold samples further ahead. The Gateway fake now
   refuses far-future ends, so the missing-end-bound test is no longer vacuous.
3. No data and no bound found (whole read empty) => GoodNoData, no values —
   the pre-bounds answer (6.5.3.2: success only if one or both bounds exist).
   A one-value page whose start bound is missing first checks there is data or
   a found end bound.
4. ServerHistorian:HonorReturnBounds (default true); false ignores ReturnBounds
   as before #691, including the modified-read Bad_InvalidArgument.
2026-10-12 17:52:34 -04:00
Joseph Doherty ebc6b8d4d9 chore(historian): #691 comment order in ServeRawPagedAsync 2026-10-12 17:52:34 -04:00
Joseph Doherty 60febbf2dc feat(historian): #691 HistoryRead-Raw ReturnBounds — Part 11 bounding values on every backend
The node manager composes the start/end bounds from each backend's existing raw
reads (one authority, HistoryBounds): at-or-before = a capped reverse read
(native, or ReverseRawReader's walk-back on a forward-only backend — never
AVEVA's synthetic boundary row), at-or-after = a capped forward read. Bounds
count toward NumValuesPerNode, the start bound leads the first page, the end
bound ends the last (carried in the continuation point). Missing bound =
Bad_BoundNotFound (null, at the bound time; one-sided reads ±1 s from the
previous value, Table 1 footnotes a/b). A modified read with ReturnBounds is
Bad_InvalidArgument (Part 11 6.5.3.3).

The .NET SDK's ReadRawModifiedDetails defaults ReturnBounds to TRUE; the
existing tests built details without setting it and now set it false.
Oracle: every row of Part 11 Table 1 over three backend shapes, plus paging
concatenation, ties, Bad bounds, microsecond truncation; conformance pins the
lookup primitives on SqlServer + QuestDB.
2026-10-12 17:52:34 -04:00
Joseph Doherty 574c3aa1b0 fix(historian): #691.4 review — Null reports SupportsReverseEvents = true, like raw
The null source has no history in either direction; reporting false
made the server walk an empty reverse events read back through three
empty forward reads. It now answers the reverse overload with the empty
result itself (guard test unchanged: it still declares both members).
2026-10-12 17:37:32 -04:00
Joseph Doherty ccda27d220 fix(historian): #691.4 review — uncapped Gateway events reads no longer pre-size to int.MaxValue
GatewayHistorianDataSource.ReadEventsAsync sized its result list to
maxEvents, which is int.MaxValue for NumValuesPerNode = 0: every
uncapped Gateway events read (forward on master, and the new walked
reverse one) threw 'Array dimensions exceeded' and counted as a backend
failure. Capacity is bounded at 1024. The events walk delegate moved to
ReverseEventsReader so it is tested over the real Gateway source
(forward uncapped, reverse uncapped, reverse capped).
2026-10-12 17:20:41 -04:00
Joseph Doherty d277f34a96 fix(historian): #691.4 walk reverse events reads back on forward-only backends (Gateway)
A reverse or end-only capped events read on a backend without
SupportsReverseEvents (Gateway, Null) was refused
BadHistoryOperationUnsupported (explicit reverse) or answered with the
OLDEST N ascending (end-only, #675). It is now served newest-first by
the same ReverseWalk as raw reads: page-sized ms-aligned segments,
never answered from a read the source flagged truncated, ties sorted by
EventId (the gateway orders by event time only - the fake shuffles
them per call), one extra event walked to signal 'more', uncapped = one
forward read reversed, and the width + proven-empty stretch carried in
the events cursor. The oversized-cluster over-fetch keeps exactly the
events at T in the same order. A tie cluster over the budget is
BadTooManyOperations. Docs note the gateway's EventReadMaxRows must be
unlimited (it truncates silently).
2026-10-12 17:19:05 -04:00
Joseph Doherty 186b7d5ea5 fix(historian): #691.5 a reverse walk proves the rest empty once and carries it page to page
The last pages of a deep reverse page-through on a forward-only backend
each walked down to 1970 one x4 growth step at a time (~14-16 reads,
~2.5 s live). After two empty segments the walk now asks once per page
for the whole remainder at the page budget (a truncated answer is
discarded and the walk carries on), and the stretch the page's last
complete read proved empty rides the continuation point
(HistoryContinuationState.EmptyFromTicks/EmptyToTicks, session-held, so
no versioning) so the next page steps over it. Fuzz now pages with the
walk's own hints and with true random empty stretches.
2026-10-12 17:19:05 -04:00
Joseph Doherty d33992034d fix(historian): #689 size reverse-walk segments from the page, not the tie budget
ReverseRawReader aimed every segment at half of MaxTieClusterOverfetch
(~32k rows by default) and carried that width page to page, so a paged
'latest N' read on a dense Gateway tag fetched ~32,770 rows per page
(2-5 s live) to return N. The walk now lives in a generic ReverseWalk
core: segments are aimed at ~N+8 rows (page budget 2N+16), a discovery
segment (first without a hint / after an empty one) may read 1,024, and
the tie budget is only the ceiling a single-timestamp cluster escalates
to. A truncated read never re-enters the stretch below its newest row
until the gap above it is bisected. Tests: per-page row counts on a
dense tag (node manager + walk), randomized exactness fuzz (walk and
node-manager paging), dense-tag-below-a-long-gap read bound.
2026-10-12 17:19:05 -04:00
Joseph Doherty cf1af55720 fix(historian): #686 review — bounded, density-aware reverse raw walk; uncapped Gateway raw reads
1. Uncapped reverse raw read on a forward-only backend = ONE forward read of the
   window, filtered and reversed in memory (was a segment-by-segment walk).
2. Segment width is aimed at ~half the row budget from the density just observed
   (overflow: from the truncated read's own sample span) and carried page to page
   in HistoryContinuationState.SegmentWidthTicks.
3. The walk starts no later than now + 1 day and never reads before the epoch.
4. Request bounds are rounded outward to the millisecond, so a row AVEVA
   fabricates at a rounded-up query start cannot land inside the segment.
5. Budget M = max(1, MaxTieClusterOverfetch); reads ask M + 2 rows, completeness is
   'short OR last row past the segment', and a read crowded by rows before the
   segment is re-read once (<= 2M + 4). A cluster of exactly M, neighbouring
   clusters each within M, and M <= 1 are all served; only one timestamp over M
   is refused. ServerHistorianOptions text updated.
6. The raw and events tie-path oversized-cluster refusal is BadTooManyOperations
   (was BadHistoryOperationUnsupported), consistent with the walk.
7. GatewayHistorianDataSource.ReadRawAsync: a caller cancellation is not a backend
   failure; NumValuesPerNode = 0 is sent as int.MaxValue (the client package and
   the gateway reject max_values <= 0, so every uncapped Gateway raw read failed);
   a MaxValuesCeiling refusal is a HistoryReadLimitExceededException.
8. Tests: upper-edge filter, epoch floor, look-ahead, ms alignment, exact/adjacent
   clusters, minimal budget, carried density, and the walk over the real Gateway
   source + FakeHistorianGatewayClient (OpcUaServer internals visible to it).
2026-10-12 15:11:21 -04:00
Joseph Doherty 410c8d8976 fix(historian): #686 serve reverse/end-only raw reads on forward-only backends (Gateway)
#669 turned an end-only capped HistoryRead-Raw into a reverse read (start > end)
for every backend; the gateway rejects start > end (InvalidArgument), so on the
default Gateway backend every end-only capped and every explicit reverse raw read
failed the node and was counted as a backend health failure.

- IHistorianDataSource.SupportsReverseRaw (default false; SqlServer/QuestDB/Null
  true, Gateway false; guard test pins that every production backend declares it).
- OtOpcUaNodeManager sends a reverse window only to a backend that declares it;
  otherwise ReverseRawReader walks (lo, hi] back in bounded forward segment reads
  (1 min, x4 while sparse; each <= MaxTieClusterOverfetch + 1 rows; a full read is
  re-read narrower, never answered truncated; filtered on sample timestamps, which
  also drops AVEVA's fabricated query-start rows; stops at the Unix epoch). A
  one-timestamp tie cluster over the bound => BadTooManyOperations.
- ServeRawPagedAsync maps HistoryReadLimitExceededException to BadTooManyOperations.
- GatewayHistorianDataSource.ReadRawAsync refuses start > end unsent, without
  recording a health failure.
2026-10-12 14:42:27 -04:00
Joseph Doherty c019d0fc2b fix(historian): trim only HistoryServerCapabilities/AggregateFunctions; null set = all (#682 review)
ServerCapabilities/AggregateFunctions (i=2997) is Part 5's list of every
aggregate the server supports, and live AggregateFilter subscriptions still
compute all 37 SDK aggregates, so it keeps the SDK's full set; only i=11201
follows the backend. A null SupportedAggregates is treated as the interface
default (all) by both the advertisement and HistoryRead-Processed (it used to
NRE inside the batch). Docs: i=2997/i=11201 split and the boot window.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 09:18:53 -04:00
Joseph Doherty 6ea3ba44da feat(historian): advertise only the backend's aggregates under AggregateFunctions (#682)
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:52:47 -04:00
Joseph Doherty 945ec2c069 Merge feat/677-aggregates: reverse processed reads SDK-anchored, start-of-data Partial (forward), AggregateConfiguration accept-defaults, IntegralDivisor=1, SDK parity tests (#677) 2026-10-11 11:47:52 -04:00
Joseph Doherty 029a916288 Merge feat/674-historyupdate-posture follow-ups: HistoryUpdate closed until first redundancy snapshot, node-level v1 refusal, host wiring tests, plan-before-gate, UTC skew (#674 review) 2026-10-11 11:41:36 -04:00
Joseph Doherty 232c03ed1b fix(historian): #674 review follow-ups — HistoryUpdate boot-window gate, node-level v1 refusal, plan-before-gate, UTC skew compare
- HistoryUpdate Primary gate now reads IHistoryUpdateRoleView (PrimaryGatePolicy.ShouldServeHistoryUpdate):
  the data-plane verdict but CLOSED until this node's first redundancy snapshot, so both pair nodes cannot
  accept in the pre-join boot window. Single nodes open on their own singleton's first snapshot; admin-only
  nodes register no view => no gate. DriverHostActor publishes it at every Primary-verdict publish point.
- SqlServer v1: the updater throws NotSupportedException on every arm; the node manager maps it to a
  node-level BadHistoryOperationUnsupported (was per-item under a Good node for UpdateData/DeleteAtTime).
  HistoryWrite bit still advertised on v1 (not knowable at materialise time) — documented.
- The update plan is evaluated before the Primary gate: an unservable request (Remove, IsDeleteModified,
  not historized, no updater) is BadHistoryOperationUnsupported off-Primary, not BadInvalidState.
- Local-kind SourceTimestamp converted to UTC before the future-skew compare and sent as UTC.
- Hosted-service wiring extracted to BuildHistoryUpdatePosture and tested (gate source, skew, clock).
- Docs: Historian.md (boot window, ordering, node-level v1, HistoryWrite bit), CLAUDE.md.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-11 11:41:11 -04:00
Joseph Doherty 93d6c1da67 fix(historian): #675 review follow-ups — SupportsReverseEvents capability, no catch-as-unsupported
- IHistorianDataSource.SupportsReverseEvents (default false; SqlServer/QuestDB true; Gateway/Null declare false).
  The node manager checks it BEFORE calling the source: an end-only capped events read becomes reverse only on
  a supporting backend (Gateway, the default, keeps the pre-#675 forward [MinValue, EndTime) paging), and an
  explicit reverse read on a non-supporting backend is BadHistoryOperationUnsupported without a call.
- Removed the broad catch (NotSupportedException) in the events path: a NotSupportedException from inside a
  backend is now logged as a backend fault (Error), not an 'unsupported' Warning.
- Null declares the reverse overload itself; guard test: every production IHistorianDataSource declares both
  the reverse overload and the capability (positive-control scan of all four backends).
- Conformance: capability reported + oversized reverse tie cluster (forward over-fetch reversed == reverse
  read order; capped reverse read stopping inside the cluster flags truncation) on SqlServer + QuestDB.
- docs/Historian.md: end-only capped events read is backward on SqlServer/QuestDB, forward on Gateway.
2026-10-11 11:36:12 -04:00
Joseph Doherty 9120182314 Merge master into feat/677-aggregates
# Conflicts:
#	docs/Historian.md
2026-10-11 11:30:46 -04:00
Joseph Doherty 27ca6cd791 feat(historian): #677 Part 13/SDK reverse anchoring, start-of-data Partial, AggregateConfiguration, IntegralDivisor
- BucketPlan: reverse processed reads anchored at the later time for every
  aggregate: window (end, start], partial interval earliest, results stamped
  with the later edge, latest first (SDK CreateSlice/GetTimestamp/backward
  GetValues). SqlServer adds @bucketAdjustTicks + flips the window predicate;
  QuestDB binds BucketBaseUnixTicks. IndexOf is the C# statement of the math.
- Partial bit on the interval the tag's data starts inside (SDK m_startOfData),
  every aggregate but Average; per-sample reads fetch the earliest sample.
- Node manager: non-default client AggregateConfiguration =>
  BadAggregateConfigurationRejected per node (ServerAggregateConfiguration,
  pinned to the SDK AggregateManager defaults).
- GatewayTagProvisioner sends IntegralDivisor = 1.
- SDK parity oracle (Opc.Ua.Server calculators) in unit + both backends'
  conformance suites, forward and reverse; remaining divergences documented.
- docs/Historian.md + CLAUDE.md release note (client-visible).
2026-10-11 11:26:00 -04:00
Joseph Doherty 8e6ac4b344 Merge feat/675-reverse-event-paging: reverse event HistoryRead paging, end-only backward, Gateway MaxValue pin (#675) 2026-10-11 11:23:51 -04:00
Joseph Doherty 8d154c6a81 feat(historian): #675 reverse HistoryRead-Events paging + end-only backward reads
- IHistorianDataSource.ReadEventsAsync(..., bool reverse, ...) overload: reverse = Part 11 window (lo, hi]
  newest-first by (time, id) DESC. Default forwards forward reads and throws NotSupportedException for
  reverse (Null); Gateway refuses explicitly (ascending-only stream). Server answers
  BadHistoryOperationUnsupported instead of silently returning the earliest N ascending.
- SqlServer / QuestDB serve reverse natively (ORDER BY time DESC, id DESC + TOP/LIMIT).
- Node manager: reverse event reads page with continuation points; direction stored in the cursor
  (HistoryContinuationState.Reverse) so a resume continues in the cursor's direction; reverse tie clusters
  over-fetched forward and sliced from the newest end (SliceTieCluster(reverse:)); end-only reads with a
  cap page backward from EndTime (mirrors raw).
- Resumed event pages over-fetch by the boundary skip, so every non-final page is a full cap page.
- Tests: node-manager reverse paging (ties, clusters, coarse resolution, overfetch bound, resume
  direction both ways, end-only, unsupported backend); shared EventReadConformance on SqlServer +
  QuestDB; over-the-wire reverse paging; offline Gateway pin of the UTC DateTime.MaxValue bound through
  the published client's proto conversion (in-process gRPC capture) and FILETIME.
- docs/Historian.md event paging section.
2026-10-11 11:19:54 -04:00