Commit Graph

2994 Commits

Author SHA1 Message Date
Joseph Doherty efaf3435ea fix(historian): gateway AGGREGATE_NOT_SUPPORTED_FOR_TAG -> BadAggregateNotSupported, no health failure (#900)
v2-ci / unit-tests (push) Has been cancelled
A node declared numeric whose AVEVA tag is discrete (or string) is not refused up front by #897, so its native
TimeAverage / Total still reach AVEVA, which refuses them. A HistorianGateway at or after historiangw#135
(a220ec0) answers that FailedPrecondition with a google.rpc.ErrorInfo (reason AGGREGATE_NOT_SUPPORTED_FOR_TAG,
domain historian_gateway.v1) in grpc-status-details-bin. The Gateway backend's native processed read now maps
exactly that to HistoryAggregateNotSupportedException (BadAggregateNotSupported for the node) without recording
a health failure or treating it as transport. Any other FailedPrecondition and every Internal (an older gateway's
answer for the same case) keep today's handling.

Detection lives in one place, GatewayErrorReasons. The HistorianGateway.Client pin stays 1.0.0 (1.1.0 is not on
the feed): 1.0.0 keeps the gateway's RpcException, trailers included, as the InnerException of the
HistorianGatewayException it throws, so the reason is read with Grpc.StatusProto 2.76.0 (added via central
package management; nuget.org; same version as Grpc.Core.Api; depends on Google.Api.CommonProtos 2.16.0, which
accepts Google.Protobuf 3.34.1). With Client 1.1.0, ReasonOf becomes
`(ex as HistorianGatewayException)?.ErrorReason` and the StatusProto reference can go.

Tests:
- GatewayAggregateNotSupportedWireTests: in-process gRPC gateway sending the real status + ErrorInfo shape
  (copied from historiangw HistorianRpcFaults.ToRpcException), read through the PUBLISHED Client 1.0.0 via the
  production adapter: TimeAverage / Total -> HistoryAggregateNotSupportedException, 0 health failures; controls
  (Internal; FailedPrecondition without ErrorInfo; other reason; other domain; the reason on Internal) are
  rethrown, not IOException, and counted; positive control served.
- DiscreteTagTimeWeightedWireTests: over a real SDK session, a Double / Int32 node on the discrete tag ->
  BadAggregateNotSupported with 0 failures; the older gateway's Internal answer keeps its status and counts.
- AvevaGatewayAggregateRefusalLiveGateTests: opt-in read-only live gate (OTOPCUA_AVEVA_GATE_900=true).
Mutations (catch removed; status ignored; domain ignored; reason ignored; inner-exception walk removed) each
fail at least one test. Release build 0 warnings; Historian.Gateway.Tests 601/607 (6 skipped), OpcUaServer
IntegrationTests 323/361 (38 skipped).

Live, read-only: private HistorianGateway container from a220ec0 (API-key auth disabled, live writes off,
removed after) against wonder-sql-vd03: Double and Int32 nodes on SysPulse, TimeAverage and Total ->
BadAggregateNotSupported, health failures unchanged (0); SysTimeSec TimeAverage Good (2 rows); SysPulse Average
served. Gateway traffic was ReadAggregate / ReadRaw / GetTagMetadata only.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 01:31:52 -05:00
Joseph Doherty 6c4d29902e fix(historian): read-only historian bindings + ProvisionTags policy; provision OPC UA type names (#899, #898)
v2-ci / unit-tests (push) Has been cancelled
#899 — EnsureTags is an upsert, so binding a node to an existing AVEVA tag via
historianTagname re-typed/re-described it on every add and host start.
- TagConfig `historianReadOnly: true` (TagConfigIntent -> RawTagPlan.HistorianReadOnly):
  HistoryRead from an existing historian tag that OtOpcUa never provisions and never
  records into (AddressSpaceApplier.IsOwnedHistorizedValue gates both the EnsureTags
  hook and the recorder feed). Node-inert: surgical, not mirrored on UNS refs, in the
  RawTagPlan equality/hash. A non-bool value is the deploy error HistorianReadOnlyInvalid.
  AdminUI: CSV column HistorianReadOnly (strict), TagHistorizeConfig reads via the
  runtime parse.
- ServerHistorian:ProvisionTags = CreateOrUpdate (default, unchanged) | CreateOnly
  (Gateway only: TagExistsAsync first, existing tags left alone, failed lookup not
  sent) | Off. Unknown value / CreateOnly on a SQL backend fail host start.
- Default for an explicit historianTagname stays OWNED: overrides pin owned history
  across renames, a read-only default would silently stop provisioning + recording.

#898 — the hook parsed Tag.DataType (an OPC UA name) as DriverDataType, so historized
Float/Double/Byte/SByte tags were never provisioned. New shared
HistorianDataTypeNames.TryResolve: Float->Float32, Double->Float64, Byte/SByte->Int16
(lossless, provisioned as Int2); unmapped names logged at Information. Gateway now skips
Int64/UInt64 (HistorianGateway cannot create Int8/UInt8: lossless-cast-or-reject).

UPGRADE: the first host start after this provisions (upserts) every existing historized
Float/Double/Byte/SByte tag. Mark foreign-tag bindings historianReadOnly, or set
ServerHistorian:ProvisionTags=CreateOnly/Off, before upgrading (docs/Historian.md).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 01:16:24 -05:00
Joseph Doherty eec87d3f74 fix(ablegacy): libplctag refuses MicroLogix function files locally instead of a comms fault (#592)
v2-ci / unit-tests (push) Has been cancelled
libplctag 2.6.0 cannot parse RTC/HSC/... names; tag creation failed with
PLCTAG_ERR_BAD_PARAM, which the status translation reported as BadConnection:
evicted handle, degraded health, counted toward marking a healthy device down.
The libplctag runtime now refuses function-file addresses before creating a
native tag (Unsupported -> BadNotSupported, health untouched), matching native.
Docs: function files are unsupported on both transports, not "unverified".

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-13 01:00:47 -05:00
Joseph Doherty ee39c169df fix(historian): refuse aggregates not valid for the node's DataType up front (#897)
v2-ci / unit-tests (push) Has been cancelled
TimeAverage / Total on a Boolean node over the Gateway backend (AVEVA discrete tag, live
SysPulse) went to the gateway since native TimeAverage/Total became default-on (#694). AVEVA
refuses both modes at StartQuery; the gateway reports that only as Internal /
HistorianServerOperationException (the same answer as a historian storage fault), so the
node answered BadHistoryOperationUnsupported and the read counted as a backend failure.

HistoryRead-Processed now checks the node's scalar DataType before calling any backend and
answers BadAggregateNotSupported (Part 11 6.5.4: "If the Aggregate is not valid for the Node
then the StatusCode shall be Bad_AggregateNotSupported"). Nothing is sent, nothing is counted.
- Shared, every backend: HistoryAggregateDataTypes (Core.Abstractions) applies Part 13
  5.4.2.3 Table 50: the numeric-only aggregates (Average, TimeAverage(2), Total(2), Min/Max,
  ActualTime, Range, Delta, DeltaBounds, Interpolative, stddev/variance) on a String or
  DateTime node are refused. Client-visible: the SQL backends used to answer every interval
  BadAggregateInvalidInputs.
- Per backend: IHistorianDataSource.SupportsAggregateFor(aggregate, TypeCode?) (default true;
  every production backend declares it, guard-tested). Gateway: false for native
  TimeAverage/Total on a Boolean node. SqlServer/QuestDb: true, since they serve Boolean as
  0/1 (stepped), so Boolean stays served there, as do the Gateway's raw-evaluated aggregates.

The residual case (a node declared numeric whose AVEVA tag is discrete) cannot be told apart
from a historian fault on the wire and still counts; that needs the gateway to surface
AVEVA's native error code as a deterministic status.

Tests: HistoryAggregateDataTypesTests, guard test, node-manager String/DateTime and per-type
refusals, Gateway capability matrix, DiscreteTagTimeWeightedWireTests (production Gateway
backend over an AVEVA-shaped fake: unsent, 0 health failures, positive controls). Each
mutation (no check, no shared rule, no backend hint, Gateway hint true, shared rule always
true, SQL inheriting the default) fails at least one. Live gate re-run read-only against
AVEVA 2023 R2 via a private HistorianGateway fa9cf31 container: SysPulse TimeAverage on the
wire = BadAggregateNotSupported, health failures unchanged; #801 gate 244 cases pass.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 00:41:33 -05:00
Joseph Doherty 2e537dc6ca docs(heidenhain): honest provenance for alarm class/group codes (#853)
v2-ci / unit-tests (push) Has been cancelled
No public source exists for DNC_ERROR_CLASS / DNC_ERROR_GROUP or their numbers
(the table was written from memory). The iTNC 530 Technical Manual's PLC error
table documents the same class and group NAMES without codes; three real
iTNC 530 records corroborate 7/4, 7/3 and 1/3. DNC and LSV2 numberings are
known to differ (ProgramStatus vs R_RI 26), so the numbers stay unproven.
Docs/comments only; no behaviour change.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-13 00:36:11 -05:00
Joseph Doherty a4ce417d11 Merge branch 'heidenhain-pylsv2-parity'
v2-ci / unit-tests (push) Waiting to run
2026-11-13 00:32:52 -05:00
Joseph Doherty 53978ca638 feat(historian-gateway): native TimeAverage/Total on by default, SendEvents batches, server-side max_events (#694, #746)
#694 item 3 — historiangw#27 is fixed and now live-verified through OtOpcUa:
- ServerHistorian:Gateway:EnableNativeTimeWeighted defaults to true (false is
  the kill switch for a gateway older than 9f711c1). TimeAverage / Total are
  served and advertised again (12 aggregates).
- The gateway sends the interval as max(1, round(ms)) and AVEVA clips its last
  interval at the request end (live). NativeAggregatePlan therefore sends a
  multi-interval grid only for whole-ms intervals; a fractional interval is read
  one interval per request (at most 1,000, else BadTooManyOperations), and every
  one-interval request goes out rounded UP to whole ms so AVEVA answers one
  clipped row. Live, a 3.0004 s clipped interval sent at its own width lost its
  last 0.4 ms (Total 123 vs 123.0172).

#746 — HistorianGateway Contracts/Client 1.0.0 surface:
- Alarm history: GatewayAlarmHistorianWriter sends SendEvents batches of <=200.
  A failed per-event result is re-sent alone via SendEvent (classified as
  before); a refused batch (InvalidArgument etc.) goes event by event so a
  poison event dead-letters alone; a systemic fault retries the rest without
  sending more; Unimplemented (older gateway) falls back to SendEvent for good.
  IHistorianGatewayClient.SendEventsAsync defaults to Unimplemented.
- Events reads: a capped page sends cap + 1 as ReadEventsRequest.max_events, so
  the gateway completes the stream instead of the client cancelling it. An
  ambiguous capped answer (foreign rows filtered out of the server's cut) or a
  refused cap is re-read uncapped with the existing EventDrainBudget drain.

Tests: GatewayServerCappedEventsTests, GatewayAlarmWriterBatchTests,
GatewayNativeWholeMillisecondGridTests (gateway-faithful rounding fake), gate
tests flipped to the new default; every new test mutation-checked (each
mutation of the production change fails at least one). Live harness
AvevaGatewayHistorianFollowupsLiveGateTests (read-only) for #694/#700/#746.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 00:09:34 -05:00
Joseph Doherty f8785cfa2c Merge branch 'worktree-agent-a71443b71c4c01c13' into heidenhain-pylsv2-parity
# Conflicts:
#	.claude/rules/drivers.md
#	docs/drivers/Heidenhain.md
#	tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Heidenhain.Tests/HeidenhainPyLsv2ParityTests.cs
2026-11-12 23:57:06 -05:00
Joseph Doherty 3055646688 Merge branch 'worktree-agent-ab2193d7e99435def' into heidenhain-pylsv2-parity
# Conflicts:
#	.claude/rules/drivers.md
#	docs/drivers/Heidenhain.md
2026-11-12 23:56:31 -05:00
Joseph Doherty a842e8c31e Merge branch 'worktree-agent-a6ace38a85443d6d6' into heidenhain-pylsv2-parity 2026-11-12 23:56:08 -05:00
Joseph Doherty 09b373e4db fix(address-space): no UNS value Variable for a reference to an alarm-bearing tag (#896)
v2-ci / unit-tests (push) Has been cancelled
A UnsTagReference to a raw tag that carries an "alarm" object produced a UNS
value Variable that read BadWaitingForInitialData forever (G6 live gate,
TestAlarm002Condition on testmachine-015). BuildUnsReferenceVariables now skips
those references; the equipment gets the alarm only as the raw condition wired
as an event notifier of its folder, which is unchanged (the reference still
feeds RawTagPlan.ReferencingEquipmentPaths).

Why condition-only and not "feed the mirror":
- The raw side has no value node for such a tag: AddressSpaceApplier
  materialises an alarm tag as a condition OR a value variable, never both.
  There is no raw value for the UNS mirror to mirror.
- DriverHostActor skips alarm tags when it builds the value fan-out and write
  routing maps (`continue` before the UNS registration), and the driver leaves
  them out of its value subscription ("subscribed to 5 refs" with 6 tags in
  G6). Feeding the mirror would mean subscribing a value the design says the
  tag does not have.
- Docs already state condition-only: ScriptedAlarms.md ("instead of a value
  variable"), AlarmTracking.md ("only the condition, with no sibling value
  variable"), the v3 design (native alarm = one condition at the raw tag,
  surfaced at equipment via notifiers).
- The composer tests that asserted the mirror were incidental: f4f3e17e
  (v3 B4-WP1) put the alarm on the fixture's Speed tag to test raw alarm
  attachment, and later tests reused Speed for writable / historian / array
  checks of the mirror. No commit or doc chose a value mirror for alarm tags.
- ScadaBridge does not read it: its OtOpcUa alarm path is event subscriptions
  on the equipment/raw folders (v3 alarm B/C live gate), and a node that never
  had a value cannot be a working binding. G6 itself authored the alarm's
  Boolean as a separate value tag (TestAlarm002), which is the supported way to
  read it.

Tests: the composer fixture moves the alarm onto its own HighTemp tag
(Speed is now a plain historized value tag), with a new test asserting no UNS
Variable for the alarm reference, with positive controls (the raw tag still
carries the equipment path; the value references still emit). New real-SDK
test Alarm_bearing_reference_surfaces_as_the_condition_only_against_real_SDK:
no Variable at the UNS path, the Speed mirror present, one condition, the
equipment folder wired as its notifier. Both new tests fail with the skip
removed. Docs: Uns.md, ScriptedAlarms.md, G6 evidence note.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:55:49 -05:00
Joseph Doherty 4cee05c2f8 feat(heidenhain): R_RI runtime-info parity with pyLSV2 — state names, refusal codes, timer/unit docs
Compared every R_RI decoder we use (22-28, 30-34, 51) byte-for-byte with pyLSV2 (MIT,
97a3d691) and ran the captured iTNC 530 replies through pyLSV2 itself: no decode
disagreement, so no published value changes.

- Additive read-only String leaves Status/ExecutionStateName, Status/ProgramStateName and
  Tool/AxisName (pyLSV2 ExecState / PgmState / decode_tool_info names; out-of-table code =
  "Unknown"). They ride the raw leaf's feature, so no extra wire read; raw Int32 leaves unchanged.
- Refusals: 52/72 -> BadOutOfRange, 58 -> BadTimeout, 56 -> BadInvalidState, 57 (outside the
  alarm walk) -> BadNoData. IsStable and the 57-only alarm-walk end are unchanged.
  Lsv2RefusalCodes names every code (pyLSV2 LSV2StatusCode) in exception/log/probe text.
- Timers 30-34 documented as uint32be seconds (lifetime counters) per pyLSV2, hardware-unproven;
  TimersEnabled stays false. Axis/tool units: none on the wire, none assumed.
- Lsv2ReadOnlyPolicy untouched (pinned by a test); R_RI 20/21 left out.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-12 23:55:34 -05:00
Joseph Doherty 50933d57fc feat(heidenhain): PLC/DATA parity with pyLSV2 — FF bits, INPUT/OUTPUT_DWORD, PLCDEBUG write-up
- PLC bit areas (MARKER/INPUT/OUTPUT/COUNTER/TIMER) accept 00/01/FF: real iTNC 530s
  return FF for the same operands over R_DP, and pyLSV2 decodes them as non-zero = true.
  02..FE stay BadDecodingError.
- New PLC:INPUT_DWORD / PLC:OUTPUT_DWORD areas (pyLSV2 MemoryType 13/12): signed
  int32le, element index x 4 over the input/output-word byte pools.
- R_PR offsets agree with pyLSV2. Its count units (BYTE x2, WORD/IW as word counts)
  conflict with the captured controls, so we keep the captures and pin them in a test.
- Docs: pyLSV2 correspondence table, native-operand mapping (no aliases), and why
  raw R_MB is refused (pyLSV2 logs in as the write-capable PLCDEBUG role). That stays
  an owner decision; Lsv2ReadOnlyPolicy is unchanged.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-12 23:53:16 -05:00
Joseph Doherty ba9015fac8 feat(heidenhain): classify the control family from R_VR; DATA gated to iTNC 530 (per pyLSV2)
- HeidenhainControlFamily + HeidenhainControlFamilies.Classify (Contracts): one authority, ported from
  pyLSV2 (MIT) dat_cls.py VersionInfo.control (strip spaces, case-insensitive prefix, ITNC530 first).
  Families: ITnc530, Tnc (640/620/320/128), Tnc7, Lathe (CNC PILOT 640 / MANUALplus 620), MillPlus,
  LegacyTnc (426/430, not a pyLSV2 class), Unknown.
- New additive read-only leaf Controller/Family (String, Identity feature, derived from the same R_VR
  reply: no extra request). Lsv2Versions.Family is computed, so record equality and the #831
  identity-change rediscovery signal are unchanged. Probe message + /hosts diagnostics (<host>.family).
- DATA gate now goes through SupportsDataPath(family) (pyLSV2 client.py read_data_path is iTNC-only):
  non-iTNC and Unknown => Capabilities/DataLogin Unsupported, DATA login never sent. Replaces the
  "model contains itnc" substring check (same result on the real iTNC530 controls).
- No per-family selector pre-gates: pyLSV2's TNC 640 observations (27/28/51/30-34 unknown telegram,
  25/26 need DNC) are already handled by the reply-driven refusal/suppression path; tests pin it.
- Lsv2ReadOnlyPolicy, HeidenhainTagWritable and the no-IWritable rule are untouched.
- Simulator: per-family ControlProfiles (strings from the pyLSV2 README tables / protocol.rst), both
  R_VR field counts, 124-byte R_PR; HeidenhainControlFamilyTests (classification, gates, decoding,
  wire log inside the read-only set, refused selectors asked once).
- docs/drivers/Heidenhain.md: Control families matrix (iTNC 530 hardware-proven; the rest per pyLSV2,
  hardware-unproven) + open questions (C_CC buffer size, encrypted passwords, R_RI 20/21).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-12 23:52:06 -05:00
Joseph Doherty e113faf5de fix(galaxy): picker-authored native alarms resolve via the feed's source reference (#895)
The gateway alarm feed names an alarm "Galaxy!<Area>.<tag>.<Attr>". The AdminUI Galaxy
picker authors attributeRef "<tag>.<Attr>". OnAlarmFeedTransition reverse-mapped only the
full reference, so a picker-authored native alarm never matched its condition and never
went active. Its ack also dialled the bare reference, which AcknowledgeAlarm refuses.

ResolveAlarmRawPath now matches the full reference, then SourceObjectReference, then the
full reference without its Provider!Area. prefix. A fallback match records the full
reference per RawPath, and AcknowledgeAsync dials it, so the gateway accepts the ack and the
#869 confirmation watch matches. Found and re-verified live in the #615 G6 gate on the
windev test Galaxy.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:37:11 -05:00
Joseph Doherty fa8c3d1a1a fix(adminui): bodyless POST /api/deployments answered 404 on fused admin+driver nodes (#761)
Found on the docker-dev rig during the #761 Wave-4 gate run (2026-11-12): the documented
bodyless headless deploy (docker-dev/deploy.sh, README "Headless deploy") returned 404 on
both centrals, while the same POST with a JSON body was accepted.

Cause: the optional body was an inferred `DeployRequest?` parameter, which stamps
Accepts(application/json) metadata on the endpoint. For a request with no Content-Type the
routing AcceptsMatcherPolicy prefers endpoints that declare no consumes metadata, and on a
fused admin+driver host MapOtOpcUaMeshGrpcServices has mapped grpc-dotnet's
`{unimplementedService}/{unimplementedMethod}` catch-all, so the request was routed away
from the deploy handler and 404'd. The deploy API on its own does not reproduce it.

The handler now reads the optional body itself: no body (absent or zero-length) is the
headless form, a non-JSON body answers 415 and malformed JSON 400, as the framework's
binding did. New DeployApiBodylessPostTests compose the deploy API with the mesh gRPC
services and drive real HTTP; with the fix reverted the two bodyless cases fail with 404.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:44:29 -05:00
Joseph Doherty 412da27c5f feat(telemetry): Telemetry 0.3.0 - service.version, site.id/node.role, logs over OTLP (scadaproj#55)
v2-ci / unit-tests (push) Has been cancelled
Pins ZB.MOM.WW.Telemetry(.Serilog) 0.1.0 -> 0.3.0. The OTel Resource now names the
node: site.id = its application Cluster id (the cluster-<ClusterId> role suffix,
omitted on a legacy node), node.role = its plain roles in canonical order
(admin / driver / admin,driver), service.version = the entry assembly's informational
version (the git SHA). Program passes its resolved EffectiveRoles. The whole
OtOpcUa:Telemetry section goes through the shared ApplyConfiguration, so SiteId /
NodeRole / ServiceVersion / ResourceAttributes:* are operator overrides, and
Exporter=Otlp now also sends the Serilog logs to the same collector.

ObservabilityResourceIdentityTests cover the mapping and the built container's
Resource. Host.Tests 288/288, Host.IntegrationTests 325/327 (2 skipped).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 21:13:23 -05:00
Joseph Doherty 0b90379f70 fix(telemetry): export the Galaxy driver's spans and galaxy.events.* counters (#892)
ZbTelemetryOptions.Meters/ActivitySources are strict allowlists. The host named only
ZB.MOM.WW.OtOpcUa and the LocalDb meter, so the in-process Galaxy driver's galaxy.* spans
and its event pump's galaxy.events.* counters (both on "ZB.MOM.WW.OtOpcUa.Driver.Galaxy")
were silently dropped from /metrics and OTLP. Found exercising OTLP end to end against a
collector (scadaproj#13).

- Galaxy driver: public GalaxyDriverTelemetryNames (MeterName / ActivitySourceName);
  GalaxyTelemetry and EventPump reference it.
- Host: ObservabilityExtensions.ObservedMeters / ObservedActivitySources include it.
- Host.IntegrationTests: ObservabilityAllowlistTests assert on the listener side (span
  sampled, instrument enabled) with an unregistered-name negative control; dropping the
  Galaxy entries fails 3 of 8.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:31:14 -05:00
Joseph Doherty e303165444 chore(deps): LocalDb 0.3.4 -> 0.3.7 (#889, scadaproj#53)
v2-ci / unit-tests (push) Waiting to run
Pins ZB.MOM.WW.LocalDb and .Replication at 0.3.7 (published 2026-11-12).
Wire-compatible; no code change needed.

- 0.3.6 (scadaproj#36): the library sync service enforces
  LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
  its refuse-every-stream-when-blank rule; doc comment updated. docker-dev
  site-a pair: both nodes now read one variable,
  OTOPCUA_SITE_A_LOCALDB_SYNC_KEY (same dev default as before), so an
  override can never split the pair.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
  node offline past TombstoneRetention no longer brings a pruned
  deployment back. The ledger grows while a peer is down; optional
  TombstoneHardCap (documented in Configuration.md) bounds it.

Runbook, Configuration.md and the cluster-topology rule updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 13:44:55 -05:00
Joseph Doherty b61f3e0560 fix(galaxy): AVEVA priority 1 is the most urgent: Galaxy alarm Severity follows AVEVA's priority bands, live and history alike (#887)
v2-ci / unit-tests (push) Has been cancelled
GalaxyAlarmPriority, the one table live Galaxy alarms (MxAccessSeverityMapper)
and AVEVA A&E history (EventMapper, #886) map the AVEVA priority through, ranked
a higher number as more severe. AVEVA ranks priority 1 as the most urgent, so a
priority-1 alarm read 250 and a priority-999 system event 1000.

It now uses AVEVA's default priority ranges, most urgent first:

  AVEVA priority  AVEVA severity  bucket    OPC UA Severity
  1-250           1 Critical      Critical  1000
  251-500         2 High          High       750
  501-750         3 Medium        Medium     500
  751-999         4 Low           Low        250
  (history only)  0 not an alarm  -            1

Out-of-range priorities clamp (<= 0 Critical, > 999 Low). Evidence, vd03
Runtime.dbo.Events read-only: priority 1/100 alarms carry AVEVA severity 1,
400/500 severity 2, 600/700 severity 3, and every priority-999 System.* /
User.* row severity 0 with IsAlarm 0.

- Live and history now use the same number. GalaxyDriver passes the table's
  exact value as AlarmEventArgs.OpcUaSeverity. Before, it passed only the bucket,
  which NativeAlarmProjector re-mapped through its generic 200/500/700/900
  ladder, so live and history disagreed even before the direction fix.
- History: a row whose AVEVA severity is 0 (System.Start/Stop/Deploy,
  User.Write...) is not an alarm and reads Severity 1, below every alarm band.
  For alarms AVEVA's 1-4 severity is still never the value: the live feed
  carries only the priority, so it cannot use it, and history must agree.
- Docs: docs/Historian.md gains "Galaxy alarm priority -> Severity
  (lmxopcua#887)"; docs/AlarmTracking.md updated.
- Live re-run of the #801/#886 harness against vd03 HistorianGateway
  ea9f21c (read-only, network-only): 224 of 233 cases identical to the #886
  run, the 9 others differ only in Severity (priority-700 alarm rows 750 ->
  500, 19 system/user events 1000 -> 1). Record: aveva-history-801.md
  "#887 re-run", report aveva-gate-sdk2-887.json.gz.

Client-visible: every OPC UA client that ranks or filters Galaxy alarms by
Severity sees the order reversed (live and HistoryReadEvents). ScadaBridge has
no code that depends on the old values; its own mixed severity directions are
ScadaBridge#129.

Tests updated because they pinned the inverted table:
- Driver.Galaxy.Tests MxAccessSeverityMapperTests (rewritten: edges 1/250/251/
  500/501/750/751/999, 0/-100/int.MinValue/1000/int.MaxValue clamps,
  monotonic-decreasing sweep, equals GalaxyAlarmPriority for -2..1001)
- Driver.Galaxy.Tests GatewayGalaxyAlarmFeedTests (750 and 600 -> Medium)
- Driver.Historian.Gateway.Tests EventMapperTests (band table, priority 999
  -> 250, 1 -> 1000; system events -> 1)
- OpcUaServer.IntegrationTests GalaxyEquipmentEventHistoryWireTests (700 ->
  500, 100 -> 1000, System.Stop -> 1; the fake now stores AVEVA's real 0-4
  severity)

New tests:
- Runtime.Tests NativeAlarmRestatementEndToEndTests
  .Galaxy_live_condition_severity_is_the_AVEVA_priority_band (8 rows: the real
  feed -> real GalaxyDriver -> host -> projector; the same rows EventMapperTests
  pins on the history path)
- EventMapperTests.Aveva_severity_zero_marks_an_event_that_is_not_an_alarm
  (5 rows, incl. the priority-999 alarm positive control)
- GalaxyDriverAlarmSourceTests: OpcUaSeverity passes through

Mutation check, 8 mutants, all killed: the pre-#887 inverted table; the
driver dropping OpcUaSeverity; no not-an-alarm rule; each of the three band
edges off by one; history ignoring priority; a missing AVEVA severity read
as 0.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 12:14:49 -05:00
Joseph Doherty fa811feb3d fix(opcua): drop the reflective folder-event verdict invalidation; deny the SDK channel under enforcement (#878)
v2-ci / unit-tests (push) Waiting to run
The SDK memoizes INodeManager3.ValidateEventRolePermissions per (item, event type,
SourceNode). #826 kept it fresh by reflecting into the private
MonitoredNode2.InvalidatePermissionCaches (plus a detach/re-attach fallback). Since #855
condition events never take that channel, and the search for #878 found nothing
production-raised that does except a condition's SDK method audits, which reach it only
with auditing on, SignAndEncrypt and a SecurityAdmin session (nothing grants that role).

Under enforcement the channel verdict is now a flat deny for items on our nodes, so no
cached verdict depends on a grant, and InvalidateFolderEventVerdicts, the reflection,
the fallback, its counter and the guard test are removed. Method audits still reach
Server-object items, decided per event.

Tests: FolderEventVerdictInvalidationTests -> FolderEventVerdictFreshnessTests; new
revoked-grant test (positive control: a still-granted session) and method-audit test
(SecurityAdmin mapped for the test; positive controls: Server-object item, flag off);
the plain-event test runs each mode on a fresh server. Mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 11:45:54 -05:00
Joseph Doherty 3a312ffdd1 fix(history): a folder's event history reads the Galaxy objects behind its native conditions (#886)
v2-ci / unit-tests (push) Has been cancelled
A notifier folder read one event source, its NodeId identifier (a RawPath or an EquipmentId). AVEVA stores a
Galaxy object's A&E history under the object's tag_name (Source_Object), so AVEVA's own alarm and event
history reached no production folder. Its lower-case property keys were also missed, so every row read
Severity 1 with no ConditionName.

Sources. A folder's event history now reads its own source plus the Galaxy object of every native condition
that fires on it live: those parented under it and those WireAlarmNotifiers wired to it. Live and history use
the same membership. The object comes from the condition's Galaxy alarm reference, its raw tag's attributeRef,
up to the first '.':
- The composer reads it with the driver's own parser (GalaxyAttributeRef, shared with GalaxyDriver), only for
  a GalaxyMxGateway tag with an alarm (RawTagPlan.NativeAlarmReference, part of plan equality).
- The applier binds it after materialising the condition (IOpcUaAddressSpaceSink.BindNativeAlarmHistory,
  forwarded by the deferred sink). A rebuild or the condition's removal forgets it.
Each object is read with the #882 rule: its own events and <Object>.<Attribute> rows, never a prefix sibling.

Fan-out. historiangw's ReadEvents filter is a single predicate, so there is one request per source; the wire
contract is unchanged. EventSourceFanOut merges the reads as one source:
- order is (time, source position, arrival), reversed for a reverse read; ties are never re-sorted by EventId,
  so the boundary skip stays stable;
- duplicates by EventId are dropped and the merge is cut to the cap;
- a truncated source bounds the page at the time it stopped.
Paging, continuation points (bound to the whole source set), the walked-back reverse read and the tie-cluster
over-fetch are unchanged. A folder with one source is read exactly as before.

ACL. Every source is read inside the folder's own HistoryRead, after AuthorizeHistoryRead (#826). A denied
folder reads nothing and fails closed.

Mapping (EventMapper, property names now case-insensitive):
- Severity: an AVEVA row's priority goes through GalaxyAlarmPriority, the table live Galaxy alarms use
  (MxAccessSeverityMapper now delegates to it), so history matches live. Otherwise severity (OtOpcUa's own
  rows, which AVEVA stores lower-cased), clamped. AVEVA's own 1-4 severity is ignored. The table ranks a
  higher priority as more severe, which inverts AVEVA's convention: filed as #887.
- An AVEVA row whose source_conditionvariable names a native condition that fires on the folder takes that
  condition's SourceNode and live ConditionName. Otherwise ConditionName is alarmname or the condition
  variable's attribute.
- alarm_id/alarmid map to AlarmId.

Tests:
- GalaxyEquipmentEventHistoryWireTests (4, real SDK session, production Gateway backend, AVEVA-shaped fake
  gateway): equipment EqA backed by objects Oven_007 + OvenDoor_07. Covers object and attribute events from
  both objects, the prefix sibling excluded, cross-object and in-object ties, forward paging at 1-4 a page,
  reverse whole and paged, SourceNode/ConditionName/Severity, the raw device folder, ACL deny with no backend
  call plus a granted positive control, and a rebuild forgetting the bindings.
- EventSourceFanOutTests (8), EventMapper AVEVA cases (11), GalaxyAttributeRefTests (16), a composer test and
  an applier ordering test.
- Mutation-checked, 17 mutants, all killed. Covered: parent-only membership, no scope, a case-sensitive
  condition match, no condition rewrite, no truncation bound, EventId tie order, no sort, no cap, no stop at
  the bound, unreversed source order, no dedupe, no '.' boundary, priority ignored, case-sensitive keys, no
  applier bind, no deferred forward, and a composer that ignores the driver type. The forget-on-rematerialise
  guard survives as belt-and-braces: the rebuild clear covers it.
- Live, read-only, no WinRM, vd03 HistorianGateway ea9f21c: equipment folder EQ-886-ZipsenLeft (objects
  DevWinPlatform + Left_001). It read 0 events before and 92 now, equal to the unfiltered read's rows for both
  objects. 1/7-event pages and reverse whole/5-event pages agree, and 73 rows resolve to their live condition.
  diff_runs vs the #882 run: 221/226 identical. The 5 that differ are expected: DevWinPlatform Severity
  1 -> 750/1000, and #881 capabilities.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 11:45:48 -05:00
Joseph Doherty a83bf1fa27 fix(opcua): a configured OpcUa:EnabledSecurityProfiles list replaces the default, so None can be dropped (#879)
v2-ci / unit-tests (push) Has been cancelled
ConfigurationBinder appends configured collection items to an existing list, so
the pre-filled [None, Sign, SignAndEncrypt] default made "drop None" a no-op:
every deployment that listed only the secure profiles still advertised an
unsigned/unencrypted None endpoint.

EnabledSecurityProfiles is now null by default; ResolveEnabledSecurityProfiles()
applies the unchanged three-profile default only when nothing is configured.
A set-but-empty list fails startup validation and BuildSecurityPolicies no
longer falls back to a None endpoint on empty input (it throws). Sweep of all
options classes found no other non-empty pre-filled collection default.

Tests: options binding (indexed, JSON, env forms, default, empty, bad name), a
booted-server wire test (no None endpoint, None connection refused with
BadSecurityPolicyRejected, secure connection works; default case is the control).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 11:38:42 -05:00
Joseph Doherty be28aa4432 fix(opcua-client): send UserName passwords in the legacy RSA format so 65-char API keys log in to SDK 1.5 servers (#880)
SDK 2.0.1's UserNameIdentityTokenHandler encrypts a password longer than 64 bytes
(a private RsaEncryptedSecretPasswordThreshold, no setting) as an RsaEncryptedSecret
with EncryptionAlgorithm=null. SDK 1.5.378 (.106 and .182) decrypts only the legacy
format under RSA policies; its SecurityPolicies.Decrypt answers a null algorithm with
BadIdentityTokenInvalid ("Unexpected encryption algorithm") before any key check.
Every production API key is 65 chars, so no SDK 2 OtOpcUa client could use one
against QA (b8b4c18f), a rollback target or an older third-party server.

LegacyRsaUserNameIdentityTokenHandler (Driver.OpcUaClient.Session, plugged in through
the public UserIdentity(IUserIdentityTokenHandler) ctor) always uses the legacy
format, RSA-OAEP(len|password|serverNonce) under the server certificate, under an RSA
policy. That is what SDK 2 sends for passwords of 64 bytes or less and what SDK 1.5
clients always send. The legacy format has no size limit (the plaintext is
length-framed over as many RSA blocks as it needs), and an SDK 2 server reads both
formats. Policy None, the ECC policies and a missing certificate go to the SDK
handler unchanged.

OpcUaUserIdentityFactory.UserName is now the one UserName identity the driver, the
AdminUI browse and Client.Shared (CLI/UI) present. Client.Shared references .Session
for it.

Tests: handler unit tests (all three RSA policies, the 1.5-required algorithm URI,
SDK 2 server-side decrypt, the session Copy() keeps the type, None unchanged, and an
SDK control pinning the 64/65-byte switch); a Client.Shared identity test; and an
SDK2<->SDK2 integration test where a 65-char key is admitted in both formats and a
wrong 65-char key reaches the key check (BadIdentityTokenRejected). The #800 Windows
gate's SDK 2 CLI 65-char case now gates against the 1.5 server too.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 11:12:27 -05:00
Joseph Doherty 3fff439b11 fix(redundancy): advertise ServiceLevel 0 until the node has a redundancy level (#885)
v2-ci / unit-tests (push) Has been cancelled
The SDK starts Server.ServiceLevel at 255 (full service), and OpcUaPublishActor publishes nothing
until the node's first redundancy snapshot arrives, so a restarting pair node advertised 255 for the
~10-13 s it took to join, out-ranking the real Primary (250) for any client that picks the highest
ServiceLevel.

OtOpcUaSdkServer now writes StartupServiceLevel = 0 in OnNodeManagerStartedAsync: after the SDK's
CreateServerObjectAsync wrote 255 and before the server goes Running, so no session can ever read
255. 0 is the design's existing Down / Detached tier (ServiceLevelCalculator returns 0 for a member
that is not Up/Joining, and a node with no snapshot has no role yet). It is below the SDK 2
session-headroom floor (200), so it stands through session churn before the #884 claim, and the
claim's bind replays any level already requested. Single-node deployments are unaffected beyond
the window: the first snapshot publishes 250 as before.

New Node_with_no_requested_level_advertises_the_startup_level_until_one_is_published: reads 0
before the bind, 0 after a bind with nothing to replay, then 250 once published. Without the
startup write it reads 255 on the first session (mutation-checked). docs/Redundancy.md updated for
#884 and #885.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 11:11:11 -05:00
Joseph Doherty f66b9daec0 fix(redundancy): claim Server.ServiceLevel so SDK 2 stops overwriting 250/240 with 255 (#801)
SDK 2.0.1 recomputes Server.ServiceLevel from session headroom on every session create/close and
overwrites any value >= 200 unless a provider has claimed it through
IServerServiceLevelControl.ClaimServiceLevelControl. SdkServiceLevelPublisher never claimed it, so on
the docker-dev pair both nodes read 255 as soon as a client connected while their own metrics showed
250 (Primary) and 240 (Secondary); the Stale tier's 200 was lifted to 255 too. Levels below 200 (100,
0) were untouched, which is why the bootstrap 100 showed briefly.

The publisher now claims the value at construction and writes through the SDK's writer; a server
without the claim (test doubles) keeps the diagnostics-guarded write. New
Published_level_survives_later_sessions (250/240/200) fails without the claim (reads 255) and passes
with it.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 10:55:49 -05:00
Joseph Doherty da948e5fb5 fix(history): a folder's event history includes its object's attribute alarms (#882)
v2-ci / unit-tests (push) Has been cancelled
HistorianGateway answers a per-source ReadEvents (Source_Object = DevWinPlatform) with the
object's own events (SourceName DevWinPlatform) AND its attribute alarms, which it reports as
<Object>.<Attribute> (Source_ProcessVariable, e.g. DevWinPlatform.CPULoadAvg). The Gateway
backend's defensive client-side source filter compared SourceName for equality, so it dropped
the attribute alarms: 70 of the 72 DevWinPlatform events on vd03.

The filter (and the past-the-cap "more events" signal) now uses EventSourceMatch: the source
itself, or <source>.<anything> on the '.' boundary, ordinal. DevWinPlatform never matches
DevWinPlatform2 or DevWinPlatform2.X. This needs no wire change and still makes one request.
Each event keeps its own SourceName. SourceNode stays the condition its stored alarm id names,
which is null for AVEVA rows. The attribute events ride the notifier's own read, so the
per-folder HistoryRead verdict (#826) governs them: a denied folder answers
BadUserAccessDenied before any backend call. The SqlServer and QuestDB backends still match
exactly; they hold only OtOpcUa's own events.

Tests:
- GatewayAttributeEventSourceTests (16): the boundary theory, attribute-in/sibling-out through
  the data source, the drain signal, and an exact attribute-named read.
- AttributeEventHistoryWireTests (2, over a real SDK session with the production Gateway
  backend on an AVEVA-shaped deterministic gateway): object + attribute events with their own
  SourceName, a prefix sibling excluded, paging, and the ACL-on deny with no backend call plus a
  granted positive control.
- Mutation-checked: exact match fails both wire tests; dropping the '.' boundary fails the
  wire sibling check and 6 unit tests.
- Live AVEVA harness re-run against vd03 HistorianGateway ea9f21c (read-only, no WinRM):
  folder DevWinPlatform 2 -> 72 events (whole, 72 1-event pages, reverse); the new #882 check
  passes; 221 of 225 shared cases are identical to the #801 SDK2 baseline, and the 4 that
  differ are exactly the DevWinPlatform event cases. Recorded in aveva-history-801.md.

Follow-up #886: production folders name RawPaths/EquipmentIds, not Galaxy objects, plus the AVEVA
severity/condition property mapping.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 10:34:29 -05:00
Joseph Doherty b27b5b9cc1 fix(history): advertise HistoryServerCapabilities from the wired backend (#881)
v2-ci / unit-tests (push) Has been cancelled
Every Boolean under Server/ServerCapabilities/HistoryServerCapabilities read
false (the SDK default) even though data and event history were served, so a
spec-following client skipped HistoryRead entirely.

Each backend now declares what it serves through its own interface:
- IHistorianDataSource.ReadCapabilities (Data / Events / ServerTimestamps):
  Gateway Data|Events; SqlServer + QuestDb add ServerTimestamps (they store
  ServerTimeUtc, the Gateway stamps read time); Null declares None. A
  reflection guard requires every production source to declare it itself.
- IHistorianDataUpdater.Capabilities (required member): SqlServer and QuestDb
  declare Insert/Replace/Update/DeleteRaw/DeleteAtTime; the Gateway provider
  has no updater.

The node manager writes the flags on every source/updater assignment
(AdvertisedHistoryCapabilities), and the same declarations drive dispatch:
an update the updater does not declare is BadHistoryOperationUnsupported
without reaching it, the HistoryWrite AccessLevel bit follows a non-empty
declaration, and event-history notifier promotion follows the Events flag.
Event and annotation update flags stay false (nothing dispatches them).

Tests: booted-server wire test per backend + Null + a SqlServer->Gateway
rewire pairing on one session; dispatch gating unit test; the #801 live
gate now asserts the Gateway flags (not run here).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 10:31:29 -05:00
Joseph Doherty f264977fb0 feat(galaxy): wait on the gateway's stream-attached ack instead of the 1 s best-effort wait (#877)
v2-ci / unit-tests (push) Has been cancelled
mxaccessgw#227 (Contracts/Client 0.4.0) adds an opt-in attach acknowledgement on
StreamEvents: a StreamAttached sentinel MxEvent sent first, only after the
stream's subscriber is registered on the session. This closes the #865 attach
race that waiting for "stream requested" only narrowed.

- Bump vendored MxGateway.Client/Contracts 0.3.0 -> 0.4.0 (nuget-packages/,
  pulled from the Gitea feed; local-mxgw stays the mapped source).
- IGalaxySubscriber.SupportsStreamAttachedAck (default false, so fakes and older
  seams keep the fallback); GatewayGalaxySubscriber reads it from the live
  session's capabilities (stream-events-attached-ack) and passes
  emitStreamAttached accordingly; TracedGalaxySubscriber forwards it.
- EventPump: records AttachAckExpected before the stream opens (published
  before StreamRequested completes); intercepts ev.StreamAttached ahead of the
  ReplayGap check - completes StreamAttached and continues: never dispatched,
  never counted as received, never a resume cursor (worker_sequence 0). The
  signal also completes when the loop ends so no waiter hangs. One pump = one
  stream, so a reconnect's new pump has a fresh signal. New counter
  galaxy.events.stream_attached.
- AwaitCurrentEventStreamAsync: with the capability, waits on StreamAttached
  within the same StreamRequestWait budget (1 s); a missing ack logs a
  rate-limited Warning and proceeds instead of hanging. Without the capability
  the old StreamRequested wait is the fallback.

Tests (StreamAttachedAckTests, 6): ack-gated subscribe gets its initial value;
capability absent uses the fallback; missing ack times out with a warning; the
sentinel is never dispatched; a dead stream releases ack waiters; reconnect
gives each stream a fresh signal. With the ack wait skipped, 3 of them fail.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 07:38:09 -05:00
Joseph Doherty 633f43170a test(galaxy): #872 pin the detected_by rule per source
v2-ci / unit-tests (push) Has been cancelled
The rule fix itself landed in 1a08b4f7 (detail-0 failure rows route on
category); its test only exercised raw_detected_by = 3. Add per-source
coverage for every MxStatusSource value mxaccessgw's MapSource emits
(-1, 0..5, and an out-of-range value) across the failure categories, plus
the live-captured shape the old rule got wrong (OperationalError detected
by the responding automation object, mxaccessgw docs/AlarmProbeFindings.md)
and a RequestingLmx CommunicationError positive control. All 10 new cases
fail against the old 'detail 0 && raw_detected_by != 0 => comms' rule.

Comment in FromMxStatus now cites MapSource and says why NMX (2/3) is not
a transport signal. Item 1 (Warning/Pending) stays open pending a capture.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-12 07:17:53 -05:00
Joseph Doherty f127f45428 fix(opcua): don't register with a discovery server unless configured (#876)
v2-ci / unit-tests (push) Has been cancelled
OpcUaApplicationHost left ServerConfiguration.MaxRegistrationInterval at the
SDK default (30000) with a null RegistrationEndpoint, so SDK 2.0.1's
StandardServer armed its registration timer against opc.tcp://localhost:4840,
which on an OtOpcUa node is the server itself (no LDS). It answered
BadServiceUnsupported and the SDK retried forever, visible since #874.

Registration is now opt-in: ApplyDiscoveryRegistration sets
MaxRegistrationInterval = 0 (the value that keeps StandardServer from arming
the timer) unless OpcUa:DiscoveryServerUrl is set, in which case the server
registers against that URL every OpcUa:DiscoveryRegistrationIntervalMs
(default 30000) over SignAndEncrypt/Basic256Sha256, like the SDK's own
default endpoint. The validator requires an absolute opc.tcp:// URL and a
positive interval when the URL is set. Documented in Configuration.md and
OpcUaServer.md.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-12 06:29:35 -05:00
Joseph Doherty 03ac57d5d6 fix(scripting): #789 review — reader faults end the worker, write stalls kill it, Cancel off the worker reader
- Host reader: the frame read and its handling are caught separately. Only the
  read's I/O exceptions mean the pipe ended; anything HandleFrame's inline
  completion work throws now kills that worker as ProtocolFault (outstanding
  runs -> worker restarted / BadResourceUnavailable) instead of escaping a
  dedicated thread (process crash) or being swallowed as a clean end (an
  InvalidOperationException used to be, leaving the worker unread until a Crash
  2 s later).
- Worker reader: the same catch-all around HandleFrame (exit ProtocolFault).
- Worker: Cancel() runs on the pool, so a script's blocking
  ctx.CancellationToken callback cannot stop the worker reading its pipe.
- Host write-stall check: ScriptWorkerFrameWriter.WriteInProgressFor; the
  heartbeat monitor kills a worker whose pipe write has been blocked longer than
  max(10 s, silence limit) as KilledUnresponsive, which fails the write and
  releases the producer (seam submit, deploy load, Cancel). Fixes the wrong
  'heartbeat monitor ends it' comment on the pipe buffer quota.
- StopAsync sends Shutdown off the caller so its 1 s budget holds on a full pipe.
- WindowsBlockingPipeStream refuses the async surface (Stream's base versions
  serialise a write behind a waiting read).
- Tests: PipeFaultTests (throwing completion callback -> ProtocolFault and a
  BadResourceUnavailable sibling, shard serves again; a worker that stops reading
  but heartbeats is killed within the bound and the blocked producer returns,
  via the Hello.test_stall_reader_on_diag seam); BlockingPipeStreamTests dispose
  with a write blocked on a full pipe, and with a read and a write both pending;
  Category=ScriptWorker on the protocol, handshake and pipe test classes.
  Mutation checks: no stall kill -> stall test red; catch-all without the fault
  -> callback test red (Crash).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 20:26:02 -05:00
Joseph Doherty 82da1ecbc8 perf(scripting): #789 give the Windows worker pipe a 1 MiB buffer quota each way
With a zero quota an overlapped write stays pending until the peer has read the
bytes, so the new blocking writer waited out a peer wake-up per frame (windev
closed loop 54k -> 26k evals/s). With a quota the write completes once buffered.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 19:30:07 -05:00
Joseph Doherty 3c0e5ea4d7 perf(scripting): #789 Windows worker pipe I/O blocks on dedicated threads, off the completion port
A .NET pipe opened Asynchronous completes even its synchronous Read/Write
through the thread pool's completion port, so every frame on Windows cost a
completion packet plus a pool work item (and the pool's spin-wait). New
WindowsBlockingPipeStream issues its own overlapped ReadFile/WriteFile with the
OVERLAPPED.hEvent low bit set (no completion packet) and waits on that event on
the calling thread; reads and writes stay concurrent, unlike a synchronous
handle. ScriptWorkerFrameCodec.BlockingStream picks it on Windows and the pipe
itself on Unix, so both ends now use the dedicated blocking reader + synchronous
combining writer on every OS (UseBlockingIo and the async read loops are gone).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 19:11:10 -05:00
Joseph Doherty 95bde828eb fix(galaxy): #875 Galaxy initialize gets a 20 s budget via per-driver-type capability defaults
Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.

DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 18:44:44 -05:00
Joseph Doherty 2e948d3139 fix(opcua): #874 route the server's OPC UA SDK log output to the host logger
The host handed the SDK a provider-less telemetry context, so SDK-side
refusals — identity-token decryption (the SDK 2 RsaEncryptedSecret clock
window, #873), certificate validation, secure-channel faults — left no server
log line (found on windev during the #800 Windows validation). The hosted
service now creates one ServerSdkTelemetry context from its ILoggerFactory
(Warning and above, SDK categories, rate-limited 20/min per category) and
shares it between OtOpcUaSdkServer and the ApplicationInstance; released on
dispose. Test: an untrusted client's rejection reaches the captured log.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 16:37:03 -05:00
Joseph Doherty e57e1324c3 Merge #870 ring-buffer event monitored-item queue
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:20:31 -05:00
Joseph Doherty bcde93056a perf(opcua): #870 ring-buffer event monitored-item queue
Since #855 condition events are queued to folder event items under the node
manager Lock. At a full DiscardOldest queue the SDK's EventQueueHandler
dequeues the head of EventMonitoredItemQueue, a List.RemoveAt(0) shift of the
whole queue (~1 us/item at QueueSize 1000, ~10 us at 10000).

OtOpcUaSdkServer.CreateMonitoredItemQueueFactory now wraps the SDK factory in
RingEventQueueFactory: event queues are RingEventMonitoredItemQueue (O(1)
discard), data-change queues, durable requests and disposal go to the SDK
factory unchanged. A factory set through StandardServer.MonitoredItemQueueFactory
is used as-is.

The SDK queue is storage only (decompiled 2.0.1): DiscardOldest, overflow-event
positions and discarded/overflow counters live in EventQueueHandler, which is
unchanged. The ring reproduces QueueSize 0 initial + BadInternalError enqueue,
drop-oldest at Count == QueueSize, FIFO dequeue, SetQueueSize keep-newest /
keep-oldest, the newest-1000 duplicate window via IsEventRaisedFor, no-op
Dispose. RingEventMonitoredItemQueueParityTests drive both queues, and the SDK
EventQueueHandler over both factories, with the same random sequences.

FolderEventDeliveryPerfTests gains Ring_vs_sdk_event_queue (A/B in one process
via the internal OtOpcUaSdkServer.RingEventQueues switch).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:15:45 -05:00
Joseph Doherty 1a08b4f746 fix(galaxy): #872 detail-0 failure rows route on category, not detected_by
Every raw detected_by value names an MXAccess layer (requesting/responding
LMX, NMX, automation object); the gateway's own failures arrive as
ProtocolStatus. The 'non-zero source = transport failure' rule made a failure
the responding object detected a comms error. Item 1 of #872 (Warning/Pending
categories) stays open pending a captured row.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:15:05 -05:00
Joseph Doherty 970f2159ad docs(galaxy): IsSuccess checks the status category only
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:14:22 -05:00
Joseph Doherty 875b82175d fix(galaxy): #866 grade OK-status data changes by the item's OPC DA quality
The gateway fills Statuses on every data change (live OnDataChange, value-cache
replay and session snapshot all carry Quality too), and EventPump used the status
row whenever one was present, so Quality was never read: an off-scan, Bad or
Uncertain attribute with an OK status row was published Good. A failure status
row still wins; otherwise the quality byte grades the value (legacy parity:
quality >= 192 Good).

Live, read-only on VD03 (DEV): healthy ScanState and _TestArea attributes stay
Good (0x00000000) via quality 192; the stopped JoePlatform/JoeAppEngine ScanState
now publish BadCommunicationError (0x80050000) instead of generic Bad, with no
"unrecognised quality byte" warning.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:14:11 -05:00
Joseph Doherty 71b3b7f35f fix(galaxy): #866 map MxStatus failure details by the MxStatusDetail vocabulary, not as OPC DA quality
The gateway copies MXSTATUS_PROXY.detail verbatim, so a failure row's detail is
always an MxStatusDetail code. FromMxStatus routed any detail <= 0xFF through the
OPC DA quality table: MX_E_PlatformCommunicationError (2) on a stopped platform's
ScanState logged "Unrecognised MXAccess quality byte 0x02" and published generic
Bad (VD03 live capture), MX_E_InvalidObjectId (8) became BadNotConnected,
MX_E_Overflow (20) BadCommunicationError, and a detail in the Uncertain band
published Uncertain for a failure.

Every non-zero detail now maps per the gateway.md vocabulary (timeout,
communication, configuration, type, range, access, initialising), then by
category; the category-fallback warning logs once per (detail, category).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 15:06:26 -05:00
Joseph Doherty efb7b21508 Merge #855 shared per-report event projection: Lock hold back near pre-#855 at N=50
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 14:34:23 -05:00
Joseph Doherty 74b2ce170b fix(opcua): #855 perf review 4 — registry sweep threshold capped; re-sweep at the cap
- The sweep threshold never rises above the 4096 cap (a threshold the count
  could not reach stopped sweeping, and sharing, for good); at the cap a
  sweep for dead shapes runs after every 256 refusals.
- The per-report "queued" mark is taken after the filter group resolves,
  so a failure there leaves the item unmarked.
- The bucket key also carries TypeDefinitionId and IndexRange.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 14:31:46 -05:00
Joseph Doherty 733dfe94da fix(opcua): #855 perf review 3 — weak filter-group registry; where clause validated; sharing pinned
- The filter-group registry holds groups weakly (the filter->group table is
  the only strong reference), sweeps dead entries amortised and counts only
  live shapes against the 4096 cap; filters past the cap are counted on the
  new meter otopcua.opcua.condition_events.filter_groups_unshared.
- A group validates its where clause too (restored, never-validated filters
  evaluate like validated ones).
- Test: value-equal filters cost one projection per locale set per report.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 14:20:19 -05:00
Joseph Doherty b77bb2ded1 fix(opcua): #855 perf review 2 — where clause in the group bucket key; pin dedupe, modify and keys
- Filter-group buckets include the where clause's (value-based) hash, so
  per-source filters sharing a select list are not all scanned under Lock.
- Locale keys are cached per session array.
- Tests: a notifier reached twice in one walk (Line above two wired
  equipment) gets each event once on both paths (fails without the
  per-report set); queued events re-projected by a modify are identical on
  both paths; locale keys unambiguous and where clauses hash by value.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 14:10:02 -05:00
Joseph Doherty ca0e73fbca fix(opcua): #855 perf review 1 — groups evaluate their own filter clone; lazy fields; tighter equivalence tests
- A filter group owns a deep clone of its first filter, validated once and
  evaluated under the group's lock: no item-owned filter is mutated or raced
  (Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
  only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
  registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
  item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
  modified mid-run, a session with other locales, an ungranted anonymous
  Server-object item, per-rig counter; plus a small overflowing queue with
  DiscardOldest true and false.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:59:22 -05:00
Joseph Doherty 98d962b24b perf(opcua): #855 — shared per-report projection for condition events under Lock
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.

ProjectedEventDeliveryWireTests compares both paths over the wire.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:42:29 -05:00
Joseph Doherty c63233b654 Merge #868 fourth review fixes: poison only by SQL error allowlist, memory-counted acks survive restart, Stale-recovered revisions unconfirmed
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:35:42 -05:00
Joseph Doherty ec3b835e4a fix(controlplane,runtime): #868 fourth review — SQL poison allowlist, memory-counted acks survive restart, adopted revisions unconfirmed
M1 IsPoison classified every SQL Server transient as poison:
   Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
   (1205), timeout (-2 / Win32 258), dropped connection or AG failover was
   counted from memory and the node's row left Applying. Poison is now an
   ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
   whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
   retries, bounded by the deadline deferral cap. IsPoison is public for tests.
   Acks counted only from memory (poison branch and the deferral cap) are kept
   in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
   restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
   same-revision dispatch re-applies instead of short-circuiting a clean
   Applied over an empty address space: TryRecoverFromStale (restores only the
   revision marker) and RestoreApplied when the artifact read fails (null
   blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
   restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
   TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
   reach the ConfigDb acks a same-revision re-dispatch Failed
   (PartiallyFailed, not Sealed); Stale recovery and failed restores are
   unconfirmed too.

Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.

Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:31:49 -05:00