Compare commits

..

304 Commits

Author SHA1 Message Date
Joseph Doherty dc0a24f224 feat(galaxy): operator-authenticated alarm ack — send Galaxy credential to mxaccessgw (#749)
v2-ci / unit-tests (push) Failing after 1h8m43s
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:

1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
   held ONLY in the encrypted Secrets store
   (otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
   no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
   FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
   env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
   operator_user. Refusals are excluded from the resilience retry/breaker.

- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
  nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
  RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
  is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
  ("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
  account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
  acknowledger scrubs it from every gateway message it echoes and withholds
  any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
  Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
  path is fire-and-forget after the Part 9 method returns).

Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.

Closes #749
2026-11-06 16:23:00 -05:00
Joseph Doherty 7ec5b95d24 docs: point markdown backlogs at Gitea issues (status lives in Gitea)
v2-ci / unit-tests (push) Waiting to run
2026-11-06 15:31:43 -05:00
Joseph Doherty 2f7f891636 Merge lane D-b: AdminUI rendering, leaks, stubs, deployed-set /hosts eviction (#745)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 18:34:51 -04:00
Joseph Doherty 2e64af26ef fix(adminui): /hosts eviction keys on authored + DEPLOYED drivers (review D-b r1 I1)
Staleness is not a liveness proof: DriverInstanceActor publishes on a 30 s
tick and dedups on a health fingerprint, so a running Faulted, idle or
tagless driver deleted in /raw (undeployed) went stale and was evicted.
DeployedDriverSet reads, per node, the latest and latest-Applied
NodeDeploymentState deployments plus the latest Sealed one, parses each
artifact's DriverInstances once (cached by deployment id, pruned to the live
set, only the blob column of an uncached row is read), and /hosts unions it
with the authored set. The staleness clause stays as a second guard. An
unreadable artifact or a failed read means 'unknown' and skips eviction.
The Hosts.razor comment claiming '~12 health publishes' is corrected.

N-new-2: the CSV download's Content-Disposition carries an RFC 5987
filename* via ContentDispositionHeaderValue.SetHttpFileName.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 18:30:23 -04:00
Joseph Doherty 0f1f8ad8fa fix(adminui): lane D-b review r0 fixes (I1, M1, M2, M5-M9, N1-N3, N5, H3)
I1: /hosts eviction keyed off the AUTHORED DriverInstances, so a driver
deleted in /raw but not yet deployed - still running, still publishing -
vanished every tick. The stores' sweep is now EvictUnconfiguredStale(set,
olderThanUtc): an entry goes only when unconfigured AND its PublishedUtc is
older than 60 s. Tests cover unconfigured+fresh kept (positive control),
unconfigured+stale evicted, configured+stale kept, and the deploy hand-off.

M1: Test Connect falls back to the driver's documented ProbeTimeoutSeconds
default (Galaxy 30, OpcUaClient 15, ...) when the key is absent.
M2: the budget never undercuts the config's root timeoutMs (+1 s).
M5: RawBrowseModal compares the open generation captured before the await,
so a close-then-reopen mid-open closes the stale session.
M6: /alarms-historian keeps a 30 s re-poll on 'not registered' (a driver
node may still be starting) instead of stopping for good.
M7: /scripts source fetch failure shows in the dialog, not the error boundary.
M8: the export modal maps only its preview rows; the endpoint writes via
Results.Text (no second byte[] copy).
M9: /hosts config queries are ordered, so SequenceEqual change detection holds.
N1 tooltip, N2 CRLF-safe regex, N3 throttled Warning for subscriber faults,
N5 equipment-import help text now describes RFC 4180 quoting.
H3: MonacoEditor documents that themes are page-global.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 18:21:46 -04:00
Joseph Doherty 01f7e3d62e Merge master 4fb04a9f (lane D-a) into remed/lane-D-b
ScriptEdit.razor conflict resolved keeping both D-a's on-demand Monaco sync +
Problems list and D-b's delete confirm.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 18:14:58 -04:00
Joseph Doherty 21cb50c51a feat(adminui): refresh the browse tree after a rebirth request (G2-F3)
RequestRebirthAsync now returns only once the requested births have arrived
and gone quiet, but the modal still told the operator to re-expand the tree in
a few seconds. ConfirmRebirthAsync re-keys the tree against the same session
(as the Refresh button does) when the session and modal are still current,
drops the now-stale armed scope, and keeps the outcome line.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 18:02:57 -04:00
Joseph Doherty 15bd409cdd fix(adminui): say why a Sparkplug array/unsupported metric cannot be committed (G2-F2)
Since R06:M-5 the Sparkplug browser states no address for a metric the
ingestor will not read and labels its descriptor 'not bindable', but the commit
refusal still said the attribute lookup returned nothing. RawBrowseModal keeps
the attribute's descriptor on the selection, and DescribeUncommittableLeaf
reports a 'cannot be bound in this version' message echoing it. A lookup that
really returned nothing keeps the old text. A test pins the marker to the
browser's own string.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 18:01:44 -04:00
Joseph Doherty fc3178cca8 feat(adminui): link /fleet rows to node diagnostics; show authored driver names (A-F4, A-F5)
/fleet/diagnostics was reachable only by URL: each driver-role row on /fleet now
links to it (FleetDiagnosticsLinks.For escapes the host:port node id). The
diagnostics table showed the string instance id because the wire carries no
display name; the page now joins the authored DriverInstances.Name from the
ConfigDb (enrichment only - a failed lookup falls back to the id) and keeps the
id beneath the name.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:59:32 -04:00
Joseph Doherty f12573d3fa fix(adminui): register the /dev/context-menu-demo route only in DEBUG builds (R04 S7)
The dev-only demo page shipped routable in the production assembly. Its route
moves from @page to a code-behind [Route] under #if DEBUG, so a Release build
keeps the component type (nothing referencing it breaks) but has no /dev/*
route. The page-authorization guard classifies it under #if DEBUG, and
DevRouteGatingTests asserts each configuration's half (Debug: exactly the demo
route; Release: none) with a positive control. Both configurations pass.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:57:54 -04:00
Joseph Doherty 746267c2a2 fix(adminui): equipment CSV import reads RFC 4180 via CsvParser (R04 S6)
ImportEquipmentModal split each line on commas, so a quoted Manufacturer with
a comma silently shifted every later column. EquipmentCsvParser reads through
the shared Commons CsvParser (the raw-tag importer's authority): quoted
fields, doubled quotes, CRLF and blank lines parse; malformed quoting is an
error, not a guess. Header/shape messages are unchanged.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:56:26 -04:00
Joseph Doherty 4fb04a9fab Merge lane D-a: Monaco on-demand sync, sequenced diagnostics, bounded marker payload, Problems list (#745)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:55:04 -04:00
Joseph Doherty 4cfa369354 refactor(adminui): delete the dead LoadTagsForEquipmentAsync surface (R04 S4)
v3 retired the equipment-Tag binding; IUnsTreeService.LoadTagsForEquipmentAsync
always returned empty, had no production caller, and EquipmentTagRow plus the
always-zero tagCounts / EquipmentRow.TagCount plumbing existed only to feed it.
All removed; the equipment badge count (virtual tags) is unchanged.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:54:53 -04:00
Joseph Doherty a4cdedc31c fix(adminui): confirm equipment-page and script deletes (R04 H1)
The equipment page's Remove reference / Delete virtual tag / Delete alarm row
buttons and the ScriptEdit Delete ran on the first click. They now go through
RawConfirmDialog, the same confirm /raw uses. The ScriptEdit change is limited
to the delete button and its dialog.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:53:06 -04:00
Joseph Doherty a9e8f5abfe fix(adminui): bound the diagnostics marker payload on both the endpoint and the circuit (D-a review r1)
I-1 - a non-C# paste produced thousands of Roslyn markers. The Problems-list report sent
them all as one JS->.NET message, which could exceed the 1 MB hub limit and kill the
circuit. The JS now sends at most MaxReportedMarkers (200) markers, errors first, each
message cut to 300 chars, plus the total. ScriptProblemReport.FromEditor clamps again
on arrival, and the Problems list shows "showing N of Total" / "+N more". The diagnostics
endpoint also bounds its response: Distinct, errors first, MaxDiagnosticMarkers (1000),
messages cut to 1000 chars. An error is always kept, so editor-accepts == publish-accepts.

M-1 - drop the per-instance compilation lineage; analyses derive from BaseCompilation.
M-2 - the VirtualTag modal clears its problems (and editor ref) on close and on a script
reload.
M-5 - an over-limit text is announced at blur (OnSyncRefused), not only at Save.

Refs #745

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:51:35 -04:00
Joseph Doherty e16146fa0c fix(adminui): close a browse session that opens after its picker is gone (R04 L4)
Every picker and RawBrowseModal stored the OpenAsync token after the await; if
the host was disposed (or the modal closed) meanwhile, its teardown had already
run with no token, so the plant-side session lived until the reaper's idle TTL.
BrowseOpenGuard.CloseIfOrphaned closes it at once; each host sets _disposed
first thing in DisposeAsync (RawBrowseModal also treats !Visible as gone).

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:51:34 -04:00
Joseph Doherty 1313b94097 fix(adminui): /alarms-historian says when the role runs no historian; Fleet-style lifecycle (R04 L3)
A bare catch {} turned every failure - including an admin-only node with no
historian actor - into 'Loading...' forever, and a tick queued after a sync
timer dispose still issued an Ask. HistorianSinkStatusReader classifies a poll
as Available / NotOnThisRole (permanent: shown once, polling stops) /
Unavailable (shown with the last good status and its read time). The page
follows Fleet.razor: IAsyncDisposable, _disposed, overlap guard, page CTS.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:47:30 -04:00
Joseph Doherty 17c7d05696 Merge commit '20c7d13d' into remed/lane-D-a 2026-10-22 17:47:26 -04:00
Joseph Doherty 4b594ec525 fix(adminui): Test Connect honours the configured probe timeout; drain its clear timer (R04 L2, G1-F5, F-FU2)
DriverTestConnectButton always sent a fixed 10 s, and the Galaxy/OpcUaClient
probes run under min(caller, configured), so ProbeTimeoutSeconds - documented
on every driver as the Test Connect timeout - could shorten a probe but never
lengthen it (Galaxy's 30 s default was silently 10 s). TestConnectBudget reads
the root ProbeTimeoutSeconds from the merged config at click time, clamped to
the actor's [1, 60] s, falling back to TimeoutSeconds. DeviceModal (the only
host) needs no change. The Galaxy/OpcUaClient form hints are reworded.

The 30 s result-clear timer moves from System.Timers.Timer + an async void
Elapsed lambda to System.Threading.Timer, drained in DisposeAsync, with a
generation so a superseded fire cannot clear a newer result.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:45:39 -04:00
Joseph Doherty b0959f697b perf(adminui): cache the modals' DynamicComponent parameter dictionaries (R04 H2)
DriverConfigModal, DeviceModal and RawTagModal built a new parameter dictionary
and new EventCallbacks every render. ParameterDictionaryCache reuses the
dictionary while the values it carries are unchanged and rebuilds it with the
current value when one changes (a build-once cache would strand edits); the
callbacks are created once. Blazor still re-sets an EventCallback-bearing
child's parameters per parent render, so the forms' _lastParsed guards stay.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:43:01 -04:00
Joseph Doherty 36e6ab590b perf(adminui): virtualize the CSV import grid; download exports from an endpoint (R04 P6)
The import review grid rendered every parsed row and re-scanned all rows three
times per render for its counts; RawCsvReviewView recomputes rows + counts once
per change and the grid renders through Virtualize. The export modal embedded the
whole CSV twice (textarea + base64 data: URI); it now shows a 50-line preview
and links to GET /api/raw/tags.csv (ConfigEditor, like /raw), which runs the same
RawTagCsvExport builder and sends it as an attachment.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:40:40 -04:00
Joseph Doherty 20c7d13d74 Merge Wave-2 integration: Modbus array-write rows + FC15/FC16/FC23 PDU length fix (#741)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:40:15 -04:00
Joseph Doherty da2578833d fix(adminui): Monaco text syncs on demand, diagnostics are sequenced, Problems list (D1)
R04:P1 - the editor no longer ships its full text JS->.NET on every keystroke. It reports
at blur, and ScriptEdit / the VirtualTag modal read the authoritative value with
MonacoEditor.GetValueAsync at Save (validation runs after the pull). The Blazor hub's
MaximumReceiveMessageSize is raised from 32 KB to 1 MB. The JS refuses a value that
would not fit instead of sending an oversize frame, and Save surfaces that refusal
rather than persisting stale source.

R04:P9 - diagnostics carry a generation counter and an AbortController. Every edit aborts
the in-flight request, and only the response for the current text is applied.

R04:S2 - the always-empty inlay-hints provider, endpoint, service method and contracts
are removed.

R04:S3 - MarkersChanged is typed (DiagnosticMarker) and feeds a new ScriptProblemsList.
Clicking a problem calls RevealLineAsync. Markers cross to .NET only when a list
subscribes.

Refs #745

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:38:56 -04:00
Joseph Doherty af0cd50cdc Merge lane K: clients, CLI, AB transport (#747)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:38:08 -04:00
Joseph Doherty ce9679b671 feat(adminui): GET /api/deployments/{id}, the status resource the 202 Location names (R04 S5)
The deploy API's 202 pointed at a route that did not exist. The GET reads the
deployment (projected, no artifact blob) and its per-node apply states, and
carries the POST's own gate: API key in X-Api-Key, AllowAnonymous against the
cookie fallback, 503 while Security:DeployApiKey is unset.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:36:54 -04:00
Joseph Doherty 0f305f1827 fix(modbus): multi-write PDUs carry no trailing byte past their byte count
The FC15, FC16, BitInRegister[] RMW FC16 and FC23 builders allocated one byte
more than they filled (6 + 1 + n / 10 + 1 + n, data copied at offset 6 / 10), so
every multi-write request ended in a stray zero byte beyond the length its
byte-count field declares. A strict device or gateway that checks the frame
length against the byte count refuses it. Each PDU is now exactly 6 + n
(FC15/FC16) or 10 + n (FC23).

Tests: ModbusWritePduFramingTests asserts the exact length and byte-count field
for FC15 (1/8/9 coils), FC16 (array, Float64 array, scalar multi-register), the
BitInRegister RMW FC16 and FC23 (all red before the fix); the parity suite's
conformant device now refuses a frame-length mismatch on FC15/FC16/FC23.
Live: Driver.Modbus.IntegrationTests against 10.100.0.35:5020 (standard
profile) passes, including the FC16 and FC23 multi-register round trips.

Refs #741

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:35:40 -04:00
Joseph Doherty 2a788ed8ac perf(adminui): RawTree renders a flat Virtualize list, pages 500 per level, caches menus (R04 P5)
The recursive RenderNode rendered every row of an expanded device, and built a
fresh context-menu list + closures per node per render. RawTreeRows flattens the
expanded tree (explicit stack) into rows rendered by one Virtualize; each level
shows 500 children, then a 'Show more' row. RawNodeMenuCache keys menus weakly
on the node instance and rebuilds on a picker-mode or Enabled change.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:33:45 -04:00
Joseph Doherty cf62007678 fix(arrays): Modbus row honours the FC16/FC03 protocol ceilings, the RMW read cap and the address-space end
Review r0 of Integration-W3:
- I1: the effective FC16 cap is min(configured maxRegistersPerWrite, 123). The
  driver trusts a larger configured cap and sends an over-ceiling FC16 (past 127
  registers its byte-count byte wraps), which every conformant device refuses, so
  such an array is never advertised writable.
- I2: a BitInRegister[] write reads its span first in one unchunked FC03, so it is
  also bounded by min(configured maxRegistersPerRead, 125) (absent/0 = 125).
- m2: every span (registers and coils) must end at or before address 65535.
- m1: the String-node wording says it converts only when the text parses, which
  the server cannot promise (DateTime never converts).
- m4: the parity device now enforces FC01/02 2000, FC03/04 125 (or a smaller
  device read limit), FC15 1968, FC16 123, FC23 121/125, the FC15/FC16 byte-count
  field (exception 03) and the address-space end (exception 02).

Tests: parity rows for each ceiling/cap/span boundary against the real write
path, and Core.Abstractions row tests. The configured-cap test that asserted a
cap of 200 lifts the ceiling now asserts the opposite.

Refs #741

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:32:57 -04:00
Joseph Doherty 8e3bbf91fb fix(adminui): script analysis reuses one compilation, honours cancellation, indexes the catalog (D2)
R04:P2 - every analysis derives from one process-wide bound base compilation via
ReplaceSyntaxTree (the scoped service keeps its current compilation); every entry point
takes the request token, the endpoints bind RequestAborted and answer 499 on abort, and
the Monaco providers abort their fetch when Monaco cancels the request.

R04:P3 - the catalog snapshot carries a sorted, de-duplicated string[] (binary-search
prefix completion) and an Ordinal path->info dictionary, built once per snapshot.

R04:S1 - Diagnose emits an OTSCRIPT_SETVTAG_INERT warning for each ctx.SetVirtualTag
call. It is a warning, so editor-accepts == publish-accepts still holds.

Refs #745

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:31:54 -04:00
Joseph Doherty 10b83d69c5 perf(adminui): /scripts lists a projection and loads source per dialog (R04 P8)
The list loaded every script entity including SourceCode to render names and
hashes. ScriptListQueries projects {ScriptId, Name, Language, SourceHash} for
the list and fetches one script's source when its Source dialog opens, with a
generation guard against a superseded open.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:30:50 -04:00
Joseph Doherty 7e958cd750 perf(modbus): pooled request ADUs, a reused MBAP header and single-buffer RTU deframing
R07:L5 (Modbus half, former F2): ModbusTcpTransport builds the MBAP ADU in a
pooled buffer and reads the reply header into one per-transport buffer (sends are
single-flight under the gate). ModbusRtuOverTcpTransport writes its ADU into a
pooled buffer (ModbusRtuFraming.WriteAdu, byte-identical to BuildAdu), and
ReadResponsePduAsync reads and validates the whole frame in one pooled buffer
instead of five arrays. Only the returned PDU is allocated. The exchange stays one
async method: splitting it cost a state-machine box that ate the saving. Scratch
A/B per op: RTU deframe 352 -> 184 B, RTU send 640 -> 408 B, TCP send 864 -> 800 B;
timings within noise; Driver.Modbus.Tests 2 s before and after. NormalizeWordOrder
lives in ModbusDriver (not a transport) and is left alone.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:28:53 -04:00
Joseph Doherty 8e69118200 perf(adminui): coalesce /script-log and /alerts renders with a ring buffer (R04 P7)
Each received event used to Insert(0) into a List and render the full page.
LiveTailBuffer queues events from the bridge thread into a bounded newest-first
ring; CoalescingRenderScheduler (TimeProvider-driven, drained on dispose)
flushes and renders at most once per 200 ms window. ScriptLog's filtered rows
are recomputed on flush/filter change instead of three times per render.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:28:23 -04:00
Joseph Doherty 348ffaec1c perf(adminui): /hosts rebuilds driver rows on the tick, not per health publish (R04 P4)
OnSnapshotChanged used to re-project every cluster group and re-render the
whole page for each driver publish, per open tab. It now only marks a
CoalescingDirtyFlag; the 5 s tick rebuilds when the flag is set or the config
load changed (or evicted) anything.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:28:23 -04:00
Joseph Doherty a04196d725 fix(adminui): evict unconfigured drivers from status stores; isolate throwing subscribers (R04 L1, L5)
The driver-status and resilience stores had no removal path, so a deleted
driver stayed on /hosts as a frozen row until restart. Both stores gain
RetainOnly, which /hosts calls after each successful config load (never on a
failed one).

The stores and InProcessBroadcaster now raise their events one subscriber at a
time, so a throwing handler cannot starve later subscribers or escape onto the
bridge actor. The alert/script-log bridges publish inside their try, and the
resilience bridge catches a store failure instead of restarting.

Refs #745
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:24:04 -04:00
Joseph Doherty 004c09fa13 fix(ablegacy): an Unsupported the device replied with is an answered refusal, not a local one
Review I2: Driver.Ab.Transport raises statuses that came off the wire in an
Execute PCCC reply (envelope CIP status, PCCC STS) as AbDeviceReplyException.
AbLegacy's IsLocalRefusal now excludes it, and the new IsAnsweredRefusal (used
by the read, write, probe and indirect-index paths) counts a wire Unsupported -
CIP 0x08/0x1C/0x20 on the envelope, STS 0x10 - as answered: per-tag
BadNotSupported, Healthy, LastSuccessfulRead advanced. Before, they landed in
the 'refused before the wire' neutral class. Pre-wire refusals are unchanged
(still neutral). A socket test covers all four codes (red before); transport
tests pin the marker and a pre-wire builder refusal as the positive control.
The AbCip Forward Open BadConnection path is untouched.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:21:42 -04:00
Joseph Doherty 8699614d2f Merge commit '82380d2b' into remed/integration-w3 2026-10-22 17:21:38 -04:00
Joseph Doherty 0c9e4f4e3d fix(client): a failover abandons itself once the user disconnects or reconnects; paging and teardown minors
Review I1: ConnectAsync, DisconnectAsync and Dispose each start a new connection
epoch under _subscriptionLock. A keep-alive failover runs under the epoch it
started in and abandons itself the moment it moves: it installs its session only
if the epoch still matches (checked and assigned under the lock, otherwise the
session is disposed), its state transitions are epoch-guarded, it stops trying
endpoints, and it never flips a user's new connection to Disconnected. A late
keep-alive from a previous connection's session starts no failover. Gate-driven
tests cover the success and all-fail arms, the stale keep-alive, with a genuine
failover as the positive control (4 of them red before).

Minors: m8 only the CompareExchange winner sets LastFailover; m5 the endpoint
loop stops early, and a failed Unsubscribe still drops its bookkeeping; m1 a read
cancelled mid-paging releases its cursor (bounded token) and rethrows, and the cap
release no longer rides the caller's token; m2 results are trimmed to the cap;
m3 a non-positive cap is refused (CLI and UI already bound it to >= 1); m6 the
subscription adapter's Dispose uses one bounded bridge for delete + detach.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:18:26 -04:00
Joseph Doherty 82380d2bee Merge lane G1: OpcUaClient + MTConnect driver remediation (#743)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:16:20 -04:00
Joseph Doherty 31605e547d Merge commit 'd1f24cf6' into remed/lane-K 2026-10-22 17:13:56 -04:00
Joseph Doherty 5533de6281 feat(arrays): Modbus String[]/BitInRegister[] array writes reachable; table follows the resolver
The Modbus lane (#741 U3) gave the driver String[] and BitInRegister[] holding-register array encoders, and
moved address resolution to ModbusTagAddressResolution (keys first, a bit region coerced to Bool, strict key
reads), but DriverArrayWriteSupport still listed both array shapes read-only (follow-up F1), so a client
write was BadNotWritable before it reached the driver.

Modbus row now mirrors the driver end to end:
- Keys shape only (region + address both present). A string-only addressString tag is resolved by the
  driver with its own family through a parser Core.Abstractions cannot reference, so it stays read-only
  (pinned: the conservative direction).
- Strict reads as the resolver: address an integer 0-65535; a present region / dataType / byteOrder /
  stringByteOrder string must name a member.
- Coils: any dataType (coerced to Bool), <= 1968 coils (FC15).
- HoldingRegisters (FC16, never Bool): numeric, String ((stringLength+1)/2 registers, stringLength >= 1),
  BitInRegister (1 register, bitIndex 0-15, span ending <= 65535), all within maxRegistersPerWrite.
  FC23 is opt-in and never changes the answer.
- Elements convert (ScadaBridge int[] -> Int16 stays writable); only a String / DateTime node over a
  numeric or bit element is left out, since such an element never converts.

Tests: the parity test's PendingF1 exemption and its guard are gone; the parity matrix is now region x
element type x node type against the real WriteAsync with a spec-conformant device (FC15 > 1968 refused),
plus shape edges at every cap boundary (numeric, String, BitInRegister, coil), strict key reads, FC23
on/off, string-only, the ScadaBridge int[] shape and U3 wire bytes. Core.Abstractions row tests and a
composer test (String[] and BitInRegister[] writable; over-cap, no-stringLength and string-only read-only).
Docs: table XML doc, Uns.md, modbus-addressing.md, drivers/Modbus.md, data-plane rules.

Refs #741

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:10:00 -04:00
Joseph Doherty d64c77b62d fix(opcuaclient): shutdown cancels session work before the gate wait; dispose skips a redundant nested shutdown (review r2 R2-1, R2-2)
- R2-1: the ConditionRefresh sweeps (RefreshRequired, post-outage recovery) run
  under the session's work token, and ShutdownAsync cancels session work first
  thing - so a sweep holding the subscription gate against a stalling upstream no
  longer eats the whole teardown budget the deletes and close need.
- R2-2: DisposeAsync skips its nested ShutdownAsync when a shutdown already started
  for the current lifecycle generation (nothing initialised since), so PostStop's
  dispose does not re-wait the gate past its budget. Test: with the gate held, the
  shutdown waits (positive control) and the dispose does not wait again; red with
  the skip removed.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:08:44 -04:00
Joseph Doherty 419ecb3ce9 fix(mtconnect): a delivered /sample chunk clears an unanswered-write degradation (review r2 N-5)
Coordinator ruling: a delivered chunk proves the Agent is alive (the analogue of
OpcUaClient's good keep-alive), so TryMarkStreamDelivered - generation-checked
under _subscriptionLock - also clears _writeDegraded. The degradation no longer
sticks forever on a subscription-only driver. Test: unanswered write -> Degraded,
stays Degraded with nothing else happening (positive control), the next chunk ->
Healthy; red with the clear removed. Round-2 doc text corrected.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:08:44 -04:00
Joseph Doherty d1f24cf6a7 Merge lane E5: alarm comment audit table (#748)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 17:06:37 -04:00
Joseph Doherty dad28d37a2 fix(alarms): never drop a comment-trail entry on pair divergence (E5 review r1)
I-1: a node whose in-memory trail is one short of the stored trail (it missed
a command the peer applied, whose entry then replicated in) silently dropped
its own next entry — in failover, the survivor's newest ack — because the
positional "past MAX(seq)" rule treated it as already stored.

SaveAsync now reconciles the last 16 in-memory entries against the stored
rows from that position on, by content (Kind/User/Text, timestamps within a
5-minute skew window; matched rows are consumed so repeats count). The peer's
copy of the same command is recognised and not re-appended; an unmatched
entry is inserted at its own ordinal when free, otherwise appended at
MAX(seq)+1 with a Warning — kept, never silent. Bounded read, constant write.

m1: MigrateInlineComments copies only — it no longer UPDATEs the condition
row (a fresh-HLC re-stamp could beat a newer peer ack under LWW); the next
save clears comments_json. m2: it returns unparseable alarm ids and OnReady
logs a Warning per id (+ an Information tally). m3/m4: documented in the
store remarks (Remove vs lagging peer; in-memory index == durable ordinal).

Tests: store-level failover sequence (missed command -> survivor's ack and
next entry persisted once, re-save writes 0 rows, one Warning) with the
dual-write positive control (peer copy present -> 0 rows, no Warning) and a
repeated-identical-comment case; a loopback pair failover test (link down,
survivor acks, rejoin -> both entries on both peers, held); migration
no-row-rewrite, partial fill and unparseable reporting; OnReady Warning.
Both failover tests fail against the round-0 store.

Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:58:46 -04:00
Joseph Doherty bc1f341e8a fix(mtconnect): Unsupported latch inside the checked section; live-index reads do not clear a write degradation (review r1 N-4, N-5)
- N-4: the /sample Unsupported latch is set inside TryDegradeStream's
  generation-checked critical section (latchUnsupported), so a superseded pump can
  no longer latch it on the session that replaced it.
- N-5: a read served from the live pump index clears only the read degradation -
  it is stream evidence, like a chunk, which by the I-3 ruling does not clear an
  unanswered write. The 'cannot stick' wording in the XML doc and
  docs/drivers/MTConnect.md now states the latch-until-next-answered-/current-read-
  or-write behaviour, incl. the subscription-only consequence. Tested (chunk +
  live-index read keep Degraded; an answered write clears it).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:58:37 -04:00
Joseph Doherty c39bcd6e24 fix(opcuaclient): unresolvable acks warn instead of tripping the ack breaker; shutdown holds subscription management (review r1 N-1, N-2, N-3)
- N-1 (Important): m3's BadNodeIdUnknown throw for an ack with no resolvable
  condition counted toward the host's AlarmAcknowledge breaker (threshold 5), so
  re-acks of one unresolvable condition blocked every genuine ack. It now logs a
  Warning and returns; only a no-reply transport failure throws. Test through the
  real CapabilityInvoker: six unresolvable acks, then a genuine ack that is still
  delivered (red with the throw restored).
- N-2: ShutdownAsync holds _subscriptionGate for the whole teardown (in-flight
  subscribes finish first, bounded by the budget), so a subscribe arriving during
  the delete phase finds the session released instead of creating a subscription
  on the closing session. Red-tested.
- N-3: the shutdown-side cert-validation detach is generation-checked under
  _probeLock with the map / endpoint clear.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:58:37 -04:00
Joseph Doherty 92dea29547 Merge lane Modbus: Modbus driver remediation (#741)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:53:49 -04:00
Joseph Doherty 863a00c25d Merge master (75b7effb) into remed/lane-G1
Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:52:43 -04:00
Joseph Doherty abbaac3615 perf(ab-transport): pool CipWriter, reassembly and frame-header buffers
R07:L5 (AB half): CipWriter rents its working buffer from ArrayPool and returns
it in ToArray (which now completes the writer; later writes throw). A single-reply
Logix read is one copy of the data (no List<byte> + ToArray); a fragmented read
reassembles in a pooled buffer. EncapsulationClient reads the frame header into a
per-session buffer (reads only happen under the session gate) instead of a new
array per reply. Measured per op (scratch A/B, Release): frame encode 784 -> 200 B,
Logix read 1994 -> 1655 B, 3-fragment read 11.9 -> 8.2 KB, EIP round trip
2.75 -> 2.15 KB; encode time 41-46 -> 28-39 ms per 500k, other timings within
noise; Ab.Transport.Tests 3 s before and after. Modbus transports are left to a
follow-up (its lane is still in review).

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:52:26 -04:00
Joseph Doherty 75b7effbff Merge lane G2: MQTT + Sql driver remediation (#744)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:47:32 -04:00
Joseph Doherty 13f1357ad2 Merge master d520f77b into remed/lane-E5
Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:46:30 -04:00
Joseph Doherty 92749402b5 fix(modbus): review r3 - one address authority (ModbusTagAddressResolution)
N3: a keyless blob carrying the addressResolved marker was served from its
string by the driver but saved as HR[0] by the editor model / CSV path. The
precedence rule now lives in ONE pure resolver in Driver.Modbus.Contracts
(ModbusTagAddressResolution.Classify/Resolve): keys when region+address are
both present, else a string-only addressString resolved with the driver's
family (refused when it cannot parse), else refused. The marker never decides.

Consumers that now ask it instead of restating it: the driver's tag reader
(ModbusTagDefinitionFactory.FromTagConfig), the deploy gate (Inspect /
DescribeRejection), the editor model (AddressShape / IsStringOnly /
HasKeyAddress / ResolveWith, ToJson stamping, Validate), the CSV export cells
and the editor hint. The model writes keys only when it loaded them or a field
was authored - never defaults it did not resolve (also closes two divergences
the new matrix found: malformed-string+marker and non-string addressString were
saved as HR[0]).

One table-driven test (ModbusAddressAuthorityAgreementTests) runs 15 persisted
shapes through driver, deploy gate, model load/save/Validate and the real CSV
export->import under all four families; breaking the CSV export or reverting
the model to the marker rule fails it.

Minors: m1 CSV address cells keep the string+marker while it still describes
them (lossless unchanged round trip), drop it when contradicted, and the review
grid reports the drop; m2 the pre-lane test uses the real export residual; m3
unmarked string-only + typed-cell case added; m4 stale docs/summary and the
unused factory parameter removed.

Refs #741

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:46:21 -04:00
Joseph Doherty dfc62cef14 fix(ab-transport): map CIP 0x1C/0x20/0xFF-0x2104 as refusals; remap the PCCC STS table
K-F1 (AbCip #737 F1): FromCip maps general 0x1C and 0x20 to Unsupported and
0xFF/0x2104 to OutOfBounds, so a controller refusal is per-tag Bad instead of a
BadReply comms fault. A Forward Open refused with one of them is still
BadConnection. Every CIP code is now listed with its meaning; 0x01/0x07 stay
opaque on purpose.

K-F3 (AbLegacy F8, review 07 r0 I4): FromPccc follows the DF1 STS table:
0x20/0x30/0x90 -> BadConnection (a bridge reporting an unreachable END node),
0x50 and EXT 0x06 -> NotFound, EXT 0x07 -> OutOfBounds, 0x60 -> NotAllowed,
0x70/0xB0 -> DeviceStateConflict. Golden ParseReply tests per STS, and an
end-to-end AbLegacy test: STS 0x30 degrades with LastSuccessfulRead preserved;
the 0x60 positive control stays Healthy and advances it. Both were red before.
Driver edits are doc remarks and the two runtime tests the old table pinned.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:43:37 -04:00
Joseph Doherty f0d807e7b7 fix(opcuaclient): review r0 minors m1-m10
- m1: the form hint now says ProbeTimeoutSeconds can only shorten Test Connect's
  10 s budget (passing the configured value as the button's budget is shared code -
  follow-up).
- m2: a superseded init no longer overwrites _options (generation-checked under
  _probeLock) or the cert-validation handler (handed back and published with the
  session; detached when not installed; a previous one detached on replace).
- m3: an ack none of whose condition ids resolves throws BadNodeIdUnknown + Warning.
- m4: an ack with no reply (transport failure / non-caller timeout) degrades.
- m5: no session before the gate is the same ServiceResultException(BadNotConnected).
- m6: the caller-cancelled-write deviation is documented.
- m7: a reply during a published outage advances the stamp but does not write
  Healthy; the keep-alive recovery ends the outage.
- m8: test for a non-caller OperationCanceledException write -> Degraded.
- m9: the browse session owns and disposes the user certificate it loaded
  (OpcUaUserIdentityFactory.LoadCertificate; a key-less cert is disposed before the
  throw); AdminUI-host certificate path documented.
- m10: a subscribe overtaken by a shutdown / re-init removes its half-built
  subscription instead of registering it (data and alarm).
- Sync Dispose drains each gate for 500 ms (was 2 s).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:42:29 -04:00
Joseph Doherty e7e84e0577 fix(mqtt): review minors - QoS 0 is not answer evidence, caller cancels stay cancels, rebirth wait cap, bound/trie pins (R06 G2 MQTT review minors 2 3 4 6 7 8)
2: only a PUBACKed (QoS >= 1) publish advances LastSuccessfulRead; a QoS 0 Sparkplug
command completing means only that the bytes left the local socket.
3: a no-verdict SUBSCRIBE caused by the caller's own cancellation now throws
OperationCanceledException (refs still Bad + pending), so the breaker no longer
counts a pipeline timeout as a failure.
4: the rebirth wait is the window capped at 15 s, below the AdminUI 20 s per-call
timeout.
6: a 10-topic test pins the 8-in-flight publish bound. 7: partial-level wildcards and
$share filters are pinned never to confirm. 8: the intended init-vs-shutdown
supersede is commented at the re-check.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:39:24 -04:00
Joseph Doherty 846c8d6335 fix(alarms): move scripted-alarm comment trail to its own append-only table (R05:4)
Every Part 9 transition used to re-serialize the alarm's whole comment trail
into alarm_condition_state.comments_json and replicate it to the pair peer:
O(n) per transition, O(n^2) over an alarm's life (measured: one ack at 10k
comments shipped 1,080,553 oplog bytes; now ~1.6 KB, the same as at 10).

- New replicated table alarm_condition_comment, PK (scripted_alarm_id, seq),
  seq = the entry's ordinal in the trail. Ordinal (not content) keying makes
  the two pair nodes' copies of one logical entry (each stamped by its own
  clock, since both nodes apply every command) collapse to one row under LWW.
- LocalDbAlarmConditionStateStore.SaveAsync appends only entries past the
  stored MAX(seq) (INSERT OR IGNORE, append-only) and upserts a fixed-size
  condition row with comments_json = "[]", in one transaction. Remove drops
  the trail with the row. Load merges the table with any inline trail.
- comments_json column kept (always "[]") so the condition table's shape and
  replication digest are unchanged and old-shape rows still load exactly.
- AlarmConditionStateSchema.MigrateInlineComments: idempotent per-row move of
  inline trails, run by LocalDbSetup.OnReady after RegisterReplicated so the
  moved entries replicate. Comment table joins ReplicatedTables (now five).

Tests: constant transition payload at 10 vs 10k comments (+ instrument
positive control), old-shape back-compat load, migration idempotency /
partial re-run / lazy move, OnReady boot migration captured in the oplog,
and two loopback pair tests (A->B and B->A trail convergence; the shared
entry converges to one copy, with a distinct-alarm positive control).

Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:36:33 -04:00
Joseph Doherty 099f253321 fix(sql): review minors - superseded-reader polls, rebind faults, disposed-reader guard, JSON position (R06 G2 review m-1 m-2 m-4 m-6 m-7)
m-1: a poll that captured a reader a rebind then disposed retries once on the current
reader; with no newer reader the typed SqlPollReaderDisposedException is marked
classified, so HandlePollError no longer degrades for it. Groups report a disposed
reader as a sentinel so WhenAll surfaces the typed exception, not a bare cancel.
m-2: TryAddRef refuses new groups once Dispose ran, matching the doc comment.
m-4: a rebind / tag-table failure in InitializeAsync now writes Faulted with a
sanitized reason (generation-checked) and host Stopped instead of latching
Initializing with no LastError.
m-6: SecurityClass-as-descriptor is documented at the Sql site (Commons field is a
follow-up). m-7: a position-less JsonException renders '?' instead of blanks.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:35:52 -04:00
Joseph Doherty edd2e23b7a test(mtconnect): de-race the raised-count step-back test
The one-shot /sample failure was enqueued while the FIRST enumeration could still be
before its own one-shot check (the fake releases the call-count waiter first), so
under suite load the failure hit the first stream instead of the raised one
(intermittent). It is now enqueued after the first stream has consumed a chunk.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:35:34 -04:00
Joseph Doherty f9fbd4ee0d fix(cli): AbCip subscribe serialises and guards console writes; Ctrl-C exits cleanly; arrays render
R07 U7: Driver.Cli.Common gains RunCancellableAsync (moved up from AbLegacy) and
CreateChangeLineWriter (one lock-serialised line per change; a failing console
write is logged and swallowed so it cannot fault the poll loop). AbCip
probe/read/write print Cancelled. on Ctrl-C instead of a stack trace, and AbCip
subscribe uses the shared writer. SnapshotFormatter renders arrays/lists as
[a, b, c] instead of System.Int32[].

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:34:48 -04:00
Joseph Doherty e03f343f14 fix(mtconnect): read-path restart publishes the new cursor; document the review trade-offs (review m6, m1, m4, m5, m8)
- m6: a /current read that sees a new Agent instanceId now also publishes the new
  process's cursor (generation-checked, captured with the read's session), so a
  later subscription does not open at the dead instance's sequence and flicker
  through a clear + re-baseline. Pinned in the T-2 test.
- Documented (code + docs/drivers/MTConnect.md): m1 the bounded stale-read window of
  a stream stalled without failing; m4 a refused multi-pair PUT's retry may re-send
  an already-applied pair (harmless); m5 a non-Agent 200 now reads reachable to the
  ping; m8 a refused raised count takes one transient blip before the step-back.
- m10 confirmed: no host key is persisted (DriverHostStatus dropped, #521) - the
  redacted HostName only changes an in-memory /hosts key for a query-bearing URI.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:33:14 -04:00
Joseph Doherty c1b9c7ec5c fix(sql): attribute a duplicated key to its own tags; key-value cap gets 2x headroom (R06:S-1 review I-1, m-3)
One stray duplicate row used to fail every tag in a key-value group or none of them,
depending on whether an unrelated key was absent. Duplicated keys are now refused
per key: only that key's tags publish BadConfigurationError and the rest of the group
reads normally. The key-value cap reads up to 2x the distinct keys (+1 to detect
excess), so a few stray duplicates are attributed per key and only a gross excess
(history/events table) fails the whole group. Duplicated-key refusals count as
answered refusals for health. The row-cap, duplicate-key and multi-row warnings now
use the per-plan limiter (m-3); the reader-wide limiter is removed. Unit tests pin
both the all-keys-present and absent-key cases with Good controls; live SQL Server
gate added (23/23).

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:32:23 -04:00
Joseph Doherty b2c064bec3 fix(mtconnect): an unanswered write degrades; one success writer for reads and writes (review I-3, m7)
Coordinator ruling on F3: a write with no reply (transport failure / deadline)
sets Degraded with the stamp preserved and a type-only LastError, through a new
_writeDegraded flag folded into PublishHealthy. One success writer is shared:
RecordReadAnswered (a /current answer, an Agent refusal, a live-index read)
clears the read AND write degradations; RecordWriteAnswered clears the write
one; neither clears the other path's stream degradation. Tests both directions
(answered write clears it, good read clears it) plus the converse (an answered
write does not clear a failed read); red with each half removed.
m7: a good /current resets the refusal-dedup latch, so a refusal recurring after a
recovery is logged again (tested).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:31:26 -04:00
Joseph Doherty 2d8bb037f5 fix(client): failover runs under the subscription gate; ConnectAsync refuses when connected
R08 4.5: every subscription operation that awaits I/O and the failover's
teardown + reconnect + replay share one async gate, and every read/write of the
subscription fields (failover's null-outs included) is under _subscriptionLock.
A subscribe in flight when a failover starts is recorded first and then replayed
onto the new session instead of landing on the dying session's subscription. The
dead session is disposed before the gate is taken so a blocked call fails fast.
A DisconnectAsync during a failover is no longer undone by it. ConnectAsync on a
connected or reconnecting client throws instead of overwriting (and leaking) the
live session.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:30:35 -04:00
Joseph Doherty 31af7acde0 test(mtconnect): make the STAB-20 retention test deterministic again (review I-2)
29c829b5 let the test accept 'already finished' for the stale-session pump, which
since T-1 is nearly always true, so deleting the retention reintroduced STAB-20
silently. A new CannedAgentClient.HoldSampleExit knob keeps that pump alive past
the retire point; the test now asserts strictly that it is still running
(positive control) AND retained, then releases it and asserts shutdown joined
it. Red with the _supersededPumps.Add removed. The pump's own self-exit on a
gone generation is covered by MTConnectStaleCursorTests (pump A completes, the
current pump keeps streaming).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:29:04 -04:00
Joseph Doherty 13b29e8d4a Merge commit 'd520f77b' into remed/lane-Modbus 2026-10-22 16:28:49 -04:00
Joseph Doherty 8f0046ed05 Merge branch 'master' into remed/lane-G2 2026-10-22 16:28:15 -04:00
Joseph Doherty 540b1632c5 perf(client): lazy child detection in BrowseAsync, NodeId-indexed UI notifications
R08 3.6: BrowseAsync no longer issues one HasChildren Browse per Object child; an
Object reports HasChildren (may have children) and is confirmed when expanded or
recursed. ISessionAdapter.HasChildrenAsync is removed. The UI subscriptions panel
routes each notification through a NodeId index kept in step with
ActiveSubscriptions (add/remove/replace/reset) instead of scanning every row.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:27:24 -04:00
Joseph Doherty f48a065c21 fix(mtconnect): a superseded pump writes no health, flags or index state (review I-1, m2, m3)
- The pump's delivered/degraded writes (stream flag, live flag, PublishHealthy /
  Degrade) now happen in TryMarkStreamDelivered / TryDegradeStream: one critical
  section under _subscriptionLock that re-checks the session generation. A pump
  wedged in a subscriber past its last check can no longer report a replaced
  session's (down) stream Healthy and make the read path trust it.
- m3: the index and condition tracker are installed in the same lock as the
  session + generation and captured under it (TryCaptureSessionState), so a stale
  chunk cannot fold into the new session's index.
- Teardown also bumps the generation, so an abandoned pump cannot write over the
  shutdown's health either.
- m2: an Agent restart clears the live flag before clearing the index, so reads
  go to /current until the re-baseline lands.
Red test: the wedged-pump harness with the new stream failing, plus a positive
control (the current pump's chunk restores Healthy).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:27:23 -04:00
Joseph Doherty a74a18e49a fix(client): offer exactly the served aggregates in the CLI and UI
R08 2.3: AggregateType drops StandardDeviation (no backend serves it; always
BadAggregateNotSupported) and gains Range, Delta, MinimumActualTime,
MaximumActualTime, PercentGood and PercentBad. A parity test pins the enum to
AggregateFamilies.Served and each member to its same-named Part 13 ObjectId.
The CLI parses every enum name (plus aliases) and the UI picker lists them all.
Existing members keep their numeric values; 6 is retired.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:27:08 -04:00
Joseph Doherty d520f77ba0 Merge Wave-2 integration: array writes reachable for AbCip/AbLegacy/TwinCAT (#737 #738 #739)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:25:16 -04:00
Joseph Doherty 0200619322 fix(modbus): review r2 - region+address keys always win; string-only means no keys
N2: pre-#741 rows carry region/address beside an addressString with no marker
(the old driver never read the string, the old model preserved it as an
unknown key). The r1 rule resolved those from the string - a silent remap or
BadNodeIdUnknown on upgrade. The driver now loads from region+address whenever
both are present (classic, editor-pinned, pre-lane); only a blob with NO keys
is string-only and resolved through the parser. This is safe because the model
no longer stamps keys onto a string-only tag. The model distinguishes
string-only / legacy-string-beside-keys / pinned, and the editor hint says so.
Test matrix over every persisted shape (keys only, pre-lane keys+string,
pre-lane stamped HR/0+string, string only, marker+keys, marker no keys) under
every family; the pre-lane row also through a model save and a CSV
export->import.
n5: a typed CSV address cell is an authored address - it drops addressString
and the marker (new CsvTypedColumn.ClearsKeys) instead of being discarded.
n6: Validate skips the region/dataType/byteOrder enum checks on a string-only
tag (the driver ignores them). n7: docs/v2/modbus-addressing.md restates the
key-presence rule.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:23:37 -04:00
Joseph Doherty b943f48e01 Merge master (7feffc96) into remed/lane-G1
Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:22:30 -04:00
Joseph Doherty 57ae69a05e fix(client): release a capped HistoryRead cursor, read (not release) follow-up pages, detach deleted subscriptions
R08 1.4: raw and modified history reads share HistoryReadPaging. A read stopped by
the caller's cap releases the server continuation point. The raw path also passed
releaseContinuationPoints = (cp != null), which released page two instead of reading
it, so every raw read was silently truncated to its first server page.

R08 1.5: DefaultSubscriptionAdapter removes its Subscription from the SDK Session
on delete/dispose (and a failed CreateAsync removes it too), so re-targeted alarm
subscriptions no longer accumulate in Session.Subscriptions.

Refs #747
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:21:22 -04:00
Joseph Doherty 1f82ecfc0d Merge commit '7feffc96' into remed/lane-Modbus 2026-10-22 16:19:17 -04:00
Joseph Doherty 44ea6595e5 feat(arrays): advertise AbCip/AbLegacy/TwinCAT array writes in DriverArrayWriteSupport
The Wave-2 lanes gave AbCip, AbLegacy and TwinCAT real whole-array write
encoders (review 07 U10/U9), but the capability table still listed all three
as scalar-only, so every array node materialised read-only and a client write
was BadNotWritable before it reached the driver.

Rows added, each listing only what WriteAsync encodes end to end:
- AbCip: SInt/USInt/Int/UInt/DInt/UDInt/Real/Dt/LInt/ULInt/LReal, declared
  length, not a safetyTag. BOOL/STRING/Structure arrays stay read-only.
- AbLegacy: Int/AnalogInt on N/B/A, Long on L, Float on F (file-width match),
  plain <file><n>:<word> address, <= 256 elements; not /bit, .sub, indirect,
  S:/I:/O:, nor native PLC-5.
- TwinCAT: every codec element type, declared length; not a BOOL bit-in-word
  path.
Elements convert in all three encoders, so the rows never key on an exact
node DataType (ScadaBridge int[] -> Int16 stays writable); they leave out
only node types whose elements never convert (DateTime/String over numeric,
Boolean on AbLegacy, STRING under DateTime on TwinCAT: culture-dependent).

Tests: driver-level parity for each driver (every element type x node type
plus shape cases, against the real write path), the AbLegacy parity tests
un-skipped, Core.Abstractions row tests, a composer test (three drivers
writable, refused shapes read-only), and the existing AbCip negative cases
moved to a BOOL array. Docs: Uns.md table, per-driver docs, data-plane rules.

Refs #737 #738 #739

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:17:48 -04:00
Joseph Doherty d77a538e17 test(mqtt): pass the test cancellation token to the wedged-init lifecycle test
Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:17:09 -04:00
Joseph Doherty 7feffc963a Merge lane S7: S7 driver remediation (#742)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:11:24 -04:00
Joseph Doherty a7506e7ed4 test(sql): live SQL Server gate for the row cap (R06:S-1)
An events table with 40 rows for one where-pair selector: the capped SELECT TOP (2)
runs as real T-SQL, the ambiguous tag publishes BadConfigurationError, a single-row
station reads Good through the same statement, and the driver stays Healthy. Live
run 2026-10-22: SQL Server 22/22, Postgres 7/7 and Oracle 6/6 (LIMIT n / FETCH FIRST n
ROWS ONLY after WHERE) all green on 10.100.0.35.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:10:12 -04:00
Joseph Doherty f51055b509 test(opcuaclient,mtconnect): live-fixture cases for the lane G1 fixes
Against opc-plc (opc.tcp://10.100.0.35:50000): a shutdown with three live
subscriptions is bounded and the same instance re-initialises and reads (O-9 +
lifecycle generation through the real SDK); a server-refused write
(BadTypeMismatch) passes through and keeps the driver Healthy with its stamp
advanced (health contract); the browse side-panel reports a live variable's type,
shape and access (O-5).
Against the MTConnect fixture Agent: the headers-only ping reaches it and fails on
a dead port (T-7); a delivering /sample stream serves reads and reports its lag and
effective count on the diagnostics bag (T-4/T-5).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:07:01 -04:00
Joseph Doherty 3531834cce fix(modbus): review r1 - resolved address keys win only when the editor pinned them
N1: the m2 precedence trusted region/address whenever the keys were present,
but the AdminUI model stamps defaulted HoldingRegisters/0 on every save, so a
string-only tag (CSV import, or any editor toggle) silently became HR[0] -
reads AND writes. The resolved keys are now authoritative only when the editor
marked them ("addressResolved": true, written by Resolve); an unmarked
addressString is always resolved through the parser with the driver's family.
The model no longer stamps address fields for a string-only tag, CSV export
leaves its address cells blank, and the editor hint distinguishes string-only
from pinned tags. Tests: CSV import and editor save of a string-only tag, a
stamped-default blob, and the pinned positive control across families.
n1: docs/v2/modbus-addressing.md updated (FC23 opt-in, maxReadGap semantics,
coalescing scope, addressString/family rules, coil coercion, refusals).
n2: form round-trip tests for UseFC23ForMultiRegisterWrites /
DisableReadCoalescing. n4: the editor keeps unresolved address text when the
parent echoes back its own emit. n3: noted, no change.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:06:02 -04:00
Joseph Doherty 2a28c06fc7 fix(sql): a skipped tag reports the deploy gate's reason, pinned for the deferred Query model (R06:S-4)
A model:Query tag reached the runtime skip ledger as a generic 'did not map'. The
ledger reason (and the skip warning) now comes from SqlTagDefinitionFactory.Inspect -
the deploy gate's own explanation over the same parser - so /hosts names the
deferral. A test pins that the deploy gate names Query through the parser. Making it
a hard deploy error in Warn mode is a ControlPlane change (follow-up); MySql/Odbc and
allowWrites remain feature issues.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:04:59 -04:00
Joseph Doherty ac01c88c0e fix(mtconnect): an Agent refusal of /current is a reply, not a comms fault
Health contract: an MTConnectError document (NO_DEVICE, INVALID_REQUEST, ...) is the
Agent answering. The parsers now mark such an InvalidDataException
(MTConnectXml.IsAgentRefusal - the type is sealed, so a Data marker, every other
handler unchanged); ReadAsync reports the batch Bad but does not degrade the read
path, advances LastSuccessfulRead, clears a prior read degradation and warns once
per distinct refusal. An unparseable answer stays a comms fault.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:03:38 -04:00
Joseph Doherty a7d21d5a65 feat(sql): IBatchSubscribable and lifecycle-generation-stamped health (R02:P1, R06:S-9)
R02:P1: SqlDriver runs PollGroupEngine, so it now implements IBatchSubscribable and
raises one OnDataChangeBatch per poll burst (DriverInstanceActor forwards one message
per cycle instead of one per tag).
S-9: InitializeAsync/ShutdownAsync take a lifecycle generation; their health writes
apply only while current, and a poll that started under an older generation (one that
outlived the engine's teardown wait) no longer writes its verdict onto the new
configuration. Poll-driven updates can no longer end Initializing - only the
initialize's own final write does. Lifecycle writes now preserve LastSuccessfulRead.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:03:23 -04:00
Joseph Doherty 0ad460c5c4 Merge lane B: OPC UA address space remediation (#732)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:01:44 -04:00
Joseph Doherty d67634cc4f fix(mtconnect): redact the Agent URI everywhere it is shown (R06:T-8)
The driver logged and surfaced the raw AgentUri (which may carry user:pass@) as
HostName (the /hosts key), in ~20 log lines and in exception messages that reach
DriverHealth.LastError, and the client's request-URI messages (timeouts, stream
end, not-a-stream, framing) embedded it too - while the Test-Connect probe
deliberately redacted it. New MTConnectUriRedaction (user-info, and for the
configured string its query, removed; a clean URI is unchanged so host keys do
not move); the driver computes its display form once per options document.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:00:40 -04:00
Joseph Doherty ab29b76be0 perf(sql): rate-limit per-poll warnings and cache poll plans per reference list (R06:S-6, S-7)
S-6: a frozen group's BadTimeout warning and a failing group's query-error warning
(with SQL text) logged every poll per group indefinitely; both are now rate-limited
per plan (once a minute), and an uncoercible cell warns once per tag instead of once
per poll. Every poll still publishes the Bad status.
S-7: plans and the slot map were rebuilt every poll although a poll loop passes the
same reference list each cycle. They are cached per list instance (weakly keyed) and
rebuilt only when a reference resolves to a different definition instance, which a
catalog-gate swap produces.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 16:00:25 -04:00
Joseph Doherty 081fa693af fix(s7): access-denied is an answer; review round-1 minors (R07:H4 follow-through)
I-1: a PUT/GET-disabled CPU (every read denied) stayed Faulted with
LastSuccessfulRead frozen, so the comms-loss watchdog reconnected and
re-subscribed it every 90 s forever. Both access-denied arms now also record
an answer: Faulted is kept (worst state wins), the stamp advances, and the
sticky write fault still clears only on a real success. Pinned end to end in
Runtime.Tests: the real S7Driver under the real DriverInstanceActor watchdog
is not reconnected across 2x the threshold; the control (every read timing
out) is.

Minors:
- m-1: injectable pass clock (UtcNow); "LastSuccessfulRead advanced" is now
  asserted strictly, including the write-refusal case.
- m-2: the editor uses S7TagShapeRules.IsByteAnchored instead of its own list.
- m-3: a release that races an init publish also cancels the probe loop the
  publish started after the release took its snapshot.
- m-4: a retained poll failing during a lifecycle operation writes no health.
- m-5: a multi-var chunk that fell back memoises its members refused alone.
- m-7: access-denied needs S7.Net's exact wording and no transport fault in
  the chain.
- m-8: sync teardown clears the write-side Degraded owner; a sync Dispose
  racing a running shutdown is tested.

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:57:53 -04:00
Joseph Doherty d4b9fa1c29 perf(mtconnect): headers-only connectivity probe (R06:T-7)
The connectivity probe parsed the full /probe device model every 5 s and discarded
it. It now calls a new IMTConnectAgentClient.PingAsync: one GET /probe read with
ResponseHeadersRead and a status check; the body is dropped unread. Same failure
surface as ProbeAsync (the default implementation delegates to it). The probe-tick
tests now count pings and pin that no tick issues a full /probe; the shutdown
test's absence assertion gained a positive control.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:57:43 -04:00
Joseph Doherty 0a5aa9fc43 fix(sql): row-cap key-value and where-pair plans with a TOP/LIMIT N+1 dialect seam (R06:S-1)
Key-value and where-pair plans had no row limit, so a selector pointed at a history
or events table materialised every matching row on every poll. ISqlDialect gains
RowLimitPrefix/RowLimitSuffix(n) (SQL Server TOP (n), Postgres LIMIT n, Oracle FETCH
FIRST n ROWS ONLY); plans carry a RowCap (distinct keys, or 1) and ask for RowCap+1
rows; the reader also stops client-side at RowCap+1. Getting the extra row proves the
selector ambiguous: the group publishes BadConfigurationError with a rate-limited
warning. The reader reports those answered refusals so the driver counts them as
answer evidence, and any refusal-only poll now restores Healthy after a comms fault.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:57:13 -04:00
Joseph Doherty f05e3bf6f4 perf(mtconnect): one Agent write request per device (R06:T-6)
Writes were one POST per slot although the Agent's PUT surface accepts several
id=value pairs. The slots surviving the local refusals now go out as one
WriteObservationsAsync(device, pairs) per device (new IMTConnectAgentClient method;
the default loops the single-pair call). A batch closes at another device or a
repeated DataItem so same-item writes keep request order; a refused multi-slot
request is retried per slot so one refused DataItem costs only its own slot.
Health contract: any Agent answer (accepted or refused) advances
LastSuccessfulRead with the state following the read / stream flags.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:55:37 -04:00
Joseph Doherty fd6ecf8293 fix(modbus): review r0 - FC23 opt-in, explicit maxReadGap 0 = off, legacy coil types load, one health writer
Coordinator rulings and Important findings from review r0:
1. FC23 is opt-in (new UseFC23ForMultiRegisterWrites, default false); a config
   without it - including the persisted "disableFC23": false every form-saved
   config carries - keeps the FC16 wire. DisableFC23 stays the hard kill-switch.
2. An explicit maxReadGap 0 means coalescing off again (its pre-lane meaning);
   the gap-8 default applies only to an absent key.
3. Coils/DiscreteInputs tags persisted with a register dataType (the old
   editor saved Int16) are coerced to Bool, never refused, with no deploy
   warning (a warning can be promoted to a blocking error). The editor loads
   them as Bool. Register region + Bool is still refused.
4. One comms-fault slot for reads and writes: a good read clears a write's
   transport fault and vice versa.
Minors: Dispose writes its own Unknown health (m1); persisted region/address
are authoritative over addressString so a Family change cannot remap a live
tag (m2); a malformed-but-delivered reply no longer fails the unit fast (m3);
a fresh socket published after a concurrent dispose is torn down (m4); the
editor resets address text per tag (m5); master merged (m6); the FC23 latch
log says re-init restores it (m7). Back-compat tests use the old persisted
JSON shapes and fail without the fixes.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:55:32 -04:00
Joseph Doherty cb9a48ee96 Merge lane AbLegacy: AbLegacy driver remediation (#738)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:50:46 -04:00
Joseph Doherty 21b4f00f46 feat(mtconnect): /sample lag measurement + adaptive count (R06:T-5)
Subscription throughput was hard-capped at SampleCount / SampleIntervalMs with no
lag detection: a machine producing more fell behind the ring buffer forever,
re-baselining every buffer-length with every value backlog-stale.

- The parser carries the header's optional lastSequence; each chunk measures
  lag = lastSequence + 1 - nextSequence and whether it was saturated (full count).
- Three consecutive saturated chunks with a positive (or unknown) lag reopen the
  stream at the cursor reached with count doubled, up to 16x SampleCount (max
  65536), through a new IMTConnectAgentClient.SampleAsync(from, count, ct) overload
  (default implementation ignores count). A raised stream that delivers nothing
  steps back to the last good count; at the ceiling one Warning per pump.
- MTConnectDriver implements IDriverDiagnostics: sampleCount, sampleLag,
  sampleSaturatedChunks, sampleCountRaises on the /hosts bag. The cap is
  documented on SampleCount.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:50:42 -04:00
Joseph Doherty 7f5d959fb9 feat(sql): surface IS_NULLABLE on browsed column attributes (R06:S-5)
Every dialect projects IS_NULLABLE and nothing read it. The browse session now reads
it (tolerantly: a dialect that does not project it reads as unknown) and the picker's
descriptor line says 'nullable' / 'NOT NULL', so an operator knows a cell can come
back NULL (Uncertain, or Bad under nullIsBad) before committing the tag. The runtime
catalog loader keeps reading names only - nothing at runtime consumes nullability.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:49:24 -04:00
Joseph Doherty c12269c221 Merge commit '5135a036' into remed/lane-Modbus 2026-10-22 15:47:50 -04:00
Joseph Doherty 91b7e9541b fix(sql): withhold the JSON parser's text on a malformed driver config (R06:S-8)
The factory quoted System.Text.Json's message (and attached the JsonException) into
the failure that reaches InitializeFailed -> health LastError -> /hosts; a pasted
connection-string literal is exactly the malformed text that lands there. The
message now names the instance and the line/byte position only, as the probe and
browser already do.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:47:18 -04:00
Joseph Doherty 04ad774235 Merge commit '5135a036' into remed/lane-S7 2026-10-22 15:46:38 -04:00
Joseph Doherty ff4e210a27 fix(opcua): replay kept scripted conditions after a rebuild; fit window values onto re-shaped nodes (Lane B fix round 3)
N3 (critical): the node manager carries only NATIVE conditions' state over a
full rebuild; scripted conditions come back fresh and rely on the engine's
re-assert, which since the off-thread plan is handled in the window and lands
on the discarded instance. On a rebuild the condition replay scope is now
AddedConditions plus every enabled scripted condition in the new composition
(native kept conditions stay carried-over, not replayed). The test sink again
models the real rebuild: it clears non-native condition state only.

N4: a surgical in-place shape edit (DataType / IsArray / ArrayLength) resets the
raw node and its UNS mirror to BadWaitingForInitialData without ensuring them,
so the window's value was not replayed and the two NodeIds disagreed. The
applier records re-shaped nodes (MaterialisationRecording.Reshaped); their
deferred value replays only once fitted to the new shape (VariableShape.TryFit:
as is, or converted invariantly; never an old-typed value - WriteValues assigns
the value verbatim with no type check), otherwise it is dropped at Debug and the
node keeps waiting. Status-only entries are not replayed onto a re-shaped node.

Minors: m1/m2 documented on _replayScope (degraded-path only); m3 corrected the
false carry-over premise in the scope doc (the ScriptedAlarmHostActor ordering
remark is outside this lane - F8); m4 the replay Debug log reports alarm-state
and alarm-quality counts again.

Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:46:29 -04:00
Joseph Doherty 16f401b9e7 fix(sql): dispose the poll reader's concurrency gate once no group holds it (R06:S-3)
SqlPollReader's SemaphoreSlim was never disposed and a new reader is built per
rebind. The reader is now IDisposable with a reference count (owner + one per
in-flight group): the gate is disposed only when the last group - including a
timed-out one still wedged in the provider - releases it, so nothing ever releases
into a disposed semaphore. ApplyBinding disposes the replaced reader; the final
DisposeAsync disposes the live one (Shutdown does not, since a re-init without a
rebind keeps polling through it).

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:46:21 -04:00
Joseph Doherty 29c829b540 test(mtconnect): a stale-session pump may finish before the restart retires it
Since the T-1 generation guard a pump stops on its own once its session generation
is gone, so the STAB-20 test's 'the retired pump is retained' assertion raced it
(intermittent failure). The test now accepts either outcome - retained for shutdown
to join, or already finished - which is what the retention exists to guarantee.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:45:54 -04:00
Joseph Doherty e3cb4f0c7b fix(ablegacy): opaque replies are neither answers nor faults; round-2 minors
Review r1 N1 (Important): the I1 fix made IsDeviceRefusal default-open and
mapped libplctag's ERR_REMOTE_ERR to an answer. libplctag folds EVERY non-zero
PCCC STS and CIP status into that code, including a routing hop's "unconnected
send timed out" / "connection lost" / "remote node missing", and native
BadReply covers truncated / mismatched / unmapped replies - so a powered-off
PLC behind a bridge read as reachable: Healthy, LastSuccessfulRead advancing,
the comms-loss watchdog unable to fire.

Three classes now, each pinned both ways:
- no reply (Timeout/BadConnection/Abort/Pending/unknown): comms fault;
- answered - an ALLOW-LIST again (NotFound/NotAllowed/OutOfBounds/TooLarge/
  TooSmall/TypeMismatch/DeviceStateConflict): proves reachability;
- OPAQUE (BadReply/Partial; ERR_REMOTE_ERR/ERR_BAD_STATUS now translate to the
  existing BadReply, OPC UA BadCommunicationError as before r0): per-tag Bad,
  handle kept, coalescing fallback, never fail-fast, never an answer. Option
  (a): a device whose pass/call produced only opaque outcomes is Degraded with
  LastSuccessfulRead not advanced, so the watchdog reaches a dead routed PLC.
  The probe and index reads classify the same way.

Minors:
- N2: a mismatched array (type vs data file) READS BadTypeMismatch instead of
  decoding the file at the wrong width as Good.
- N3: a swept indirect resolution is never reconnected under a forgotten gate;
  that one call answers neutrally.
- N4: an index read answered Unsupported is neutral, not a transport fault.
- N5: ArrayFileMismatch checks indirect addresses (the file letter is static).
- Test nit: the evicted-while-waiting test signals gate arrival via a seam
  instead of sleeping.

Live: 51/51 on ab_server (slc500/micrologix/plc5 x both transports), incl. an
opaque-only libplctag device staying Degraded with LastSuccessfulRead frozen.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:45:04 -04:00
Joseph Doherty c529948745 perf(mqtt): publish a write batch concurrently, serialised per topic; writes feed health (R06:M-8)
WriteAsync published one awaited QoS 1 round-trip per request, so 50 writes to a
stalled broker cost 50 publish deadlines. Different topics now publish concurrently
(at most 8 in flight); same-topic writes stay in request order. A caller cancel still
fills every slot. A successful publish advances LastSuccessfulRead. MiniBroker now
frames packets properly and records/PUBACKs PUBLISHes.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:42:50 -04:00
Joseph Doherty 994c0b3b19 refactor(mtconnect): delete dead parsed fields (R06:T-3)
MTConnectObservation.SampleRate was parsed per observation and never read;
MTConnectTagDefinition.MtType/MtSubType/Units were carried (factory blob DTO,
tags[] DTO, probe inference) and only MtCategory was ever read. Deleted. Stored
configs and TagConfig blobs still carrying mtType/mtSubType/units keep loading
(System.Text.Json skips unmapped members) - pinned by a new test. The AdminUI tag
editor reads those keys straight off the blob and is unaffected.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:42:32 -04:00
Joseph Doherty b548cc46e6 perf(mtconnect): read through the live pump index; instance-id check on the /current fallback (R06:T-4, T-2)
- T-4: every ReadAsync batch fetched and parsed the whole device /current and
  built a fresh index even while the /sample pump held a live one. A read is now
  answered from the pump's index (read only - the pump stays its single writer)
  while the current-generation pump is running, has delivered since it started and
  is not degraded; /current stays the fallback, and its per-batch index is built
  over a tag map frozen once per binding instead of per call. A served read clears
  the read-path degradation a failed /current left.
- T-2: the /current fallback checks the answer's instanceId; a change (an Agent
  restart seen by a browse-only driver) drops the cached /probe model and raises
  rediscovery once per distinct id, only while the read's session is still installed.
- The pump also acts on no verdict (Degrade / latch) once its session generation
  is gone, so an abandoned pump cannot write health over the new session (T-1 follow-through).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:39:58 -04:00
Joseph Doherty 6a51025012 perf(mqtt): wildcard topic trie and a shared strict UTF-8 decoder (R06:M-6, M-7)
Wildcard-authored tags were matched by one MqttTopicFilterComparer call per tag per
inbound message on the dispatcher thread. They now live in an allocation-free level
trie whose candidates are confirmed by the same comparer (semantics unchanged; pinned
differentially against the comparer). TryDecodeUtf8 reuses one static throw-on-invalid
UTF8Encoding instead of allocating one per Raw/Scalar delivery.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:38:29 -04:00
Joseph Doherty 6ffd881d42 fix(modbus): a failed read or write drops the WriteOnChangeOnly suppression for the tag
After a comms fault the device value is unknown, so re-asserting the same
setpoint must reach the wire - the write-side twin of the value-only
recovery dedup trap.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:37:41 -04:00
Joseph Doherty 77bfadbdda fix(mtconnect): generation guard on the pump's cursor publish, chunks and re-baseline (R06:T-1)
A pump the lifecycle retired or abandoned that finished after the restart
installed the new session published its stale cursor onto it (PublishAgentCursor
only checked for a null or identical session), so the next pump read its first
chunk as an Agent restart: index cleared, a Bad flicker on every subscribed tag,
a needless re-baseline. The pump now carries the session generation it was
started against; the cursor publish is checked under _subscriptionLock (where the
session install and generation bump are atomic) and a superseded pump also stops
folding late chunks / re-baselines into the new session's index.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:36:24 -04:00
Joseph Doherty 8a5b4434a4 Merge local master 5135a036 into remed/lane-AbLegacy
Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:36:12 -04:00
Joseph Doherty 4a3d7a9361 feat(modbus): wire Family / MelsecSubFamily / DisableFC23, and the form's timing keys
U1: a tag's addressString (Modicon / mnemonic / DL205 V-memory / MELSEC
D-X-Y grammar) now resolves through ModbusAddressParser with the DRIVER's
Family and MelsecSubFamily at every initialize and is authoritative; the
family-unaware gates (deploy Inspect, AdminUI parse consult) accept what
some family resolves. The tag editor gets an address-text field that
resolves with the owning driver's family and persists both the string and
the resolved region/offset/type fields; a hand edit drops the string.
DisableFC23 now gates FC23: by default a 2..121-register holding write is
one FC23 write + read-back (a unit answering exception 01, or failing FC23 at
the transport, moves to FC16 for the rest of the generation); RTU framing
deframes the FC23 reply. Live-checked: the pymodbus fixture implements FC23.
Also: the AdminUI driver form serialises ModbusDriverOptions (TimeSpan
strings), which the factory DTO never read - every timing field on the form
was dead. The DTO now accepts both shapes (*Ms keys win).

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:35:58 -04:00
Joseph Doherty c14a3388f1 fix(mqtt): Sparkplug array and unsupported-type metrics are not bindable from the browser (R06:M-5)
The ingestor refuses array metrics and unmapped types, yet DescribeMetric still stated
their binding tuple, so a browse-commit produced a tag stuck at
BadWaitingForInitialData forever. Those leaves now state no address (the commit
refuses them) and say why in the descriptor line.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:35:55 -04:00
Joseph Doherty d67396cc6d feat(mqtt): wire sparkplug.birthObservationWindowSeconds into the browse surface (R06:M-4)
The authored window was persisted and consumed by nothing. It now (clamped 1-120 s)
extends the Sparkplug OpenBudget, bounds a passive UntilStable-style settle on open
(return after a 1 s quiet spell in NBIRTH/DBIRTH traffic), and bounds how long an
operator rebirth request waits for the births it asked for before returning, so the
re-rendered tree carries them. A caller deadline during the wait still reports the
publish.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:33:47 -04:00
Joseph Doherty a0b635f673 fix(s7): probe gate contention is no verdict; Test Connect honours timeoutMs (R07:H7)
- The background probe reported the host Stopped whenever a read pass held
  _gate past the probe deadline: contention is the connection being used,
  not lost. A contended tick now gives no verdict (counted for tests); a
  real probe failure still reports Stopped. A tick that races a teardown no
  longer reopens a connection on an uninitialised driver.
- S7DriverProbe deserialised S7DriverOptions directly, whose Timeout is a
  TimeSpan no config names, so "timeoutMs" was ignored and every handshake
  waited the whole Test Connect budget. The handshake and test read are now
  bounded by timeoutMs, capped by the caller's timeout.

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:33:38 -04:00
Joseph Doherty fcdc44f9f7 feat(opcuaclient-browser): real AttributesAsync + certificate-auth browse (R06:O-5)
- AttributesAsync was a stub returning nothing; it now reads NodeClass /
  DisplayName / DataType / ValueRank / UserAccessLevel in one request and maps them
  through UpstreamAttributeMapping, the same mapping discovery registers with (moved
  into the shared .Session leaf; the driver forwards to it). Objects have no value
  attributes.
- Certificate auth was refused "in v1". The browse now builds its identity through
  OpcUaUserIdentityFactory (shared .Session leaf, the driver forwards to it),
  including the secret: resolution of Password / UserCertificatePassword through the
  host's ISecretResolver (optional ctor param, DI-injected). Fail-closed: a secret:
  ref with no store throws instead of being sent as the literal (the browse used to
  send a secret: Password verbatim). The identity is built before any network I/O.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:33:08 -04:00
Joseph Doherty 1f7ab80b81 fix(s7): a refused span falls back per member and is not regrouped (R07:P2)
A span's block is the union of its members' ranges, so one member that runs
past the DB's real end (or over-declares a STRING) made the CPU refuse the
whole block, and every neighbour within it was Bad on every poll. The
planner's stated invariant (a grouped read cannot fault where per-tag reads
would succeed) was false; the multi-var path already fell back, spans did not.

- A refused multi-member span hands its members back; they are read one at a
  time for that pass, bypassing the multi-var pass.
- _noGroup remembers the members refused on their own (or every member when
  none was) and keeps them out of later plans, so the rest group again. It is
  cleared by every initialize publish. Non-refusal span failures are
  unchanged (every member stamped).
- The once-per-tag refusal Warning memo (R07:H4) is now also cleared by the
  publish, and shutdown resets the write-side Degraded owner, as documented.
- Live gate: MW10 + MD254 (2 bytes past the 256-byte M area) merged into one
  refused block; MW10 keeps reading 4097 on every pass.

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:30:12 -04:00
Joseph Doherty 9c8e70c5b7 Merge commit '5135a036' into remed/lane-B 2026-10-22 15:29:36 -04:00
Joseph Doherty 57ffbf79e5 fix(mqtt): prune LastValueCache to the authored set on Register (R06:M-2)
Both ingest paths now drop cache entries whose RawPath is no longer authored when a
deploy re-registers, so a removed tag's value and every non-authored subscribe
reference's BadNodeIdUnknown marker no longer accumulate for the driver's lifetime.
Still-authored tags keep their observed values.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:28:33 -04:00
Joseph Doherty d98f4a5bf1 fix(opcuaclient): wire ProbeTimeoutSeconds into the Test Connect probe (R06:O-4)
The knob was authored by the AdminUI and parsed, but consumed by nothing. It now
caps the probe deadline (shorter of the caller's budget and the config; never
extends it; 1 s floor), the same rule as the Galaxy probe.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:28:20 -04:00
Joseph Doherty 1adc9d65e1 feat(modbus): String[] and BitInRegister[] writes
U3: both array shapes read fine but every write failed BadInternalError.
String[] now encodes each element at its register width (StringLength chars,
2 per register); BitInRegister[] reads the span once, flips the addressed bit
of each register and writes the span back with FC16 (per-register RMW locks
taken in ascending order). A length mismatch is BadOutOfRange and sends
nothing. DriverArrayWriteSupport (outside this lane) still advertises these
nodes read-only - recorded as follow-up F1 in the parity test.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:26:59 -04:00
Joseph Doherty 43f45bb978 fix(mqtt): dispose an unpublished connection and stamp lifecycle generations (R06:M-3, M-10)
A refused initial connect (or a failed Sparkplug establish) now disposes the
MqttConnection it built. Every lifecycle operation takes a generation (initialize
inside the gate; shutdown/dispose before waiting for it): a wedged initialize that
completes after a timed-out ShutdownAsync's ungated teardown releases its session
instead of publishing it, and a superseded operation's health/host-state writes are
refused. TeardownAsync captures every field before disposing so an ungated teardown
never reaches into a later initialize's state.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:26:55 -04:00
Joseph Doherty cba3b2e960 fix(opcuaclient): writes feed health; a good keep-alive clears a stale Degraded
Health contract: a write response (per-node refusals included) is the server
answering, so it restores Healthy and advances LastSuccessfulRead; a write with no
reply (transport failure / timeout) writes Degraded with the stamp preserved; a
caller-cancelled write touches nothing. A good keep-alive with no outage or dead
subscription latched now clears a Degraded left by a failed read/write instead of
only moving the stamp (a subscription-only driver otherwise stayed Degraded until
its next re-initialise). All writes are generation-stamped.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:26:43 -04:00
Joseph Doherty 1d83cde6bc perf(s7): fail the rest of a pass fast once the handle is dead (R07:P1)
No read or write loop checked _plcDead between items: on a frozen-but-
established peer the first op blew its deadline and marked the handle dead,
and every remaining span / tag / write still went to the wire and could wait
its own deadline under _gate (a 20-span pass ~100 s), with Degraded and the
comms-loss watchdog's clock deferred until the pass ended.

- The span loop, the multi-var chunk loop, the per-tag loop and the write
  loop stamp BadCommunicationError without I/O once _plcDead is set.
- A multi-var chunk that dies on a transport loss (socket / IO / timeout /
  ConnectionError) marks the handle dead instead of falling back to one
  doomed per-tag read per member. The framing class is deliberately excluded
  there: snap7/S7.Net report a multi-var with one refused item as
  ReadData(WrongNumberReceivedBytes) over a healthy stream.
- Live gate via an in-test freezable TCP proxy (counts request frames, not
  seconds). Mutation shows S7.Net 0.20 already self-closes on a cancelled op,
  so the gate pins the end-to-end property; the driver guard itself is proven
  by the unit tests (an op that ignores its token keeps the socket open).

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:26:14 -04:00
Joseph Doherty 5135a0366e Merge Wave-1 integration: cross-lane scripting/runtime follow-ups (#731, #734)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:25:43 -04:00
Joseph Doherty 66d100a9e9 perf(opcuaclient): split the read gate from subscription management; no gate on writes or the discovery browse (R06:O-6)
One SemaphoreSlim serialised reads, writes, subscribes, acks and the whole recursive
discovery browse, so a stalled write (up to the operation timeout) blocked every
read and subscription create, and a discovery browse would block the data plane for
its full duration. Reads/acks keep the read gate; writes and DiscoverAsync are
ungated (the SDK session is safe for concurrent service calls, the host bounds write
concurrency); subscription management takes its own gate. Dispose drains both
(DisposeAsync's drain no longer over-releases on a timed-out wait).

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:24:40 -04:00
Joseph Doherty 2ef3cd0df2 feat(modbus): block-read coalescing on by default; coils/DI/BitInRegister/String fold
U2: MaxReadGap defaults to 8 and 0 now means adjacent-only rather than
off; a new DisableReadCoalescing switch (factory DTO + AdminUI form) turns it
off entirely. The planner also merges Coils/DiscreteInputs (FC01/FC02, gap in
bits), String scalars and BitInRegister tags - sixteen bit tags on one status
word are one FC03 instead of sixteen. Auto-prohibition and its re-probe /
bisection use the region's own function code (a coil range was re-probed
with FC04).

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:24:38 -04:00
Joseph Doherty efa040813a fix(ablegacy): coalesceReads absent means default; the form never pins it
Review r0 M9. The AdminUI form wrote coalesceReads:true on every save, pinning
today's default into every edited driver. AbLegacyDriverOptions.CoalesceReads
is now bool? (null = default ON, never serialised, same encoding as
Transport); the form omits the key when the switch is on and writes only an
explicit false. Persisted coalesceReads:true keeps loading as on.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:24:30 -04:00
Joseph Doherty 01efd39177 fix(mqtt): retry failed SUBSCRIBEs via a pending set; throw on no-verdict subscribe (R06:M-1, M-11)
A failed SUBSCRIBE (refusal or no broker verdict) used to drop its topic from every
retry set while SubscribeAsync returned a live handle, leaving tags dark behind a
Healthy connection until a redeploy. Failed topics now join a pending set retried by
the next SubscribeAsync and by every reconnect (not counted toward the reconnect's
total-failure verdict, so an ACL-denied topic cannot reconnect-loop). A no-verdict
subscribe degrades its refs then throws so the actor's reconcile re-subscribes. A
subscribe with no transport attached is recorded pending, never established (M-11).
MiniBroker now answers SUBSCRIBE.

Refs #744
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:22:17 -04:00
Joseph Doherty 16c2045cab fix(ablegacy): never operate on a handle evicted while waiting for its gate
Review r0 M3. The indirect sweep disposed stale resolutions without holding
their gates, and any reader/writer that fetched a handle before waiting on its
gate could then use an instance a sibling had evicted and disposed - a
spurious comms fault and Degraded. The sweep now evicts under each key's gate,
and read/write re-resolve the handle once the gate is held (reconnecting
single-flight if it was evicted; a retired state never reconnects).

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:22:17 -04:00
Joseph Doherty afafd483dd fix(opcuaclient): an acknowledge with no reply fails with its transport status (R06:O-8)
AcknowledgeAsync swallowed every non-cancellation transport failure (and returned
normally when the session was lost while the ack queued), so the host logged the
Part 9 ack as applied while upstream still showed it unacknowledged. Both now throw
(ServiceResultException with the transport status / BadNotConnected). A reply,
including per-ack refusals, still returns normally and counts as the server answering.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:21:19 -04:00
Joseph Doherty a1f20e78f9 fix(s7): a CPU refusal is per-tag Bad and counts as answered (R07:H4)
A deterministic PLC refusal (object does not exist, address out of range,
type inconsistent...) degraded driver health, and an all-refused driver (a
wrong DB number) sat Degraded with LastSuccessfulRead frozen, so the 90 s
comms-loss watchdog reconnected it forever against a reachable PLC.

- IsRefusal / IsAccessDenied classify over the whole exception chain. Live
  probe on the snap7 fixture: S7.Net's READ paths raise the item error as a
  BARE Exception ("Received error from PLC: ..."), its write paths as
  PlcException(WriteData). The old PlcException-only match missed the read
  shape entirely, so a real read access-denied was BadCommunicationError +
  Degraded instead of the Faulted PUT/GET alert; the Test Connect probe had
  the same blind spot.
- A refusal (read or write) is BadDeviceFailure with a once-per-tag Warning,
  no health write and no reopen; it counts as answered, so it advances
  LastSuccessfulRead and restores Healthy after a comms fault. An undecodable
  buffer (S7ValueRangeException) also counts as answered.
- A write batch the CPU answered without a failure advances
  LastSuccessfulRead and retires a Degraded the write side itself published.
- Tests that used ReadData as their generic "device fault" now use SendData
  so their Degraded leg is still exercised.

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:20:51 -04:00
Joseph Doherty d758dd1e16 fix(ablegacy): encode array writes at the data file's element width
Review r0 I3. The array encoder used the AUTHORED dataType's width while both
runtimes size their buffer from the FILE: an Int array on F8 wrote four shorts
into a 16-byte float buffer and sent it all (garbage neighbours, answered
Good); Long/Float on N overran it.

- The write path resolves the element width from the data file the address
  names (N/B/A/I/O/S 16-bit word, L 32-bit, F float) and encodes at THAT width.
- An authored type that does not match its file is refused BadTypeMismatch
  before any runtime exists (warned once, no health write).
- The same rule (AbLegacyTagDefinitionFactory.ArrayFileMismatch) backs an
  Inspect deploy warning and the AdminUI model's Validate error; FromTagConfig
  still accepts the tag so persisted configs keep loading.
- M6: the skipped F1 parity test gains the shapes the write path refuses
  (/bit, .sub, indirect, type/file mismatch, PLC-5 native).

Tests: one case per file type, a mismatch per pairing with nothing created,
and a positive control.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:20:10 -04:00
Joseph Doherty 6c3a754be9 fix(opcuaclient): lifecycle generation, close the local session on init throw, one teardown budget (R06:O-2, O-9)
- Every InitializeAsync/ShutdownAsync bumps a lifecycle generation. The session,
  namespace map and endpoint are installed under _probeLock only while the init's
  generation is current, so an init the host abandoned that finishes after the next
  shutdown discards its session instead of resurrecting it. Every health write
  (lifecycle, keep-alive, publish-error, read path) is generation-stamped, so a late
  writer can no longer overwrite newer state.
- O-2: the init catch closes and disposes the LOCAL session it opened (it closed the
  still-null Session property, orphaning an activated session + keep-alive timer).
- O-9: ShutdownAsync spends ONE budget (4 s, inside the host's 5 s PostStop budget) on
  wire calls: every subscription delete concurrently, then the session close with
  what remains; waits are abandoned when the budget fires even if the SDK ignores the
  token. It used to be 5 s per subscription, serially.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:19:48 -04:00
Joseph Doherty f827b3a134 fix(modbus): enforce region/type compatibility at the tag factory
P6: Coils/DiscreteInputs carry only Bool and register regions never Bool;
the factory (and so the deploy-gate Inspect and the AdminUI parse consult)
refuses a mismatch, and an absent dataType defaults per region (Bool for a
bit region). A HoldingRegisters+Bool tag used to throw out of the coalescing
planner and stall every tag of the subscription; the planner also skips any
incompatible pair defensively. The tag editor keeps the type compatible when
the region changes; the CLI refuses the pair up front. The array-write parity
test records the Coils divergence from DriverArrayWriteSupport (owned outside
this lane) as follow-up F1.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:19:28 -04:00
Joseph Doherty 17ae2eea0b fix(ablegacy): any answer restores Healthy; writes feed health and the backoff
Review r0 I2 + M1/M2/M4/M7/M10.
- I2: one health writer (ApplyHealth) for reads AND writes, worst wins per
  call: a comms fault => Degraded; otherwise any answer (Good value, device
  refusal, Good or refused write) => Healthy; LastSuccessfulRead advances on
  any answer. A refusal-only read used to leave a recovered driver Degraded,
  and writes never touched health at all.
- M10: a no-reply write STATUS now degrades like the exception path did; a
  Good/refused write records an answer on the device's backoff, a no-reply
  one a fault (bit RMW included — it now reports its raw status).
- M2: a write inside the device's backoff window fails fast with
  BadCommunicationError even when its handle is cached.
- M1: ShutdownAsync/Dispose write their final Unknown only while the epoch
  they retired is still current (a newer initialise owns health otherwise).
- M4: an indirect bit write releases the previous index value's parent-word
  handle (the per-word RMW gate is deliberately kept: removing a gate another
  writer may hold would let two RMWs of one word race).
- M7: backoff decisions read an injectable Clock; the isolation tests drop
  their stopwatch asserts and cross the window on a manual clock.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:16:49 -04:00
Joseph Doherty b455685331 test(adminui): classify /fleet/diagnostics in the page authorization guard
PageAuthorizationGuardTests.EveryRoutablePage_isClassifiedAndMatchesExpectedPolicy
was red on master: the FleetDiagnostics page (lane A, 953bd275) was
routable but unclassified. It declares AuthenticatedRead, the same
policy as /fleet, and is classified as such. Count comment 16 -> 17.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:16:43 -04:00
Joseph Doherty d7c0135d0b docs(scripting): ctx.CancellationToken also fires on caller cancellation
Review m-6. The XML doc said the token is cancelled on timeout only.
ScriptRunGuard links it to the caller's token
(CreateLinkedTokenSource(ct)), so it is also cancelled on shutdown or
when the evaluating actor stops mid-run. The doc now says so.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:15:49 -04:00
Joseph Doherty f5a392c684 docs(scripting): note that dot-completion walks the base chain for every receiver
Review m-5. The ctx.CancellationToken completion fix (d2cc0715) made
TryGetDotMembers include inherited instance members for EVERY receiver,
not only ctx. The visible side effect beyond the item's scope: struct
and enum receivers now also complete the System.ValueType / System.Enum
instance members they inherit (CompareTo, HasFlag, ...). These are
valid, compilable members, so the change is kept. This commit records
it here and in the code comment.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:15:41 -04:00
Joseph Doherty 54d7a88862 docs: name the batched value path in ReadWriteOperations
Review m-4. The status-code paragraph still traced the value path
through the deleted per-value AttributeValuePublished. It now reads
AttributeValueBatchPublished -> AttributeValueBatchUpdate ->
IOpcUaAddressSpaceSink.WriteValues, which is what OpcUaPublishActor
calls.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:15:31 -04:00
Joseph Doherty 620ecc41dd fix(modbus): refusals are per-tag Bad, batch health, generation-stamped lifecycle, write status mapping
H4: exception 01/02/03 (and undecodable data) are device refusals - per-tag
Bad with a once-per-tag Warning, no Degraded, and they advance
LastSuccessfulRead (the device answered), so an all-refused driver no longer
trips the 90 s watchdog. 04/05/06 still degrade (answered); 0A/0B degrade
without advancing. Health is one worst-wins write per batch instead of
last-writer-wins per tag. A comms failure on one unit fails the rest of
that unit's batch fast; an unreachable endpoint fails the whole batch fast.
H7: write transport failures map to BadCommunicationError, value overflow and
array length mismatch to BadOutOfRange, conversion failures to
BadTypeMismatch; writes report write-side health (a good write or a refusal
clears a write-side Degraded; a good read does not).
Lifecycle: every health write and the transport/table publish carry a
generation; a superseded init releases its transport and publishes nothing,
a read straddling a re-init cannot degrade the new transport, and the
shutdown's final write cannot clobber a newer init. The tag table is
build-and-swapped (re-init no longer merges into the old table). Teardown
joins the probe/re-probe loops (bounded); a probe answer advances
LastSuccessfulRead and an exception PDU from the device counts as Running.
Transports latch dispose atomically and never reconnect after it.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:15:20 -04:00
Joseph Doherty 5dabafecdc fix(calc): log a clamped per-driver run-guard value; document the Scripting ranges
Review m-2. A Calculation driver's own maxAllocBytesPerRun /
quarantineAfterConsecutiveTimeouts used to be clamped silently by
ScriptRunGuard. ResolveScripting now sanitizes the layered options
itself and returns the adjustments, and the factory logs one Warning
per clamped value, naming the driver. docs/ScriptEditor.md "Runtime
safety" gains the key table: defaults, the 1 MB allocation floor, the
1000 caps, NaN and disable semantics, and the per-driver override.
docs/Configuration.md has no Scripting section, so these keys have no
other home.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:15:11 -04:00
Joseph Doherty 4d8de20050 test(scripting): pin the Scripting section bind behind one helper
Review m-1. The bind moves from Program.cs into
Host.Configuration.ScriptingOptionsBinding.Bind, which binds the
section and clamps it. Program.cs calls it, and
ScriptingOptionsBindingTests drives it with in-memory configuration.
Every documented key takes a non-default value, so a misspelled
property or key goes red instead of silently falling back to the
defaults. The tests also cover a clamped bind (1e300 / 1024 / 0, three
adjustments), an absent section, and the sibling LogFilePath /
LogTopicMinLevel keys leaving the bind untouched.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:14:02 -04:00
Joseph Doherty 89204ac0fa fix(alarms): re-apply scripted alarms after a supervision restart of their host
Review I-2. Under factory Props a supervision restart of
ScriptedAlarmHostActor builds an empty engine, and nothing re-sent
ApplyScriptedAlarms. Until the next deploy the host had no mux interest
(inputs stopped), and its ownership filter silently dropped every
operator ack and shelve.

The host now tells its parent ScriptedAlarmHostRestarted from
PostRestart. DriverHostActor keeps the last plans it applied and
re-sends ApplyScriptedAlarms when the signal comes from the current
host, in Steady, Applying and Stale. This mirrors how a driver child's
ReprovisionRequested is answered. The OnAlarmsLoaded comment, which
assumed a re-apply that never happened, now says where it comes from.

DriverHostActorScriptedAlarmRestartTests deploys one scripted alarm
through a real host, crashes the alarm host, does NOT re-apply by hand,
and asserts that interest is re-registered, a dependency push activates
the alarm, and an operator ack reaches the engine. It is red without
the signal (times out waiting for the re-registration).

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:13:02 -04:00
Joseph Doherty 8a80dcc787 fix(ablegacy): a status the PLC returned is an answer, never a comms fault
Review r0 I1. On the default libplctag transport a PLC refusal (non-zero PCCC
STS, e.g. an element past the end of the file; or a CIP error) arrives as
ERR_REMOTE_ERR / ERR_BAD_STATUS, which fell into the BadConnection default: a
no-reply comms fault. A coalesced block with one such member then failed as a
group every poll, and two refused tags in a row opened the device's fail-fast
window against a device that was answering.

- LibplctagLegacyStatusTranslation maps ERR_REMOTE_ERR / ERR_BAD_STATUS to the
  existing AbStatus.DeviceStateConflict (BadDeviceFailure): "reachable, refused
  the operation". No new AbStatus value (Driver.Ab.Transport is Lane K's).
- IsNoReply (Timeout / BadConnection / Abort / Pending / unknown codes) is now
  the ONLY comms-fault class; IsDeviceRefusal is every other non-Ok status
  except Unsupported (BadReply / Partial included: a reply arrived).
- A coalesced block refused with ANY returned status falls back to per-member
  reads in the same pass; the shape is remembered only after a repeat, and the
  refused block's handle is always evicted (M5).
- A runtime status Unsupported is neutral (keeps the handle); the probe no
  longer counts it as a fault.
- Refusal warnings log the status actually answered (M8).

Live: the new past-end-of-file ab_server case fails on libplctag with the old
translation and passes with it, on slc500/micrologix/plc5 x both transports.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:12:21 -04:00
Joseph Doherty 4d0d5d229d fix(opcuaclient): remove a half-built data subscription when its create throws (R06:O-1)
SubscribeAsync left the SDK Subscription on the session (server-side when
CreateAsync succeeded, item handlers attached, tracked by nothing) when
CreateAsync/CreateItemsAsync threw, so the host's per-tick reconcile retry
accumulated one live subscription per retry. Mirror the alarm path: detach
the item handlers, bounded RemoveSubscriptionAsync, rethrow.

Refs #743

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:12:03 -04:00
Joseph Doherty ecf1e86464 fix(s7): lifecycle gate + generation-stamped health and publishes (R07:H5)
Overlapping InitializeAsync calls (the actor's fire-and-forget init plus a
retry tick) could leak an opened PLC connection and orphan a probe loop: no
gate guarded the lifecycle, only _gate for I/O.

- _lifecycleGate (the Modbus pattern) serialises Initialize / Reinitialize /
  Shutdown / Dispose; teardown waits a bounded time, then tears down anyway.
- Every initialize and teardown starts a lifecycle generation. Health writes
  (init, read/write passes, poll errors, shutdown) and the init publish carry
  it and are dropped once it has moved on, so an abandoned init never
  publishes over a later shutdown and a shutdown's final write never clobbers
  a newer init. A superseded init disposes what it opened and throws.
- The init publish (options, tag tables, Plc, probe loop) happens under _gate;
  the tag tables are built fresh and swapped by one reference write instead of
  being cleared in place under live poll passes.
- Read/write passes re-check initialization under _gate, so a pass that waited
  out a teardown cannot reopen a connection on a shut-down driver.
- The semaphores are no longer disposed: Dispose can race a running shutdown.

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:10:21 -04:00
Joseph Doherty 9c0f59a3ab Merge lane AbCip: AbCip driver remediation (#737)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:09:58 -04:00
Joseph Doherty a8284f499e fix(opcua): scope the deferred-write replay to nodes the apply created (Lane B fix round 2)
N1: a full alarm-state snapshot also sets the condition's Quality, so it now
supersedes an earlier quality-only write for the same id in the deferred
buffer; a quality arriving after the state is kept and replays after it.

N2 / M1: the replay is not inert on a node the apply KEPT - an SDK-applied
Acknowledge/Confirm/Shelve or an optimistic client Write in the window would
be rolled back (with spurious Part 9 events). The replay is now scoped to
nodes the apply created: variables the applier ensured during this apply
(AddressSpaceApplier.BeginRecordingEnsuredVariables) or every variable after a
full rebuild; conditions in the applied composition but not the previous one
(also on a rebuild, which carries a kept condition's state over). Option (b),
a sink that reports whether a live write landed, needs the Commons
DeferredAddressSpaceSink seam outside this lane (follow-up F7).

M2: the benchmark remark now matches its Category=Benchmark trait.
M3: TestHooks and the seam-carrying constructor are internal; PropsForTests
produces the actor through an IIndirectActorProducer.
M4: pinned replay-nothing-and-drop-the-buffer after a failed plan.
M5: PreRestart answers every queued rebuild requester (degraded).

Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:08:44 -04:00
Joseph Doherty 20875ea939 Merge local master 0d49eaf7 into remed/lane-AbLegacy
Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:06:32 -04:00
Joseph Doherty acdf6d83a3 fix(runtime): drop value bursts from driver children the host already stopped
Review I-1. Context.Stop is asynchronous, so a removed child kept
publishing through the rest of the apply handler and its own stop.
Those bursts were forwarded after SourceRemoved and re-retained the
removed driver's last Good in the mux, where nothing would ever clear
it while a replay subscriber held the ref (a removed driver never
publishes again, and its PostStop Bad fan resolves no refs once the
index is rebuilt without it).

StopChild and HandleRestartDriver now retire the child (_retiredChildren
+ Context.Watch) before stopping it. ForwardBatchToMux drops a burst
whose Sender is retired, and Terminated (handled in Steady, Applying and
Stale) removes the entry. Akka delivers Terminated after the watched
child's last message, so the set is bounded and a live incarnation is
never filtered. ConnectivityChanged from a retired child is
deliberately not filtered, because the respawn path relies on its Bad
fan.

Tests (review m-3 folded in): a stale burst spoofed as the removed child,
held alive in PostStop by a new RecordingSubscribableDriver.ShutdownGate,
no longer re-retains (red with the guard disabled). New cases cover a
config-change respawn and an operator Restart, each with the new
driver's value replaying as the positive control. The arrange step now
waits for the real subscription set and fishes for both drivers'
values at once (two sequential fishes could discard the other burst).

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:06:23 -04:00
Joseph Doherty ae20519e9d fix(modbus): deadband baseline on (value, status) so the Good recovery publishes
H2: a Bad/null publish drops the deadband baseline and any status change
publishes, so a device that returns holding the identical value is no longer
swallowed by |new - old| < deadband (the #527 value-only-dedup trap).
Integration case drives a real drop/restore through a loopback relay.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:05:25 -04:00
Joseph Doherty f37cb2a4a3 fix(abcip): only a reply type settles the BOOL-array layout (review r2)
N1: a refusal of the DWORD-addressed request permanently settled the layout
as per-element, so a tag deployed before the online edit that creates its
BOOL array later read DWORD n and published byte 0 as Good. The native runtime
now keeps an explicit layout (Unsettled / Packed / PerElement) that ONLY a
reply type changes: 0xD3 settles packed (Bits[n/32], bit n%32), 0xC1 settles
per-element (Bits[n]). Refusals and comms faults leave it unsettled, so the
next read asks again. A reply whose type disagrees with the addressing it was
asked under is discarded and re-read once under the shown layout (it would be
some other element), also when an online edit re-types a settled array. A
scalar BOOL answered 0xD3 with no element bit, or not settled packed, is
refused BadTypeMismatch instead of decoding byte 0.

m1: the write-owned Degraded marker is set together with the CAS that installs
it, under one lock, so concurrent failing writes cannot leave it naming a
different instance (a good write then clears it); a refusal-only read batch
still keeps it write-owned.
m2/m3/m5: documented (breaker accounting of a superseded init; libplctag
throw-only NotImplemented assumption; the TypeMismatch write status word is
libplctag-only).
m4: test for a write on an unsettled layout (settles by a read first).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:05:00 -04:00
Joseph Doherty 0d49eaf77b Merge lane TwinCAT: TwinCAT driver remediation (#739)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:04:51 -04:00
Joseph Doherty 87a7414df7 fix(s7): refuse unservable tag shapes per tag at every gate (R07:H3, R07:U4)
The init-time tag guards THREW: one mis-authored tag (DB1.DBZ0, DB1.DBW0 typed
Float64, T5 typed Int32, a writable Bool array) faulted the whole driver and the
actor retried the same init every 10 s forever, darkening every tag on the PLC.
Neither the editor nor the deploy gate checked any of it.

- S7AddressParser moves to S7.Contracts (same namespace) and a new
  S7TagShapeRules there is the single authority on servable shapes.
- S7TagDefinitionFactory.FromTagConfig applies it, so the AdminUI editor, the
  deploy gate (Inspect, with the specific reason) and the driver init refuse
  the same tags. Init now skips a refused tag (ISkippedTagReporter) instead of
  throwing; the throwing RejectUnsupportedTagConfigs is gone.
- New rules on top of the old guards: a narrow type at the wrong access width
  (every read failed) and a Bool array not starting at bit 0 (U4: DB1.DBX0.3
  silently read bit 0 of bytes 0..n).
- AdminUI test fixtures that used addresses the driver could never serve
  (DB1.DBS0, a browse "SomeAddress", Int32/Float32 on DBW) now use real ones.

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:04:19 -04:00
Joseph Doherty 530f1b2f0b fix(modbus): transport self-heals with one throttled in-gate connect; PDU length check
P3: a torn-down socket is re-dialled inside the gate (one attempt per backoff
window) instead of failing fast with 'Transport not connected' until the 90 s
comms-loss watchdog re-initialised the driver.
L3: ConnectAsync disposes the previous/failed TcpClient, DNS runs under the
per-op timeout, and the reconnect never sleeps inside the gate (the old
10-attempt backoff held it for ~81 s).
H9: an MBAP response with no PDU, or an exception PDU with no code, is a
TruncatedFrame desync rather than an IndexOutOfRangeException.

Refs #741
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:02:11 -04:00
Joseph Doherty 7aca052e3c Merge lane FOCAS: FOCAS driver remediation (#740)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:01:43 -04:00
Joseph Doherty a8d313ed5a fix(twincat): review round 2 - teardown confined to its generation; write health; re-arm epoch
- I1: the poll engine belongs to a teardown generation. ShutdownAsync/Dispose
  retire the current engine and install a fresh one in the same swap as the
  device/tag tables, then dispose the retired one (possibly in the background
  after a cancelled or over-budget shutdown). A subscription made after the
  next InitializeAsync can no longer land in an engine whose late dispose tail
  would dispose its CTS without cancelling it (a loop no UnsubscribeAsync could
  find). Poll handles are generation-qualified (TwinCATPollHandle) because each
  engine numbers handles from 1; SubscribeAsync re-homes a subscription that
  raced a retirement.
- M1: a write's status feeds per-device health - Good clears that device's
  comms fault and advances LastSuccessfulRead, a comms status degrades, a
  refusal keeps the state.
- M2: a symbol-version re-arm bumps a per-registration epoch; an in-flight
  retry that fails afterwards no longer re-imposes the backoff.
- M3: a re-arm restarts the refusal backoff at 30 s.
- M4: documented that PLC-not-in-RUN detection needs the probe.
- M5: WSTRING and multi-dimensional array encode-plan tests; the cancelled-
  shutdown test asserts Unknown health and awaits its background teardown.
- M6: an infinity spelled as a string ('Infinity', '-inf') passes like a
  float/double infinity; a finite overflow stays BadOutOfRange.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:00:23 -04:00
Joseph Doherty 09e0fe1841 fix(focas): a returned status never ends a device's batch; round-2 minors (#740 review round 2)
N1: linkDown is set only when NO reply arrived, i.e. a timeout, a transport exception or
a connect failure. A status the CNC returned proves it answered. EW_REJECT and every
unlisted rc map to BadCommunicationError, so a deterministic rc on one tag used to fail
every later tag of a reachable control on every poll. The comms-classed status still
degrades health. The coalescer already treated a returned refusal this way; it is
unchanged.

Tests: a returned EW_REJECT on tag 1 leaves tags 2 and 3 read and Good (red before). A
timeout still fails the rest without reading them (positive control).

Minors:
- m1: the _lifecycleLock doc now covers the health transitions it also guards.
- m2: the NoteFixedTreeFailed summary says a refusal credits the answer.
- m3: ReadAsync stamps a failed batch's exception with the generation the batch started
  under, and HandlePollError degrades that generation only.
- m4: the fixed-tree "blip" absence check runs unconditionally under a wide budget, and
  asserts that both ticks landed inside it.
- m5: direct unit cases for the coalescer link-down paths. A transport fault on the first
  range fails the second range and the per-tag tag without reading them. A connect
  failure costs one connect per batch. CncRelay.Up() retries the re-bind.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:55:37 -04:00
Joseph Doherty bc985f8857 test(ablegacy): array-write gate takes a base address and restores the words
Makes the N-file array-write case usable as the real-controller hardware gate:
AB_LEGACY_ARRAY_WRITE_BASE names the first of six consecutive words (default
N7:1), each word is read first and written back in a finally, and the
neighbours are verified per word with coalescing off. Live-run green on
ab_server slc500/micrologix/plc5 x libplctag/native (39/39), and with a moved
base (N7:3).

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:53:46 -04:00
Joseph Doherty 459cbb444e fix(abcip): review fix round 1 - BOOL-array fallback, local refusals, minors
I1: the U11 Bits[n] -> Bits[n/32] rewrite was unconditional. The native
runtime now keeps the original element path until the first reply settles the
layout: 0xD3 confirms DWORD addressing (bit n%32); any other reply type (e.g.
a 0xC1 per-element BOOL array) or a refusal of the DWORD index falls back to
Bits[n] and re-reads, and a write settles the layout by a read first, so it can
never decode or write element n/32 of an element-addressed array.

I2: libplctag ErrorNoMatch/ErrorBadDevice no longer fold into NotFound (they
now read as BadConnection: comms, Degraded, LastSuccessfulRead preserved).
ErrorNotImplemented - and the native runtime's own pre-send refusals - raise
AbCipLocalRefusalException: BadNotSupported, no health, LastSuccessfulRead NOT
advanced, no backoff record, not reachable for the probe or Test Connect.

M1: the bit RMW's parent read/write status words follow the health contract
(refusal: per-tag Bad + Warning; comms: evict the parent + Degraded).
M2: device Unsupported read refusals return Refused (advance
LastSuccessfulRead); device write refusals call RecordWriteAnswered (not the
pre-send native type gate).
M3: a good write clears a Degraded a write comms fault recorded (a read-side
Degraded is still left for the read path).
M4: a superseded init throws AbCipInitializationSupersededException (not an
OCE); ShutdownAsync stamps Unknown only if no newer init published meanwhile.
M5: the sync-Dispose timing assert is 2 s against a 4 s probe read; the
unencodable-array test asserts health.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:53:06 -04:00
Joseph Doherty 991d277c03 Merge commit '7bcb6331' into remed/lane-TwinCAT 2026-10-22 14:53:01 -04:00
Joseph Doherty 95b75c2e58 Merge local master 7bcb6331 into remed/lane-FOCAS
Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:51:47 -04:00
Joseph Doherty 9496f061af Merge commit '7bcb6331' into remed/lane-B 2026-10-22 14:49:17 -04:00
Joseph Doherty 34e455f412 test(ablegacy): dead-device gate tolerates a coalesced single-unit pass
Live run against ab_server (slc500/micrologix/plc5 x libplctag/native): with
coalescing on, the blackholed device's 20 tags are one block, i.e. one comms
fault per poll, so its backoff window opens on the second poll rather than the
first. The gate now asserts the actual contract: at most two timeouts across
three polls, the live device Good on every poll, and the third poll fast.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:48:35 -04:00
Joseph Doherty f179829ca6 fix(twincat): review round 1 - PLC-not-in-RUN degrades, per-device health, retry backoff
- IMPORTANT: a successful probe records the PLC's ADS state; a PLC not in Run
  keeps the driver Degraded with 'PLC in {state}' while LastSuccessfulRead
  still advances (the device answered). RecordDeviceAnswered never restores
  Healthy while any device is out of Run.
- Health is per device, rolled up worst-of: a comms fault is remembered on the
  device it was seen on and only that device answering clears it (device A
  answering no longer clears device B's fault).
- Refused (non-comms) dead registrations back off 30 s doubling to 10 min and
  are re-armed by a symbol-version change or retried by the reconnect replay;
  comms-class failures keep the every-tick retry. A retry whose failure class
  changes republishes the tag's status, once.
- Bit-RMW gates are kept across generations and teardowns (never cleared).
- ShutdownAsync honours its CancellationToken: it stops waiting, finishes the
  bookkeeping and throws OperationCanceledException.
- Remarks on why a superseded InitializeAsync throws OperationCanceledException.
- REAL / LREAL values (scalar or array element) outside the target range are
  OverflowException -> BadOutOfRange instead of being written as Infinity.
- String / array write encoding factored into AdsTwinCATClient.PlanStringWrite
  / PlanArrayWrite; the fake runs them so a driver-level test covers the real
  encode path against a fake resolved symbol.
- Timing-fragile tests use condition waits / measured probe-tick windows; the
  dispose-during-shutdown test asserts ordering with a 1 s bound.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:47:24 -04:00
Joseph Doherty 61dd498eab test(ablegacy): pin the array-write capability row against the encoder
DriverArrayWriteSupport (Core.Abstractions, not this lane's) still lists
AbLegacy as scalar-only, so every AbLegacy array node is materialised
read-only and the new array encoder is unreachable from OPC UA. Adds the
Modbus/S7-style parity test, skipped until follow-up F1 adds the AbLegacy row
(Int/AnalogInt/Long/Float with a declared length), plus an active test pinning
exactly which element types the encoder accepts.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:46:04 -04:00
Joseph Doherty 6626c99c71 Merge branch 'master' into remed/lane-AbCip 2026-10-22 14:45:40 -04:00
Joseph Doherty ffa5b2a4fa fix(ablegacy-cli): Ctrl-C prints Cancelled instead of a stack trace
Review 07 U7 (AbLegacy part): probe/read/write had no OperationCanceledException
arm, so cancelling a slow read printed an unhandled-exception stack trace. The
driver work now runs through AbLegacyCommandBase.RunCancellableAsync, which
reports an operator cancellation as "Cancelled." and still propagates anything
else. (The AbCip subscribe lock and SnapshotFormatter array rendering are
other lanes' files.)

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:44:51 -04:00
Joseph Doherty de54b23b1a test(focas): wire-level cases for fixed-tree health and the legacy alarm layout (#740 review I2)
Against focas-mock:
- H15: a local TCP relay in front of the shared mock takes "the CNC" away and brings it
  back without stopping the fixture. A FIXED-only driver goes Degraded with
  LastSuccessfulRead frozen and its value Bad, then returns to Healthy with Good values
  (the positive control).
- H20: the mock answers cnc_rdalmmsg2 in the 80-byte legacy layout. Nineteen alarms
  (1,520 B = 20 x 76) now decode as the 19 seeded alarms; the old length test gives 20,
  verified red. A 3-alarm reply is the control.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:44:42 -04:00
Joseph Doherty fd64fed217 fix(focas): a stalled device fails fast for the rest of its batch; generation-stamp health (#740 review round 1)
I1: Task.WhenAll waits for the slowest device, so concurrency alone still let a device
that accepts TCP and then stalls cost (its tag count x call timeout) per batch. After the
first timeout, transport failure or comms-class status on a device, its remaining tags in
the batch now fail BadCommunicationError immediately, the same as the coalescer's
FailMembers, which now also stops after a failed range. FIXED: tags are still served
from the cache. A malformed address is BadNodeIdUnknown and does not count as a link
failure. Test: a 10-tag stalled device beside a healthy one completes in under 3 call
timeouts with the healthy value Good. The P9 test now also asserts per-client read
counts and that two reads overlapped across devices.

Minors:
- 1: a superseded init releases its table only if it was never published (flag).
- 2: a fixed-tree refusal credits LastSuccessfulRead; a value-path refusal after a comms
  fault restores Healthy.
- 5: alarm/fixed/refusal health is reset inside the publish lock, after the generation
  check.
- 6: FocasFixedTreeHealthTests wait on served-failure/refusal counters, not wall-clock.
- 8: data-path health writes (reads, coalescer, writes, poll errors, refusals) carry the
  generation they started under and are dropped once a lifecycle call has moved on. Each
  write is a check-and-set under the lifecycle lock.
Also adds a unit case where 20 vendor records with type 0 and empty messages stay vendor.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:44:42 -04:00
Joseph Doherty 66927a3faa fix(ablegacy): wire the authored probeTimeoutSeconds into Test Connect
Dead config field (user decision: wire them all): the AdminUI form authors
probeTimeoutSeconds ("Admin UI probe timeout", default 5) but AbLegacyDriverProbe
never read it, so Test Connect always ran on the caller's budget. It now
tightens that budget (never raises it), the same rule as the Galaxy probe; a
non-positive value keeps the caller's budget.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:43:23 -04:00
Joseph Doherty 582e82899b feat(ablegacy): implement IBatchSubscribable
Review 02 P1 / Wave 2 rule (d): AbLegacy runs PollGroupEngine but raised only
the per-tag OnDataChange, so every changed tag paid its own trip through the
actor chain. The engine's per-cycle batch is now raised as OnDataChangeBatch
(one per poll cycle), alongside the per-tag event for legacy consumers.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:42:23 -04:00
Joseph Doherty 6636de1199 refactor(runtime): delete the per-value AttributeValuePublished message
It had no production sender: every driver data change is already a
batch, and the comms-loss fan was converted earlier. DependencyMuxActor
drops its per-value receive, DriverInstanceActor drops the record, and
DependencyMuxActorTests' 13 sends move to the ValuePublishes.Single
one-item batch (same semantics through the batch path). XML docs and
the living docs (DriverLifecycle, VirtualTags, ScriptEditor) are
reworded; historical plan docs are left as written.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:42:08 -04:00
Joseph Doherty 530f4bdcd4 perf(ablegacy): coalesce adjacent N/B/F/L scalar reads into one PCCC request
Review 07 P7 (rest): adjacent words on one data file were never coalesced
(one round trip per tag) although the array path proves the transport can read
a span. Within a device pass, scalar tags on one N/B/F/L file within 8
elements of each other (block <= 200 data bytes, under the 236-byte typed-read
limit) now ride one multi-element read and are decoded from their offsets with
exactly the CLR types the single-tag path returns (int / float / bool).

Only shapes the single path decodes identically are coalesced: indirect,
sub-element, array, file-number-less (S:/I:/O:) and width-mismatched tags keep
the single path. A block the device refuses (an authored element past the end
of the file, a native PLC-5 non-N file) falls back to per-tag reads for its
members and its shape is remembered; a comms fault fails the members together.

New driver-config key coalesceReads (default true) with an AdminUI switch
(Razor change: AbLegacyDriverForm, for the Wave 4 live check). Adds an
env-gated ab_server parity case (coalesced == per-tag values).

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:41:20 -04:00
Joseph Doherty 0475603192 refactor(runtime): drop DriverHostActor's unused ILoggerFactory slot
The parameter was retained only for signature stability after the
EfAlarmConditionStateStore fallback that consumed it was retired. It is
removed from Props, the constructor and the positional Props.Create
forwarding list together, and from its one caller in
WithOtOpcUaRuntimeActors. No test passed it.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:38:33 -04:00
Joseph Doherty 6b863cd524 test(abcip): ab_server cases for the probe blackout, refusals, array writes
Integration cases (both transports, skipped when the fixture is unreachable):
- H10: a typo'd probe tag leaves the host Running and a cold data tag created
  after many probe ticks still reads Good;
- H1/L4: a missing tag reads BadNodeIdUnknown every poll beside a Good sibling,
  the driver stays Healthy and LastSuccessfulRead advances even when only the
  refused tag is read;
- U10 (AB_SERVER_ALLOW_WRITES=1): an int[16] written to TestArray (DINT[16])
  reads back element for element through a fresh driver; a 15-element write is
  BadOutOfRange.

Not run here: the 10.100.0.35:44818 fixture was down during this lane (Wave 4).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:37:18 -04:00
Joseph Doherty 5de1d661bb fix(runtime): evict a stopped driver's retained values from the dependency mux
DriverHostActor now tells the mux SourceRemoved(driverInstanceId)
wherever a driver object ends: StopChild (deploy removal and the stop
half of a config-change respawn) and an operator Restart. It is not
sent on a supervision restart, which re-uses the same driver. Before
this nothing sent it, so a removed driver's last Good stayed in the
mux and replayed to the next replay-opting subscriber as current.

DriverHostActorSourceRemovedTests drives a real mux with two real
driver children: after a deploy drops DI-1 only DI-2's value replays
(positive control: both replay before the removal), and a supervision
restart of DI-1 leaves its value replayable.

Residual (documented on EvictRetainedValues): a burst the old child
enqueued just before it stopped can still be forwarded after the
eviction; the next publish, a comms-loss fan or the grace sweep
bounds it.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:36:54 -04:00
Joseph Doherty 8a941de4cd fix(abcip): keep a sync Dispose, but one that never blocks on the teardown
F10 follow-through: removing IDisposable outright broke a synchronous
drv.Dispose() call site outside this lane (Runtime.Tests
AbCipAlmaNativeAlarmEndToEndTests). The sync Dispose() is back, but it starts
the same teardown as DisposeAsync and returns instead of parking the caller on
DisposeAsync().AsTask().GetAwaiter().GetResult(); a background fault is
logged. The runtime still takes the IAsyncDisposable path (preferred, never
both).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:34:53 -04:00
Joseph Doherty e5527d0201 fix(ablegacy): isolate a dead device — parallel devices, per-device backoff, single-flight connect
Review 07 P7 (High): reads walked every reference of every device serially
with no backoff, so a dead device cost Timeout per tag per poll (50 tags x 2 s
= 100 s) and starved every healthy device long enough to trip the comms-loss
watchdog against a reachable one.

- ReadAsync buckets references by device and reads up to 8 devices
  concurrently; each device's tags are still read in order.
- Each device has a ConnectionBackoff (1 s doubling to 30 s). Two consecutive
  comms faults mark it down: the rest of that pass and every later pass fail
  fast with BadCommunicationError and touch no wire until the window expires or
  any answer (data path or probe) resets it. One timing-out tag does not
  blacken its answering siblings.
- Cold connects are single-flight per device (ConnectGate), replacing the
  per-key creation locks; a cached handle never touches the gate.
- A status/exception proving the device ANSWERED with a deterministic refusal
  (not found, not allowed, out of bounds, type/size mismatch) is per-tag Bad,
  keeps its handle, warns once per tag and advances LastSuccessfulRead instead
  of evicting and degrading. Writes classify the same way.
- Health is written once per read: Degraded if any device faulted, Healthy if
  anything read Good, LastSuccessfulRead advanced when any device answered.

Adds an env-gated ab_server case with a blackholed sibling device.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:33:54 -04:00
Joseph Doherty a33145fe71 refactor(alarms): spawn the scripted-alarm host through its factory Props
DriverHostActor now hands ScriptedAlarmHostActor an engine factory
instead of a pre-built engine, so every incarnation (a supervision
restart included) builds, owns and disposes its own upstream + engine.
The legacy overload was already restart-safe in the detach sense, so
this is hygiene rather than a live hole; the factory path is the one
ScriptedAlarmHostActorRestartTests covers end to end. The closure
captures locals only and carries the node-wide ScriptingOptions.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:32:53 -04:00
Joseph Doherty aabe027d10 test(abcip): native end-to-end proof a NotFound tag causes no lease churn
L4 evidence over the real native runtime and a fake CIP target: a renamed tag
that is the device's only lease holder is created once and Forward-Opened once
across five polls, with no Forward Close, every poll BadNodeIdUnknown and the
driver Healthy.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:32:31 -04:00
Joseph Doherty fc94b87634 perf(abcip): cache the declaration-only UDT layout per member list
P11: AbCipUdtReadPlanner called AbCipUdtMemberLayout.TryBuild(parent.Members)
for every member reference on every poll - O(members^2) per parent per tick,
a dictionary allocation each time. GetOrBuild caches the layout (including a
"not groupable" verdict) in a ConditionalWeakTable keyed by the member list's
identity, which is immutable for the life of the tag table; a re-init's new
lists replace the entries and the old ones are collected with them.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:31:32 -04:00
Joseph Doherty d2cc071517 feat(scripting): expose the run token to scripts as ctx.CancellationToken
ScriptContext.CancellationToken is backed by an AsyncLocal that
ScriptEvaluator.RunAsync scopes around the compiled delegate, so it is
exactly the token ScriptRunGuard cancels on timeout. A polling script
can now end its own abandoned run and free the slot. AsyncLocal rather
than a field because the scripted-alarm engine reuses one context per
alarm: an orphan keeps seeing its own cancelled token. The scoping
entry point is internal (IVT to Core.Scripting) so a script cannot
replace its own token.

Editor parity: ScriptAnalysisService dot-completion now walks the
receiver's base chain (instance members only, stopping before
System.Object). Before this, a member declared only on ScriptContext
compiled and published but never completed. DiagnoseTests pins the
member as allow-listed; CompletionTests pins the completion.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:31:03 -04:00
Joseph Doherty f966efcfad fix(opcua): replay writes deferred during an off-thread plan; cache-ref and guard fixes (Lane B fix round 1)
C1 (critical): B6 computes the plan off the pinned thread while B1 made writes
non-creating, so a value/status/alarm write for a node the in-flight plan adds
was skipped, and change-driven drivers never republish it (BadWaitingForInitialData
indefinitely; on a full rebuild every re-subscribe value landed on discarded
nodes). OpcUaPublishActor now applies writes immediately AND remembers the
latest per (realm, id) while a plan is in flight or queued (value+status
compose; alarm state/quality coalesce to the latest, which the node's delta
gate makes inert on an existing condition), replays it after each apply before
answering, and drops it once the pipeline is idle. Bounded (250k keys by
default; beyond it new keys are not held, logged once).

I2: the data-plane-authority parse runs in its own try inside the plan task; a
throw is logged and never degrades the address-space apply.

I3: the SDK's DeleteMonitoredItems releases an item's component-cache
reference on a Good delete; a tombstoned item's reference was already
released, so the delete took one from the re-created node's entry. Released
items are remembered and the orphan delete takes one back first. Event items
never held a reference (EventManager + SubscribeToEvents), so the three
event-item releases are gone.

I4: direct tests for the HistoryUpdate override's per-request updater check
and the HistoryReadEvents override's source guard (both reached by restoring
the bit after a reset - the B7 race window); fixed the false remark.

Minors: the MaterialiseAlarmCondition kind-swap tombstones the dropped
instance; a bytes-less rebuild arriving mid-plan queues behind it (replies in
order); a PlanReady/PlanFailed from a previous actor incarnation is ignored and
its requester answered degraded; the tie-cluster end bound of an uncapped read
fits against the backend ceiling (defence in depth - guarded by a sweep test);
the lock-time benchmark stays in the default run, headroom noted.

Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:30:11 -04:00
Joseph Doherty 7cc8415310 feat(abcip): array writes encode count elements at the element width
U10 (user decision: implement array encode now): config, the AdminUI and
discovery all advertised array tags Operate-writable, but both runtimes encoded
a scalar - an int[] threw InvalidCastException (BadTypeMismatch AND Degraded),
a scalar wrote element 0 natively, and libplctag re-sent its whole last-read
buffer with element 0 patched (stale elements 1..n-1).

- AbCipArrayCodec encodes count elements little-endian at the element width,
  CONVERTING each element (ScadaBridge writes int[] to INT arrays; overflow is
  BadOutOfRange, unconvertible BadTypeMismatch, both before anything is sent).
- The driver checks the shape before any runtime exists: not a list ->
  BadTypeMismatch, length != elementCount -> BadOutOfRange, BOOL/STRING/
  structure element types -> BadNotSupported; and, before sending, a controller
  buffer that is not width x count -> BadTypeMismatch. Refusals warn once and
  write no health.
- Native: ONE Write Tag of count elements with the controller's type code (the
  client fragments a large one); libplctag: the codec's bytes copied into the
  elem_count-sized buffer, refused TooSmall if it does not fit.
- IAbCipTagRuntime.EncodeArray (default: NotSupportedException).

Real-controller proof is a hardware userGate (see the lane report recipe).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:30:11 -04:00
Joseph Doherty 8de5947e2d test(twincat): XAR-gated wire cases for H11/H12/H18/U9
Native-subscribe isolation of an unknown symbol, LastSuccessfulRead advance
from pushes alone, a WORD bit write flipping exactly one bit (bit 16
refused), a whole-array REAL write round-trip (length mismatch refused) and a
STRING(80) write round-trip (overflow refused). Every writing case restores
the seed it touched. Skipped without the TwinCAT XAR fixture.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:30:05 -04:00
Joseph Doherty 6201bafea4 test(twincat): pass the test token to the abandoned-init wait
Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:28:53 -04:00
Joseph Doherty 8ffa6178b5 test(focas): integration case for re-initialise under live wire reads (R07:H13)
Against the focas-mock fixture: the fixed-tree loop is running and reads are in flight
while the driver re-initialises five times. No reader throws and reads keep returning
Good. After shutdown, a fresh driver still connects and reads.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:27:38 -04:00
Joseph Doherty 82a485e42e fix(abcip): a BOOL-array element reads its own bit, not "low byte != 0"
U11: a scalar BOOL authored over Bits[n] requested element n and decoded
"any of bits 0-7 of the reply set", so Bits[n] read Good with the subscript
ignored. A Logix BOOL array is stored as DWORDs (0xD3) and its element segment
addresses them, so the native runtime now requests DWORD n / 32 and decodes bit
n % 32 when the controller answers 0xD3 (a plain BOOL reply still decodes its
byte). libplctag exposes no type code and so cannot say which DWORD it read: a
4-byte reply for such a tag is refused BadNotSupported (warned once) instead of
publishing a wrong value as Good. Real-controller proof is a hardware gate.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:27:34 -04:00
Joseph Doherty 2724d7dae8 fix(focas): a CNC refusal is per-tag Bad, not a driver Degraded (lane health checklist)
Applies the driver-lane health contract to the value path. Every non-Good read status used
to write Degraded on every poll, so one vacant macro (BadNoData), one EW_NUMBER address
(BadOutOfRange) or one fractional macro on an integer tag made a reachable CNC look sick
forever. Now:
- a comms-class status (BadCommunicationError / BadInternalError (EW_HANDLE) / BadTimeout)
  still degrades, with LastSuccessfulRead preserved;
- any other non-Good status is a refusal: the tag reads Bad, one Warning is logged per tag
  (re-armed when it next reads Good, cleared at init), driver state is untouched, and
  LastSuccessfulRead advances because the CNC answered (CAS, since devices now read
  concurrently);
- a write refused with NotSupportedException no longer degrades the driver.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:26:28 -04:00
Joseph Doherty 9beeed7a76 feat(twincat): implement IBatchSubscribable (R02:P1)
The poll engine already builds a per-cycle batch; TwinCAT now forwards it as
OnDataChangeBatch, so DriverInstanceActor gets one message per poll cycle
instead of one per changed tag. Because the actor subscribes ONLY the batch
event of a batch-capable driver, every native-path data change (pushes, the
BadNodeIdUnknown / refusal / unavailable snapshots) is raised through a single
RaiseDataChange that fires both the per-tag event and a fresh one-item batch.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:25:20 -04:00
Joseph Doherty 517837e798 feat(ablegacy): encode array writes element by element
Review 07 U10 (user decision: implement array encode now). Both runtimes
encoded a scalar for an array tag: an int[] threw InvalidCastException into
BadTypeMismatch + Degraded, a scalar wrote element 0 natively, and libplctag's
SetInt32(0, ..) re-sent the stale last-read buffer for elements 1..n-1.

IAbLegacyTagRuntime gains EncodeArray; both runtimes write every element at
the element width (Int/AnalogInt 2 bytes, Long/Float 4). The driver converts
the written value before any runtime exists: each element is converted, not
exact-matched (ScadaBridge writes int[] to Int16 nodes), a length other than
the tag's element count is BadOutOfRange, an overflowing element is
BadOutOfRange, a non-array or non-numeric payload is BadTypeMismatch, and a Bit
array (bit-packed words; an encode would clobber neighbour bits) is
BadNotSupported. None of these touch health or the wire. Adds an env-gated
ab_server round trip for an N-file array write.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:25:12 -04:00
Joseph Doherty 4615495f01 feat(scripting): bind the Scripting section node-wide and clamp its knobs
Program.cs binds ScriptingOptions from Scripting:* once, clamps every
numeric knob through ScriptingOptions.Sanitized (logging a Warning per
adjustment) and registers the one instance. It now reaches all three
script seams: RoslynVirtualTagEvaluator, the scripted-alarm engine
built in DriverHostActor (new trailing Props parameter, wired in
WithOtOpcUaRuntimeActors) and every Calculation driver (node options
as the base, per-driver config keys override one knob on a copy).

ScriptRunGuard sanitizes again in its ctor and computes the stuck
threshold with a saturating multiply, so an absurd
StuckOrphanQuarantineFactor can no longer throw OverflowException on
a refused run.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:53 -04:00
Joseph Doherty cbacaee4ee fix(twincat): bounded, race-safe lifecycle; drop counter; post-unsubscribe purge (L8)
- Device and tag tables are build-and-swap: InitializeAsync builds a fresh
  generation and publishes it by one swap under a lifecycle lock, then
  releases the replaced generation; teardown swaps in empty tables. A
  retained poll loop never sees a table mid-Clear (it used to read
  _tagsByRawPath while a re-init only ever added to it), and each device
  set is released by exactly one caller.
- A lifecycle generation makes an InitializeAsync overtaken by a shutdown
  (or a newer init) release what it built instead of installing it - the
  Lane A abandoned-init-overlaps-shutdown case.
- ShutdownAsync joins all probe loops together within 2 s (it awaited each
  unbounded - a probe wedged in the blocking ADS Connect hung the actor's
  teardown) in parallel with the poll-engine dispose, the whole bounded at
  4.5 s; DisposeAsync may run concurrently and returns at once. A connect
  that completes on a released device disposes its client.
- Releasing a generation disposes its native registrations, so a direct
  re-init no longer leaves live notifications bound to dead DeviceStates.
- Connect gates and RMW semaphores are no longer disposed under in-flight
  users (no kernel handle is ever allocated); a fault on a released device
  is not recorded as driver health.
- The notification channel counts DropWrite overflows
  (ITwinCATClient.DroppedNotificationCount); the probe tick warns with the
  delta. A queued value whose registration was disposed is purged in the
  client dispatch loop and dropped driver-side for any client.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:10 -04:00
Joseph Doherty 895cc28886 fix(abcip): write-side health neither latches nor overwrites
H19: Format/InvalidCast/Overflow on a client value never touched the wire but
latched Degraded; a good write set Healthy and so undid a Degraded the poll
loop had just recorded for other tags (the batch read path is worst-wins for
exactly this reason).

- The three conversion failures map to BadTypeMismatch/BadOutOfRange with a
  once-per-tag Warning and no health write (AbLegacy's rule).
- A good write (scalar or bit RMW) advances LastSuccessfulRead only - the
  device answered - and leaves the state alone.
- A write comms fault (thrown, or a non-refusal status word, which used to
  write no health at all) is Degraded with LastSuccessfulRead preserved and
  never downgrades Faulted.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:06 -04:00
Joseph Doherty 29b566d51b fix(focas): disambiguate the cnc_rdalmmsg2 record layout at 1,520-byte multiples (R07:H20)
ParseAlarms took the 76-byte vendor layout whenever Length % 76 == 0, so a legacy 80-byte
reply of exactly 19 records (1,520 B = 20 x 76) decoded as 20 garbage vendor records.

The response block carries no record count (its 16-byte header is length/class/path/
command/rc/payload length), so the layout is chosen from what is known: the record count
the request asked for (a reply never holds more), then each layout's field invariants on
every record. For legacy records, the int32 type/axis/length must fit a short and the
length must be at most 64. For vendor records, the i16 length must be at most 64. If both
still fit, the layout whose declared lengths match the decoded text wins. Vendor stays the
default when both readings are equally plausible, and unambiguous lengths are unchanged.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:05 -04:00
Joseph Doherty cb16ebc75e fix(abcip): bit-in-word writes take the parent width from its read
H17: the .n read-modify-write hard-coded a DINT parent. On an INT or SINT host
(both legal in Logix, and reads of Flags.3 there already worked) the DINT
decode of a 2-byte buffer threw OutOfBounds, the generic catch evicted the BIT
tag's (non-existent) runtime and reported BadCommunicationError + Degraded.

The parent width now comes from the parent read's size (1 SINT / 2 INT /
4 DINT / 8 LINT; unknown keeps the DINT assumption). Narrow hosts use unsigned
arithmetic so a set top bit encodes without overflow; a bit past the host's
width is BadOutOfRange and a non-integer width BadTypeMismatch, both refused
before anything is written, warned once, no health write. A transport failure
in the RMW now evicts the parent runtime it actually used. Native end-to-end:
an INT host is written as ONE INT (type 0xC3, 2 bytes).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:21:43 -04:00
Joseph Doherty 4c4dfaadcc fix(focas): recycle the device handle under ConnectGate and drain it first (R07:L6)
The handle-recycle loop disposed the device client with no gate. It could null a client a
concurrent EnsureConnectedAsync had just installed (orphaning a live two-socket session),
or dispose one a poll was mid-read on (ObjectDisposedException -> BadCommunicationError +
Degraded on every recycle interval).

The recycle now takes the device's ConnectGate, detaches the client, and disposes it via
SynchronizedFocasClient.DisposeWhenIdleAsync, which waits (bounded by the call timeout plus
slack) for ops that already hold the client. The gate stays held through the drain, so no
fresh session opens beside the old one.

SynchronizedFocasClient no longer disposes its call gate. Releasing a disposed semaphore
threw out of reads that had otherwise completed. Ops that reach a disposed client now get
ObjectDisposedException and are never run on the dead socket.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:21:02 -04:00
Joseph Doherty 8b50b43048 fix(ablegacy): deterministic NotSupported refusals are BadNotSupported with no health write
Review 07 H16: an array of String/Timer/Counter/Control passed the mapper, so
every poll created a handle (one init read), the decode threw
NotSupportedException, the generic catch evicted the handle and latched
Degraded. FromTagConfig now rejects that combination (the skip carries the
reason, Inspect/deploy gate and the AdminUI model state it), and a runtime
NotSupportedException on read or write is BadNotSupported: handle kept, no
health write, one Warning per tag and operation. Write refusals (native PLC-5
and status-bit writes, libplctag translated addresses) now warn once per tag too.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:20:17 -04:00
Joseph Doherty 4e9bd9d966 perf(abcip): native read window >= 64; cold waiters hold no window slot
P8: on the native transport the bounded-parallel read window is also the cap
on services per Multiple Service Packet (the device's shared scheduler packs
whatever is queued, within the connection's request/reply byte budget), so the
default 8 cost a 1000-tag controller ~125 round trips per poll instead of ~10.
The effective window (AbCipDriver.ReadWindow) is now max(64, option) when the
native tag factory serves reads; libplctag and injected factories keep the
option exactly. Option doc + AdminUI help text updated (Razor: Wave 4 live
check).

P7 addendum: a work item now takes its window slot only after its runtime
exists. A cold create queues on its device's single-flight connect gate, and
holding a slot there let one dead device's backoff-expiry attempt occupy the
whole window for a full Timeout, stalling every other device's reads.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:19:38 -04:00
Joseph Doherty 256915b7e4 perf(focas): read devices concurrently within a batch (R07:P9)
ReadAsync walked the whole batch sequentially, so one SLOW CNC (TCP-accept-then-stall,
the live 31i-B case) cost Timeout per tag per poll while no other device was read.
ReadAsync now resolves every reference against one table snapshot, buckets the rest by
device and reads the devices with Task.WhenAll; each device's tags still go in order on
its own serialised socket (the PMC coalescing pre-pass runs per device, first). Results
land at their own indices.

Also precomputes each device's fixed-tree key prefix (it was built per reference, twice
per poll).

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:18:16 -04:00
Joseph Doherty 7377daa9b0 fix(ablegacy): prune per-key gates when an indirect index moves
Review 07 L7: each index value an indirect tag (N7:[N7:0]) swept through left a
per-runtime and a creation SemaphoreSlim in the device's gate tables until
shutdown. A moved index now drops the stale resolution's gates along with its
handle, and sweeps any other stale resolution of the same tag a concurrent
reader cached, so the tables stay bounded by the authored set.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:17:16 -04:00
Joseph Doherty 48b0173ae9 fix(abcip): a reader racing a re-init neither leaks a runtime nor degrades
Found by the H13 stress test (200 re-inits under a concurrent reader): a read
that resolved the device just before the table swap could cold-create a
runtime into the retired state after its teardown had disposed it (leaked
native handle / lease), and could hit a per-tag SemaphoreSlim the teardown had
disposed, which the generic catch reported as a comms fault (Degraded).

- DeviceState.Retire marks the state retired before disposing handles; each
  handle is removed-then-disposed exactly once. A creator re-checks the flag
  after its TryAdd (fenced, Dekker-paired with Retire) and disposes the orphan.
- A failure on a retired device is the teardown's doing: BadCommunicationError
  for that one call, no evict, no health signal.
- Per-tag/RMW gates are no longer disposed (async-only waits hold no handle),
  matching the ConnectGate rationale.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:17:16 -04:00
Joseph Doherty 22d4680e7b fix(abcip): a refused probe tag proves the device reachable (no blackout)
H10: the probe loop recorded EVERY exception - including the controller
answering NotFound for a typo'd or download-deleted Probe.ProbeTagPath - as a
Forward-Open failure. At a 5 s tick against a 30 s backoff cap ShouldAttempt
was never true again, so every cold or evicted data tag on a reachable device
read BadCommunicationError indefinitely and the host showed Stopped.

The probe now decides the backoff verdict and the host state separately:
- a device refusal (thrown or as a status word) is backoff success + host
  Running + one Warning per device naming the probe tag; an initialised probe
  runtime is kept so its lease is not re-opened every tick;
- a malformed probe tag name (AbCipTagConfigurationException) never reached
  the device: no backoff signal, no host transition, one Warning;
- a genuine transport failure is unchanged (backoff failure, host Stopped).

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:17:03 -04:00
Joseph Doherty cc140593a0 fix(focas): refuse String on PMC/PARAM/MACRO in the factory, editor and CLI (R07:U8)
FocasDataType.String was authorable on any address, but no area read returns text: PMC
fell through to a byte read, PARAM to Int32, MACRO to a double, and the String node
received a boxed number at Good.

One rule, FocasTagDefinitionFactory.DescribeDataTypeMismatch, now backs every path:
- FromTagConfig rejects the pair, so the driver skips + reports it (BadNodeIdUnknown,
  never a Good number) and Inspect names the reason at deploy;
- FocasTagConfigModel.Validate refuses it on the typed fields;
- the AdminUI editor only offers String where it is readable and shows the rule's text
  inline (kept visible for a loaded blob that already carries it);
- the CLI read/subscribe/probe/write commands refuse it at the boundary.
String stays valid on FIXED: paths (Identity/Program name/mode text) and is ignored on
ALARM: tags; an unparseable address keeps its own diagnosis.

Razor change (FocasTagConfigEditor.razor) needs the Wave 4 live check.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:15:35 -04:00
Joseph Doherty f17ba8e258 feat(twincat): array and STRING writes through the codecs (U9)
Array tags were advertised writable but every write threw InvalidCastException
in ConvertForWrite (reported BadTypeMismatch); STRING writes went through the
Beckhoff marshaller with no length check against STRING(n) and the host's
ANSI code page, which the read side deliberately avoids.

- Array write: TwinCATArrayCodec.ToWriteElements converts exactly ArrayLength
  elements to the declared type (CLR lists convert - ScadaBridge writes int[]
  to Int16 nodes; a length mismatch or out-of-range element is BadOutOfRange, a
  scalar value BadTypeMismatch) before any wire traffic; the ADS client plans
  the layout from the resolved PLC array (TwinCATTypeGuard.CheckArray, as the
  read does), encodes with TwinCATArrayCodec.Encode - the exact inverse of
  Decode - and writes the raw bytes (new ITwinCATClient.WriteArrayAsync).
- STRING / WSTRING write: encoded at the PLC's declared width by
  TwinCATStringCodec.EncodeString/EncodeWString (Windows-1252 or UTF-8 per
  target, NUL-padded, strict: an over-long value or unrepresentable character
  is OverflowException -> BadOutOfRange) and written as raw bytes.
- TwinCAT CLI read/write gain --array-length (+ --separator) for the hardware
  gate.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:15:29 -04:00
Joseph Doherty 6423ed272f fix(focas): the fixed-tree loop reports device silence to driver health (R07:H15)
The fixed-tree loop swallowed every failure at Debug and never touched health, so a
driver serving only FIXED: tags (Identity/Axes/Program/Status, the typical FOCAS shape)
read Healthy on /hosts with every node Bad and the #527 watchdog could never fire.

The fixed tree is now a third health path beside the value and alarm paths, combined in
GetHealth (worst state, latest LastSuccessfulRead):
- an answered bootstrap / axis pass clears the device's failure and advances
  LastSuccessfulRead;
- a transport failure degrades the driver once the device has been silent past the
  same staleness budget that turns its values Bad (from its last answer, or from loop
  start when it never answered), with LastSuccessfulRead frozen;
- a CNC refusal (non-OK rc) is not a comms loss: it clears the failure and credits nothing.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:11:25 -04:00
Joseph Doherty 40c7493829 fix(twincat): BOOL-within-word writes RMW the parent at its own width (H18)
The bit write read and wrote the parent word as a 32-bit UDInt with no parent
check: a bit beyond a 16-bit parent was written into neighbouring bytes, and a
signed parent with its sign bit set read back >= 2^31, so the checked
conversion turned any bit write into BadOutOfRange.

The write now goes through ITwinCATClient.WriteBitAsync: the ADS client
resolves the parent symbol, checks the bit against it (TwinCATBitAccess.
CheckParent, as the notification path already did), reads exactly
parent.ByteSize raw bytes, flips one bit (TwinCATBitAccess.ApplyBitWrite) and
writes the bytes back by index group/offset or variable handle. A bit beyond
the parent is BadOutOfRange, a non-integer parent BadTypeMismatch, a short
parent read BadCommunicationError (never written back), all without a health
write. The per-parent RMW lock stays in the driver.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:10:12 -04:00
Joseph Doherty 7bcb6331ed Merge lane C: control plane, host, security (#733)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:09:08 -04:00
Joseph Doherty 5097444d56 fix(controlplane): condition the dispatch flip on the guard's RowVersion; one deadline source; align IdentityModel pins
Re-review 1: HandleDispatch's terminal guard and its AwaitingApplyAcks flip read the row separately, so a
lost-dispatch closure committed between them was silently reopened. The flip now carries the RowVersion the
guard read; a change in between is a concurrency conflict (nothing seeded, nothing published, coordinator
reset). Pinned on real SQL Server with a closure injected between the two reads (fails without the fix).

AdminOps' apply deadline and the coordinator's now both read DeployTimings.ApplyDeadline, passed explicitly
to both singletons. Microsoft.IdentityModel.Protocols(.OpenIdConnect) pinned to 8.11.0 and referenced
directly by the Security project, so the whole IdentityModel family resolves 8.11.0 (fresh restore clean
of NU190x/NU1605).

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:08:42 -04:00
Joseph Doherty f087f01d2b fix(abcip): classify every controller refusal as per-tag Bad, keep the runtime
H1/L4: one classification (AbCipRefusals.IsDeviceRefusal) of the statuses
that prove the controller answered - NotFound, OutOfBounds, NotAllowed,
TooLarge, TooSmall, TypeMismatch, Unsupported - shared by the data path, the
Test-Connect probe (IsReachableStatus widened) and the native runtime.

- A refused read/write (thrown or as a status word) is per-tag Bad with a
  once-per-tag Warning, keeps the runtime (no evict, so no lease/Forward-Open
  churn for a sole lease holder), writes no Degraded and, since the device
  answered, advances LastSuccessfulRead without touching the state.
- A refused cold create records the backoff as reachable, so siblings are no
  longer throttled for up to 30 s by one Arr[999].
- The native runtime completes InitializeAsync on any refusal (it used to on
  TypeMismatch/Unsupported only) and reports it on every later read.
- A refused READ of NotAllowed reads BadNotReadable, not BadNotWritable.
- Test Connect names the refusal ("probe tag refused: OutOfBounds").

0x1C/0x20/0xFF-0x2104 still fold into BadReply inside Driver.Ab.Transport
(Lane K follow-up); they remain comms faults until mapped there.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:07:50 -04:00
Joseph Doherty 12b981dc1b fix(focas): carry macro/parameter numbers to the CNC unnarrowed (R07:H14)
The capability matrix admits #0..99999 on 30i/31i/32i (and anything on Unknown), and
the FOCAS/2 request block has always carried 32-bit arguments, but WireFocasClient cast
every number to short: MACRO:99999 asked the CNC for #-31073 and published that variable
at Good under the authored tag. FocasWireClient's macro/parameter reads, WireMacro.Number
and WireParameter.DataNumber are now int; the request blocks come from MacroRequest /
ParameterRequest builders (byte-identical to the live 31i-B capture for #500).

Same class on PMC: a range whose last byte passes 65535 (Unknown series admits it) is now
refused BadOutOfRange instead of wrapping onto a different 16-bit address.

The wire client is in-process (the other end is the CNC itself), so this is not a
cross-repo contract change. Whether a real 30i honours cnc_rdmacro Arg1 > 32767 is the
hardware userGate.

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:07:42 -04:00
Joseph Doherty 4083b4aada fix(twincat): warn once per bad tag; a device refusal is not a comms fault (P10)
CompleteRead logged a Warning for every non-Good status on every poll and
latched Degraded for all of them. Now a bad tag warns once per bad episode
(re-armed when it reads Good, and on re-initialise), a comms-class status
(BadCommunicationError / BadTimeout) degrades with LastSuccessfulRead
preserved, and any other status - the device answered and refused this tag -
advances LastSuccessfulRead with no state change. Read-path exception
handlers now go through MarkCommsFault, so they never downgrade Faulted.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:07:03 -04:00
Joseph Doherty 92171a01ec fix(ablegacy): build-and-swap device/tag tables; teardown safe vs Dispose and abandoned init
Review 07 H13: _devices and _tagsByRawPath were plain dictionaries cleared and
refilled in place while the retained poll loop, the actor's GetHostStatuses and
server reads enumerated them. Both are now built locally and published by
reference under a lifecycle lock; teardown swaps in empty tables.

Lane A contract: ShutdownAsync and Dispose each retire a disjoint set of device
states, so Dispose running concurrently with a still-joining ShutdownAsync is
safe. A lifecycle epoch makes an InitializeAsync that the runtime abandoned
(overlapping the next ShutdownAsync) refuse to publish its tables, start probe
loops or write health. A runtime cached on an already-retired device state by an
in-flight read is disposed instead of leaked, and late reads/writes no longer
overwrite shutdown's health. Per-key semaphores are no longer disposed under
callers that may still hold them.

Refs #738
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:04:51 -04:00
Joseph Doherty 3e0606af58 Merge lane E: script runtime safety + scripted alarms (#734)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:03:54 -04:00
Joseph Doherty ecdd6dbecb fix(twincat): native/probe evidence advances LastSuccessfulRead; probe failures degrade (H12)
In native-notification mode LastSuccessfulRead never moved past the init
stamp, so one Degraded write on a connection older than 90 s latched the
comms-loss watchdog into a full reconnect (every notification on every device
deleted and re-registered). Conversely an unreachable AMS router was swallowed
by the probe and never left Healthy.

- a native push and a successful probe now stamp LastSuccessfulRead (throttled
  to one health record per 250 ms) and restore Healthy unless a device is still
  probe-unreachable;
- 3 consecutive failed probe ticks write Degraded with the stamp preserved;
- a NotSupportedException write is BadNotSupported with no health write (#611);
- a reconnect replay the PLC refuses republishes the refusal's status without
  degrading (it used to report BadCommunicationError + Degraded on every
  reconnect for a correctly-isolated type-mismatch tag).

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:03:28 -04:00
Joseph Doherty a32894e93b fix(focas): build-and-swap the device table; make the lifecycle overlap-safe (R07:H13)
_devices was a plain Dictionary cleared and refilled in place by InitializeAsync /
ShutdownAsync while the retained poll engine, the fixed-tree and alarm loops and the
actor's health tick enumerated it ("Collection was modified"). It is now built whole and
published by reference swap, and a release swaps it out to empty before tearing it down.

Lane A's runtime overlaps lifecycle calls, so the driver now tolerates both shapes:
- a lifecycle generation is bumped by every Initialize/Shutdown; an initialise that finds
  itself superseded (abandoned by the runtime, then a shutdown ran) publishes nothing,
  starts no loops, writes no health and throws OperationCanceledException;
- a shutdown overlapping another (dispose beside a still-running shutdown) waits for the
  first's teardown via a release tail instead of returning on an empty table;
- released devices are Retired: EnsureConnectedAsync refuses to reconnect them, and a
  connect that completes after retirement disposes what it opened. Clients are disposed
  under the device ConnectGate (bounded).

Refs #740
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:03:08 -04:00
Joseph Doherty 20cb2a52d6 fix(abcip): build-and-swap device tables, volatile health, async-only dispose
H13: _devices/_devicesByRoutingKey are published by reference swap under a
lifecycle lock and never mutated after publish, so the retained poll engine,
discovery and the actor's GetHostStatuses pull never read a table mid-Clear.
Every retire (init's own release, shutdown, dispose) takes the table it
replaces, so ShutdownAsync racing DisposeAsync tears each state down once, and
a generation check makes an InitializeAsync overtaken by a shutdown publish
nothing (it throws OperationCanceledException) instead of resurrecting probe
loops behind the shutdown. Probe join is one bounded 5 s wait independent of
the caller token.

H21: _health is volatile and read-modify-write updates go through a CAS.

F10: the sync Dispose() (sync-over-async) is removed; DisposeAsync is the
complete, idempotent teardown the runtime calls after ShutdownAsync.

Refs #737
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:02:00 -04:00
Joseph Doherty 17f4bf6d33 fix(twincat): isolate native registrations per tag and per device (H11)
Native subscribe was all-or-nothing: any registration failure other than a
type refusal disposed every registration and rethrew, so one typo'd symbol or
one unreachable PLC in a multi-device driver served zero data, churned
register/delete every reconcile tick, and left health Healthy.

Now each registration fails on its own: a refusal (the device answered: symbol
not found, PLC not ready, type mismatch) publishes its status with no health
write; any other per-tag failure publishes BadCommunicationError and degrades;
a connect failure marks the device unreachable for the rest of the call and
stores its tags as dead intents. Every failed tag is kept as a dead intent that
the probe-tick retry and the reconnect replay re-register. The ADS client now
classifies a failed symbol resolve / AddDeviceNotification by status instead of
throwing InvalidOperationException for all of them. Cancellation still tears
the partial subscription down.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:00:38 -04:00
Joseph Doherty a85d1c069a fix(scripting): quarantine a refusal behind a stuck orphan (fix round 2)
Ruling refinement on I1. Only timeouts are strikes, so a never-returning
while(true), which times out once and is then only refused, was never
quarantined, and that defeated E1. A refusal now quarantines immediately
when the orphan it was refused behind has been executing for longer than
ScriptingOptions.StuckOrphanQuarantineFactor x the script's timeout
(default 10, so 20 s at the 2 s default; 0 or less disables it). It uses
the same quarantine path: one Error, otopcua.script.quarantined with
reason=stuck, Bad, lifted on source change or redeploy. A young orphan
(a short overrun) is still a plain refusal.

The guard stamps the run's start (Stopwatch timestamp) when the pool
picks it up and clears it when the run really finishes.

Tests: the 30-trigger while(true) regression (paced) now ends
quarantined with reason "stuck", still with exactly one orphan and a
healthy sibling. Positive control: a 2x overrun is refused, not
quarantined, and evaluates again once the orphan completes. A factor of
0 disables the rule. Docs (ScriptEditor "Runtime safety", VirtualTags)
and the seam remarks are updated.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:59:10 -04:00
Joseph Doherty de21a54949 fix(twincat): publish BadNodeIdUnknown for unresolvable native-subscribe references (U12)
The native subscribe path silently skipped a reference it could not resolve
(unknown RawPath, or a tag routed to an unconfigured device), leaving the node
on whatever status the server seeded. It now publishes one BadNodeIdUnknown
snapshot, matching the poll path.

Refs #739
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:57:01 -04:00
Joseph Doherty dc3323fcca fix(host): publish the known-id set and its stamp atomically; never write the shared guard options
BoundedApiKeyStore kept the set (volatile) and its DateTimeOffset stamp (non-volatile, 16 bytes) in separate
fields, so a reader could pair a new set with an old stamp or read a torn stamp; they are now one immutable
snapshot behind a volatile reference. An inner store that cannot list ids now switches the set off with a
per-instance flag instead of writing KnownIdSet = false into the shared ApiKeyStoreGuardOptions.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:51:28 -04:00
Joseph Doherty 8a4cf8b823 Merge lane F: Galaxy driver teardown, probing, knobs (#735)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:50:53 -04:00
Joseph Doherty 896aaa2b02 fix(controlplane): a PSK secret deleted from the store revokes the key; bound refresh concurrency
RefreshAsync now tells "the store answered: no such secret" (revocation - drop the refreshed key AND the
lazily cached one) apart from "the read failed" (keep the previous key). Store reads fan out through
Parallel.ForEachAsync capped at MaxConcurrentRefreshReads = 8 instead of one read per cluster at once.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:49:56 -04:00
Joseph Doherty 2adeacc655 fix(controlplane): keep the revision hash byte-identical to the pre-refactor composer
The one-snapshot refactor re-sorted every collection in memory with an ordinal comparer, while the old
composer's SQL ORDER BY used the database's case-insensitive collation; mixed-case keys therefore re-hashed
an unchanged configuration (spurious drift fleet-wide, a "new" revision on a no-op redeploy). The snapshot
factory now reads every composed table ORDER BY its natural key server-side and Compose serialises in the
order given. Pinned byte-for-byte against a verbatim replica of the old composer on the real SQL Server.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:48:40 -04:00
Joseph Doherty 7c7ecff974 Merge branch 'master' into remed/lane-B 2026-10-22 13:47:37 -04:00
Joseph Doherty 0d367590db test(galaxy): isolate process-wide telemetry tests; bounded stream-cancel wait
GalaxyTelemetryTests (ActivityListener) and GatewayGalaxyDataWriterFailClosedTests
(MeterListener, filtered only by meter and instrument name) observe every Galaxy span or
measurement in the process. They now run in a collection with parallelization disabled
(F-FU6), because a parallel driver test, or its background probe seed, polluted their counts.
The stream-cancel assertion in GalaxyDriverShutdownTests goes back to a bounded wait: the
instantaneous IsCompleted check introduced in round 1 was racy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty 3e60a925dc fix(galaxy): background probe seed; generation-checked registrations
Review fix round 2 for lane F:
- N1: the init platform seed no longer runs inline under the init token. Galaxy's Tier-A
  Initialize budget is 5 s, and a cold gateway DiscoverHierarchy can spend all of it, so the
  OperationCanceledException rethrow failed init, opened the Initialize breaker and cost a full
  rebuild per retry. InitializeAsync now only resolves the platform source (no network) and
  queues a background seed tied to a per-runtime CTS that the teardown cancels. A result that
  lands after its runtime was torn down is discarded.
- N2: the seed worker serves the runtime that is live when each pass STARTS, so a request made
  under a new runtime while a stale pass is in flight always gets a pass of its own.
- N3: SubscribeAsync and the subscribe-once read capture the runtime generation before
  SubscribeBulk and register only if it is unchanged, atomically with the teardown's registry
  Clear (_registryGate). On a mismatch they release the handles and fail the call. Re-subscribe
  Rebind uses the same gate, and the production rebuild clears the registry unconditionally.
- PerPlatformProbeWatcher re-checks its stop after the SubscribeBulk await, atomically with
  the record and publish. A pass that straddles StopAsync releases the handles it acquired
  instead of writing the probe entry back into a cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:47:34 -04:00
Joseph Doherty 9466b8b242 fix(controlplane): close a lost dispatch from the in-flight check; guard terminal rows; blob-free status writes
I7: a Dispatching row committed after the coordinator's PreStart whose Tell never arrived had no owner and
refused every later deploy. The in-flight check (now fully off the admin singleton's thread, serialised by the
actor-local compose flag alone) closes a Dispatching row with zero NodeDeploymentState rows that is past the
apply deadline as TimedOut (optimistic on RowVersion) and proceeds.

HandleDispatch ignores a dispatch for a Sealed/PartiallyFailed/TimedOut row, so a late or duplicated dispatch
cannot flip it back. Coordinator status/reason writes attach a key+RowVersion stub instead of loading the
multi-MB ArtifactBlob; proven on the real SQL Server (blob untouched, seal lands).

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:46:54 -04:00
Joseph Doherty f7e7ba0263 fix(controlplane): audit writer drops only already-persisted EventIds, never the whole SQL Server batch
SQL Server's DbUpdateException for one duplicate names every entry in the statement batch, so detaching
ex.Entries dropped the whole flush. On a row-level rejection the writer now asks the database which of the
batch's EventIds it already holds, drops exactly those, re-saves the rest, and falls back to row-by-row only
if a non-duplicate row is still refused. Fresh context + entities per attempt. Pinned against the real
central SQL Server (new SqlServerTestDatabase harness) and a faithful whole-batch interceptor.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:42:29 -04:00
Joseph Doherty ef2eebe6b6 fix(scripting): review minors: configured limit in summaries, dispose gate/CTS, validator, docs (fix round 1)
- ScriptLogRateLimiter.LimitPerWindow. The ScriptLogTopicSink and
  VirtualTagActor summaries now print the configured limit, not
  DefaultLimitPerWindow. Tests assert "limit N/window" for a non-default
  N.
- CalculationDriver.Dispose cancels, then takes _evalGate (bounded 2 s)
  and disposes the gate and _evalCts. The pump and timer ticks enter the
  gate through TryEnterEvalGateAsync, which stops cleanly once the
  driver is disposed. Test: Dispose with a 6 s runaway run in flight
  returns promptly, disposes both, and a late trigger is harmless.
- ScriptedAlarmEngine captures _shutdownToken once and disposes
  _shutdownCts in Dispose when every drained task has completed. A task
  stuck past the grace leaves the CTS to the GC. Lifecycle tests assert
  the disposal.
- DraftValidator: CalculationNoEffectiveTrigger is skipped for a
  change-triggered tag whose scriptId does not resolve. That case is
  already CalculationScriptMissing or CalculationScriptNotFound, and
  there are no reads to count. Both existing tests now assert the
  absence (red before the change).
- docs/VirtualTags.md: System.Threading is denied as a whole namespace,
  with an allow-list; the type list is now accurate (Unsafe,
  RuntimeHelpers, and the ForbiddenMembers on System.Type); the resource
  limits section covers the run guard and the allocation budget.
  docs/ScriptEditor.md: accurate deny-list and a new "Runtime safety"
  section covering timeout, refusal, quarantine after 3 timeouts,
  allocation budget, lifting, and rate-limit summaries that appear only
  on the source's next entry after a burst then quiet.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:41:15 -04:00
Joseph Doherty f327dd6cdb Merge lane A: fix hung-write test fixture (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:58 -04:00
Joseph Doherty f1c52385ed fix(controlplane): retention keep window counts Sealed rows; the newest Sealed blob is never pruned
The window used to count every terminal row, so 50+ failed deploys pushed the last good artifact out of it.
Now the newest KeepSealed (>= 1) Sealed rows keep their blob, failed rows are prunable unless referenced in
the ReferencedDays window. Pinned: 60 TimedOut + 1 old Sealed -> Sealed retained, 60 TimedOut emptied.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:34 -04:00
Joseph Doherty 9e180b72a3 test(runtime): de-flake the hung-write release test; name the real write bound
The fixture wedged EVERY write on one shared token-ignoring release gate, so
the later same-tag write was also hung, and the 300 ms pessimistic bound
abandoned it while the test was still polling before ReleaseWrites(). Its
reason was "write did not complete within" (the bound), not "queued for" (the
queue budget), and it had already entered the driver, so the tag had been
released. The release path is correct. Only the first write now hangs, and
later writes complete at once.

The abandon reason formatted the bound with TotalSeconds:F0, which printed a
300 ms bound as "0s". It now uses milliseconds, and the test asserts it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:33 -04:00
Joseph Doherty 88268ea3a6 fix(host): OPC UA endpoint probe is Ready-only on a fused admin+driver node
Coordinator ruling (I6): until ServiceLevel consumes IOpcUaEndpointState (F-C1), the probe must not sit on
the Active tier of a fused node, because Traefik routes the AdminUI on /health/active. Driver-only nodes keep
it on Ready + Active; admin-only nodes do not register it. Pinned per role.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:38:03 -04:00
Joseph Doherty c0a0e7aa81 fix(security): audit actor falls back to HttpContext when the server auth-state provider has no circuit
ServerAuthenticationStateProvider throws InvalidOperationException from GetAuthenticationStateAsync in any
scope where no circuit set its state (minimal-API requests, hosted-service scopes). The accessor now treats
that as "no circuit identity" and falls back to the HTTP principal. Pinned with the real provider on both
sides (unset -> HTTP fallback; set -> circuit principal wins).

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:36:51 -04:00
Joseph Doherty 3ddb6876db fix(security): first-party JwtBearer with HS256 pinned; /auth/token mints for a cookie session only
The hand-rolled bearer handler is replaced by AddJwtBearer (Microsoft.AspNetCore.Authentication.JwtBearer
10.0.7, pinned on the same servicing line as the other AspNetCore packages). Validation parameters come
from JwtTokenService.BuildValidationParameters, which now pins ValidAlgorithms = HS256 for both the
bearer scheme and TryValidate; MapInboundClaims is off so the minted Role/zb:username keys resolve.
/auth/token requires the cookie scheme, so a bearer can no longer mint a fresh token for itself.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:35:54 -04:00
Joseph Doherty ad6be56850 fix(runtime): bound DependencyMux last-value retention to replay-opting refs (fix round 1 I2)
DependencyMuxActor retained the last value of every ref ever registered,
and never removed the entries. The continuous-historization recorder
registers every historized ref, so on a historizing node the map grew
with the historized address space and never shrank. The class remark
claimed a bound that did not hold.

- Retain only refs that at least one subscriber registered with
  ReplayRetained: true names (VT actors, the Calculation mux adapter, the
  scripted-alarm host). A non-opting subscriber is still delivered to but
  retains nothing.
- When a ref's last replay-opting subscriber leaves, its value is kept
  for a grace (default 2 min, Props(retentionGrace)) so a restart
  (old adapter gone, new one not yet registered) still replays. A
  periodic sweep then drops it along with the entry. Entries with no
  subscriber and nothing retained are removed.
- New SourceRemoved(DriverInstanceId) evicts the values a removed driver
  produced, so its last Good is never replayed. The mux records the
  publishing driver per retained ref. No production sender exists yet:
  DriverHostActor is outside this lane (follow-up).
- Remark rewritten. Tests: retention bounded to replay-opting refs,
  release after grace (with an in-grace replay as the positive control),
  SourceRemoved eviction, and the restart-replay scenario still green.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:31:41 -04:00
Joseph Doherty 4d17b346c6 Merge branch 'master' into remed/lane-C 2026-10-22 13:31:03 -04:00
Joseph Doherty 8a0acc4649 Merge master into remed/lane-F
Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:15 -04:00
Joseph Doherty 77ca10aa37 test(galaxy): de-flake writer-lane and supervisor tests; align EffectiveTimeout doc
The writer concurrency fake parks on its release signal (the 300 ms delay could release the
gate early on a starved pool), the supervisor test's cross-thread hold flag is Volatile, and
the EffectiveTimeout doc now states the 1 s floor the code and tests implement.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 3456756cd1 fix(galaxy): seed the probe set at init, refuse pumps mid-teardown, bound the close
Review fix round 1 for lane F:
- The ScanState probe watcher is seeded at InitializeAsync from a platform-only
  DiscoverHierarchy query (IGalaxyPlatformSource / GatewayGalaxyPlatformSource over the owned
  repository client) and re-seeded on every deploy-watcher rediscovery. The runtime never
  calls DiscoverAsync on a live driver, so per-platform host status was unreachable in
  production. A failed fetch logs a Warning and never fails init; re-seeds coalesce so a
  deploy during an in-flight pass still gets a pass that starts after it.
- EnsureEventPumpStarted refuses while the runtime is stopping, BEFORE reusing the live pump
  (that pump belongs to the dying subscriber), and the production rebuild disposes any stale
  pump before wiring the new subscriber.
- TeardownRuntimeAsync closes the MX session under the caller's token linked with a 5 s
  SessionCloseDeadline, so a blackholed gateway cannot hold DriverInstanceActor.PostStop for 30 s.
- A runtime generation counter, bumped by every teardown, is re-checked before each Rebind
  in ResubscribeAllAsync, so a replay-gap re-snapshot that straddles a teardown cannot
  resurrect handles into the cleared registry.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 05a0e51fc1 fix(galaxy): GalaxyMxSession.CloseAsync(ct) bounds the gateway CloseSession
The SDK's DisposeAsync closes with no token, i.e. under the 30 s DefaultCallTimeout, so a
blackholed gateway parked the driver's teardown for the whole of it. CloseAsync(ct) runs the
close under the caller's token, drops the channel on a cancelled/failed close (the gateway
reaps the session on its idle lease) and leaves the session re-openable. A close followed by
the final dispose closes the gateway session once.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:11 -04:00
Joseph Doherty 0117e1501b fix(opcua): trace every rebuild path; align history tests with the B7 contract
B6 moved the address-space-rebuild span into ApplyPlan, so the two rebuilds
that complete synchronously on the actor thread (no applier -> raw-sink
rebuild; no artifact bytes -> held) stopped emitting it
(OtOpcUaTelemetryHookTests.RebuildAddressSpace_starts_an_address_space_rebuild_span).
Both now open their own span.

B7 (late HistorianDataSource / HistorianDataUpdater re-evaluate the
capability bits) reversed three tests that pinned the old "fixed at
materialisation" contract. Re-expressed against the new one, end to end:
- events: a folder promoted without a source is unsupported, then served
  once a source is wired (source reached);
- in-process HistoryUpdate: clearing the updater retracts HistoryWrite and
  the old updater is never reached;
- over the wire: a late updater lights HistoryWrite and an insert reaches
  it; a reset retracts the bit and the insert is BadHistoryOperationUnsupported.

Refs #732
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:29:07 -04:00
Joseph Doherty 6aa5edaf30 fix(scripting): only timeouts are quarantine strikes; alarm Bad annotation once per bucket change (fix round 1 I1, I3)
- I1 ScriptRunGuard: a run refused because the previous one is still
  executing is a Bad publish plus otopcua.script.run_refused, and no longer
  a quarantine strike. Only a timeout (and an allocation breach) counts;
  a refusal neither adds to nor resets the timeout streak; a completed run
  (value or throw) resets it. On a 1 Hz trigger with a 2 s budget a single
  4 s overrun is one timeout plus two refusals and must not quarantine.
  Tests updated across the guard, the alarm engine, the Calculation
  evaluator and the VT evaluator, plus
  Refusals_neither_count_toward_nor_reset_the_timeout_streak.
- I3 ScriptedAlarmEngine: the quality bucket is tracked only after the
  predicate completes (or in the inputs-not-ready path), so the tracked
  bucket is the last one annotated. A refused or quarantined predicate
  used to re-track the input bucket Good before the attempt and then flip
  it Bad, emitting QualityChanged on every push. The run-guard tests now
  count emissions (exactly 1 across N pushes, refused and quarantined
  cases), with a healthy sibling's dispatch count as the positive control
  that every push was processed. Both were red before the fix.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:26:12 -04:00
Joseph Doherty 909b061d3a Merge branch 'master' into remed/lane-E 2026-10-22 10:48:41 -04:00
Joseph Doherty be22b48b1d feat(opcua): advertise the historian raw cap as MaxReturnDataValues on source sync
Lane H1 follow-through: SyncAdvertisedAggregates now also calls
HistoryPaging.AdvertiseMaxReturnDataValues(Server, source
.MaxRawValuesPerRead) so HistoryServerCapabilities/MaxReturnDataValues
(i=11273) follows the historian source on every assignment (the Null
default advertises 0 = no limit); a false return logs a Warning (the cap
is enforced by HistoryRead regardless).

Test: NodeManagerMaxReturnDataValuesSyncTests reads i=11273 back through
the SDK master node manager after setting a capped source (7000, 250)
and after resetting to the Null source (0).

Refs #732 #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty 8a1116e658 fix(opcua): fit the raw end bound against the page's effective cap
H1 fuzz finding (routed to Lane B because AppendEndBoundAsync lives in
OtOpcUaNodeManager.cs): a bounded HistoryRead-Raw page sized BELOW
NumValuesPerNode by the backend's row ceiling (RawPageLimit.Plan) took
the end bound whenever its data fitted, because the fit was judged
against NumValuesPerNode instead of the page's effective cap — a full
ceiling-sized page came back with cap + 1 values. The bound is now
fitted against the page allowance (the ceiling-adjusted data cap plus
the start bound a first page already carries); when the page is full
the bound goes to the next page through the continuation point.

Test: NodeManagerHistoryReadBoundsTests
.A_full_ceiling_sized_page_carries_the_end_bound_on_the_next_page
(ceiling 4 ⇒ 3 data rows; first page = start bound + 3 with a CP, second
page = the end bound alone; control: no ceiling ⇒ one page of 5).

Refs #732 #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty d90f35b6be fix(opcua): refresh historian capability bits on late source/updater assignment (B7)
Review 01 U3. EnsureFolderIsEventNotifier granted the EventNotifiers
.HistoryRead bit (and registered the event-history source) only when a
real HistorianDataSource was wired at promotion time, and EnsureVariable
composed the AccessLevels.HistoryWrite bit from whether a
HistorianDataUpdater existed at materialise time — so an assignment
after the first apply left existing folders without event history and
existing nodes refused by the SDK's HistoryUpdate admission until a full
rebuild, and a source reset to Null never took the bit back.

The two setters now re-evaluate: HistorianDataSource walks every promoted
folder (bit + source registration, both directions), HistorianDataUpdater
recomposes the AccessLevel of every historized variable (Writable read
back off the node's own CurrentWrite bit). The Host's boot ordering is no
longer load-bearing.

Tests: NodeManagerCapabilityRefreshTests (late source lights the bit on
an already-promoted folder and a reset takes it back, live-event
subscription untouched; late updater grants HistoryWrite to historized
nodes only and a reset takes it back, authored Writable preserved).

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty deaa31479b perf(runtime): parse + plan a deploy off the pinned OPC UA thread (B6)
Review 02 P5. OpcUaPublishActor ran the whole apply — DeploymentArtifact
.Open, ParseComposition, AddressSpacePlanner.Compute and the sink
materialisation — on the opcua-synchronized-dispatcher, so a large
artifact serialised every live value batch (and, because the node
manager lock is the SDK's, every client Publish) behind the parse.

StartRebuild now runs Open + ParseDataPlaneAuthority + ParseComposition
+ Compute in Task.Run and pipes a PlanReady (or PlanFailed) back to Self;
only ApplyPlan — the sink passes — runs on the actor thread. Rebuilds are
applied strictly in arrival order (a plan is a diff against _lastApplied,
which only the apply advances): one in flight, later ones queue in
_pendingRebuilds while value/status batches keep flowing. The apply is
closed (CompleteAddressSpaceApply) and the RebuildCompleted reply sent on
every path, PlanFailed included. The bytes-less ConfigDb fallback
(LoadArtifact / LoadLatestArtifact) is deleted: it had no production
producer and would have blocked the pinned thread on central SQL; the
dbFactory Props/ctor parameter is kept for wiring compatibility (F4).

Tests: OpcUaPublishActorOffThreadPlanTests — a value batch sent right
after a 50k-tag rebuild lands before that rebuild's first materialise
call (positive control: all 50k tags still materialise); back-to-back
rebuilds apply in arrival order (the second diffs against the first);
unreadable bytes answer degraded with nothing torn down. The rebuild /
apply-failure suites now hand the actor the artifact bytes every
production producer carries (SealArtifact records them; Rebuild(dep)).

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty 118933d5d5 refactor(opcua): delete the dead v3 equipment-tag materialisation (B5)
Review 01 U2. The composer never emits EquipmentTags in v3 (device values
live in the Raw subtree and reach equipment as UnsReferenceVariables), so
the equipment-tag plan diff sets (Added/Removed/ChangedEquipmentTags,
EquipmentTagDelta), the planner's diff over them, the classifier's
TagDeltaIsSurgicalEligible and its Rebuild clause, the applier's
MaterialiseEquipmentTags pass, surgical ChangedEquipmentTags loop,
removed-tag teardown loop and add-announcement term, and the publish
actor's equipment-tag pass were unreachable, maintained dead weight.
IsEmpty keeps its semantics (an otherwise-empty plan is still Empty).

Kept, deliberately: AddressSpaceComposition.EquipmentTags and the
EquipmentTagPlan record — Runtime's DeploymentArtifact (outside this
lane) still assigns the empty set and the artifact parity tests assert
it; follow-up F2 removes both together.

Tests: the equipment-tag vehicle tests for PureRemove / AddRemoveMix /
the rebuild ratchet are re-expressed over raw tags (same applier paths);
the equipment-tag surgical and classifier tests are deleted (their raw /
UNS-reference equivalents already exist); SubscriptionSurvivalTests now
materialises its tags through MaterialiseRawSubtree.

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty 1578efd4a9 fix(opcua): write-path robustness, injected clock, configurable quotas (B4)
Review 01 L3/H5/H4/P6/P4.

- L3/H5 OnEquipmentTagWrite's dispatch is factored into DispatchDeviceWrite:
  a gateway that throws synchronously is refused inline with
  BadDeviceFailure (it used to escape the handler under the SDK Lock), and
  a faulted/cancelled dispatch task has its exception observed and logged
  and carries the real reason into the revert/audit event instead of
  surfacing as UnobservedTaskException at finalization.
- H4 every timestamp the manager stamps (condition quality, timed-shelve
  expiry, method-transition events, carry-over, revert, revalidation)
  reads the injected Clock; RemainingShelveMs takes the clock reading.
- P6 OpcUa:Quotas:* (OpcUaQuotaOptions on OpcUaApplicationHostOptions):
  Min/MaxRequestThreadCount, MaxQueuedRequestCount with the former
  hard-coded 5/100/200 as defaults, and the TransportQuotas members as
  nullable overrides over the SDK defaults.
- P4 the SDK line exposes IHistoryReadAsyncNodeManager but CustomNodeManager2
  offers no async virtual to override (adopting it means re-implementing
  the base dispatcher), so the lever is the admission cap:
  MaxConcurrentHistoryReadBatches defaults to clamp(2*ProcessorCount, 4, 16).
  The Host still overrides it from ServerHistorianOptions (default 16) —
  follow-up F3.

Tests: NodeManagerWriteDispatchTests (sync throw refused; faulted task
reverts; success leaves the value; clock-stamped revert),
OpcUaApplicationHostQuotaTests.

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:32 -04:00
Joseph Doherty dca822dc59 perf(opcua): de-quadratic the lock-time teardown paths (B3)
Review 01 P1/P2/P3/P5 — work done under the manager-wide SDK Lock on the
pinned publish thread, where every Read/Browse/Publish waits.

- P1 RebuildAddressSpace detaches the shared root's top-level children
  once instead of every leaf from its parent (NodeState.RemoveChild is a
  linear scan + RemoveAt, ~k²/2 compares per k-child folder); root
  notifiers are cleared as a list instead of one O(F) RemoveRootNotifier
  each; ForgetPredefinedSubtree / TombstoneMonitoredSubtree are iterative
  with one child buffer.
- P2 RemoveEquipmentSubtree walks a parent→children index maintained by
  EnsureFolder/EnsureVariable/MaterialiseAlarmCondition (keyed by the
  parent id the caller named) instead of prefix-scanning six maps per
  removed equipment; a surviving condition's wiring into a removed folder
  is found through the folder's own notifier list; descendants are not
  detached from a folder that is itself being discarded.
- P3 LowerAllEventsMonitoring(condition) replaces the full O(C) reconcile
  on RemoveAlarmConditionNode / RemoveEquipmentSubtree / the kind-swap.
- P5 AlarmNotifierWiring.Folders is a reference-identity HashSet and an
  already-wired pair skips the SDK's linear AddNotifier dedup.

Benchmark (NodeManagerLockTimeBenchmarkTests, M-series Mac under load,
three runs each): rebuild of a 10k-tag folder 75–150 ms → 5–6 ms;
removing 200 equipment (250 signals each) from a 50k-node space
471–681 ms → 58–64 ms; all-events bookkeeping visits for 10 condition
removals over 50 conditions 900 → 10. The two timing legs assert < 1 s;
the P3 leg asserts the visit count (machine-independent).

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:31 -04:00
Joseph Doherty eaac65286d fix(opcua): tombstone MonitoredNode2 on every node removal (B2a)
Review 01 L2. The SDK keys its fan-out lists by NodeId and its component
cache hands back the CACHED NodeState for a NodeId it already holds, so a
rebuild or a surgical remove+re-add left a NEW item on the same NodeId —
from any session — bound to the DEAD node: created Good, fed its stale
initial value, never again, until every session had deleted its items.

Every removal path (RebuildAddressSpace, RemoveVariableNode,
RemoveAlarmConditionNode incl. the condition's instance-scoped children,
RemoveEquipmentSubtree) now queues BadNodeIdUnknown to the node's data
items, drops the MonitoredNodes entry, releases each item's component
cache reference and forgets the _liveItems/_eventItems entries. Items
the ACL pass stood down (on no fan-out list) are swept by NodeId. A later
client delete of an orphaned item is completed by the DeleteMonitoredItem
override / the SubscribeToEvents unsubscribe arm instead of answering
BadMonitoredItemIdInvalid or NRE-ing in MonitoredNode2.Remove. The
revalidation restore re-resolves the node by NodeId and no longer
re-attaches to Handle.Node blindly. Timestamps now come from the injected
Clock (UtcNow).

Tests: StaleMonitoredNodeWireTests (real SDK, three legs: rebuild,
surgical remove+re-add, second session; each keeps the old item alive,
proves the new item receives a value written after its creation, the old
one gets BadNodeIdUnknown, and its delete succeeds — all three fail with
the tombstone neutered), NodeManagerStaleMonitoredNodeTests (stood-down
item swept on surgical remove and rebuild; sibling removal leaves it
restorable).

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:31 -04:00
Joseph Doherty 00300d0f60 fix(opcua): no write ever creates a node; cross-kind id guard (B1)
Review 01 L1/L4/H2. WriteValue/WriteValues are non-creating (an unknown
id is skipped, counted and Debug-logged, as WriteStatuses always was), so
a driver publish racing a surgical remove can no longer resurrect the id
as a flat BaseDataType phantom under the shared root — nor hand a later
re-add of the same tag the phantom's parent/DataType/writability through
EnsureVariable's early return. WriteAlarmCondition for an unmaterialised
id is a logged no-op instead of minting the legacy bool[2] placeholder
(AddPredefinedNode is AddOrUpdate, so it replaced the SDK lookup entry of
any folder sharing the id). The applier no longer writes a condition
state against every removed equipment id (a folder, never a condition).
EnsureFolder/EnsureVariable/MaterialiseAlarmCondition refuse an id already
held by another node kind in the same realm. CreateVariable is deleted.

Tests: NodeManagerPhantomNodeTests (publish racing a remove leaves no
node; the re-add gets the authored shape and is live; unknown alarm id
is a no-op; the guard in all three directions, realm-qualified); the
sink / batch-write / write-statuses tests now declare their variables.

Refs #732

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:47:31 -04:00
Joseph Doherty d8c39933ed Merge lane A: runtime driver lifecycle (#731)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:44:22 -04:00
Joseph Doherty 7ae74ef147 fix(runtime): pessimistic write bound, incarnation-stamped write completions (fix round 2)
- RunWriteAsync wraps the invoker call in WaitAsync(writeTimeout) (Props,
  default 10 s = 2x the 5 s write cap): a token-ignoring hung write now
  answers BadTimeout, releases its slot and tag, and its orphan outcome
  is observed and discarded - later writes to that tag reach the device.
- WriteCompleted carries the launching incarnation (from the process-wide
  generation source); a late completion from a previous incarnation after
  a supervision restart is ignored instead of freeing the new
  incarnation's slot/tag.
- Test: a hung token-ignoring tag-a write is abandoned and a later tag-a
  write reaches the device while the abandoned one is still inside it.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:43:45 -04:00
Joseph Doherty d422eb747b Merge lane H1: HistoryRead raw cap (#736)
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:54 -04:00
Joseph Doherty 3e62683517 chore(historian): review cleanups for the raw cap
- GatewayHistorianDataSource.ReadRawWindowAsync keeps the cap + 1 probe budget
  (budgetCap) separate from the client's cap: a strict-ceiling or crowding
  refusal now reports Limit = the server cap that applied and advises setting
  NumValuesPerNode at or below it (it said "lower NumValuesPerNode" with
  Limit = cap + 1 to a client that sent 0).
- Fuzz: the oracle-source over-cap assertion was vacuous (cap 0 never counts);
  it now checks no reverse window reached the forward-only shape. Class docs
  name the post-floor cap set (3 ... 1000) and distinguish NumValuesPerNode
  1 / 2 from the smallest server cap.
- HistoryPaging.IsFullPage doc says it judges the REQUEST cap (page + ties +
  lookahead), which is finite under any limit; AdvertiseMaxReturnDataValues
  swallows any exception (returns false for the caller to log), as its doc
  promised.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:41:37 -04:00
Joseph Doherty e04156d8f7 fix(host): refuse unknown API-key ids in memory and budget store misses; document the mesh gate's boot window
BoundedApiKeyStore's negative cache was keyed by key id, so a flood of DISTINCT well-formed
otua_<random>_x credentials missed it every time and cost one SQL query per attempt under the
SDK's SessionManager-wide lock, no credential needed (R03:3.3).

- IApiKeyStore.ListKeyIdsAsync (default null = unsupported; ConfigDbApiKeyStore lists every
  ApiKeyId): the bounded store keeps a known-id set refreshed every StoreCacheSeconds
  (single-flight, failed refresh keeps the previous set) and refuses an unknown id without a store
  read ({outcome=unknown_id}).
- A process-wide miss budget (token bucket, Security:ApiKeys:MissBurst=100 /
  MissRefillPerSecond=20, ApiKeyStoreGuardOptions) bounds lookups that reach the store and miss
  while no set is available; a hit refunds its token. Per-endpoint accounting needs the
  OpcUaServer impersonation seam to carry the client address — follow-up.
- docs/security.md: the new knobs, and the known-cluster gate's documented fail-open window at
  central boot (R03:4.2) plus the async PSK refresh (R03:3.4).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:39:12 -04:00
Joseph Doherty a01f5e6705 build: pin patched System.Security.Cryptography.* / SignalR to clear fresh-restore NuGet audit
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:38:07 -04:00
Joseph Doherty 953bd27533 fix(runtime,adminui): per-tag write ordering, pessimistic init bound, unique generations, PostStop budget (fix round 1)
- Writes: a second write to a tag already inside the driver waits in the
  queue behind it (_tagsInFlight) while other tags still run under the
  cap of 4; queued writes are stamped and failed BadTimeout at dequeue
  once older than the 8 s host Ask budget (Props queuedWriteBudget)
  instead of being written late.
- Initialise attempts are bounded pessimistically (initializeTimeout,
  default 30 s = 2x Tier C) on top of the invoker's token-only timeout:
  an overrun is logged as abandoned, reported InitializeFailed so the
  retry loop continues, and its late result is dropped by generation.
  Generations come from a process-wide Interlocked counter so a prior
  incarnation's late result cannot collide (F6).
- PostStop shares one shutdownTimeout budget across shutdown + dispose.
- TelemetryDialSupervisor cancels the establishment deadline on
  StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside
  the try (a malformed node degrades to "unavailable"); the diagnostics
  page labels the column "Last health publish".
- Tests: same-tag serialisation with a distinct-tag positive control;
  stale queued write BadTimeout; token-ignoring hung init -> later
  attempt connects; stopwatch assertion demoted to a ceiling.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:36:30 -04:00
Joseph Doherty 63714e430a fix(controlplane): evict retired mesh channels and refresh PSKs off the request path
- MeshGrpcCommandDialer.RetainOnly (IMeshCommandDialer, default no-op): CentralCommunicationActor
  hands the dialer the current endpoint set on every contact refresh, so a renamed host or moved
  port releases its cached GrpcChannel and idle HTTP/2 connection instead of living for the
  process (R03:1.3).
- IMeshPskProvider.RefreshAsync (default = the old InvalidateAll): MeshPskProvider pre-resolves
  every known cluster's key concurrently, bounded like the sync path, and swaps an immutable
  snapshot in atomically; a failed read keeps that cluster's previous key. The actor calls it from
  ContactsLoaded (after SetKnownClusters, off-actor via PipeTo) instead of InvalidateAll, so the
  first ack per cluster per ContactRefreshSeconds no longer blocks a gRPC thread on a 3 s-bounded
  store read or gets denied past the bound (R03:3.4).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:34:39 -04:00
Joseph Doherty 18352442ab fix(controlplane): audit writer drops only the rejected row, not the whole batch
AuditWriterActor documented per-row dedup on the UX_ConfigAuditLog_EventId index but flushed one
statement batch: a single duplicate EventId (any at-least-once producer across a singleton
hand-over) threw for all 500 rows (R03:2.6). SaveChanges now detaches the entries a
DbUpdateException names (2601/2627 logged at Debug as the expected duplicate, anything else at
Warning) and re-saves, bounded by the batch size; a failure naming no entry stays a whole-batch
failure.

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:30:03 -04:00
Joseph Doherty 1bf5ef916c feat(controlplane): deployment artifact retention on the admin singleton
Every deploy stored the whole gzip'd fleet snapshot in Deployment.ArtifactBlob forever (R03:1.2).
DeploymentRetentionActor (admin singleton, proxy-less) sweeps every Deployments:Retention:
IntervalMinutes (60): terminal rows (Sealed / PartiallyFailed / TimedOut) older than the newest
KeepSealed (50) and not referenced by a NodeDeploymentState started/applied within ReferencedDays
(30) get their blob EMPTIED — never deleted, never NULLed: the column stays NOT NULL and every
reader already treats zero bytes as "no artifact" (#485), so no schema migration and no reader
change. In-flight rows are never touched; MaxRowsPerSweep (200) bounds one sweep.

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:27:59 -04:00
Joseph Doherty 4d3fc9c816 fix(runtime): restart-safe alarm host, mux last-value replay, VT log rate limit, Calculation remediation (E4)
- R02:H1 ScriptedAlarmHostActor gains a factory Props
  (Func<(upstream, engine)>) so every incarnation builds and owns its own
  engine; a supervision restart no longer re-creates the actor around a
  disposed engine. The legacy instance-taking Props stays source-compatible
  and is now restart-safe in the weaker sense (PreRestart marks the stop as
  a restart, PostStop then detaches instead of disposing). DriverHostActor
  still spawns through the legacy overload (not owned by this lane — F3).
- R02:H3 OnAlarmCommand runs every engine op under a 10 s token linked to
  the actor's lifetime (DefaultCommandTimeout / Props commandTimeout) and
  logs a Warning on timeout instead of suspending the mailbox on a stalled
  store.
- R02:L3 DependencyMuxTagUpstreamSource.RetainOnly(loaded dep-ref union),
  called after every successful load.
- R02:P6 VirtualTagActor's per-failure ScriptLogEntry is rate-limited per
  tag (ScriptLogRateLimiter, 20/min + one summary); the Bad publish and the
  metric are untouched.
- R07:H6 DependencyMuxActor retains the last (value, status, ts) per
  ever-registered ref (through unregisters) and replays them in one batch to
  a RegisterInterest(ReplayRetained: true) — VirtualTagActor, the Calculation
  mux adapter and the alarm host opt in; the historization recorder does not
  (a replay there would record a stale sample per deploy). CalculationDriver
  records inputs regardless of _running so a replay landing before
  InitializeAsync still satisfies the change-gate.
- R07:P4 CalculationEvaluator negative-caches compile failures by source
  hash (one Roslyn attempt + one Warning per source until redeploy); the
  driver's script-log entry fires on the Good→Bad transition or a change of
  reason, matching the once-per-transition Bad publish.
- R07:H8 array results dedup structurally.
- R07:L2 ShutdownAsync drops the compiled scripts (collectible ALCs).
- R07:U5 DraftValidator rule CalculationNoEffectiveTrigger: a tag with no
  timer whose change trigger is off or whose script has no literal
  ctx.GetTag read can never evaluate.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:27:49 -04:00
Joseph Doherty f72af76e99 fix(security): validate the API-key pepper at start, finish JWT bearer, circuit-aware audit actor
- ApiKeyOptionsValidator (ValidateOnStart on any driver/admin node): a blank or placeholder
  Security:ApiKeys:Pepper fails host start instead of rejecting every key after a green boot (R03:2.3).
- JWT is finished and opt-in by key (R03:2.4): with Security:Jwt:SigningKey set, a Bearer scheme
  (JwtBearerAuthenticationHandler over JwtTokenService.TryValidate / BuildValidationParameters, fail
  closed) accepts the tokens /auth/token mints; the default scheme is a cookie-or-bearer policy scheme
  and the fallback policy names it, so /api/* and every RequireAuthorization endpoint take a browser
  session OR a machine token. JwtOptions are validated at start (JwtOptionsValidator). Without a key,
  /auth/token is not mapped (404, not the former constructor 500) and no bearer scheme exists.
  A direct AddJwtBearer needs Microsoft.AspNetCore.Authentication.JwtBearer in
  Directory.Packages.props (not this lane's file) — the handler is a drop-in to swap.
- AuthenticationStateAuditActorAccessor is the registered IAuditActorAccessor: the Blazor circuit's
  principal first, HttpContext as the fallback (R03:2.5). HttpAuditActorAccessor shares its resolver.
- CookieAuthenticationStateProvider (unregistered, fire-and-forget ping loop) is deleted (R03:1.5/2.5).
- /auth/login logs the LDAP exception at Warning before answering 503 (R03:2.7).
- Admin operations reply with a fixed message + correlation id instead of ex.Message (R03:4.1).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:23:14 -04:00
Joseph Doherty 6cbf763719 fix(alarms): engine cache prune + bounded dispose, source Warning, cached dead-letter depth, script-log rate limit (E3)
- R05:1/12 ScriptedAlarmEngine: _valueCache is cleared in LoadAsync (it is
  re-seeded for the referenced paths) so retired tags no longer accrete across
  deploys; OnUpstreamChange no longer writes the cache after Dispose.
- R05:3 ScriptedAlarmEngine.Dispose drains in-flight work for at most 5 s
  (DisposeDrainTimeout), then cancels it through a shutdown token every
  DrainDirtyAsync/ShelvingCheckAsync now runs on, logs a Warning and gives
  one second of grace — a stalled IAlarmStateStore.SaveAsync can no longer
  hang host shutdown. A save that completes inside the bound is still drained
  (positive control test).
- R05:6 ScriptedAlarmSource logs a throwing OnAlarmEvent subscriber at
  Warning (naming the alarm) instead of swallowing it.
- R05:7 LocalDbStoreAndForwardSink keeps a cached dead-letter counter in step
  with every dead-letter/revive/purge path and resyncs it with the queued
  counter on the same cadence; GetStatus no longer runs a COUNT(*) scan.
- R05:9 ScriptLogRateLimiter (20 per source per minute, fixed window, one
  summary entry naming the dropped count when the window rolls) behind
  ScriptLogTopicSink (key = script|vtag|alarm) and ScriptLogCompanionSink
  (key = script name).

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:17:30 -04:00
Joseph Doherty 7e8941fbdc test(historian): paged-vs-unpaged raw-cap fuzz, MaxReturnDataValues advertisement, cap floor of 3
- NodeManagerHistoryReadRawCapFuzzTests: 10k seeds (env OTOPCUA_PAGING_FUZZ_ITERATIONS)
  through a real SDK node manager against wire-faithful fakes of SqlServer / QuestDB /
  Gateway that enforce the backend half of the cap; random series with ties, random
  windows (empty, on/past a tie, between samples, forward/reverse/end-only, bounds),
  N in {0,1,2,small,huge}, cap in {off,3..1000}, tie budget {3,8,65536}. Pages must
  concatenate to the cap-off one-shot answer (checked against the series), terminate
  within values + 2 (+ one empty step per oversized boundary cluster), never exceed
  the cap, never ask the backend for 0 / over-cap; refusals only past the budget.
- HistoryPaging.AdvertiseMaxReturnDataValues writes HistoryServerCapabilities/
  MaxReturnDataValues (i=11273), read back through the SDK master node manager.
- MinMaxRawValuesPerRead raised to 3: at 2 a resumed page has no room (value + its
  boundary tie + lookahead) and every value cost an empty page + CP (seed 15).
- RawPageLimit rows pin the default-cap arithmetic (9 999 first page, 9 998 resumed).

Fuzz findings recorded as follow-ups: a bounded read whose data fits one page
returns cap + 1 (end bound fitted against N, not the cap; node manager, F2);
the Gateway walk-back's cost on a sparse series under a tiny tie budget is
pre-existing (seed 320: 82 reads uncapped, 93 capped).

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:21 -04:00
Joseph Doherty f5579e6ef2 fix(host): make a failed OPC UA SDK start visible and stop the SDK on StopAsync
A failed SDK start (port in use, PKI unwritable, bad OpcUa:* override) was one Error line: the
node stayed /health/ready 200, joined its mesh, could be elected driver Primary and acked deploys
through the Null sink while every client got connection-refused.

- IOpcUaEndpointState / OpcUaEndpointState: the start outcome is a fact the hosted service
  maintains (NotStarted / Started / Failed / Stopped + reason).
- OpcUaEndpointHealthCheck ("opcua-endpoint", Ready + Active) is Unhealthy unless Started;
  registered by AddOtOpcUaHealth(hasAdmin, hasDriver) on driver-role nodes.
- A driver-only node rethrows the start failure (Program.cs's fatal handler exits non-zero so
  the supervisor restarts it); a fused node keeps its AdminUI up but is visibly not ready.
- StopAsync detaches the sinks and then STOPS the SDK (the port closes, clients get
  connection-refused rather than a stale, inert endpoint); DisposeAsync is idempotent with it.
- The ServiceLevel calculator / redundancy election live in Cluster + Runtime (not this lane);
  the fact is exposed for the Wave 3 follow-up to consume.

Tests: port-in-use ⇒ state Failed, probe Unhealthy, real /health/ready and /health/active 503
(positive control: 200 once Started); driver-only rethrows; StopAsync closes the port and
DisposeAsync twice is a no-op; the probe registration is on both tiers for driver nodes only.

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:05 -04:00
Joseph Doherty 59b72fe385 fix(runtime): diagnostics Name stays the string instance id; order-tolerant write-cap test (A6 follow-through)
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:01 -04:00
Joseph Doherty 2e9160317f feat(scripting): deny System.Threading, Type reflection entry points and RuntimeHelpers in the sandbox (E2)
R05:10 — the namespace deny-list covered System.Threading.Tasks but not
System.Threading, so SemaphoreSlim / Monitor / SpinWait / Mutex /
EventWaitHandle could park a script's pool thread past its budget (or touch a
system-global kernel object); System.Type.GetType(string) / GetMethod* /
InvokeMember and System.Runtime.CompilerServices.RuntimeHelpers resolved and
reach a forbidden type without naming it syntactically.

ForbiddenTypeAnalyzer now denies the whole System.Threading namespace with a
type-granular allow-list (CancellationToken, Interlocked, Volatile — the
sync-free primitives), adds a member deny-list for System.Type's reflection
entry points (GetType/GetMethod(s)/InvokeMember/GetProperty(ies)/GetField(s)/
GetMember(s)/GetConstructor(s)/GetEvent(s)/GetDefaultMembers; instance
object.GetType() stays allowed) and puts RuntimeHelpers on the type deny-list.
Thread/ThreadPool/Timer leave the type list — the prefix rule reaches them.

R05:11 — the allocation budget landed with E1's ScriptRunGuard
(Scripting:MaxAllocBytesPerRun, default 64 MB): sampled around each run,
breach quarantines. It is observation after the fact; a pre-emptive OOM guard
still needs an out-of-process runner (follow-up issue in the lane report).

The Monaco editor shares the analyzer, so DiagnoseTests pins every new form
as an OTSCRIPT_FORBIDDEN marker and every allow-listed form as clean.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:45:03 -04:00
Joseph Doherty d1cc9e2cfc feat(runtime,adminui): real GetDiagnostics from child health + /fleet/diagnostics consumer (A6)
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.

IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.

Review ref: 02/U1.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:44:38 -04:00
Joseph Doherty 6dedf5158f feat(galaxy): wire DiscoverPageSize and ProbeTimeoutSeconds; concurrent first-touch write lanes
Review 2026-10-21 G-5, G-6, G-8 ("wire them all").

- G-5: GatewayGalaxyHierarchySource now pages the DiscoverHierarchy RPC
  itself (DiscoverHierarchyRawAsync) at Repository.DiscoverPageSize. The
  mxaccessgw client's typed DiscoverHierarchyAsync overloads hard-code a
  5000-row page and DiscoverHierarchyOptions carries no page size, so the
  authored knob reached nothing; the raw request takes page_size on the
  wire, so it is honoured here (non-positive → 5000). The loop keeps the
  client's repeated-page-token guard.
- G-6: GalaxyDriverProbe reads ProbeTimeoutSeconds from the config and
  runs under min(caller budget, configured cap) — the knob can shorten a
  probe, never extend it past the host's budget. The TCP preflight now
  shares that deadline instead of hanging on the caller's token alone.
- G-8: GatewayGalaxyDataWriter runs a batch as one lane per distinct tag,
  concurrently and bounded at 8 (MaxConcurrentWriteLanes); same-tag
  writes keep their authored order inside one lane and results land at
  their request's index. Acks stay serial by design.
- GalaxyDriverForm: help text now says what the two knobs actually do.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:41:47 -04:00
Joseph Doherty ae84253773 fix(controlplane): keep the admin singleton's mailbox free during Test Connect and deploy compose
ReceiveAsync suspended the mailbox for the whole of every handler, so one Test Connect against a
black-holed PLC (60 s clamp) or one large deploy compose stalled every other admin operation
fleet-wide — AdminUI alarm acks/shelves, driver restarts, the next deploy.

- TestDriverConnect runs the probe on the thread pool and pipes a TestDriverConnectResult
  straight to the requester (every path yields a result, never Status.Failure).
- StartDeployment serialises only the in-flight check (DB row + an actor-local compose-in-flight
  flag, because the Deployment row is written at the END of the compose); the snapshot read,
  validation, gzip and row write run off-actor and return as DeployComposed /
  DeployComposeFailed, where the coordinator is told and the requester answered.

Tests: an alarm ack completes in < 1 s while a 60 s probe is pending and while a compose is
parked on a gated DbContext (positive controls: both still complete afterwards); a second
StartDeployment during an in-flight compose is refused.

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:41:35 -04:00
Joseph Doherty eb9269c337 feat(scripting): ScriptRunGuard quarantines runaway scripts across every seam (E1)
A timed-out script is abandoned, not stopped, and every seam re-ran it on the
next trigger, leaking one pool thread per trigger (R03:1.1, R05:2, R07:L1).
A shared ScriptRunGuard, keyed per script instance, now:

- refuses a run while that script's previous orphan is still executing
  (Bad "previous evaluation still running" + otopcua.script.run_refused);
- quarantines after 3 consecutive non-completions or an allocation-budget
  breach (sampled with GC.GetAllocatedBytesForCurrentThread; default 64 MB):
  no further dispatch, Bad quality, one Error, otopcua.script.quarantined,
  cleared on source change / redeploy / reset;
- starts the budget clock when the pool picks the run up, so starvation is
  no longer charged to the script (R07:P5), and cancels the evaluator token
  on timeout.

Wired into RoslynVirtualTagEvaluator, CalculationEvaluator (new async
EvaluateAsync; the driver's pump and timer ticks now await the run behind a
SemaphoreSlim gate) and ScriptedAlarmEngine (a non-completing predicate
holds state and annotates the condition BadInternalError). ScriptingOptions
(Scripting:MaxAllocBytesPerRun, Scripting:QuarantineAfterConsecutiveTimeouts)
carries the knobs; the Calculation driver config accepts the same two keys.

Tests: guard unit suite (incl. the 30-trigger runaway leaving one orphan with
a healthy sibling as positive control), per-seam refusal/quarantine/reset
tests. The S12 race test now uses one vtag id per loop: twelve concurrent
evaluations of ONE id is a shape production never produces and the guard
refuses by design.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:39:43 -04:00
Joseph Doherty c6d4b14c93 feat(telemetry): wire TelemetryDial:CallTimeoutSeconds as the dial-establishment deadline (A5)
CallTimeoutSeconds was bound, defaulted and documented but consumed
nowhere. TelemetryDialSupervisor now arms a per-generation deadline on
every dial: a dial that has delivered no event by the deadline is
cancelled and treated as a transport failure (reconnect with a failure
streak). A live node replays its snapshots on subscribe, so this bites
only a dial that never connects (a black-holed host, where the socket
connect has no timeout). An established stream is deliberately not
bounded — the wire has no heartbeat, keepalive pings own its liveness.

Review ref: 02/U2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:38:32 -04:00
Joseph Doherty 73e74e270d fix(controlplane): validate and compose a deployment from ONE snapshot read
StartDeployment read the whole config DB twice with no transaction — DraftSnapshotFactory for
the validator, then ConfigComposer for the artifact, plus clusters/nodes and Script.SourceCode
again in between. An AdminUI save landing between the two shipped an artifact the DraftValidator
never saw.

- DraftSnapshot carries ServerClusters, ClusterNodes and ApiKeys; DraftSnapshotFactory reads
  every table inside one SNAPSHOT transaction on a relational provider (no-op on InMemory or
  under an ambient transaction).
- ConfigComposer.Compose(DraftSnapshot) is pure over the snapshot (ordinal-sorted keys);
  SnapshotAndFlattenAsync(db) delegates to it, byte-identical.
- AdminOperationsActor validates topology, estimates compile cost and composes the artifact
  from the one snapshot it holds.
- Migration EnableSnapshotIsolation: ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON
  (suppressTransaction; no model change).

Tests: the SQL Server snapshot read ignores a row committed mid-read (positive control: a later
read sees it); an edit injected between the Equipment and Tags reads is absent from the shipped
artifact and rejected by the next deploy; Compose(snapshot) == SnapshotAndFlattenAsync(db).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty 79d6b0a4b2 fix(runtime): lifecycle housekeeping — reprovision stamp, Stale drops, off-thread DB probe, tombstone sweep (A4)
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
  (and its deferred-reprovision timer): the map grew per removed id and a
  re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
  AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
  letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
  outcome back (single-flight), with a 2 s command timeout on relational
  providers, so a hung SQL Server no longer pins the actor thread and the
  cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
  10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
  AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
  half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
  driver already uses UnsubscribeAsync (no change).

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty 1b3169273d fix(galaxy): wire ScanState probing, single-sweep discovery, liveness stamp, dispatch-loop fault
Review 2026-10-21 G-4, G-7, G-11, G-10.

- G-4: PerPlatformProbeWatcher publishes its probe handles into the
  driver's SubscriptionRegistry under the reserved ProbeSubscriptionId
  (-1), so the EventPump's fan-out finally reaches the watcher — every
  ScanState event used to be dropped as "stale after unsubscribe" and
  per-platform host status never updated. ResubscribeAllAsync delegates
  the reserved entry to the watcher's new ResubscribeAsync, so the probe
  survives a reconnect; StopAsync removes the entry. Probe traffic is
  routed by the reserved handle and never reaches the public OnDataChange.
  The watcher is also built on the seam path (subscriber only), which is
  what makes the wiring testable without a gateway.
- G-7: GalaxyDiscoverer.DiscoverAsync returns the hierarchy it fetched and
  the driver selects its $WinPlatform/$AppEngine probe targets from that
  same sweep instead of asking the gateway for the whole galaxy twice.
- G-11: OnPumpDataChange stamps LastSuccessfulRead from inbound stream
  data, throttled (1 s default), so the comms-loss watchdog has honest
  liveness evidence under a Degraded supervisor.
- G-10: an EventPump dispatch-loop exit that is not a cancellation now
  reports a stream fault so the supervisor recreates the pump, instead of
  leaving a deaf pump reporting Healthy.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:35:48 -04:00
Joseph Doherty 6e2febbf3d fix(runtime): run driver writes off the mailbox under an in-flight cap (A3)
WriteAttribute was a ReceiveAsync, suspending the child's mailbox for the
whole device round-trip (up to 5 s): writes serialised, the second of two
operator writes to a slow device missed the host's 8 s Ask and was
reverted although the device took it, and every data change queued behind
the write. Writes are now a synchronous Receive that launches the device
call on the thread pool (sender captured, reply from the task, slot freed
by a self-sent WriteCompleted) under a per-driver cap (default 4; excess
writes queue in order). Queued writes fail fast on every exit from
Connected. Subscribe/unsubscribe stay ReceiveAsync.

Review ref: 02/P2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:57 -04:00
Joseph Doherty 2d25c916b6 fix(galaxy): make ShutdownAsync a real runtime teardown so re-init rebuilds
GalaxyDriver.ShutdownAsync was a no-op: the MX session, both owned gRPC
channels, the EventPump, the alarm feed, the DeployWatcher, the probe
watcher and the reconnect supervisor were released only by DisposeAsync,
which the host never calls — one leaked gateway session per respawn,
removal and /raw browse, and a watchdog Shutdown→Initialize that neither
tore down nor rebuilt anything (review 2026-10-21 G-1, G-2, G-9, G-3).

- ShutdownAsync and DisposeAsync share one ordered TeardownRuntimeAsync
  (supervisor first, then deploy/probe watchers, pump, alarm feed,
  session, channels); owned seams are nulled so the next InitializeAsync
  rebuilds; injected seams survive a shutdown and drop only on the final
  dispose. Lifecycle calls are serialised behind a gate.
- A failed production connect releases its partial runtime (no orphaned
  client per watchdog retry) and lands Faulted before and after.
- StartDeployWatcherAsync stops the previous watcher before starting a
  replacement, and now starts on a production instance's FIRST
  InitializeAsync (the old guard returned whenever discovery had not run
  here, which on a runtime instance is always).
- ReconnectSupervisor releases each cycle's CancellationTokenSource when
  its loop exits and gains an awaitable StopAsync; PerPlatformProbeWatcher
  gains an awaitable StopAsync.
- EventPump treats any exception under cancellation as a clean stop, so a
  teardown never wakes the supervisor.
- Internal seams for tests: an IGalaxyDeployWatchSource ctor parameter and
  a session-open override forwarded to the owned GalaxyMxSession.

Refs #735

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:30:14 -04:00
Joseph Doherty 6669fae7e5 feat(historian): ServerHistorian:MaxRawValuesPerRead caps raw pages and backend reads
Review R08:1.1: an uncapped HistoryRead-Raw (NumValuesPerNode = 0) read a
tag's whole window into memory on every backend (SQL TOP(int.MaxValue),
QuestDB LIMIT, Gateway max_values = int.MaxValue + a second snapshot copy).

New ServerHistorian:MaxRawValuesPerRead (default 10 000; <= 0 unbounded with
a Validate() warning; 1 raised to 2). Each backend data source takes it,
reports it as IHistorianDataSource.MaxRawValuesPerRead (the Gateway: min
with a strict gateway's learned ceiling), so the node manager's existing
RawPageLimit paging serves N = 0 and N > cap as pages of at most the cap
with a continuation point. Independently, a backend raw read asking for 0
or more than the cap reads cap + 1 rows and is REFUSED
(HistoryReadLimitExceededException => BadTooManyOperations) when the window
holds more - never answered truncated, since the server's paging reads a
short answer as the last page. Bound through the Host registry, the
SqlServer/QuestDb providers and GatewayHistorian.CreateDataSource.

Tests: options (default/binding/warnings), provider binding, Gateway unit +
through a real SDK node manager (uncapped -> 999/998/998/605 + CPs), and a
conformance case for the SQL backends.

Refs #736

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:29:10 -04:00
Joseph Doherty bfd5b7b015 fix(runtime): single-flight reconnect attempts, bounded pre-reconnect teardown (A2)
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.

Review ref: 02/H2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:26:45 -04:00
Joseph Doherty eb23053700 fix(runtime): dispose drivers on final stop, bound PostStop shutdown (A1)
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.

DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.

Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:23:39 -04:00
Joseph Doherty b437c34f92 fix(controlplane): recover a lost dispatch instead of wedging the deploy pipeline
A Deployment row left in Dispatching with no NodeDeploymentState rows (HandleDispatch threw
before its SaveChanges, or the process died between the AdminOps row write and the
coordinator's Tell) was recovered with an empty expected set; the deadline then hit the
0 == 0 "race" branch and returned without touching the row, so every later StartDeployment was
refused AnotherDeploymentInFlight until someone hand-edited SQL.

- PreStart: a recovered row with no expected-ack rows is a lost dispatch. Inside the apply
  deadline it is re-driven through HandleDispatch (bounded: CreatedAtUtc never moves); past it
  the row is closed TimedOut with a reason naming the condition.
- HandleDispatch is idempotent on a re-dispatch (rows already seeded ARE the expected set).
- HandleDeadline takes the race branch only when _expectedAcks.Count > 0.
- The PreStart recovery read projects ArtifactBlob away (R03:3.5).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:22:20 -04:00
Joseph Doherty 7513ce5144 docs(review): record pre-remediation baseline test run
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:11:55 -04:00
Joseph Doherty ad2b5c4044 docs: add 2026-10-21 system review + remediation plan
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:04:58 -04:00
Joseph Doherty d51d35bd64 Merge branch 'docs/xml-doc-cleanup'
v2-ci / unit-tests (push) Failing after 3m14s
2026-10-15 06:25:09 -04:00
Joseph Doherty 8c6067b62e docs: move subsystem gotchas out of CLAUDE.md into path-scoped .claude/rules
Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology,
data-plane auth, alarms, historian) only when matching files are touched.
2026-10-15 06:14:58 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
dohertj2 ebcbd81fca Merge pull request 'Ensure driver test cleanup preserves primary failures' (#730) from test/task-assertion-return-discard into master
v2-ci / unit-tests (push) Successful in 1h1m21s
2026-10-13 21:06:31 -04:00
Joseph Doherty 4f537a5652 Merge remote-tracking branch 'origin/master' into test/task-assertion-return-discard
v2-ci / unit-tests (pull_request) Successful in 1h1m9s
2026-10-13 19:03:52 -04:00
dohertj2 c89912a93b Merge pull request 'Fix DB-less ServiceLevel freshness and authorization remarks' (#724) from fix/dbless-servicelevel-freshness into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 19:03:42 -04:00
Joseph Doherty c4b4046832 test: verify missing-snapshot ServiceLevel timer behavior
v2-ci / unit-tests (pull_request) Successful in 1h0m42s
2026-10-13 17:19:50 -04:00
Joseph Doherty eed7cc7d75 test: guarantee driver fixture cleanup without masking failures 2026-10-13 16:36:38 -04:00
Joseph Doherty 5310a7e07a Discard asserted probe-task return values without joining loops 2026-10-13 16:12:31 -04:00
Joseph Doherty e1feb9fbb6 Merge remote-tracking branch 'origin/master' into fix/dbless-servicelevel-freshness
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 15:58:35 -04:00
dohertj2 1bf8c614a9 Merge pull request 'Retain ServiceLevel requests across binding and publisher failures' (#720) from fix/servicelevel-binding-replay into master
v2-ci / unit-tests (push) Waiting to run
2026-10-13 15:58:19 -04:00
Joseph Doherty bc80b53e0e Merge ServiceLevel replay fixes for freshness integration 2026-10-13 15:11:20 -04:00
Joseph Doherty f478b844a1 docs: clarify anonymous session ACL enforcement 2026-10-13 14:29:00 -04:00
Joseph Doherty 1e71a25e3d fix: age DB-less ServiceLevel snapshots periodically 2026-10-13 14:29:00 -04:00
Joseph Doherty 8828e999c4 Merge remote-tracking branch 'origin/master' into fix/servicelevel-binding-replay
v2-ci / unit-tests (pull_request) Successful in 1h1m55s
2026-10-13 13:56:26 -04:00
dohertj2 4f7a07fbd3 Merge pull request 'Count individual skipped TRX results' (#714) from fix/trx-skip-reporting into master
v2-ci / unit-tests (push) Has been cancelled
2026-10-13 13:56:04 -04:00
Joseph Doherty da2e441bb9 Merge branch 'fix/trx-skip-reporting' of https://gitea.dohertylan.com/dohertj2/lmxopcua into fix/servicelevel-binding-replay
v2-ci / unit-tests (pull_request) Has been cancelled
2026-10-13 13:54:54 -04:00
Joseph Doherty c8e3e279b4 Retain and replay ServiceLevel across publisher binding and failures
v2-ci / unit-tests (pull_request) Waiting to run
2026-10-13 13:26:27 -04:00
1066 changed files with 72903 additions and 12421 deletions
+56
View File
@@ -0,0 +1,56 @@
---
paths:
- "src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/**"
- "docs/ScriptedAlarms.md"
- "docs/AlarmTracking.md"
---
# Scripted alarm ack/shelve and Part 9 condition method behaviour
> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area.
> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file.
## Scripted Alarm Ack/Shelve
Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` role — conferred by the session's API-key principal's `NodeAcl` grants (or the `anonymous` principal's), carried by `RoleCarryingUserIdentity`; LDAP is AdminUI-only; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`.
⚠️ **One condition event per accepted method call is OURS, not the SDK's (Gitea #587).** The SDK's own
report is gated on `AutoReportStateChanges` (left `false`), so before #587 a client Acknowledge emitted **no**
condition event — the old "SDK auto-fires its own event (E2)" comments were wrong. `WireMethodTransitionReporting`
wraps every method's SDK handler (Ack/Confirm/AddComment/Enable/Disable/OneShotShelve/TimedShelve/Unshelve) and,
on Good, reports once via `ReportConditionEvent` — **under `Lock`**, so the engine's/driver's re-projection
(routed by a fire-and-forget Tell *before* the SDK applies state) waits and then hits the delta-gate with no
delta. Things to keep: the SDK's shelve handlers never apply shelving, so the wrapper does; only a real
before/after change is reported (a re-ack is silent and restores the Confirmed/Retain the SDK flips);
`MaxTimeShelved` is set to `int.MaxValue` ms at materialise because `Create()` left it 0 and every TimedShelve
failed. ⚠️ **Native alarms support Acknowledge and AddComment only** — Confirm, the shelve methods and
Enable/Disable return `BadNotSupported` (no upstream path, no pair replication, and `NativeAlarmProjector`'s
Confirmed/Unshelved constants own those fields); a native ack re-asserts `ConfirmedState = true` because the
SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by
`ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`).
Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a
**no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real
remaining time from `AlarmConditionSnapshot.UnshelveAtUtc`); a **commented** re-ack is stored as an
`AddComment` (durable, replicated) and reports one event, an uncommented re-ack stays silent; a **native
ack is refused `BadInvalidState` off-Primary** via `NativeAlarmAckPrimaryGate` (fed by
`IDriverPrimaryRoleSource`; `DriverHostActor` publishes its Primary verdict at host start and on member
changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT
Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design).
**Condition `Severity` is carried verbatim, clamped to 1..1000** — `ToPart9Severity`; the node manager used to re-bucket onto
the five `EventSeverity` points (so an MTConnect Fault read 900, scripted Critical 1000 read 900, native Low 200 read 300).
ScadaBridge only clamps, so exact values pass through. ⚠️ **Never select a two-state id with `AddSelectClause(type, "X/Id")`** —
the string becomes ONE QualifiedName named `X/Id`, which no event carries; the Client.CLI/Client.UI filter did this since
`a2883b82` and every alarm read inactive+unacked. Use `new SimpleAttributeOperand(type, [X, Id])` (ScadaBridge already does).
**Condition children are instance-scoped (#629).** Every child of a materialised condition carries
`{ConditionId}/{browse path}` in the condition's namespace (e.g. `…/temp_hi/ShelvingState/UnshelveTime`), assigned by
the `OtOpcUaNodeManager.New` override — the SDK default kept each child on the TYPE's ns=0 id, so a direct Read hit
the type (or `BadNodeIdUnknown`), translate failed, and children sat outside the ACL (the slash join now puts them
under their condition's scope). ⚠️ The shelve methods resolve by the `AlarmConditionType_ShelvingState_*` ids only
because `New` adopts those (a fixed set) as their `MethodDeclarationId`; calls by the `ShelvedStateMachineType_*` id
are no longer supported (pinned by a test). Condition teardown must forget the whole `PredefinedNodes` subtree
(`ForgetPredefinedSubtree`) or a removed condition's children keep answering.
+423
View File
@@ -0,0 +1,423 @@
---
paths:
- "src/Core/ZB.MOM.WW.OtOpcUa.Cluster/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Host/**"
- "docker-dev/**"
- "docs/Redundancy.md"
- "docs/Configuration.md"
- "docs/Telemetry.md"
---
# Cluster topology: redundancy, role authority, per-cluster meshes, MeshTransport, ConfigSource/ConfigServe, telemetry transport, LocalDb, DB-less driver nodes
> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area.
> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file.
## Redundancy
The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is CLOSED** — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (`c50ebcf7`; `docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard".
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
## Role authority (`EffectiveRoles`)
**A node's roles are resolved in exactly one place** (archreview R3-04, closing 01/S-12 = 03/C5 and
01/C-9): `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`), with one precedence —
`Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`, else empty — and two-layer normalization: a
plain role (`admin`/`driver`/`dev`) is trimmed and lower-cased, but a cluster role's ClusterId suffix
(`cluster-SITE-A`) is kept verbatim, because lower-casing it would rename the Akka role of a running
deployment. An unknown role from **either** source fails host start. `Program.cs` resolves one
`EffectiveRoles` instance at the top of the composition root and registers it as a singleton; that
SAME instance drives both the conditional host wiring (`hasAdmin`/`hasDriver`) and
`AkkaClusterOptions.Roles` (what Akka gossips) — there is no second derivation for the two to
disagree about, which is why no runtime "do the views agree" assert exists. Before this, `Program.cs`
read `OTOPCUA_ROLES` only, so a node configured solely through `Cluster:Roles` gossiped the right
Akka roles while `Program.cs`'s own wiring mounted none of them — the inverse of the pre-#532 field
incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`).
- **A role overlay may not itself set `Cluster:Roles`.** Roles are resolved *before* the per-role
appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing
`Cluster:Roles` would leave the bound options disagreeing with the registered instance.
`AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views — a no-op
today since none of the four shipped overlays carries a `Cluster` section.
- **Both sources set to different normalized role sets logs a Warning** at boot (`Cluster:Roles`
still wins outright) — the boot banner also names which source won (`source=Configured` /
`Environment` / `None`).
- **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of
production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through
`EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role
during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:**
`RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`,
which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the
library nor with a local case-insensitive pre-filter, which would split the Primary election from
`ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade
window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)").
- **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to
stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s
— leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException`
(design-time tooling) passes through untouched.
- **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement:
a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads
`OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal —
with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same
English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI
RBAC role).
See `docs/Configuration.md` §`Roles`, `docs/ServiceHosting.md` §"Role gating", and
`archreview/plans/R3-04-one-role-authority-plan.md`.
## Mesh command transport (`MeshTransport`)
**The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md),
Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension,
`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails
validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values)
rather than silently falling back to `Dps`. What remains is exactly two transports, selected by
`MeshTransport:Mode`:
- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**:
it works only when central and the node share a gossip ring, which after Phase 6 means within one
2-node mesh.
- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node
carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a
node on anything else.
Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`,
served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev
rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port
of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is
every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort`
and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and
no rotation: each command goes to the specific node dialled, which is what makes per-node delivery
provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer
applies to either direction.
Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role
config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than
claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and
stops at the first success.
Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per
application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets
store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed
on every unresolvable case — constant-time compare, no key material logged. Production keys are
`${secret:OT-MESH-PSK-<clusterId>}`; the docker-dev rig supplies three literal dev keys via env
(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`).
Things that carried over unchanged from the `ClusterClient` era, because they were never
`ClusterClient`-specific to begin with:
- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node
and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across
nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery.
The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that
set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a
DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`,
`ScriptedAlarmHostActor`) accept a command from either transport identically.
- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a
deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient
deletion — it was always about the deploy path, not the transport.
`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which
makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md`
§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) /
[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) /
[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history —
those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`
is the original Phase 2 design doc and is likewise left as a historical record of the transport this
section now describes as deleted.
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again.
## Config source (`ConfigSource` / `ConfigServe`)
Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
fail-closed). Things worth knowing before touching it:
- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
`RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
**The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** —
`ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and
hashes the result, so the invariant holds unchanged and nothing between the composer and
`DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever
looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so
pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every
decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already
handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after
upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be
upgraded BEFORE central starts emitting compressed blobs.**
- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
`FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
**Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and
the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does
**not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches).
- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
SQL today; `StoreAsync` is idempotent and the cache replicates).
- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
`Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.
On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching
key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is
no `OTOPCUA_CONFIG_MODE` env var in the compose file. See
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
## Live telemetry transport (`Telemetry` / `TelemetryDial`)
Per-cluster mesh **Phase 5** (merged `35552a96`; live gate PASSED `f134935f`) added a second
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
touching it:
- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
`driver-health` (`DriverHealthChanged`), `driver-resilience-status`
(`DriverResilienceStatusChanged`) — feeding the exact same central sinks
(`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
either mode.
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport
(`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542):
`FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end**
(no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its
`DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just
central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above.
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
`TelemetryDial:Mode` on central actually switches anything: which of the two always-available
sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
[MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side
stores are not evicted). The node-local hub
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
node stream is up) — the same semantics the DPS pill has always had.
See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
§`Telemetry`/`TelemetryDial` for the config keys, and
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
gate procedure.
## LocalDb pair-local store (Phases 1 + 2)
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
**Phase 1** caches the
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
never register it.** The cache can optionally replicate to the node's redundant pair peer over the
library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
exception).
**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
same event accepted on both nodes — which happens in every boot window, since
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
**The library pin is `0.3.4`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and
`.Replication`), bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15).
The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3
upgrades in place on open, so no rig or volume action was needed. `RegisterReplicated(string)` is
unchanged and everything 0.2.x/0.3.x added is additive, **except** that `RegisterReplicated` (0.3.3)
now throws `LocalDbRegistrationException` for a table that is the parent of a row-mutating ON
DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire.
0.3.0 made the snapshot-resync flag **per-table** (see the ⚠️ below, now corrected); 0.3.2 added
validated config `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, left
unset here) and a public `LocalDbOplogGapException`, neither yet wired to a call site. **Its two
new APIs — `DeregisterReplicated` and `RegisterReplicated(table, baselineExistingRows:)` — are now
both used** (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two
things are worth knowing:
- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching
**byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized
under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row
cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which
materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps
`16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the
message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap"
was corrected with the bump; do not restore that rationale.
- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).**
`LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges`
(via `LocalDbSetup.ReplicationIsConfigured`):
**`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is
not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it
alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` +
`LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only,
silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is
fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has
nothing to capture for.
- ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`:
`ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these
rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no
peer must still drain, so the two predicates are deliberately not the same.
The **passive** half (no `PeerAddress`) cannot name its peer, so it stands down for its **mesh partner**
(the other member of its `cluster-{ClusterId}` role); a legacy passive half with no mesh role still
drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the
published Primary verdict (and the live data-plane gate, which must stay the same decision) **allows** —
for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the
Sparkplug primary-host claim with its Primary peer.
- **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the
same load-bearing DDL→register order. Baselining is what makes enabling replication on a node
that has been running *without* it actually converge: capture is CDC and the snapshot streamer
pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in
used to replicate only writes made after the restart. It seeds the ledger at the LWW floor
(HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote
write of the same key and wins only where the peer has no version at all. It is idempotent, so
it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.4`) the snapshot
flag is **per-table**, not per-database — baselining one table no longer streams every
registered table in both directions. (Prior to `0.3.0`, on the `0.2.1` pin this repo carried
until 2026-09-03, the flag was per-database and baselining anything streamed *every*
registered table in full; the old advice to trim `alarm_sf_events` before enabling replication
no longer applies at the current pin.)
- **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time
self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and
prunes that table's oplog + row-version rows. Merely declining to register would leave every
pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row
inside every write transaction — on precisely the node that has no peer to capture for. The
call returns `true` when it removed something, so the Information line fires **once, and only
when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is
Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where
resolving one from DI would nest a resolution inside a singleton construction; `Program.cs`
assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe
the line rather than infer it.
- ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake
compares registered-table digests **fail-closed**, so a node that deregisters a table its peer
still replicates stops syncing with a schema mismatch. Enabling replication on an established
pair means setting the config and **restarting both nodes** — do it on both, or on neither.
- **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was
enabled from a fresh pair, so the impact of the old gap was low — central and site-b are
default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests`
pins both branches (registered-table set on a configured node, empty set + zero triggers +
the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), `docs/AlarmHistorian.md`, and the runbook
`docs/operations/2026-07-20-localdb-pair-replication.md`.
## Phase 4 — cut the driver-side ConfigDb connection
Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles`
has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs`
gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is
unaffected (it keeps ConfigDb via its admin role). Consequences:
- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator`
fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused
node may stay `Direct`.
- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not
spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db
factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and
`Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes
**240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture
— where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself
is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less
(driver-only) nodes".
- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore`
(replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces
`EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed
`ScriptedAlarmState` table **was dropped** (Task 9, `3a590a0c`, migration
`20260723183050_DropScriptedAlarmStateTable`); `EfAlarmConditionStateStore` is deleted.
- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on
a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is
the system of record.
- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty
`NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so
`OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only —
the same rows a driver node never received via config either, so this is not a regression.
Merged to master (`26fad75c`); **live gate PASSED** (`1281aebf` — driver nodes run ConfigDb-free) and
Task 9 landed (`3a590a0c`). See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and
`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4.
+222
View File
@@ -0,0 +1,222 @@
---
paths:
- "src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/**/*Acl*"
- "docs/security.md"
---
# Data-plane authentication (API keys), node ACL enforcement, anonymous principal, AdminUI LDAP
> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area.
> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file.
## Authentication: API keys on the data plane, LDAP on AdminUI (#520)
⚠️ **Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all.** It is machine-to-machine,
so a client presents an OPC UA **UserName** token with the fixed username **`apikey`** and an
`otua_<keyId>_<secret>` credential as the password. `ApiKeyOpcUaUserAuthenticator`
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/`) implements `IOpcUaUserAuthenticator`: one indexed lookup
on `keyId` plus one peppered HMAC-SHA256, `FixedTimeEquals`, **no network I/O**. The key resolves to a
**principal**; `NodeAcl.Principal` rows grant it permissions. `LdapOpcUaUserAuthenticator` is
**deleted**, along with four `Security:Ldap` keys that existed only for it (`AuthTimeoutMs`,
`MaxConcurrentAuthentications`, `OutageFailureThreshold`, `OutageCooldownSeconds`) — they bounded an
LDAP bind running inline in the SDK's synchronous, `SessionManager`-lock-holding impersonation
callback, and nothing on the remaining LDAP path has that property.
**LDAP remains, for AdminUI login only** (`Security:Ldap`, `OtOpcUaLdapAuthService` →
`ILdapAuthService`, group→role mapping via `GroupToRole` + `LdapGroupRoleMapping`). Do not "restore"
the data-plane authenticator when you see LDAP wiring in `Program.cs` — it is there for the AdminUI.
Two things to know before relying on any of this:
- ⚠️ **Node ACLs are enforced only behind `Security:Acl:Enabled` (default `false`).** With the flag
**off**, a key's grants collapse into the two server-wide role gates (`WriteOperate` on Value writes,
`AlarmAck` on Part 9 methods), **only `Cluster`-scoped grants count** — `ApiKeyInterimRoleMapper`
deliberately under-grants, because honouring a narrower grant's flags while unable to honour its
*scope* would turn "WriteOperate on Equipment X" into server-wide write authority — and **read,
browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous**. With the
flag **on** (`DataPlaneAuthorizer`), scoped grants decide **every** data-plane operation: write and
alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6).
- **The two role gates re-check the session's KEY on every call, flag off or on (Gitea #563).** Roles
are resolved once at activation, so an expired / deployment-disabled key used to keep `WriteOperate` /
`AlarmAck` on its open session until the client disconnected. `DataPlaneAuthorizer.AuthorizeKeyUsable`
runs the same `FindUsableKey` probe against the same snapshot + clock as the ACL path (one dictionary
lookup, no I/O) and denies `BadUserAccessDenied` before the role check. **Only those two gates** —
reads/subscriptions stay ungated with the flag off, by design. Anonymous carries no key and is
unaffected. No flag: a dead key keeping write authority is a defect, not a rollout choice. ⚠️ On a
fused node a key minted after the last deploy authenticates (ConfigDb) but cannot write/ack until a
deployment carries it — the flag-on path always behaved this way.
- ⚠️ **The ACL decision REPLACES the role check — the two are never chained.** Chained, a grant would
have to confer the server-wide role *as well as* pass the scope check, and only `Cluster`-scoped
grants confer a role, so every narrow grant would stay inert and the flag would appear to do
nothing. The load-bearing test is therefore an **allow** — a session carrying *no roles* writing
successfully on the strength of its grant. A deny-only test passes against a build where the ACL is
never consulted, because the role gate denies too.
- **The trie is built inside `DataPlaneAuthorityState.Apply`**, not by a second wiring at the publish
actor, and the state owns its `PermissionTrieCache` rather than being handed one. An uninstalled
trie denies everything, which is indistinguishable from working enforcement until someone checks
that an ALLOW is possible at all.
- **A deploy reaches sessions that are already open**, in both directions: each decision stamps the
*current* generation rather than the one the session activated under, and re-checks the
authenticating key — so revoking a key or a grant stops an open session immediately instead of at
the client's next reconnect. On a driver-only node the artifact is the only revocation channel.
- `UserAuthorizationState` is built **per decision walk, not per session** — and since the hot-path
change it is allocated only on a cache **miss**, so a cache hit never constructs one. Its LDAP-shaped
freshness/staleness predicates are switched off EXPLICITLY (`AuthCacheMaxStaleness = MaxValue`).
⚠️ Relying on "the stamp is always fresh" was not safe: `MembershipResolvedUtc` comes from
`DataPlaneAuthorizer`'s `TimeProvider` while `IsStale` is evaluated against
`TriePermissionEvaluator`'s, so **any divergence between the two clocks denies every request** with a
verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead.
- **Stage 6's hot seams (read/browse/subscribe) go through a per-`(session, node, operation, realm)`
decision cache**, cleared wholesale when the generation advances. It is **sharded per API key**
(`ApiKeyId ?? Principal` — despite the type name, sessions sharing one key share one shard and one
budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and
not every other key's decisions. ⚠️ It caches the **grant walk only** — key revocation and expiry are
re-checked **before** the cache lookup on every call, because expiry moves with the clock rather than
with a deployment.
- **Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors.** A
grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward
only, so `PermissionTrieNode.BrowseTransitPrincipals` marks the path. Planting a real `Browse` grant
on each ancestor instead would be collected by the additive downward walk and leak `Browse` onto
every sibling subtree beneath that ancestor. ⚠️ **The topmost ancestor is the single shared root
folder** (`s=OtOpcUa`, raw namespace) that BOTH realm trees hang under — its identifier is a display
name, not a path, so `DataPlaneAuthorizer.BuildScope` maps it onto the trie's **zero-segment
realm-root scope** (`V3NodeIds.SharedRootFolderId`, realm-qualified: in the UNS namespace that
identifier is an ordinary Area), and its Browse considers transit in **either** realm because it
fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse
gate denies the whole call when its source node is denied, so clients were stopped at the front door.
- **Revocation reaches live subscriptions** (stage 6): every other operation re-asks per request, but a
subscription is authorized once and then streams — and streaming is how machine consumers actually
read this server. `DataPlaneAuthorityState.Changed` fires `RevalidateMonitoredItems`. ⚠️ A denied
item is queued `BadUserAccessDenied` and then **detached from its `MonitoredNode2` fan-out — NOT set
`MonitoringMode.Disabled`**: disabling discards the item's queued notifications, so the client is
silenced without ever being told why (caught only by the over-the-wire test). Detaching is also
reversible, so restoring the grant resumes the feed with no client action. ⚠️ **`Changed` is no longer
the only trigger (arch-review 03/S19):** `AclRevalidationTimer`, armed by `OtOpcUaServerHostedService`
at `Security:Acl:RevalidationIntervalSeconds` (default 60 s, `0` disables, inert while
`Security:Acl:Enabled` is false), runs the same pass on the clock — because `ExpiresUtc` passes with
the clock, not with a deployment. The pass's verdict memo is **pass-local and must stay that way**;
hoisting it across ticks would serve an expiry that landed between two ticks the earlier tick's allow.
**Event/alarm items ride both triggers too (Gitea #562)** — a second map (`_eventItems`), fed by the
`SubscribeToEvents` override, holds every *direct* notifier subscription; a denied one is detached via
the SDK's own protected `SubscribeToEvents(…, unsubscribe: true)` (what a client delete does) and
re-attached with `unsubscribe: false` on restore — again never `Disabled`. Unlike a data item it is
**not told** (no value slot for a status; the feed goes quiet), and a client delete of a stood-down
item is absorbed by the override (the SDK would otherwise remove an item that is not attached).
**Server-object (`i=2253`) event subscriptions are filtered per event at delivery (Gitea #586).** The
`SubscribeToAllEvents` override attaches them to **nothing** (just records them in `_allEventsItems`);
each condition's own `OnReportEvent` (`OnConditionReportEvent`, fires once per event ⇒ still one
Server-object copy) queues the event to each recorded item whose session holds `Subscribe` on the
condition (RawPath) **or** any notifier folder it reports through — the per-folder path's question. The
root-notifier forward to the core Server object (`OnReportEvent` override) is **dropped**: it was the
ungated path, and it is the only one that served an item created before the alarm was deployed (the
SDK's `AddRootNotifier` subscribes existing all-events items with `unsubscribe: true`). Revalidation is
**implicit** — every event re-asks the authorizer, so a deploy or expiry lands on the next event with no
pass. `ConditionRefresh` on an `i=2253` item is filtered the same way. The dispatcher **must** keep
the SDK's audit guard (`AuditEventState` only while `Server.Auditing` over SignAndEncrypt/https),
because conditions report their Part 9 method audit events through themselves.
`ReconcileAllEventsMonitoring` must keep `AreEventsMonitored` raised on every condition while any
`i=2253` item exists, because `ConditionState` gates its method **audit** reports on
`EventsMonitored()` (method-driven state-change events no longer depend on it — #587, below). The
protected `SubscribeToEvents` override that no-ops all-events items is what
stops the SDK's `AddRootNotifier` (`unsubscribe: true` bug) taking that unit back. Both review
findings (review of `75e559f6`). ⚠️ The materialise path raises only the **new** condition's unit
(`RaiseAllEventsMonitoring`); a full reconcile there made a deploy of N alarms O(N²) under `Lock` with
ScadaBridge connected — keep the full reconcile to remove/rebuild/(un)subscribe.
**Model-change and audit events are gated too (Gitea #588).** The manager's own server-wide events
(`GeneralModelChangeEvent`, the rejected-write `AuditWriteUpdateEvent`) go through
`ReportToServerObject`: flag **off** ⇒ `Server.ReportEvent` as before; flag **on** ⇒ delivered per item to
`_allEventsItems` (same audience), a model change narrowed to the `Affected` nodes the session may
**Browse** (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral
one), an audit write only with `Read` on its source node plus the SDK's audit/encryption guard. Any other
event type fails closed. Auditing is off by default (never configured in this repo).
- **ns=0 standard nodes stay ungated by design, even with `Security:Acl:Enabled`** (#617) — `ServerStatus`,
`NamespaceArray`, `ServiceLevel` etc. are served by the SDK's core node manager outside our authorizer and
clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node,
event subscription and HistoryRead is still gated. `docs/security.md`.
- **Equipment-node writes: `EvaluateEquipmentWriteStructure` is the ONLY type check.** `OnWriteValue` bypasses
the SDK's own type check on every writable equipment node, so `OtOpcUaNodeManager.EvaluateEquipmentWriteStructure`
(plus `EvaluateEquipmentWriteRange`) is the whole gate. Scalars are deliberately lenient (drivers
range-check; ScadaBridge sends Double to a Float node). **Array writes** (#610): a numeric array of another
element type is **converted, checked, element by element** to the node's element type (ScadaBridge writes
CLR `int[]`/`double[]` lists) — any out-of-range, fractional-to-integer or NaN/∞ element refuses the whole
write `BadTypeMismatch` with nothing sent, so a `UInt32` > `Int32.MaxValue` can never reach a Galaxy Int32
attribute as reinterpreted bits; non-numeric mismatches are refused (`Byte[]` ≈ ByteString); wrong length ⇒
`BadTypeMismatch`; an array to a **scalar** node is refused only on Modbus/S7 (`EnsureVariable`'s
`refuseArrayOnScalar` — pass-through drivers keep their browse-committed scalar-shaped array tags writable;
a new sink wrapper must forward that argument, and the reflection guard checks only that the call arrives);
`IndexRange` writes ⇒ `BadWriteNotSupported`, `ValueRank > 1` out of scope. An
array node is writable only when the driver can write the array (`DriverArrayWriteSupport`: Modbus keys-addressed tags bounded
by `maxRegistersPerWrite` (never above FC16's 123)/coil cap/address 65535 incl. String/BitInRegister arrays
(BitInRegister also by `maxRegistersPerRead`: its RMW is one unchunked FC03) (#741; a string-only `addressString` tag
stays read-only — family unknown at compose time), S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types **with a
declared `arrayLength`** — an undeclared length stays read-only because a short whole-array write would
resize the Galaxy attribute; AbCip atomic non-BOOL types, AbLegacy Int/Long/Float on the matching data file
(no `/bit`/`.sub`/indirect/`S:`/`I:`/`O:`, not native PLC-5, ≤ 256), TwinCAT every codec type but a BOOL
bit-in-word path — #737/#738/#739; those three CONVERT elements, so they never key on an exact node type); `arrayLength >= 1` is required, and the node's `Writable && (!IsArray ||
ArrayWritable)` is projected onto the raw node and every UNS mirror.
- **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`,
preserving today's behaviour) controls *admission*; when admitted the session authenticates as the
reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as
the same `RoleCarryingUserIdentity` a key produces. Set it `false` to make keys mandatory. A key may
**not** use the `anonymous` principal, and a failure to read anonymous's grants **denies** rather
than admitting with no roles.
- **Keys reach a driver-only node inside the deployment artifact** (stage 4). `ConfigComposer` snapshots
the `ApiKey` rows — the peppered HMAC only, never a secret, and the pepper is node config that is
deliberately *not* in the artifact — and `DeploymentArtifact.ParseDataPlaneAuthority` decodes them
node-side alongside the cluster's `NodeAcl` rows. `OpcUaPublishActor` installs both on
`IDataPlaneAuthoritySource` on every apply, **before** its `plan.IsEmpty` early-return, so a deployment
that only revokes a key still lands. A node with a ConfigDb keeps `ConfigDbApiKeyStore` (per-activation
read, wrapped — see below); a driver-only node gets `ArtifactApiKeyStore`. ⚠️ **The asymmetry is
ACTIVATION-scoped, not session-scoped — this line used to say "immediate on a ConfigDb node" and that was
wrong.** Clearing `Enabled` in the ConfigDb stops the **next session activation** on an admin/fused node
(bounded by `Security:ApiKeys:StoreCacheSeconds`, default 5 s) and takes until the next deployment on a
driver-only one — but an **already-open** session re-checks its key against the in-memory *artifact
snapshot* (`DataPlaneAuthorizer.Evaluate` → `snapshot.FindUsableKey`), never against `IApiKeyStore`, so on
**both** node kinds an `Enabled` revocation reaches an open session only at the next deployment.
**Expiry is the exception**: `FindUsableKey` compares `ExpiresUtc` against the authorizer's clock on every
call, so it is immediate everywhere — and since arch-review 03/S19 it reaches live subscriptions too, on
the host's `Security:Acl:RevalidationIntervalSeconds` timer (default 60 s; the session is **silenced with
`BadUserAccessDenied`, not closed**, and recovers on re-activation). Direct event/alarm subscriptions
are revalidated on both triggers as well (#562); with the flag **off**, expiry and revocation reach the
two role gates (#563) but not reads or subscriptions.
- ⚠️ **The activation lookup is BOUNDED (arch-review 03/S18).** The SDK raises impersonation synchronously
under a `SessionManager`-wide lock, so a ConfigDb that is *slow* rather than *refusing* stalled **every**
session activation on the server, Anonymous and X.509 included. `BoundedApiKeyStore` wraps the ConfigDb
store — and only it, never `ArtifactApiKeyStore`, which is two in-memory dictionary probes — with
`Security:ApiKeys:StoreTimeoutMs` (2000; exceeding it **denies**, via the authenticator's existing
catch-all) plus a `StoreCacheSeconds` (5) read-through cache. Faults and timeouts are never cached, a
cached hit whose `ExpiresUtc` has passed is re-read rather than served, and a timeout logs a distinct
Warning + `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. Both knobs at `0` restore the old
behaviour exactly.
- ⚠️ **The auth generation is a node-local COUNTER driven by the `RevisionHash`, not derived from its
bits.** `PermissionTrieCache` treats the highest `GenerationId` as current, so a generation taken from
hash bytes would install a *lower* number about half the time and the cache would go on serving the
previous deployment's ACLs with nothing logged. Pinned by
`DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases`.
- Note `OtOpcUaServerHostedService` takes `IOpcUaUserAuthenticator` **and** `IDataPlaneAuthoritySource`
as **required** ctor parameters — the built-in DI container does not fall back to a default value for
an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a
node with neither role) or it does not boot.
**Grants are authored through a scope picker, not free text (stage 7).** `/clusters/{id}/acls` opens
`AclScopePickerModal` over the cluster's raw + UNS trees, and selecting a node supplies `ScopeKind`
**and** `ScopeId` together — so `RawDevice` + a TagId, a cross-cluster scope, a `Cluster` grant carrying
an id, and the retired `FolderSegment` kind are all unauthorable by construction rather than merely
rejected later. ⚠️ **A `Cluster` row must produce a NULL `ScopeId`** — the deploy gate rejects one that
carries an id (`NodeAclScopeIdUnexpected`), so a picker that helpfully filled in the cluster id would
author rows that cannot be deployed. `IAclScopeService.CheckScopeAsync` mirrors
`DraftValidator.ValidateNodeAclScopes` rule for rule and **blocks the save** when a scope resolves to
nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches
the picker); the list page marks such rows `unresolved`. **UNS tags are deliberately not offered** — a
UNS signal is a `UnsTagReference`, and a `Tag` grant already covers the raw node *and* every UNS
projection of it. The picker is `AclScopeTree`, a separate component from `RawTree`, which already
carries two modes.
**All eight stages are done** (`d4389f57` enforcement, stage 7 authoring after it), each live-gated on
docker-dev. The rationale for every decision above is in
`docs/plans/2026-07-31-apikey-acl-enforcement-design.md`. See `docs/security.md` for the full guide.
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)
+307
View File
@@ -0,0 +1,307 @@
---
paths:
- "src/Drivers/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/**"
- "src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/**"
- "tests/Drivers/**"
- "docs/DriverLifecycle.md"
- "docs/drivers/**"
---
# Drivers: rediscovery signal, comms-loss watchdog + quality contract, AB native transport, per-driver notes
> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area.
> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file.
### Change Detection
`DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys.
**The signal is consumed as an operator prompt, not as an address-space rebuild** (§8.2, 2026-07-27).
`DriverInstanceActor` subscribes in `PreStart` and unsubscribes in `PostStop` — the detach is
load-bearing, because the `IDriver` instance outlives the actor when the host respawns a child around
the same driver object. A raise is recorded and rides the driver-health snapshot
(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`) to the AdminUI `/hosts` page as a
**"re-browse" chip**.
⚠️ **It is advisory: a Galaxy redeploy still does NOT change the served address space.** v3 authors raw
tags explicitly through the `/raw` browse-commit flow, so an operator must re-browse the device and
commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact
records.
Two gotchas if you touch this:
- `PublishHealthSnapshot` dedups on a health fingerprint. The rediscovery timestamp **must** stay in
that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the
operator. Pinned by `DriverInstanceActorRediscoverySignalTests`.
- The shared test `StubDriver.GetHealth()` returns `DateTime.UtcNow`, so its fingerprint differs on
every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests
use a stub with stable health for exactly this reason.
**#507 is closed as superseded, and the injection path is deleted.** Its retained code read
`EquipmentNode.DriverInstanceId` and `EquipmentTags`, both *structurally empty* in v3
(`AddressSpaceComposer.cs`, `DeploymentArtifact.cs`), so removing the guard would have changed a log
line and injected nothing. Gone with it: `HandleDiscoveredNodes`, `PartitionDiscoveredByDeviceHost`,
`ApplyDiscoveredPlansForDriver`, the redeploy re-inject tail, `DiscoveredNodeMapper`,
`AddressSpaceApplier.MaterialiseDiscoveredNodes`, the connect-time discovery loop
(`StartDiscovery`/`RediscoverTick`/`DiscoveredNodesReady`/`TriggerRediscovery`), and the 18
`DiscoveryInjectionDormantV3` tests — v2 characterization tests asserting an equipment-rooted graft
(`EquipmentRootNodeId == "EQ-1"`) that v3 cannot produce. `ITagDiscovery` itself stays: the `/raw`
browse picker drives it through `Commons/Browsing/DiscoveryDriverBrowser`, which never went through
the actor.
**`IRediscoverable` is the ONLY tag-change signal, and four drivers do not raise it.** A periodic
tag-set drift check briefly existed (shipped `e77c8a35`, **removed** by Gitea **#523**): every
`SupportsOnlineDiscovery` driver was re-browsed on a 5-minute timer and the result diffed against its
authored tags. It was taken back out because it generated recurring, **unconfigurable** (interval was a
constructor parameter with no appsettings key), **never-live-gated** outbound traffic to real
PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the
design treated it as an implementation detail.
⚠️ **Consequence to know before you rely on rediscovery:** **AbCip and FOCAS browse a live backend but
implement no `IRediscoverable`, so they have no tag-change signal at all** — a PLC program re-download
or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT,
MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an
event-driven signal, or one that runs when something *else* already indicates change (a reconnect, a
deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`TagSetDriftDetector`,
`DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` /
`DiscoveryStreamIncludesAuthoredTags`.
**`IHostConnectivityProbe` is consumed (Gitea #521, closed).** `DriverInstanceActor` subscribes
`OnHostStatusChanged`, pulls `GetHostStatuses()` fresh, and the host set rides `DriverHealthChanged` to the
AdminUI `/hosts` page — no SQL: the `DriverHostStatus` table is dropped, and so is
`DriverInstanceResilienceStatus` (#524; the live data goes over the `driver-resilience-status` telemetry
channel only). **Multi-host dispatch is wired too:** `DriverInstanceActor` resolves the breaker/host key
per call via `IPerCallHostResolver` (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip).
### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed)
Until #527, **`DriverInstanceActor.DisconnectObserved` had no production sender.** The only paths into
`Reconnecting` were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected
and then lost its device stayed `Connected` indefinitely — publishing Bad for every tag, never
re-initialising, and **not recovering when the device returned** (live-confirmed: a Modbus driver against a
stopped fixture needed a node restart). `PublishHealthSnapshot` read `GetHealth()` for *reporting only*.
The `Connected` health tick now runs a watchdog that self-sends `DisconnectObserved` when a driver reports a
non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. Things to know before touching it:
- ⚠️ **The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal.**
`Degraded` means *connected but some data is bad*, and several drivers write health **per tag inside the
read loop** (`ModbusDriver.ReadAsync`), so one unreadable address latches `Degraded` forever against a
perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely.
`LastSuccessfulRead` is the honest evidence — preserved across `Degraded`/`Faulted` writes, advanced by any
success. Each conjunct has a test that goes red when the *other* is deleted.
- **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across
all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect
supervisor, OpcUaClient via a bad session keep-alive (#670 — it used to stay `Healthy` forever; its SDK
`SessionReconnectHandler` is deleted, a lost session publishes `BadNoCommunication` once per outage and the
watchdog's re-init is the recovery; good keep-alives + notifications advance `LastSuccessfulRead`). `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has
no transport. Re-check this if a new driver lands.
- **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.**
`ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable`
field inside `InitializeAsync` without disposing the old one, so re-initialising without `ShutdownAsync`
leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose
first: `ModbusDriver.InitializeCoreAsync` releases `_transport` **and** the `_probeCts`/`_reprobeCts`
probe loops before rebuilding them (`c3970a2e` + the lifecycle gate in `409a58b6`), deliberately *not*
via the shared `TeardownAsync()` because that would also drop `_poll` and every active subscription with
nothing to re-drive them; `MqttDriver.InitializeCoreAsync` calls its (idempotent) `TeardownAsync()` first,
which reaches `_connection` and the primary-host connection (`2b4b3fcb`). The path is no longer latent —
the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new
`IAsyncDisposable` field on these two, must repeat the dispose-before-overwrite.
- The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` /
`DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy
system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped
reading.
- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises
`ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient:
keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed
connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the
SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only
while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with
the watchdog's conjunction. `docs/DriverLifecycle.md`.
- **What comms loss now does downstream — one contract, six seams (archreview R3-02).** "A value consumer
never reports a quality better than its source's" is stated once in `docs/DriverLifecycle.md` §"The
quality contract" and implemented by `Core.Abstractions.InputQuality` (`WorstOf`/`IsUsable`). Three
things changed with it, all client-visible:
- ⚠️ **Value nodes of a comms-lost driver now go Bad.** `DriverHostActor.FanCommsLossToValueNodes`
stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained
server-side**, via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host
holds no value cache, so only the node can keep the value). ⚠️ Retention is **not client-observable**:
the SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read/sample, so clients see
`Value = null` + the Bad status; the retained value is what the next Good re-publish and the
`(value, status)` dedup compare against. #481 had wired comms loss to the mux and to Part 9
conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop
the tag" (ScadaBridge) will see more Bads — that is the fix working.
- **Nothing Good is written on reconnect** — recovery is the driver's own re-publish (the poll engine's
`forceRaise`, MQTT's new initial push). Instead the host **audits**: refs still latched 30 s after a
reconnect log a Warning and increment `otopcua.driver.comms_latched_refs`. Non-zero on that counter
means a driver reconnects without re-publishing.
- **Every dedup on the recovery path is keyed on (value, status), not value.** A device that comes back
usually comes back holding the same reading, so a value-only dedup swallows the recovery and the node
stays Bad forever — the v3-gate trap, now closed in the Calculation driver, the virtual-tag actor and
(via a connection-loss stale) MQTT's plain-mode suppression. **Any new consumer of driver values must
repeat this**; a regression test for it must replay the IDENTICAL value or it proves nothing.
- Consequences per seam: MQTT/Sparkplug stale on connection loss and push cached snapshots on every
(re)subscribe (`docs/drivers/Mqtt.md`); the Calculation driver and virtual tags **hold** while any
input is Bad and inherit the worst input's sub-code, evaluate through Uncertain, and hand scripts the
input's real `StatusCode`/`SourceTimestampUtc` (`docs/drivers/Calculation.md`, `docs/ScriptEditor.md`)
— so a script that relied on receiving `null` for a dead input no longer runs at all while it is Bad.
See `docs/DriverLifecycle.md` §"The comms-loss watchdog" and §"The quality contract".
## AB drivers: native transport (`Transport`)
AbCip and AbLegacy have a second, managed wire transport (`Driver.Ab.Transport`) beside libplctag, chosen by
the driver-config key `transport` = `Native` | `Libplctag` (string; takes effect when the driver is
**constructed** — every process start, or a deploy that recreates it, not `ReinitializeAsync`; AdminUI
"Transport" dropdown; CLIs `--transport`). "Default" is encoded by omitting the key; the AdminUI must never
re-emit a concrete default. The defaults differ per driver (final review, 2026-10-07):
- **AbCip: an absent key resolves to the node default, `Native` unless pinned.** Precedence (one place,
`AbCipTransportResolver`, shared by driver, probe and CLI): explicit per-driver key → node-level appsetting
**`Drivers:AbCip:DefaultTransport`** (`Native` | `Libplctag`, env `Drivers__AbCip__DefaultTransport`,
`AbCipDriverDefaultsOptions`; a bad value **fails host start**) → compiled `Native`. Bound unconditionally
because the driver factory AND the admin-node Test Connect probe both read it. ⚠️ So **the upgrade restart
alone moves no-key AbCip drivers to Native**; to stay on libplctag, set the node key — NOT a per-driver key
through the old AdminUI, whose typed form drops unknown keys (#576). Evidence: AbCip reads proven on a real
CompactLogix (Z-35162); scalar native writes (DINT/BOOL/REAL) proven on it once (2026-10-09, user-approved one-off, restored) — and native UDT member writes (DINT/BIT/REAL, #611 gate, same day) and a native STRING write (hw5, same day, restored byte-exact); and a native array-element write (`EM01_PowerUnit._SEQ_EP[31]`, user-approved, an element no alias targets, restored; neighbours unchanged); libplctag member and fragmented writes are `ab_server`-only. #566: a real Logix drops TCP on a connection idled past its timeout (no 0x0107/0x0203); the faulted session reopens on the next request.
- **AbLegacy: an absent key resolves to `Libplctag` for every family; `Native` is opt-in** (no node key). On
a real MicroLogix 1400 (Z-33323, idle, 2026-10-09) native read S/N/F/B/T/C/R/ST/O/I incl. `S:0`, T/C/R
sub-elements and status bits, and ST byte order, all correct, and wrote N/F8/B-bit (restored). libplctag (the default)
used to fail `S:`/`O:`/`I:` and `R` status bits there (incl. the default probe `S:0`) — its `pccc.c` needs explicit
file numbers and has no R status-bit mnemonics; since #659 `LibplctagLegacyName` translates (`S2:`/`O0:`/`I1:`,
`R6:0.CON/13`) and both transports read identically on that press. The authored grammar is unchanged. Read-side
only: a libplctag WRITE to any address the translation changes (`S:`/`O:`/`I:` incl. bit writes, R `EN…FD`, `C…UA`)
is refused `BadNotSupported` before any runtime is created (none could reach the PLC before — no write-surface
widening on the default transport); libplctag T/C status-bit writes it parses itself are unchanged.
L/A files, function files, SLC, PLC-5 and `logixpccc` remain unverified natively. Explicit Native on PLC-5/`LogixPccc` logs a once-per-instance Warning.
AbLegacy native refuses status-bit sub-element writes (`T4:0.DN`…, `BadNotSupported`).
Refusals are not comms faults: an Unsupported refusal (`BadNotSupported`) and an AbCip native **type
mismatch** (`BadTypeMismatch`, where libplctag returned Good garbage for a same-width mis-authored type) both
keep the tag handle, leave driver health alone (no `Degraded`, nothing for the comms-loss watchdog) and warn
once per tag — only genuine transport failures evict and degrade (a runtime `NotSupportedException` too, #611 —
it used to latch `Degraded`). AbCip width-checks every scalar **write** against the controller's bytes before
sending, on both transports (#611; libplctag used to patch part of its buffer and send it all). Controller-discovered
UDT members are writable once browse-committed and authored `ReadWrite` (BIT members included — the controller
applies the bit) — native member writes incl. a BIT member are live-gated on Z35162 (2026-10-09); sibling-bit preservation and the libplctag path are ab_server-only; whole-UDT writes are refused `BadNotSupported` by decision (`docs/drivers/AbCip.md` §Member writes). `DeviceSessionPool.Shared` logs through the
host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT factories' `Register(registry, ILoggerFactory?)` take one too, #602 CONV-3). The AbCip `browsePageDelayMs` knob has an AdminUI field (`AbCipDriverForm` "Browse page delay (ms)"). libplctag is confined to `Libplctag*.cs` adapters by
`LibplctagConfinementGuardTests`; do not reference it elsewhere. See `docs/drivers/AbCip.md` §Transport,
`docs/drivers/AbLegacy.md` §Transport + §Wire evidence, `docs/Configuration.md` §`Drivers:AbCip`, and
`docs/plans/2026-10-07-native-ab-transport-gate-results.md`.
## Other driver notes
- **S7:** connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe
tick); a write-side access-denied (PUT/GET disabled) latches a **sticky Faulted** that outranks later read
passes; driver-side array writes cover non-Bool narrow/`Int32`/`Float32` types only — Bool arrays are
refused (a one-byte-per-element encode would clobber neighbour bits).
- **MQTT:** Plain `writeTopic` writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug
primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types
`BadNotSupported`).
- **No driver-side history (#655):** `IHistoryProvider`, `DriverCapability.HistoryRead`, `HistoryReadStatusException` and
the OpcUaClient history passthrough are deleted — nothing ever dispatched HistoryRead to a driver; it is served only by the
server-wide `IHistorianDataSource`. `isHistorized` on an OpcUaClient tag means OtOpcUa records it, not that upstream
history is proxied. A stored `ResilienceConfig` still carrying a `HistoryRead` key parses (diagnostic only).
- **FOCAS / AbLegacy:** background loops are joined (≤ 5 s) on shutdown.
- **FOCAS (#257/#266):** `cnc_statinfo` flags are a `Status/` fixed-tree subtree (`FIXED:Status/Run`…, capability-probed);
adjacent PMC tags in one poll batch coalesce into one `pmc_rdpmcrng` (`PmcCoalesce` config, default on) — a refused range
falls back to per-tag reads, a transport failure fails all members without retry. Parts count stays `PARAM:6711` (docs only).
- **FOCAS alarms (#647):** an `ALARM:` address (`*`, `<Type>`, `<Type>/<n>`, `<Type>/<lo>-<hi>`, `T<code>`, `OTHER`) +
the `alarm` object = ONE condition at the RawPath (either alone is skipped + reported). Overlapping selectors all
fire; `OTHER` = what no type selector claims. ⚠️ A failed `cnc_rdalmmsg2` THROWS (it used to return `[]` ⇒ mass
clear + un-acked re-raise) — and annotates its conditions `BadNoCommunication` once per outage (quality-only) and
degrades driver health with `LastSuccessfulRead` frozen, so an alarm-only driver trips the #527 watchdog. ⚠️ A CNC
REFUSAL (non-OK rc, transport fine) is `BadNotSupported` + one Warning and does NOT touch health — else the
watchdog reconnect-loops a production control. Health = worst of the value and alarm paths, computed together. The told
record lives on the driver INSTANCE (reconnect / in-place re-init re-raise nothing); a deploy RESPAWNS the driver,
and its Raise for a still-active alarm is a host-side re-statement (below). Type names are the 16i-era table only
(30i codes unconfirmed, not invented — use `T<code>` there). Read-only; ack server-side. `AlarmProjection.Enabled`
is a no-op. E2E: `Runtime.Tests/FocasNativeAlarmEndToEndTests` (incl. a respawn). `docs/drivers/FOCAS.md`.
- **A Raise on an already-active condition with the SAME message is a RE-STATEMENT (#647 review):** `NativeAlarmProjector` keeps its ack and
`ForwardNativeAlarm` writes no second "Activated" `/alerts` row — every native driver's told-record dies with its
instance, and a deploy respawns the driver. Only `Retrigger` (or inactive → active) un-acks, so a source whose
UPSTREAM is authoritative for the ack sends Retrigger for an unacked re-statement: OpcUaClient on first sight of a
condition (no prior told), Galaxy for an unacked active snapshot entry. **Those two also flag
`AlarmEventArgs.IsRestatement` (#656)** — history-only: a flagged Retrigger on an Active+unacked condition with the
same message AND severity, or a flagged Clear on an inactive one, writes no `/alerts` row (every Galaxy reconnect used
to duplicate them); a flagged `Unspecified` on an already-active condition (AbCip ALMA governing-limit drop-out)
moves the node's text/severity, keeps active/acked and writes no row. Active+acked is the atomic **`AlarmTransitionKind.ActiveAcknowledged`** (Galaxy snapshot
`ActiveAcked`, OpcUaClient inactive→acked-active) — never Acknowledge (kept a cold host's `Active=false`: the alarm was
invisible) nor Raise+Acknowledge (an unacked intermediate). Its mirror **`InactiveUnacknowledged`** (#657, OpcUaClient)
replaces Raise/Retrigger+Clear (an Active flicker; rows only for what changed). OpcUaClient's first sight of an inactive
condition is a flagged InactiveUnacknowledged, or flagged Clear+Acknowledge for inactive+acked (a flagged Acknowledge on an
acked condition writes no row) — never `Unspecified`, which wrote "Activated" for an inactive alarm and left a
retained-Active condition Active. `NativeAlarmProjector` returns the rows to write; unflagged
sources (FOCAS/MTConnect, live Galaxy) are unchanged. E2E: `Runtime.Tests/NativeAlarmRestatementEndToEndTests`.
- **Re-Initialize without teardown (#630):** FOCAS, AbLegacy, AbCip, TwinCAT, S7 and OpcUaClient `InitializeAsync`
first release the prior device/session state (cancel, ≤ 5 s join, dispose — incl. the AbCip alarm projection;
FOCAS's is PAUSED, its told record kept, #647) — the device-state half of `ShutdownAsync` only; the poll engine is kept and the actor re-subscribes
after every successful init. AbCip publishes fresh tag tables by reference swap (a second init used to throw
"tag name collision"); S7 swaps `Plc` under `_gate`.
- **Data and alarm subscriptions are independent in `DriverInstanceActor` (#662):** a data re-subscribe (every
deploy's bulk push to a kept driver) detaches DATA only (`DetachDataSubscription`) — it used to unhook
`OnAlarmEvent` too, silently killing the native-alarm feed until a respawn. The health-poll reconcile
re-attaches/re-subscribes alarms as well as data, so a failed alarm subscribe is retried every tick.
- **A config change is a respawn — there is NO in-place path (#665).** The `ApplyDelta`/`ApplyResult`/`DeltaApplied`
seam (host `ApplyChildDelta`, `DriverSpawnPlan.ToApplyDelta`, the actor's Connected reinit and mid-connect
`AdoptConfigDuringInit`) was unreachable since #516 and is deleted; #663/#664 were latent bugs in it. `IDriver.ReinitializeAsync`
stays as a per-driver contract the runtime never calls. Every driver's `InitializeAsync` still serves the config it is
handed (Calculation adopts, Galaxy refuses a non-equivalent one — #664), and still releases prior state (#630).
`docs/DriverLifecycle.md`.
- **S7 (#632/#638):** STRING/WSTRING/CHAR/WCHAR and DTL/DATE/TOD/TIME/S5TIME codecs (`S7ValueCodecs`; TOD/TIME/S5TIME
surface as Int32 ms); an undecodable buffer or unrepresentable write is `BadOutOfRange` with no health impact.
Optional `localTsap`/`remoteTsap`/`connectionType` override the slot TSAP (`remoteTsap` ⊕ `connectionType`). Test
Connect reads 2 bytes of `MB0` so a PUT/GET-disabled CPU fails the probe. The negotiated PDU size is logged, on
`S7Driver.GetDiagnostics()`, and (#640) on `/hosts` via the optional `IDriverDiagnostics` bag (string map on
`DriverHealthChanged.Diagnostics` / proto `has_diagnostics`+`diagnostics`; its digest is in the publish fingerprint).
- **TwinCAT (#637):** TIME/TOD publish Float64 ms, DATE/DT a UTC DateTime; a tag authored `UDInt` keeps the raw counter
(and keeps historizing — DateTime is not historized).
- **TwinCAT (#645, #310/#314):** opt-in ADS sum-read (`useSumRead`, `sumReadMaxItems` default 200 / ceiling 500, 32 KiB payload
budget). It reads raw bytes and decodes as the declared type, so only a scalar passing the strict width+kind check
(`CheckNotification`) rides a batch; value-converted pairs (DInt over INT, Real over LREAL/DINT) read singly. Only overall
ADS 1793/1794 latch the device to single reads; any other non-comms overall code (1792/1795/1797/1798...) reads that chunk singly for the poll without latching, with a once-per-device Warning; a comms failure neither.
Opt-in `diagnosticsEnabled` (default off) puts ADS state/app/online-change/cycle/exec/`HeadroomMs` on the `/hosts`
`IDriverDiagnostics` bag (≤ 4 extra reads per device per probe tick once settled; TC2 falls back to `.SYSTEMINFO` /
`.SYSTEMTASKINFOARR[1]`, costing 8 on the first tick and after each reconnect). Live-gated read-only 2026-10-08 on
Z35474 (TC3) + Z34834 (TC2, which answers sum commands) via a harness driving `TwinCATDriver` — `/hosts` end-to-end
not run (a stock host has no AMS router); the gate found ⚠️ Beckhoff's `SumRead` ctor throws for a `List<>` of entities (only an array survives
its internal cast) — `AdsTwinCATClient.CreateSumRead` owns it. Flat browse stays (VirtualTree is a strict subset
on TC3). Sum-write not ported.
- **AbLegacy (#636):** `AbLegacyAddress` (in `AbLegacy.Contracts`) is family-aware: PLC-5 `I:`/`O:` indices are OCTAL
and are emitted to libplctag as decimal (libplctag's `pccc.c` parses decimal only). MicroLogix function files parse
but read `BadNotSupported`. Indirect `N7:[N7:0]` (#646) is resolved by the driver at read/write time (index re-read
every call, never cached; bad index = `BadOutOfRange`; indirect writes are not atomic). One check (`AbLegacyAddress.Validate`) backs the
editor (family-independent) and the deploy gate (family from the tag's `DeviceConfig.PlcFamily`); an unparseable
address reads `BadNodeIdUnknown` for that tag only.
- **OpcUaClient (#635):** OperationLimits are honoured by the SDK session (`SessionClientBatched`) — do not re-add
driver-side chunking. Subscription tuning keys are range-checked at `ParseOptions`. `discoveryUrl` endpoints go
AFTER configured ones; more than one discovered server without `discoveryServerUri` fails the connect. Session
events are the OTel counter `otopcua.driver.opcuaclient.events`. `HistoryAggregateType` carries the full Part 13
set; an aggregate the gateway can't serve is `BadAggregateNotSupported` per node.
- **MTConnect (#627/#628):** TIME_SERIES reads as a variable-length `double[]` (Float64 only); a count disagreeing
with `sampleCount` is `BadDecodingError`. One MTConnect driver = one agent client = at most one `/raw` Device
(`MTConnectMultipleDevices` deploy error; `RawTreeService.CreateDeviceAsync` refuses a second).
- **MTConnect CONDITION alarms (#626):** a CONDITION raw tag is a native Part 9 condition only when its `TagConfig`
carries the standard `alarm` object (opt-in — alarm tags leave the value subscription, so auto-promotion would break
existing String tags). Normal→inactive, Warning→500, Fault→800; one condition per DataItem showing the most severe
active nativeCode (ties: ordinal nativeCode, then type); ack is server-side only. ⚠️ UNAVAILABLE never clears: an
active condition stays active+acked with a quality-only `BadNoCommunication` (same as #481 transport loss). The
driver dedups against what the server was last TOLD (level+nativeCode+nativeSeverity+qualifier — not text), records
only while a handler is attached, and drops the record of only the conditions that LEFT the set when the subscribed
alarm SET changes (#641) — never on a reconnect of the same set; a replayed `/current` must never re-raise (un-ack).
`AlarmEventArgs` gained optional `OpcUaSeverity` (exact Part 9 severity) and `QualityStatusCode`; a quality-only
event goes out as `AlarmQualityUpdate` per node and never through the snapshot projection.
- **Native condition state survives a full rebuild (#641):** `RebuildAddressSpace` captures each NATIVE condition's
live node state (client acks included), the same-apply re-materialise restores it, and `CompleteAddressSpaceApply`
(called by `OpcUaPublishActor` at the end of every apply) reports ONE event per restored off-normal condition and
discards unclaimed captures. `NativeAlarmProjector` is pruned per apply, not cleared. Native routing is an exact
`ConditionId == RawPath` match — AbCip's `#active` suffix was fixed; FOCAS routes since #647 (below).
- **Driver removal (#609):** removing a driver releases its capability-invoker pipelines and resilience
tracker, *then* evicts its telemetry snapshots. ⚠️ Both are **tombstoned**, not just cleared: the removed
child stops asynchronously and its in-flight subscribe/connect still completes through the invoker (and
emits Health) on the thread pool *after* `StopChild`, which used to resurrect the state — so a removed
driver's counters were republished forever. `Release` → tracker `Retire` and hub `Evict` ignore every later
write for that id until it is spawned again (`Create` → `Reinstate`, `SpawnChild` → `Readmit`). A
Watch/`Terminated` release would not work — the late writes land after the actor terminates.
+275
View File
@@ -0,0 +1,275 @@
---
paths:
- "src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.*/**"
- "src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/**"
- "src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/**"
- "src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/**"
- "tests/**/*Historian*/**"
- "docs/Historian.md"
---
# Historian / HistoryRead (Gateway, SqlServer, QuestDb backends; alarm history; continuous historization; provisioning)
> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area.
> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file.
## Historian / HistoryRead
**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default), `SqlServer` or `QuestDb`** (see the subsection below).
As of the gateway-integration cutover, the **Gateway** backend's historian read, alarm-write, and
continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed
**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin
`IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke
Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were
retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`,
now a retired stub).
A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the
raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default
historian tagname, which is the tag's driver `FullName`.
### Pluggable backends, SQL Server and HistoryUpdate (2026-10)
- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads".
- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
- **Raw `ReturnBounds` is honoured on every backend (#691, client-visible):** Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by `HistoryBounds` from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 `ReverseRawReader` walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward `NumValuesPerNode`, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ `Bad_BoundNotFound` (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ **Breaking for SDK-default clients:** the .NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true` — a client that cleared only `IsReadModified` now gets bounds, and one that set neither is a modified read with bounds ⇒ `Bad_InvalidArgument` (§6.5.3.3); set both explicitly (Client.Shared does; tests building details must too). No data AND no bound found ⇒ `GoodNoData`, no values (the pre-bounds answer). On Gateway the at-or-after lookup ends at now + 1 day (never a year-9999 end to AVEVA). Kill switch `ServerHistorian:HonorReturnBounds` (default `true`; `false` = pre-#691). `docs/Historian.md` §Bounding values.
- ⚠️ **Raw pages are full (#692, client-visible, every backend):** each raw page asks the backend for `N + skip + 1` rows (the boundary ties it will trim + one lookahead row) — a resumed page used to ask only N, trim the ties and come back short; at N=1 that was an EMPTY page + continuation point that then fell into the `MaxTieClusterOverfetch + 1` over-fetch (live: 8 values = 17 pages / 25 reads / 524k rows ⇒ now 8/8/≤24). No lookahead row ⇒ last page, no CP (an exact multiple no longer ends on an empty page). Exception: an oversized tie cluster — its last slice is `min(N, remaining)` with a CP, and stepping past one at the end of data can give one empty final page. The over-fetch fires only when every row past the trimmed ties sits on the boundary timestamp; a cursor inside such a cluster is marked `WithinTieCluster` (next page goes straight to the over-fetch). `SliceTieCluster` resumes only strictly before EndTime (truncated to the backend resolution) — it used to resume AT the exclusive EndTime and return samples there. Bound placeholders carry an explicit marker (`HistoryBounds.IsPlaceholder` no longer matches a stored `Bad_BoundNotFound`/null row); the N=1 bounds probe carries its walk hint + resolved end bound into the cursor. Reviewer fuzz harnesses (paged vs unpaged over random tie series) are the right test for any paging change — repo tests passed while master mismatched 1-in-160.
- ⚠️ **Strict-ceiling paging (#694):** `IHistorianDataSource.MaxRawValuesPerRead` = L = learned gateway ceiling − 16 (0 on SQL backends/Null, guard-tested; 0 until a refusal names the ceiling). `RawPageLimit.Plan(want, k, L)` sizes every raw page so page + boundary ties + lookahead ≤ L (room = L − k − 1, signed; first page L−1, resumed L−k−1; uncapped `N = 0` reads page too) — a capped read above the ceiling used to be `BadTooManyOperations`, now a ceiling-sized page + CP. One re-plan when the read taught a new limit or was refused for crowding (`HistoryReadLimitExceededException.PreWindowRows`, set only above the 16-row allowance; the page shrinks by it). No room on a resumed page ⇒ the tie-cluster over-fetch, bounded min(M, L−1); the slice size is never 0. The page path passes `retryOnLearnedLimit: false` to `ReverseRawReader` (the walk re-running at the old N cost 200–3,000 reads cold); bounds lookups keep it. ⚠️ A Gateway window starting at/before the Unix epoch counts AVEVA's synthetic row as pre-window — it used to set `reachedEnd`, so a ceiling-truncated read came back Good with no CP (forward-from-1970 = first page only; end-only past the data = the OLDEST rows as "latest"; pre-existing). The #687 crowding re-read now doubles its room up to 8 re-reads, never past a known ceiling, and its room is capped at `MaxTieClusterOverfetch` (passed to the Gateway source by the factory; past it ⇒ `BadTooManyOperations` with `PreWindowRows`). A raised ceiling is not re-probed by raw paging (pages stay small until another read re-probes or restart).
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key).
- **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history.
### Read path (`ServerHistorian` section)
The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value **null** + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (`AllowStreamingBeyondCeiling=false`) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (`GatewayRowCeiling`: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (`NumValuesPerNode = 0`) read of a small window is served; a ceiling-bounded stream that is not provably complete is `BadTooManyOperations`, never a truncated answer), Processed
(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes (⚠️ #693, client-visible: Gateway AtTime is evaluated from RAW reads by the shared `Sql.Common` `AtTimeEvaluator` — exact stored value at a stored time, null + Bad at/after a stored NULL, Uncertain hold before a Bad neighbour, linear + Interpolated bit, `BadNoData` before the first sample — never AVEVA's own interpolation (which answered 107 Good at a NULL, null+Bad at a Good sample after it, 99.99999999944 for 100, no Interpolated bit); `GatewayAtTimeReader`: one ≤1,024-row window read for the batch (jumps to far-apart times use 64-row reads) + a walk-back for the first time's prev (gallop ×4, whole-remainder read after two empties, then real BISECTION seeded by any full read's last row — a dense tag stopped a year ago ≈ 25–31 reads; the first cut regrew segments after a full read and gave up at 65) + an 18-row probe for the last time's next — 10 dense times = 1 read (exact) / 3 (interpolated), 10 hourly = 12; read cap 64 + 16 per distinct time (a runaway guard: a time in a gap of a bursty tag needs ≈ 6 reads; 4/time refused a 1 h/day tag asked daily ×30) and a strict ceiling < 4 rows ⇒ `BadTooManyOperations` with a message naming the limit; DateTime hints stepped, non-finite/out-of-range values under an integer hint unusable; all request bounds whole ms, rows counted only strictly inside a request; the node manager passes the node's scalar type (`ReadAtTimeAsync(…, TypeCode? valueType, …)` default-interface overload, ignored by SqlServer/QuestDB/Null) and the Gateway converts raw rows to it before evaluating — Boolean stepped, integers interpolated + rounded half-away and returned as the node's integer type, floats Double; no hint ⇒ Double; ⚠️ the shared evaluator used to box interpolated Int64 as Double (`cond ? (long)x : double` widens) — fixed for SQL backends too, and Shouldly `ShouldBe` hides numeric-type bugs (assert `ShouldBeOfType`); kill switch `ServerHistorian:GatewayAtTimeFromRaw` (default `true`); `docs/Historian.md` §Gateway AtTime); Events over
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.
`ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`):
| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` |
| `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) |
| `ApiKey` | `""` | Peppered-HMAC key `histgw_<id>_<secret>` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** |
| `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme |
| `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) |
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
| `HonorReturnBounds` | `true` | #691 kill switch: `false` ignores HistoryRead-Raw `ReturnBounds` as before (no bounding values, no modified-read `Bad_InvalidArgument`) |
| `GatewayAtTimeFromRaw` | `true` | #693 kill switch (Gateway only): `false` = AVEVA's own AtTime interpolation passed through, as before |
| `EventReadHorizon` | `00:00:00` | #694, opt-in and LOSSY: when > 0, HistoryReadEvents never reads events older than now − horizon (fresh windows clamped, both directions, all backends; a window wholly older ⇒ `GoodNoData` unsent). Bounds the deep "rest is empty" event probe (~3.8 s live, 30 s gateway SQL timeout unfiltered). No per-source earliest-event cache by design — late S&F drains would make it stale |
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway |
| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |
### Alarm-history path (`AlarmHistorian` section)
Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).
### Continuous historization (`ContinuousHistorization` section)
When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a
durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed
`IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The
recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the
crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**)
with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries
only the recorder + outbox knobs:
| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder |
| `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage |
| `CommitMode` | `PerEntry` | `PerEntry` = fsync before the append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs`. ⚠️ `PerEntry` now means **one fsync per ingest burst, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands |
| `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` |
| `DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs `tags × latency` — while 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, letting the outbox fall permanently behind during any gateway hiccup |
| `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) |
| `DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential `tags × latency` pass becomes `⌈tags / concurrency⌉ × latency` with outstanding round-trips bounded. `1` = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass" |
| `Capacity` | `500000` | Max un-acked outbox entries before drop-oldest; `0` = unbounded. Non-zero **by default since #538** — the disk backstop for a historian refusing *every* write, the one case the per-tag quarantine deliberately cannot resolve |
| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |
| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 |
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync`
hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's
`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues`
land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a
deploy. **Gateway backend:** non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not
provisioned), the recorder likewise drops + meters non-numeric values, and continuous historization is
**numeric-analog only** (`UInt16→UInt4` is a documented fallback). The **SqlServer** backend provisions and
records every type (see "Pluggable backends" above).
The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`),
built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the
read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier.
When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
the "Boolean historization maps to Int2" section in `docs/Historian.md`.
### Gateway-side prerequisites
The target HistorianGateway OtOpcUa points at **must** run with:
- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
- `RuntimeDb:EventReadMaxRows=0` (its default, unlimited) — or larger than the reverse-events walk's biggest read
(`2·MaxTieClusterOverfetch + 4`). The cap truncates **silently**, so a walked-back reverse events read (#691.4) would
lose the newest events of a segment it cut.
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.
### Migration note (deployments upgrading from the Wonderware backend)
The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env:
| Old (Wonderware) key | New (gateway) key |
|---|---|
| `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) |
| `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) |
| `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) |
The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`)
were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
`ServerHistorian`. See `docs/Historian.md` for the full guide.
### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
**No longer a limitation.** The cutover was live-validated against the real gateway on
`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
populate `Source_Object` — not a fixable bug). Evidence:
`docs/plans/2026-06-27-otopcua-historian-followups.md`.
Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
`wonder-sql-vd03`, gateway running the prerequisites above):
```bash
export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway URI; absent ⇒ all live tests skip
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret> # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag> # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag> # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name> # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"
```
The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
gate the operator runs on the VPN before trusting the cutover.
### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04)
The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built
later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the
`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it
through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the
recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the
deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe
`Tell`; a faulting feed can never break a deploy).
**The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a
production-fatal defect** — see `docs/plans/2026-08-01-continuous-historization-live-gate.md`. Three things
to know:
- ⚠️ **`ConfigureAwait(false)` in an Akka handler destroys `Self`.** `OnValueChangedAsync` touched `Self`
after awaiting `IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false)`, which discards Akka's
`ActorTaskScheduler` synchronization context — so `ActorBase.Context` was gone and it threw
`NotSupportedException` on the FIRST value. The throw reached supervision, the actor **restarted**, and
because production seeds an **empty** ref set the restart discarded every historized ref: one value
captured, then silence. Fixed by capturing `IActorRef self = Self` before the await (the idiom the
`GetStatus` handler already used for `Sender`).
- ⚠️ **A fake that completes synchronously hides this entire class of bug**, and 13 tests passed because
of one. `ValueTask.CompletedTask` never yields, so the continuation ran inline with the context intact.
Worse, the obvious fix to the fake — `await Task.Yield()` — is *also* vacuous: `Yield()` posts back to
the captured context, so the caller still resumes on the actor thread. The fake must complete on a
**pool** thread (`await Task.Delay(1).ConfigureAwait(false)`), as a real per-entry fsync does. Only
reverting the production fix exposed that the first regression test proved nothing.
- **Both pair nodes historized the same value** — the recorder had **no Primary gate**, unlike the alarm
store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as
**#537** and **fixed** — see the paragraph below.
**Gate COMPLETE — #491 CLOSED 2026-08-04.** Outbox append, drain, `WriteLiveValues`,
**restart-convergence**, `EnsureTags` provisioning (`ensured=2 failed=0`) and **read-back** all ✅. The
read-back leg ran against the licensed `wonder-sql-vd03` with the VPN up: two registers carrying
*distinct non-zero* constants (`8224`, `120`) each read back as its own value — a `0`-valued source
would have proved nothing (the #536 trap).
**Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison
tag); evidence in the same plan doc. Two things to know before touching either.**
- **#538 — the discriminator is "did another tag write this pass?", NOT a failure count.** `DrainOnceAsync`
writes every per-tag group and acks **all-or-nothing** (deliberate: `RemoveAsync` truncates *through* an
id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned
the FIFO head while every *healthy* tag was re-submitted on every pass, forever — duplicate samples, an
outbox that never truncated (observed 8.3 MB; signature = the gateway's `ok` **and** `error` counters
climbing together). The drain now quarantines a tag after `MaxWriteAttempts` consecutive failures, but
⚠️ **only counts a failure on a pass where some other tag succeeded** — otherwise an ordinary gateway
restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug
into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so
the load-bearing test is the **falsification control** (`A_backend_that_fails_every_tag_abandons_nothing`).
Abandonment is also deferred until *every* failing tag is over budget, so nothing is discarded on a pass
that could not have advanced the head anyway.
- **#537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink.** The alarm
queue is in LocalDb and *replicates*, so the Secondary holds the Primary's rows and gating the drain
loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its
drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of
stale duplicates on promotion). ⚠️ **And the peer that matters is one recording the SAME refs, not one
holding the same rows** — a node stands down only for a Primary in its own per-cluster mesh, and a node
with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is
usually running another cluster's configuration, and deferring to it ends this cluster's historization
*silently* (the drain reports success on an empty outbox forever) — the same scope mismatch that once
suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on
`RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
#511 registration accident).
- **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary
stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the
redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and
*nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`)
subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role
source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change
re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the
failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable`
(default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT
adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay
put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address
clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the
gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real
partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each
node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates.
Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for
newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType)
provisioned nothing (`2026-06-27-historian-gateway-integration-issues.md` Issue 5). It now also fires for those
changed raw tags.
+2 -1
View File
@@ -29,7 +29,8 @@ packages/
*.pem
# Claude Code (per-developer settings, runtime lock files, agent transcripts)
.claude/
.claude/*
!.claude/rules/
.local/
+25 -1224
View File
File diff suppressed because it is too large Load Diff
+42 -3
View File
@@ -35,11 +35,28 @@
<PackageVersion Include="Grpc.Tools" Version="2.76.0" />
<PackageVersion Include="libplctag" Version="1.5.2" />
<PackageVersion Include="MessagePack" Version="2.5.301" />
<!-- First-party bearer validation for /auth/token's JWTs (R03:2.4, Lane C fix round 1). Same 10.0.x servicing
line as the other Microsoft.AspNetCore.* pins. It depends on Microsoft.IdentityModel.Protocols(.OpenIdConnect)
8.0.1; those two are pinned to 8.11.0 below (and referenced directly by the Security project, the JwtBearer
entry point, so the pin applies without transitive pinning) to keep the whole IdentityModel family on one
version with Microsoft.IdentityModel.Tokens / System.IdentityModel.Tokens.Jwt. -->
<PackageVersion Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.DataProtection" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.0" />
<PackageVersion Include="Microsoft.AspNetCore.SignalR.Client" Version="10.0.0" />
<!--
Surgical transitive pin #4. GHSA-334f-vvp4-7g6p (CVE-2026-70159, NU1903 high) flags
Microsoft.AspNetCore.SignalR.Common 10.0.0..10.0.12, which SignalR.Client 10.0.0 pulls in via
SignalR.Client.Core. 10.0.13 is the first fixed version, pinned by direct PackageReference in
AdminUI (the one project that references SignalR.Client). Bumping SignalR.Client itself to
10.0.13 was tried and rejected: SignalR.Client.Core 10.0.13 floors Microsoft.Extensions.Logging /
DependencyInjection at 10.0.13, which the AdminUI shared framework absorbs but non-Web consumers
downstream (OpcUaServer.IntegrationTests, which references Logging.Abstractions 10.0.7 directly)
do not, producing an NU1605 downgrade.
-->
<PackageVersion Include="Microsoft.AspNetCore.SignalR.Common" Version="10.0.13" />
<PackageVersion Include="Microsoft.AspNetCore.SignalR.Core" Version="1.2.0" />
<PackageVersion Include="Microsoft.AspNetCore.TestHost" Version="10.0.7" />
<!--
@@ -71,6 +88,8 @@
<PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.7" />
<PackageVersion Include="Microsoft.IdentityModel.Protocols" Version="8.11.0" />
<PackageVersion Include="Microsoft.IdentityModel.Protocols.OpenIdConnect" Version="8.11.0" />
<PackageVersion Include="Microsoft.IdentityModel.Tokens" Version="8.11.0" />
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
<PackageVersion Include="Microsoft.Playwright" Version="1.51.0" />
@@ -149,8 +168,28 @@
Microsoft.Extensions.* and, through the EFCore adapter, Microsoft.EntityFrameworkCore at
10.0.10, forcing a family-wide servicing bump (NU1605 downgrade cascade) that deserves its own
reviewed change.
2026-10-22: raised 10.0.10 -> 10.0.13. Three more NU1903 advisories (GHSA-7hqp-7c5m-4vq6,
GHSA-c399-9853-6jmj, GHSA-r955-8q5j-3h83; vulnerable <= 10.0.12) landed against 10.0.10, again
red only on a fresh restore (~67 projects). 10.0.13 is the first fixed version. Same pin site,
same reasoning against the DataProtection parent bump.
-->
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.13" />
<!--
Surgical transitive pin #3. GHSA-846f-f764-76mx (NU1903 high) + GHSA-2fmv-rv2c-59x7 (NU1902
moderate) flag System.Security.Cryptography.Pkcs <= 10.0.12 and 9.0.0..9.0.20. Two chains bring
it in, both entering at Core.Configuration:
- System.Security.Cryptography.Xml 10.0.x -> Pkcs 10.0.x (same version as Xml), and
- Microsoft.EntityFrameworkCore.SqlServer 10.0.7 -> Microsoft.Data.SqlClient 6.1.1 -> Pkcs 9.0.4.
In Web-SDK consumers (AdminUI, ControlPlane, Host, Security, ...) the ASP.NET Core shared
framework prunes the Xml chain, so the SqlClient 9.0.4 floor is what resolves there (~9
projects); everywhere else the Xml chain's 10.0.10 won (~67 projects). Raising the Xml pin alone
only fixes the second group, so Pkcs is pinned explicitly at 10.0.13 (first fixed 10.x) by a
direct PackageReference in Core.Configuration. Its net10.0 dependency group is empty, so the pin
floors nothing else. Bumping Microsoft.Data.SqlClient / EFCore.SqlServer instead would be the
family-wide servicing bump the Xml note above declines.
-->
<PackageVersion Include="System.Security.Cryptography.Pkcs" Version="10.0.13" />
<PackageVersion Include="System.CommandLine" Version="2.0.5" />
<PackageVersion Include="System.Data.SqlClient" Version="4.9.0" />
<PackageVersion Include="System.IdentityModel.Tokens.Jwt" Version="8.11.0" />
@@ -181,8 +220,8 @@
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.2.1" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Contracts" Version="0.2.1" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Contracts" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Auth.Abstractions" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Auth.Ldap" Version="0.3.0" />
+4 -1
View File
@@ -1,5 +1,8 @@
# Integration-test sweep — status + follow-ups (2026-07-14)
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
**Context:** ran the deferred `*.IntegrationTests` sweep (the round-2 STATUS.md "Full integration-suite
sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24).
**How:** tests run **on the Mac** (repo + .NET 10 SDK), pointing at **driver fixtures brought up on the
@@ -21,7 +24,7 @@ need sudo to create new ones — I deployed FOCAS to `~/otopcua-focas` on the ho
| **OpcUaClient.Browser** | ✅ 1/1 | — |
| **S7** | ✅ 3/3 (2 skip) | skips = outage tests needing `S7_TIMEOUT_OUTAGE_*` env |
| **Modbus** (profile=standard) | ✅ 6/6 | — |
| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy E2E time out under amd64-emulated SQL (OPEN); LDAP image swapped OpenLDAP→GLAuth 2026-07-15 (PR #451). The 5 skips are driver-fixture-gated E2E, not LDAP |
| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy E2E time out under amd64-emulated SQL (OPEN → #757); LDAP image swapped OpenLDAP→GLAuth 2026-07-15 (PR #451). The 5 skips are driver-fixture-gated E2E, not LDAP |
| **TwinCAT** | ⏭️ 13 skip | no TC3 XAR fixture exists (expected clean-skip) |
| **Modbus** (dl205 / mitsubishi / s7_1500 / exception_injection) | ✅ 16/13/10/17 (was 8F/5F/3F/8F) | **NOT drift** — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below |
| **AbCip** | ✅ 10P/3skip (was 6F/1P/3skip) | static-init-order bug FIXED 2026-07-15 (`326b22a7`) — see below |
+5 -2
View File
@@ -1,5 +1,8 @@
# Arch-Review Remediation — Status & Handoff
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
**Snapshot:** 2026-07-12 · **Review baseline:** master `9cad9ed0`; **re-review baseline:** master
`f6eaa267` · this file is the single source of truth for remediation progress. Update it as work lands.
@@ -161,9 +164,9 @@ removed from `v2-ci.yml`), 07/U-3 (5 alarm operator CLI commands documented —
+ blocker comment; upstream-blocked — no xunit.v3 Akka TestKit exists). 07/U-6 needed no action (the
Wonderware husk dirs are untracked build artifacts, absent in a fresh worktree). Whole-solution build
clean. **OPEN follow-ups:** (a) when a runner-reachable fleet + a real `tests/Server/…E2ETests` land,
add a fresh E2E workflow WITH a zero-match guard (fail if `Category=E2E` matches nothing); (b) S-2
add a fresh E2E workflow WITH a zero-match guard (fail if `Category=E2E` matches nothing) → #757; (b) S-2
option #2 — a fixture-backed integration leg (opc-plc `services:` block, `MIN_EXECUTED=1`) covering
what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit.
what the unit leg cannot → #757; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit → #756.
### Round-2 remediation landed
+4 -1
View File
@@ -1,5 +1,8 @@
# Deferment register — new drivers, open issues, deferred work
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
> **Scope.** A source-verified inventory of what is unfinished in this repo: the status of the
> recently-added drivers, every open tracker issue, in-code deferrals, open live gates, and the
> documentation that misstates any of it.
@@ -60,7 +63,7 @@ So everything below is **untracked work**. Nothing in it is a data-loss or outag
| ~~**G-7 residual** — Sql / MTConnect / Calculation / OpcUaClient have no Contracts-side `Inspect`~~ ✅ **CLOSED 2026-08-09** (`2947a2b9` + `1469fbb4`, merge `0695f682`). All four now ship a Contracts-side `*TagDefinitionFactory.Inspect` **wrapping the runtime parse, never mirroring the AdminUI model** — Sql wraps the pre-existing `SqlEquipmentTagParser`; the MTConnect / OpcUaClient / Calculation blob readers were **moved** into their Contracts assemblies (the drivers now call them), so never-stricter-than-runtime is structural. `Driver.Calculation.Contracts` was created (no Contracts assembly existed). The reflection coverage guard was falsified (map entry removed → red naming `SqlTagDefinitionFactory`), and `TagConfigAuthoringDeployParityTests` pins authoring-accepts ⇒ deploy-accepts. **Galaxy is now the sole unmapped type, deliberately + symmetrically** (no typed authoring editor either). ~~Found en route, left open: `MTConnectTagConfigModel` reads `dataType` leniently (misspelled name → silently `String`) while the runtime rejects it — authoring weaker than runtime, the *opposite* direction from G-7; documented inline in the parity tests.~~ ✅ Filed and FIXED same day as **#549** — and it was 6 models wide, not 1 (see the "2026-08-09 (later)" §9 entry); residuals **#550**/**#551** are ✅ **CLOSED 2026-08-10** (merges `b50523b1` + `333ea488`, see the 2026-08-10 §9 entry) — their own residuals are **#552** (Sql browse-commit emits a config the gate rejects) / **#553** (CSV typed-column merge silently rewrites typo'd enums pre-gate). | ~~The deploy gate is weaker than the authoring gate.~~ Config arriving by CSV import or API now meets the same checks the modal applies. | §1.2 |
| ~~**3 carried-open archreview findings** — the failure-visibility trio~~ **Update 2026-08-07 (truth sweep): mostly CLOSED.** 01/S-1 **FIXED** — `AddressSpaceApplier.cs:95-97` tallies swallowed removal-pass failures into `AddressSpaceApplyOutcome.FailedNodes` (comment cites 01/S-1). 03/S4 **FIXED** — `PrimaryGatePolicy.cs` default-denies on unknown role when a driver peer exists (doc-comment "Fixes archreview 03/S4"), live-proven R2-04 T15. 06/S-1 **PARTIAL** — fail-closed advise path + `galaxy.writes.unconfirmed` meter shipped in `GatewayGalaxyDataWriter.cs`; residual = gateway-side WriteComplete correlation (cross-repo, mxaccessgw). **Update 2026-08-09: the gateway half SHIPPED** — mxaccessgw main `b948e69` correlates the worker's `OnWriteComplete` onto the unary `WriteSecured` reply (`statuses[0]`; bounded wait default **1.5 s** — deliberately under OtOpcUa's 2 s Tier A Write budget, configurable via `MxGateway:Worker:WriteCompletionWaitMilliseconds`; timeout falls back to today's empty-statuses shape, still metered unconfirmed). **Update 2026-08-09 (later): 06/S-1 CLOSED — live-gated end to end, and the gate found + fixed three defects.** The user-authorized redeploy exposed (1) the correlation covered `WriteSecured` only while OtOpcUa's dominant no-login path emits plain `Write` — fixed gateway-side (worker `53f69cd`, all four unary write kinds, redeployed both hosts); (2) OtOpcUa's `StatusCodeMap.FromMxStatus` truncated the correlated failure row's MxStatus detail code to a byte (`1007 & 0xFF = 239` → OPC DA Good band), mapping a REFUSED write to Good — fixed with the not-IsSuccess-never-Good invariant + category routing (`SECURITY_ERROR → BadUserAccessDenied`); (3) the writer's borrow-from-subscription-registry optimization could never commit (a subscription handle is plain-advised; MXAccess refuses the no-login write `SECURITY_ERROR 1008` AND the advise upgrade `E_INVALIDARG` — one advise per HANDLE, not per attribute) — borrow path REMOVED, writer always owns a supervisory-advised handle. Live legs all PASSED on windev: happy write Good off confirmed `statuses[0]` + both meters flat; read-only refusal surfaces Bad-family + node-revert input; `WriteCompletionWaitMilliseconds=0` control → provisional Good + metered unconfirmed (override removed after). Pinned by `Live_correlated_write_statuses_confirm_commit_and_surface_refusal`, `Live_write_to_subscribed_tag_commits_via_own_handle`, `GatewayGalaxyWriteMeterLiveTests`, and the new `StatusCodeMapTests` rows. | ~~Residual is only the 06/S-1 live leg~~ **CLOSED 2026-08-09** — nothing remains. | §6.3 |
| ~~**1 genuinely unexecuted plan** — `2026-05-29-adminui-followups`, **19 pending / 0 completed**~~ ✅ **CLOSED 2026-08-07 by the overlap audit it asked for — it was bookkeeping drift, not backlog.** All 19 tasks are shipped, superseded by v3, or moot; **no code was needed.** The warning was right and then some: WS1's whole *premise* died with v3 (driver-embedded tag lists in `DriverConfig` are no longer the runtime source of truth — the artifact populates `RawTags`), and its capability was delivered in the v3 shape (`TagConfigEditorMap` + `DeviceFormMap`, both verified **reachable**, not merely present). WS2/WS3 shipped; WS4's target files were deleted by v3 Batch 2 + #542. Per-task evidence in the `.tasks.json`. | §6.2 |
| ~~**11 other open plan tasks**~~ **3 as of 2026-08-10** — ~~5 akka-hosting partials (likely v3-superseded)~~ *(verified superseded 2026-08-07 — F8/F9/F10/F13/F14 marked completed; `Phase7Composer`/`ScriptedAlarmActor` deleted, evaluators/SDK sinks/security policies all production-wired)*, ~~4+1 followups-batch (open *by choice*)~~ *(2026-08-10: Tasks 9 + 10 VERIFIED PASS live — Int64 authoring end-to-end on the rebuilt rig; probe E2E 12/12 with the "needs Windows-VM" premise corrected to TCP-reachability gating)*, ~~fixedtree Task 11~~ *(closed 2026-08-10: obsolete — the injection feature it gated was deleted with #507)*, ~~1 manual smoke checklist~~ *(closed 2026-08-10: obsolete — the driver pages it smokes were retired in v3 Batch 2)*. ~~**Remaining 3: F10b + SetSeverity (reconsider gates, decisions already taken — do not build without a fresh go-ahead) and the device-gated proofs (hardware-blocked).**~~ **Remaining 1 as of 2026-08-10 (later that day): F10b + SetSeverity reconsider gates put to the user and CLOSED decided-against (F10b re-open trigger: operator complaints about rebuild churn on data-type edits; SetSeverity is operationally invisible — engine overwrites on first eval); the device-gated proofs bundle trimmed to its 2 genuinely-gated sub-items (H6 native-ack→AVEVA + AbLegacy/TwinCAT/FOCAS probes).** | Hardware-blocked only. | §6.2, §9 2026-08-10 (sixth) |
| ~~**11 other open plan tasks**~~ **3 as of 2026-08-10** — ~~5 akka-hosting partials (likely v3-superseded)~~ *(verified superseded 2026-08-07 — F8/F9/F10/F13/F14 marked completed; `Phase7Composer`/`ScriptedAlarmActor` deleted, evaluators/SDK sinks/security policies all production-wired)*, ~~4+1 followups-batch (open *by choice*)~~ *(2026-08-10: Tasks 9 + 10 VERIFIED PASS live — Int64 authoring end-to-end on the rebuilt rig; probe E2E 12/12 with the "needs Windows-VM" premise corrected to TCP-reachability gating)*, ~~fixedtree Task 11~~ *(closed 2026-08-10: obsolete — the injection feature it gated was deleted with #507)*, ~~1 manual smoke checklist~~ *(closed 2026-08-10: obsolete — the driver pages it smokes were retired in v3 Batch 2)*. ~~**Remaining 3: F10b + SetSeverity (reconsider gates, decisions already taken — do not build without a fresh go-ahead) and the device-gated proofs (hardware-blocked).**~~ **Remaining 1 as of 2026-08-10 (later that day): F10b + SetSeverity reconsider gates put to the user and CLOSED decided-against (F10b re-open trigger: operator complaints about rebuild churn on data-type edits; SetSeverity is operationally invisible — engine overwrites on first eval); the device-gated proofs bundle trimmed to its 2 genuinely-gated sub-items (H6 native-ack→AVEVA + AbLegacy/TwinCAT/FOCAS probes).** (→ #614 H6; TwinCAT probe → #755) | Hardware-blocked only. | §6.2, §9 2026-08-10 (sixth) |
| ~~**Review-coverage gap** — 10 shipped projects never reviewed (Calculation, Historian.Gateway, Mqtt +Browser/Contracts, MTConnect +Contracts, Sql +Browser/Contracts) while **3 modules still review the retired Wonderware backend**~~ **CLOSED 2026-08-10**: all 14 unreviewed projects (the 10 + Mqtt.Transport, Sql.Dialect, Calculation.Contracts, OpcUaClient.Session) reviewed by 6 parallel agents — 61 findings, 26 fixed, 35 Open in `findings.md`; Wonderware reviews banner-retired to history; README at 65 modules, `--check` clean. **The recorded Opens were themselves resolved 2026-08-10 (eighth round): fleet-wide 0 pending findings** | §9 2026-08-10 (seventh, eighth) | §6.3, §9 2026-08-10 (seventh) |
| ~~**`RawBrowseModal` stale-open race** (added 2026-08-05) — `OnParametersSetAsync` never re-checks staleness after its awaits, so a superseded open's late resolve can install the wrong device's browse config. The identical shape was found by review and **fixed** in `RawTagModal`/`RawManualTagEntryModal` (post-await staleness re-check; see `docs/plans/2026-08-05-raw-modal-picker-live-config-design.md` §6); the browse modal was deliberately left out of that branch's scope.~~ ✅ **CLOSED 2026-08-09** (`87637e4e`, merge `0739f4b9`) — the sibling `_openGeneration` guard ported to every post-await write, **plus two departures the siblings did not need**: this modal resolves inside `try/finally`, so the `catch` (a stale open's failure would paint its error over the live device) and the `finally` (a stale `return` still runs the tail, dropping the LIVE open's spinner) are generation-guarded too. Four `RawBrowseModalStaleOpenTests` facts, each observed red pre-guard and each re-falsified by reverting only its own guard (1-failed/3-passed per revert — no redundant pin). | ~~Lower stakes than the fixed pair… but the fix pattern is proven and cheap to port.~~ Ported. | design doc §6 |
+5 -5
View File
@@ -512,16 +512,16 @@ same pass (a UNS reference variable, say) are visible through the index with no
⚠️ **The value fan is now a single message.** It sends one
`DriverInstanceActor.AttributeValueBatchPublished` carrying one item per RawPath, where it previously sent
one `AttributeValuePublished` per ref — 5 000 messages onto the mux that also carries the node's steady
one per-value message per ref — 5 000 messages onto the mux that also carries the node's steady
value ingress, every time one driver flapped. Delivery is unchanged: the mux filters the batch per item
against the same registered-interest map the per-value receive consulted, so only the framing moved. The
**alarm** fan is indexed but deliberately not batched — one driver ref can back several condition NodeIds,
so it still emits one `AlarmQualityUpdate` per condition.
**Consequence worth knowing: `AttributeValuePublished` now has no production sender at all.** This fan was
its last one — everything on the subscription path was already wrapped in `AttributeValueBatchPublished`.
`DependencyMuxActor` still registers the per-value receive, but in production it is dead code; it stays
for tests and direct senders.
**Consequence: the per-value `AttributeValuePublished` record is deleted.** This fan was its last
production sender (everything on the subscription path was already wrapped in
`AttributeValueBatchPublished`), so the record and `DependencyMuxActor`'s per-value receive were removed
(2026-10-22). A single value is a one-item batch; tests build one with `ValuePublishes.Single`.
#### The third fan: the value nodes (archreview 03/S15)
+2 -2
View File
@@ -39,8 +39,8 @@ This was not always true. Until #519 the publish path projected the driver's `ui
`Good`/`Uncertain`/`Bad` enum in `DriverInstanceActor`, and `OtOpcUaNodeManager` re-expanded `Bad` to
a bare `StatusCodes.Bad` (`0x80000000`). Every driver's diagnostic intent survived in logs only —
including the 16 status constants #497 had just corrected across six drivers. The value path now
carries the raw `uint` end to end (`AttributeValuePublished` → `AttributeValueUpdate` →
`IOpcUaAddressSpaceSink.WriteValue`), and severity classification is a helper
carries the raw `uint` end to end (`AttributeValueBatchPublished` → `AttributeValueBatchUpdate` →
`IOpcUaAddressSpaceSink.WriteValues`), and severity classification is a helper
(`OpcUaStatus.IsGood`/`IsUncertain`/`IsBad`) rather than a lossy projection.
**Writing a status code in engine code:** use a named constant on
+76 -8
View File
@@ -1,5 +1,8 @@
# Monaco Script Editor (Roslyn-backed IntelliSense)
> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.
The script editor gives AdminUI users a first-class C# authoring experience for
virtual-tag scripts: completions, diagnostics, hover, signature help, document
formatting, and context-aware tag-path completions — all backed by the same
@@ -162,11 +165,76 @@ user's fault.
`ForbiddenTypeAnalyzer.Analyze(compilation)` walks the wrapped syntax tree and
resolves every referenced symbol against the sandbox deny-list (namespace
prefixes `System.IO`, `System.Net`, `System.Diagnostics`, `System.Reflection`,
`System.Threading.Tasks`, etc., plus type-granular denials like
`System.Environment` and `System.Threading.Thread`). Each violation is mapped
`System.Threading` — the whole namespace, with only `CancellationToken`,
`Interlocked` and `Volatile` allowed back in — `System.Threading.Tasks`,
`System.Runtime.InteropServices`, `System.Runtime.Loader`, `Microsoft.Win32`;
type-granular denials `System.Environment`, `System.AppDomain`, `System.GC`,
`System.Activator`, `System.Runtime.CompilerServices.Unsafe` /
`RuntimeHelpers`; and member-granular denials on `System.Type`'s by-name
reflection entry points — `GetType(string)`, `GetMethod(s)`, `InvokeMember`,
and the `GetProperty`/`GetField`/`GetMember`/`GetConstructor`/`GetEvent`
families). The runtime publish gate uses the same analyzer, so a deny-list
change needs a `DiagnoseTests` case to keep editor ⇔ publish parity. Each violation is mapped
via `tree.GetMappedLineSpan(span)` to user-source coordinates and emitted as an
error marker with code `OTSCRIPT_FORBIDDEN`.
### Runtime safety (what happens after publish)
The editor cannot catch a script that compiles cleanly but never returns or
allocates without bound, so every runtime seam (virtual tags, Calculation tags,
scripted-alarm predicates) runs scripts through the shared `ScriptRunGuard`:
- **Timeout + one orphan.** A run that exceeds its budget is abandoned (it cannot
be pre-empted in-process) and the tag/alarm reads Bad. While that abandoned run
is still executing, every further trigger is **refused** — Bad
`"previous evaluation still running"`, counted on `otopcua.script.run_refused` —
so a `while(true)` costs one pool thread, not one per trigger.
- **Quarantine after 3 consecutive timeouts** (`Scripting:QuarantineAfterConsecutiveTimeouts`).
Only a timeout is a strike: a refusal neither adds to nor resets the streak, and
a run that returns (value or throw) resets it. A quarantined script is never
dispatched again; it reads Bad `"script quarantined: …"`, logs one Error and
increments `otopcua_script_quarantined`.
- **Stuck-orphan quarantine.** A never-returning script (`while(true)`) times out
once and is then only refused, so the timeout streak alone never reaches 3.
Instead, a refusal behind an orphan that has been running longer than
`Scripting:StuckOrphanQuarantineFactor` × the timeout (default 10 × — 20 s at the
2 s default) quarantines immediately, through the same path (Error log, metric
`reason=stuck`, Bad). A short overrun (a young orphan) stays a plain refusal and
the script resumes once the orphan completes.
- **Cooperative cancellation (`ctx.CancellationToken`).** The guard cancels the run
token when a run times out, and scripts see it as `ctx.CancellationToken`. A
long-running loop that polls `ctx.CancellationToken.IsCancellationRequested` (or calls
`ctx.CancellationToken.ThrowIfCancellationRequested()`) ends its own abandoned run
and frees the slot, instead of holding a pool thread until it returns. A script that
ignores the token is still an orphan. The editor completes the member, and
`CancellationToken` is on the sandbox allow-list.
- **Allocation budget.** A single run allocating more than
`Scripting:MaxAllocBytesPerRun` (64 MB) is quarantined. This is sampled after the
run returns, so a run that OOMs the process first is not stopped — that needs an
out-of-process runner (→ #750).
- **Configuration keys and their ranges.** All three knobs are node-wide (the
`Scripting` appsettings section, shared by virtual tags, Calculation tags and
scripted alarms) and are clamped when bound, with one startup Warning per
clamped value:
| Key | Default | Range after the clamp |
|---|---|---|
| `Scripting:MaxAllocBytesPerRun` | `67108864` (64 MB) | `0` (or negative) disables the budget; a positive value below `1048576` (1 MB) becomes 1 MB |
| `Scripting:QuarantineAfterConsecutiveTimeouts` | `3` | `1`–`1000` |
| `Scripting:StuckOrphanQuarantineFactor` | `10` | `0` (or negative) disables the rule; above `1000` becomes `1000`; `NaN` becomes the default |
A Calculation driver may override `maxAllocBytesPerRun` and
`quarantineAfterConsecutiveTimeouts` in its own driver config. Those keys are
layered on the node values and clamped to the same ranges, and an out-of-range
value logs a Warning naming the driver. The stuck factor has no per-driver key.
- **Lifting.** A quarantine lifts when the script source changes or on redeploy,
never by itself.
- **Script-log rate limit.** The cluster-wide script-log fan-out admits 20 entries
per source per minute; the rest are counted and replaced by one "N suppressed"
summary. The summary is emitted on that source's **next** entry after the window
rolls, so a source that bursts and then goes quiet shows its summary only when
it logs again (the full stream is always in `scripts-*.log`).
### 3. DependencyExtractor
`DependencyExtractor.Extract(code).Rejections` enumerates every `ctx.GetTag` /
@@ -200,7 +268,7 @@ prefix. The concrete implementation `ScriptTagCatalog` reads `Tag` +
| Virtual tag | Leaf `Name` (best-effort) |
**Why only resolvable keys?** The live runtime resolves a `ctx.GetTag("X")`
literal against `DriverInstanceActor.AttributeValuePublished.FullReference`,
literal against `DriverInstanceActor.AttributeValueItem.FullReference`,
which is the `FullName` field from `Tag.TagConfig`
(see `ScriptTagCatalog.ExtractFullNameFromTagConfig` + `EquipmentNodeWalker.ExtractFullName`).
The former UNS-path cascade engine (`Core.VirtualTags.VirtualTagEngine`, keyed by the
@@ -426,20 +494,20 @@ site.
- **Tag-path completion shows resolvable keys only.** Surfacing the UNS browse
path as a completion *detail* (a non-inserted hint shown alongside the
resolvable key) for discoverability is a tracked follow-up.
resolvable key) for discoverability is a tracked follow-up. → #759
- **Literal gate matches by method name, not receiver.** `anything.GetTag("…")`
would also trigger tag-path completion (not just `ctx.GetTag("…")`). Harmless
in the sandbox today — no other `GetTag` methods exist in the allowed reference
set — but tighter binding to `ctx` / `VirtualTagContext` is a follow-up.
set — but tighter binding to `ctx` / `VirtualTagContext` is a follow-up. (Reconcile: → #759)
- **Analysis runs synchronous Roslyn (CPU-bound).** Each endpoint call builds a
`CSharpCompilation` synchronously. This is intentional: the endpoints are
admin-only, client-side debounced (~500 ms), and `Task.Run` offload adds
unnecessary complexity for infrequent admin use. If concurrency becomes a
concern under heavy admin usage, wrapping each `Analyze` call in `Task.Run` is
the correct fix.
the correct fix. (Reconcile: → #759)
- **InlayHints is a stub.** `/api/script-analysis/inlay-hints` returns an empty
list. Filling it in (parameter name hints from the semantic model) is a
follow-up task.
follow-up task. (Dropped — reconcile: → #759)
- **Language providers register globally at Monaco load.** The six providers are
registered for the `csharp` language on the global Monaco instance (loaded once
per page in `App.razor`). There is no impact today because only
@@ -449,7 +517,7 @@ site.
results).
- **Creating a new script from the VirtualTagModal inline panel** is deferred.
The current inline panel loads and saves an existing `Script` row; creating a
brand-new script requires a separate create flow and is a follow-up.
brand-new script requires a separate create flow and is a follow-up. (Shipped — reconcile: → #759)
---
+22 -7
View File
@@ -1,5 +1,8 @@
# UNS — Global Unified-Namespace Management
> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.
The **UNS** page (`/uns` in the AdminUI) is the single surface for managing
the Unified Namespace across the whole fleet. It replaces the old
per-cluster **UNS**, **Equipment**, and **Tags** tabs and the standalone
@@ -323,16 +326,28 @@ carried on the plan records (`RawTagPlan.ArrayWritable`, mirrored onto every UNS
| Driver | Array writes | Element types |
|---|---|---|
| **Modbus** | Yes | `region: Coils` (FC15, any element, ≤ 1968 coils); `region: HoldingRegisters` with `Int16`/`UInt16`/`Bcd16`/`Int32`/`UInt32`/`Bcd32`/`Float32`/`Int64`/`UInt64`/`Float64` (FC16) whose `arrayLength × registers-per-element` fits `maxRegistersPerWrite` (default 123; the smallest value set on the driver or the tag's device wins) — the driver refuses rather than splits an over-cap FC16, so e.g. a 62-element `Int32` array (124 registers) stays read-only. Not BitInRegister / String / Bool-in-register; input regions are read-only |
| **Modbus** | Yes (String / BitInRegister arrays since #741) | Tags addressed by `region` + `address` keys (strict reads: an integer address 0–65535, enum strings that name a member). `region: Coils` (FC15, ≤ 1968 coils) with **any** `dataType` — a bit region is coerced to Bool, never refused. `region: HoldingRegisters` (FC16) with a fixed-width numeric type (`Int16`/`UInt16`/`Bcd16`/`Int32`/`UInt32`/`Bcd32`/`Float32`/`Int64`/`UInt64`/`Float64`), `String` (`(stringLength+1)/2` registers per element, `stringLength >= 1`) or `BitInRegister` (one register per element, read-modify-written in one FC03 + one FC16, `bitIndex` 0–15, and — because of that unchunked FC03 — also within `maxRegistersPerRead`, default and protocol ceiling 125), whose `arrayLength × registers-per-element` fits `maxRegistersPerWrite` (default 123; the smallest value set on the driver or the tag's device wins; a value above FC16's protocol ceiling of 123 never lifts it, since a conformant device refuses a longer FC16) — the driver refuses rather than splits an over-cap FC16, so e.g. a 62-element `Int32` array (124 registers) stays read-only. Every span (registers or coils) must end at or before address 65535. FC23 (opt-in) never changes the answer. Not Bool-in-register; input regions are read-only. A **string-only** tag (`addressString`, no keys) stays read-only: the driver resolves it with its own family, which the composer cannot know — use the editor's Resolve to pin the keys |
| **S7** | Yes | `Byte`/`Int16`/`UInt16`/`Int32`/`UInt32`/`Float32` (one contiguous block). Not `Bool` (would clobber neighbouring bits) nor the wide / String / DateTime types |
| **OpcUaClient** | Yes | All — forwarded upstream unchanged; the upstream server decides (shape included) |
| **Galaxy** | Yes | OPC UA `Boolean`/`Int32`/`Int64`/`Float`/`Double`/`String`/`DateTime` (the `MxValueEncoder` array variants) — **only with a declared `arrayLength`**: an undeclared length materialises an unbounded node, and a short whole-array write would *resize* the Galaxy attribute (mxaccessgw contract), so such a node stays read-only |
| AB CIP, AB Legacy, TwinCAT, FOCAS, MQTT, MTConnect, Sql, Calculation | **No** — scalar-only write encoders; array nodes stay read-only and a client write is `BadNotWritable` | — |
| **AB CIP** | Yes (#737) | `SInt`/`USInt`/`Int`/`UInt`/`DInt`/`UDInt`/`Real`/`Dt`/`LInt`/`ULInt`/`LReal` (`AbCipArrayCodec`, both transports). Not `Bool` (bit-packed DWORDs), `String` or `Structure`, nor a `safetyTag` |
| **AB Legacy** | Yes (#738) | `Int`/`AnalogInt` on an N/B/A file, `Long` on an L file, `Float` on an F file — the encode width is the data file's, so a type that does not match its file stays read-only — on a plain `<file><number>:<word>` address, ≤ 256 elements. Not Bit / String / Timer / Counter / Control arrays, `/bit` or `.sub` addresses, indirect addresses, the `S:`/`I:`/`O:` files, nor any PLC-5 on the native transport |
| **TwinCAT** | Yes (#739) | Every element type `TwinCATArrayCodec` lays out (numeric, `Bool`, `String`/`WString`, temporal); multi-dimensional PLC arrays are written row-major from the flat value. Not a BOOL bit-in-word path (`MAIN.Word.3`) |
| FOCAS, MQTT, MTConnect, Sql, Calculation | **No** — scalar-only write encoders; array nodes stay read-only and a client write is `BadNotWritable` | — |
Modbus, S7 and Galaxy additionally need a **declared length** — `isArray: true` *and* an integer
`arrayLength >= 1`. The Modbus and S7 tag factories read anything else as a **scalar** tag (whose
scalar encoder cannot take an array), even though the composer still materialises an array node;
that node stays read-only.
Every "Yes" driver but OpcUaClient additionally needs a **declared length** — `isArray: true` *and* an
integer `arrayLength >= 1`. The tag factories read anything else as a **scalar** tag (whose scalar
encoder cannot take an array), even though the composer still materialises an array node; that node
stays read-only.
**Elements convert; they are not exact-matched.** ScadaBridge writes CLR `int[]` / `List<int>` to Int16
nodes: the node manager converts a numeric array to the node's element type, and the AB CIP, AB Legacy
and TwinCAT encoders convert each element again to the controller's type (an element that does not fit
is `BadOutOfRange`), as does Modbus to the register / coil type. Those rows therefore do not key on an
exact node `DataType`; they leave out only node types whose elements the server cannot promise to convert — a `DateTime` (never converts) or
`String` node over a numeric element (converts only when the text happens to parse, `"1"` / `"true"`;
on Modbus, over a numeric or bit element; a Modbus `String` element takes any node type), and a `Boolean` node on AB Legacy. An array tag deployed read-only before its driver gained an encoder turns
writable on the next deploy (a surgical `ArrayWritable` update, no rebuild).
Each "Yes" row is pinned against the driver's real encoder by a parity test in that driver's
test project (`*ArrayWriteSupportParityTests`), so a driver gaining or losing an array encoder
@@ -516,7 +531,7 @@ equipment page.
Phase 6 did **not** implement per-driver-instance status rows on the Hosts
page. This item (H7-runtime) is **F7-runtime-blocked**: the runtime plumbing
needed to surface per-instance health rows is not yet in place. It remains on
the backlog.
the backlog. (Stale — reconcile: → #759)
## Bulk import
+10 -7
View File
@@ -1,5 +1,8 @@
# Virtual Tags
> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.
> ⚠️ **Correction (2026-07-28): `GenericDriverNodeManager` is NOT a production dispatch path.** It is test
> scaffolding — its own source says so (`Core/OpcUa/GenericDriverNodeManager.cs:71`, "verified 07/#10: zero
> production references"). The production chain is `OpcUaPublishActor` → `IOpcUaAddressSpaceSink`
@@ -28,15 +31,15 @@ User scripts are compiled via `Microsoft.CodeAnalysis.CSharp` (regular compiler,
`ScriptSandbox.Build` constructs the compile reference set in two parts. First, four pinned OtOpcUa assemblies are always included: `Core.Abstractions` (for `DataValueSnapshot` / `DriverDataType`), `Core.Scripting` (for `ScriptContext` + `Deadband`), `Serilog` (for `ILogger`), and the concrete context type's assembly. Second, the BCL subset is enumerated from the runtime's `TRUSTED_PLATFORM_ASSEMBLIES` list, restricted to filenames starting with `System.*` plus `netstandard.dll`, `mscorlib.dll`, and `Microsoft.Win32.Registry.dll` (the last needed so `ForbiddenTypeAnalyzer` can resolve and reject registry types). Pre-imported namespaces: `System`, `System.Linq`, `ZB.MOM.WW.OtOpcUa.Core.Abstractions`, `ZB.MOM.WW.OtOpcUa.Core.Scripting`.
`ForbiddenTypeAnalyzer.ForbiddenNamespacePrefixes` denies `System.IO`, `System.Net`, `System.Diagnostics`, `System.Reflection`, `System.Threading.Tasks`, `System.Runtime.InteropServices`, `System.Runtime.Loader`, and `Microsoft.Win32`. Matching is by prefix against the resolved symbol's containing namespace, so `System.Net` catches `System.Net.Http.HttpClient` and every subnamespace. `System.Threading.Tasks` is denied because scripts are synchronous predicates with no legitimate need to start background tasks — a `Task.Run` fan-out would outlive the per-evaluation timeout entirely (Core.Scripting-003). `System.Runtime.Loader` is denied to block `AssemblyLoadContext` / `AssemblyDependencyResolver` — arbitrary DLL loads into the host process (Core.Scripting-012).
`ForbiddenTypeAnalyzer.ForbiddenNamespacePrefixes` denies `System.IO`, `System.Net`, `System.Diagnostics`, `System.Reflection`, `System.Threading` (the whole namespace — `ForbiddenTypeAnalyzer.AllowedTypesInForbiddenNamespaces` lets only `CancellationToken`, `Interlocked` and `Volatile` back in), `System.Threading.Tasks`, `System.Runtime.InteropServices`, `System.Runtime.Loader`, and `Microsoft.Win32`. Matching is by prefix against the resolved symbol's containing namespace, so `System.Net` catches `System.Net.Http.HttpClient` and every subnamespace. `System.Threading.Tasks` is denied because scripts are synchronous predicates with no legitimate need to start background tasks — a `Task.Run` fan-out would outlive the per-evaluation timeout entirely (Core.Scripting-003). `System.Runtime.Loader` is denied to block `AssemblyLoadContext` / `AssemblyDependencyResolver` — arbitrary DLL loads into the host process (Core.Scripting-012).
`ForbiddenTypeAnalyzer.ForbiddenFullTypeNames` denies type-granularly: `System.Environment`, `System.AppDomain`, `System.GC`, `System.Activator`, `System.Threading.Thread`, `System.Threading.ThreadPool`, and `System.Threading.Timer`. These types require granular denial rather than namespace-prefix denial for different reasons: `Environment` / `AppDomain` / `GC` / `Activator` live directly in the `System` namespace (which is otherwise allowed for primitives), so a namespace-prefix rule cannot reach them without blocking `int` / `string` / `Math`; `Thread` / `ThreadPool` / `Timer` live in `System.Threading` (shared with allowed types like `CancellationToken` and `SemaphoreSlim`), so a prefix on `System.Threading` would block those too. `Environment.Exit` / `FailFast` terminate the host process outright (Core.Scripting-001); `Thread` and `ThreadPool` reintroduce background-fanout vectors that `System.Threading.Tasks` denial closed (Core.Scripting-010 / -012).
`ForbiddenTypeAnalyzer.ForbiddenFullTypeNames` denies type-granularly: `System.Environment`, `System.AppDomain`, `System.GC`, `System.Activator`, `System.Runtime.CompilerServices.Unsafe`, and `System.Runtime.CompilerServices.RuntimeHelpers`. These live in namespaces that are otherwise allowed — `System` (primitives, `Math`) and `System.Runtime.CompilerServices` (compile-time attributes such as `CallerMemberNameAttribute`) — so a namespace-prefix rule cannot reach them. `Environment.Exit` / `FailFast` terminate the host process outright (Core.Scripting-001); `Unsafe.As` bypasses the type system; `RuntimeHelpers.GetUninitializedObject` builds an instance of any type without a constructor. `Thread` / `ThreadPool` / `Timer` / `SemaphoreSlim` and the rest of `System.Threading` are now caught by the namespace prefix above instead of being listed one by one — a script is a synchronous expression with no use for background work or blocking primitives (Core.Scripting-010 / -012, review R05:10). `ForbiddenTypeAnalyzer.ForbiddenMembers` additionally denies `System.Type`'s by-name reflection entry points (`GetType(string)`, `GetMethod(s)`, `InvokeMember`, and the `GetProperty`/`GetField`/`GetMember`/`GetConstructor`/`GetEvent` families), which would otherwise reach a forbidden type without naming it syntactically.
#### Known resource limits (accepted trade-offs)
The sandbox cannot prevent a script from **allocating unbounded memory**. A script calling `new byte[int.MaxValue]` repeatedly, or accumulating a large LINQ enumeration, can drive the server process to `OutOfMemoryException` before the 250 ms timeout fires. Script authoring is gated behind the Admin permission as the primary control; the test-harness preview (Stream F.4) allows operators to exercise a script before publishing. Out-of-process script execution is a v3 concern.
The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need an out-of-process runner (→ #750). Script authoring is gated behind the Admin permission as the primary control. See `docs/ScriptEditor.md` §"Runtime safety".
Similarly, **`System.Threading.Tasks` is now denied** (Core.Scripting-003), which prevents `Task.Run` / `Parallel` fan-out that would spawn background work outliving the timeout. However, a tight CPU-bound loop still runs on its thread-pool thread after `WaitAsync` returns — see the `TimedScriptEvaluator` remarks for detail. The orphaned thread is reclaimed when the Roslyn runtime eventually returns; in practice the operator fixes the script once the structured timeout warning appears in `scripts-*.log`.
`System.Threading.Tasks` is denied (Core.Scripting-003), which prevents `Task.Run` / `Parallel` fan-out that would spawn background work outliving the timeout.
### Compile cache (`CompiledScriptCache<TContext, TResult>`)
@@ -93,7 +96,7 @@ Virtual-tag nodes are materialised by `AddressSpaceApplier` and fed by `VirtualT
`ITagUpstreamSource` and `IHistoryWriter` are the two ports value consumers require from their host. Both live in `Core.VirtualTags`. In the v2 actor system:
- **Upstream-tag feed.** `DependencyMuxActor` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs`) routes `DriverInstanceActor.AttributeValuePublished` events to the `VirtualTagActor` instances that declared interest in those tag refs. Each `VirtualTagActor` holds the in-memory per-tag dependency map; the `IVirtualTagEvaluator` (`RoslynVirtualTagEvaluator`) receives the dependency snapshot synchronously on the actor message thread. Reads of never-pushed dependency refs return `null` values in the dependency snapshot.
- **Upstream-tag feed.** `DependencyMuxActor` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs`) routes the items of each `DriverInstanceActor.AttributeValueBatchPublished` to the `VirtualTagActor` instances that declared interest in those tag refs. Each `VirtualTagActor` holds the in-memory per-tag dependency map; the `IVirtualTagEvaluator` (`RoslynVirtualTagEvaluator`) receives the dependency snapshot synchronously on the actor message thread. Reads of never-pushed dependency refs return `null` values in the dependency snapshot.
- **`IHistoryWriter`** — the equipment-namespace path threads `Historize` end-to-end and `VirtualTagHostActor` invokes the injected writer on historized results (H5); the writer is resolved through `DriverHostActor` DI with a `NullHistoryWriter` default. No *durable* writer ships on this seam — durable continuous historization now flows through the separate `ContinuousHistorizationRecorder` → HistorianGateway `WriteLiveValues` path (see the `IHistoryWriter` section above and [Historian.md](Historian.md)).
## Composition
@@ -104,7 +107,7 @@ In the v2 actor system, virtual-tag engine composition is owned by the driver-ro
- `AddressSpaceComposer.Compose` emits `DriverInstancePlan` / `ScriptedAlarmPlan` records; the driver-role `DriverHostActor` spawns one `VirtualTagActor` per virtual-tag expression and one `ScriptedAlarmActor` per scripted alarm.
- `RoslynVirtualTagEvaluator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs`) is injected into each `VirtualTagActor` as its `IVirtualTagEvaluator`. It holds a per-source `CompiledScriptCache` keyed by script source and compiles on first use.
- `DependencyMuxActor` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs`) receives every `DriverInstanceActor.AttributeValuePublished` event and routes it to the `VirtualTagActor` instances that registered interest in that tag ref.
- `DependencyMuxActor` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs`) receives every `DriverInstanceActor.AttributeValueBatchPublished` and routes each item to the `VirtualTagActor` instances that registered interest in that tag ref.
`ITagUpstreamSource` and `IHistoryWriter` remain available as `Core.VirtualTags` ports.
@@ -125,7 +128,7 @@ In the v2 actor system, virtual-tag engine composition is owned by the driver-ro
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags/ITagUpstreamSource.cs` — driver-tag read + subscribe port
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags/IHistoryWriter.cs` — historize sink port + `NullHistoryWriter`
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs` — actor that receives `DependencyValueChanged` from the mux and invokes `IVirtualTagEvaluator` per expression
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs` — routes `DriverInstanceActor.AttributeValuePublished` to interested `VirtualTagActor` subscribers
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs` — routes `DriverInstanceActor.AttributeValueBatchPublished` items to interested `VirtualTagActor` subscribers
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs` — production `IVirtualTagEvaluator` binding; holds a per-source `CompiledScriptCache`
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceComposer.cs` — pure data composer: config-DB entities → `AddressSpaceComposition` (UNS topology + raw subtree + driver/alarm plans)
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` — applies the composed plan into the SDK node manager
+16 -5
View File
@@ -252,7 +252,7 @@ The driver has two interchangeable wire transports, selected by one `DriverConfi
write, read back, restored byte-exact). A user-approved **native array-element write** followed: `Program:EM01_PowerUnit._SEQ_EP[31]`
0→4242→0 (the array's only alias targets `[1]`, no routine indexes or block-copies it; all 32 elements snapshotted
twice before, after the write and after the restore — only `[31]` changed). Still ab_server-only: whole-array
writes (the driver has none), libplctag member writes, and fragmented writes. #566 is answered (the controller drops TCP; see "A target-closed CIP connection"). Mixed STRING/array/scalar
writes (#737; see [1-D array support](#1-d-array-support)), libplctag member writes, and fragmented writes. #566 is answered (the controller drops TCP; see "A target-closed CIP connection"). Mixed STRING/array/scalar
Multiple Service Packet reads (#582) are proven on the press (one 5-read MSP, values equal to single reads). Results:
[native-ab-transport-gate-results](../plans/2026-10-07-native-ab-transport-gate-results.md).
- **libplctag cannot be deleted yet.** Scalar, UDT-member and STRING native writes are proven on a CompactLogix
@@ -262,7 +262,7 @@ The driver has two interchangeable wire transports, selected by one `DriverConfi
libplctag member-write path is `ab_server`-only. See [Member writes](#member-writes).
- ⚠️ RESIDUAL (#567): wide-struct **fragmented writes** (8-byte alignment) are unit-tested only. A write fragments only
above `MaxRequestBytes` (4000 B on a Large Forward Open), so it needs a ~4 KB value; Z-35162 has no safe one (a DINT[32]
or a STRING is a single Write Tag, and AbCip has no whole-array write anyway). It stays open until a bench controller
or a STRING is a single Write Tag — and a whole DINT[32] array write, #737, is 128 B). It stays open until a bench controller
or an approved large scratch tag exists.
- Native browse (#572, resolved in code): listing paging no longer assumes ascending instance order, pages
are paced, and a browse's template reads share one lease. Observed on Z-35162 (2026-10-09, 41
@@ -361,7 +361,17 @@ integration fixture is down during normal dev (Docker host fixture currently off
**Live-verify** — integration-fixture-gated (requires `ab_server` Docker fixture on `10.100.0.35:44818`).
**Deferrals** — array *writes*, multi-dimensional arrays, per-element historization. Safety-partition array tags (`SafetyTag: true`) are excluded (GuardLogix write constraints extend to array writes).
**Array writes (#737, review 07 U10)** — a whole-array write of exactly `arrayLength` elements, encoded by
`AbCipArrayCodec` little-endian at the element width on both transports. Element values **convert**
(ScadaBridge writes `int[]` to INT arrays); a value that does not fit is `BadOutOfRange`, one that cannot convert
`BadTypeMismatch`, a different length `BadOutOfRange`, all before anything is sent. The controller buffer must be
width × count (else `BadTypeMismatch`, checked before sending). Writable element types: `SInt`/`USInt`/`Int`/`UInt`/
`DInt`/`UDInt`/`Real`/`Dt`/`LInt`/`ULInt`/`LReal`. **BOOL arrays** (bit-packed DWORDs), **STRING** and **structure**
arrays are refused `BadNotSupported`, so their nodes stay read-only (`DriverArrayWriteSupport`); so does a
safety-partition array tag (`safetyTag: true`, `BadNotWritable`). An array deployed read-only before #737 turns
writable on the next deploy (a surgical `ArrayWritable` update). Proven on `ab_server` only until the hardware gate runs.
**Deferrals** — multi-dimensional arrays, per-element historization, whole-array writes on hardware.
See [Uns.md §Array tags](../Uns.md#array-tags-1-d) for the cross-driver coverage matrix
and the UI authoring flow.
@@ -454,8 +464,9 @@ when it is browse-committed in `/raw`, and that tag writes like any other atomic
`writable` key. The node is writable only once the operator sets the raw tag to `ReadWrite` (the
node-manager gate); an authored `"writable": false` still refuses at the driver (`BadNotWritable`). The
browse's `SecurityClass` (members inherit the parent symbol's read-only bit) is not consulted by the
commit. An element of a member array is authored by hand as `Seg.Points[2]`; a whole member array is
not writable (AbCip has no array encoder, `DriverArrayWriteSupport`).
commit. An element of a member array is authored by hand as `Seg.Points[2]`; a whole member array of an
atomic type is writable as an array tag (`isArray` + `arrayLength`, #737), a BOOL / STRING / structure member
array is not (`DriverArrayWriteSupport`).
- **The request names the member.** One CIP symbolic segment per member (plus an element segment), the
member's own CIP type, the member's own bytes — never the parent UDT, never a hidden host. Pinned at the
request-byte level by `NativeAbCipTagRuntimeTests.A_member_path_write_names_the_member_and_sends_only_its_own_bytes`
+20 -5
View File
@@ -1,5 +1,8 @@
# AB Legacy Driver
> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.
In-process native-protocol driver that exposes legacy Allen-Bradley PLCs —
**SLC 500**, **MicroLogix**, **PLC-5**, and Logix-via-PCCC — as OPC UA nodes. It
runs inside the OtOpcUa server's .NET 10 AnyCPU process and speaks PCCC over
@@ -274,7 +277,7 @@ Simulator evidence:
[Wire evidence](#wire-evidence-native-pccc).
- **PLC-5 native is N-file reads only.** Every other file type and all PLC-5 writes are refused before the
wire and reach the client as `BadNotSupported` (handle kept, health unchanged; #571 closed). Native is opt-in
for every family; an explicit Native on PLC-5 / `logixpccc` logs a Warning at init.
for every family; an explicit Native on PLC-5 / `logixpccc` logs a Warning at init. PLC-5 hardware proof → #752 (SLC/MicroLogix evidence → #592).
- Status-bit sub-element writes (e.g. `T4:0.DN`, `C5:0.DN`) are Unsupported natively (they would clobber the
control word); nothing goes on the wire — the driver refuses before creating the tag's runtime, so not even an
init read is sent (`NativeAbLegacyTagRuntime.WriteRefusal` is the single predicate, asked through
@@ -304,8 +307,18 @@ N values of the declared `AbLegacyDataType`. Maximum element count is 256 (PCCC
limit).
This closes the previously-noted limitation ("single-element addressing today") for the
array read path. The opt-in native transport also writes multi-element arrays (chunked); on libplctag
array *writes* remain a follow-up.
array read path.
**Array writes (#738, review 07 U10 / r0 I3)** — on both transports, a whole-array write encodes exactly
`arrayLength` elements at the **data file's** element width (N / B / A words, L longs, F floats). Element values
**convert** (ScadaBridge writes `int[]` to Int16 nodes); a length mismatch or an element that overflows is
`BadOutOfRange`, a non-numeric value `BadTypeMismatch`, before any runtime exists. Writable: `Int` / `AnalogInt` on
an N / B / A file, `Long` on an L file, `Float` on an F file, on a plain `<file><number>:<word>` address of at most
256 elements. These nodes stay **read-only** (`DriverArrayWriteSupport`): Bit arrays (bit-packed words), String /
Timer / Counter / Control arrays (rejected at `FromTagConfig`), `/bit` and `.sub` addresses, a type that does not
match its file (`BadTypeMismatch`), any PLC-5 on the native transport (writes refused before the wire), the
`S:` / `I:` / `O:` files (libplctag refuses their translated names), and — deliberately — indirect addresses, whose
element moves with the index word. An array deployed read-only before #738 turns writable on the next deploy.
**Unit test coverage** — `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbLegacy.Tests/` covers
the multi-element read path via the fake tag runtime. The AB Legacy integration fixture is
@@ -313,8 +326,10 @@ down during normal dev.
**Live-verify** — integration-fixture-gated (requires the AB Legacy Docker fixture).
**Deferrals** — array *writes* on libplctag (the opt-in native transport writes them), multi-dimensional arrays, per-element historization.
`arrayLength > 256` is rejected at init with an explicit error (PCCC payload cap).
**Deferrals** — multi-dimensional arrays, per-element historization, array writes on hardware
(`AbLegacyArrayWriteIntegrationTests` is green on `ab_server` and ready for a real controller via
`AB_LEGACY_ARRAY_WRITE_BASE`).
`arrayLength > 256` is clamped to 256 by the tag factory (PCCC payload cap), so such a node stays read-only.
See [Uns.md §Array tags](../Uns.md#array-tags-1-d) for the cross-driver coverage matrix
and the UI authoring flow.
+36
View File
@@ -85,6 +85,42 @@ Project root files:
- **`MxAccess`** — `ClientName` (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), `PublishingIntervalMs` (forwarded as `buffered_update_interval_ms` on subscribe), `WriteUserId` for ArchestrA secured-write, `EventPumpChannelCapacity` (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the `galaxy.events.dropped` metric).
- **`Repository`** — `DiscoverPageSize`, `WatchDeployEvents`.
- **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (calls the gateway's `ReplaySubscriptions` RPC after reconnect rather than re-issuing subscribe-bulk for every tag).
- **`AlarmAck`** (optional, lmxopcua#749) — `FallbackUser`, `FallbackDomain`, `FallbackPasswordSecretRef`. See [Alarm acknowledge credential](#alarm-acknowledge-credential).
### Alarm acknowledge credential
mxaccessgw `488649c`+ (client/contracts `0.3.0`) acknowledges a native alarm by running
`AuthenticateUser` as the operator and writing the comment to `<alarm>.AckMsg`, so the Historian
`ACK_ALM` row records a real Galaxy account. `AcknowledgeAlarmRequest` therefore carries
`operator_user` + `operator_domain` + `operator_password`, and the gateway rejects an empty
`operator_user`. The OPC UA data plane is machine-to-machine (API key or anonymous) and carries no
Galaxy password, so `GalaxyAckCredentialResolver` picks the account per acknowledge:
1. **The API key's own Galaxy ack identity** — set on AdminUI `/apikeys` ("Ack identity"). Stored only
in the encrypted Secrets store as `otopcua/apikeys/<keyId>/galaxy-ack-user` (`DOMAIN\user` or
`user`) and `otopcua/apikeys/<keyId>/galaxy-ack-password` (`GalaxyAckIdentitySecrets`) — never in
the config DB or the deployment artifact. A key with a user but no password secret is **refused**,
not silently downgraded to the fallback.
2. **The driver's fallback account** — `AlarmAck.FallbackUser` / `FallbackDomain` /
`FallbackPasswordSecretRef` in `DriverConfig`, used by keys without an identity and by anonymous
sessions. The password ref must be `secret:NAME` / `${secret:NAME}` (production), `env:NAME`,
`file:PATH` or `dev:VALUE`; a bare literal is refused so a password never sits in DriverConfig.
3. **Neither** → the driver throws `DriverOperationRefusedException`; **nothing is sent to the
gateway**. The refusal is excluded from the resilience breaker (it says nothing about gateway
health). ⚠️ The native ack path is fire-and-forget after the Part 9 method returns, so the OPC UA
call itself still returns `Good` and the refusal surfaces as a `DriverInstance` warning naming the
fix — a synchronous status / revert is #760.
The comment written to `AckMsg` is suffixed with the OPC UA identity —
`"<comment> [OPC UA: <principal>, key <keyId>]"` or `"[OPC UA: anonymous]"` — so the client is not
lost when the Historian's Operator is a shared fallback account. The password is never logged: the
acknowledger scrubs it from every gateway message it echoes and `GalaxyAckCredential.ToString()`
omits it.
⚠️ **Secrets are node-local** (optionally Akka-replicated within one mesh). The per-key secrets and
the fallback secret must exist on the node that serves the OPC UA session and hosts the Galaxy
driver; on a split admin/driver topology create them on the driver nodes too (`/admin/secrets` or
the `secret` CLI) under exactly those names.
Full per-field descriptions live in `Config/GalaxyDriverOptions.cs`. The full JSON skeleton is reproduced in [docs/v2/driver-specs.md §1](../v2/driver-specs.md).
+65 -7
View File
@@ -73,7 +73,7 @@ below).
## Capability surface
`MTConnectDriver : IDriver, IReadable, IWritable, ISubscribable, ITagDiscovery, IHostConnectivityProbe, IRediscoverable`
`MTConnectDriver : IDriver, IReadable, IWritable, ISubscribable, ITagDiscovery, IHostConnectivityProbe, IRediscoverable, IDriverDiagnostics`
(`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs`).
`IWritable` landed 2026-08-05 (`b29623cf`), against the Agent's `AllowPut`
@@ -87,11 +87,12 @@ request/response handshake — is a different mechanism and remains deferred; se
| Capability | Path | Notes |
|------------|------|-------|
| `ITagDiscovery` | `DiscoverAsync` — streams `/probe`'s Device→Component→DataItem tree into the address-space builder | `SupportsOnlineDiscovery = true`, `RediscoverPolicy = Once` |
| `IReadable` | `ReadAsync` → one `/current` per call | **Not the production data path** — see below |
| `IReadable` | `ReadAsync` → the live `/sample` pump's index while it is delivering and healthy, else one `/current` per call | **Not the production data path** — see below |
| `IWritable` | `WriteAsync` → `POST {agent}/{device}?{dataItemId}={value}` on the Agent's `AllowPut` surface | **Off by default** — needs `Write.Enabled` **and** the Agent's `AllowPut`; CONDITION items are refused |
| `ISubscribable` | `SubscribeAsync`/`OnDataChange` — the shared `/sample` long-poll pump | **The production data path** |
| `IHostConnectivityProbe` | periodic `/probe` under `Probe.*` | No consumer wires it today — see [Known limitations](#known-limitations) |
| `IHostConnectivityProbe` | periodic headers-only `/probe` ping under `Probe.*` — the status is read, the device model is neither downloaded in full nor parsed (review 2026-10-21 T-7; it used to parse the whole model every tick). Trade-off: a 200 from something that is not an Agent (a proxy's HTML page) now reads as reachable; the data paths still fail on it | Consumed — feeds `/hosts` (see [Known limitations](#known-limitations)) |
| `IRediscoverable` | watches the Agent's `Header.instanceId` | No consumer wires it today — see [Known limitations](#known-limitations) |
| `IDriverDiagnostics` | `GetDiagnostics()` — `/sample` effective count, lag, saturated chunks, count raises | Shown on `/hosts` (review 2026-10-21 T-5) |
| `IAlarmSource` | CONDITION observations off the same `/sample` pump → Part 9 transitions keyed by the alarm tag's RawPath | Opt-in per raw tag (`alarm` object) — see [CONDITION alarms](#condition-alarms-626) |
### Feature status at a glance (#613)
@@ -113,6 +114,25 @@ subscribes once and lives off `OnDataChange`. `ReadAsync` exists for the
Client CLI (`... read -n ...`) and for the unit/integration suites; it is
correct and tested, just not on the hot path.
**Read path (review 2026-10-21 T-4 / T-2).** While the current-generation `/sample` pump is running, has
delivered since it started (a chunk, heartbeat or re-baseline) and its stream is not degraded, a read is answered
from the pump's observation index — **read only**; the pump stays its single writer — with no Agent round trip.
Otherwise (no subscription, a pump still catching up after a resubscribe, a degraded stream) it falls back to one
`/current`, whose per-batch index is now built over a tag map frozen once per binding. **Stale window:** a
stream that stalls without failing (no chunk, no heartbeat) still serves Good reads from the index until the client's
idle watchdog fires (2 × `HeartbeatMs` + `RequestTimeoutMs`) — the same bounded window subscriptions already have. During
an Agent-restart re-baseline reads fall back to `/current`. A served read also clears a
read-path degradation an earlier failed `/current` left (the read path is answering). The `/current` fallback now
checks the answer's `instanceId`: a changed one (an Agent restart seen by a browse-only driver with no pump) drops the
cached `/probe` model and raises `OnRediscoveryNeeded` once per distinct id — it used to be detected only by the pump.
**An Agent refusal is a reply (health contract).** An `MTConnectError` document answering `/current` (e.g.
`NO_DEVICE` for a wrong `DeviceName` scope) still fails the batch (`BadCommunicationError` per ref), but it is the
Agent answering, not a comms fault: the read path is not degraded by it (that would let the comms-loss watchdog
reconnect-loop the driver over a configuration problem), `LastSuccessfulRead` advances, a prior read degradation
clears, and the Agent's text is logged once per distinct refusal. An answer that is not an MTConnect document at all
(a proxy's HTML page, truncated XML) stays a comms fault.
## Minimum deployment
```jsonc
@@ -146,11 +166,11 @@ Read off `MTConnectDriverOptions` / `MTConnectDriverConfigDto`
| Key | Default | Notes |
|---|---|---|
| `AgentUri` | — (required) | Agent base URI; the driver appends `/probe`, `/current`, `/sample` |
| `AgentUri` | — (required) | Agent base URI; the driver appends `/probe`, `/current`, `/sample`. Any `user:pass@` (and query) is **redacted** from `HostName` (the `/hosts` key), every log line and every exception / health message (review 2026-10-21 T-8); a URI without credentials is shown unchanged |
| `DeviceName` | `null` | **Double duty.** (1) Scopes requests to `{AgentUri}/{DeviceName}/...`; `null` = whole Agent. (2) The write path's fallback device — the device a write is addressed to when the probe model does not say which device declared the DataItem (see [Write-back](#write-back--observation-injection-off-by-default)) |
| `RequestTimeoutMs` | `5000` | Per-call deadline for `/probe` and `/current` |
| `SampleIntervalMs` | `1000` | The Agent's `/sample?interval=` query param |
| `SampleCount` | `1000` | The Agent's `/sample?count=` query param |
| `SampleCount` | `1000` | The Agent's `/sample?count=` query param. **Caps subscription throughput** at `SampleCount / SampleIntervalMs` — see [Subscribe](#subscribe--the-sample-pump) for the adaptive raise |
| `HeartbeatMs` | `10000` | The Agent's `/sample?heartbeat=` query param — also the watchdog's liveness window |
| `Probe.Enabled` / `Probe.IntervalMs` / `Probe.TimeoutMs` | `true` / `5000` / `2000` | Background connectivity-probe knobs (mirrors `ModbusProbeOptions`) |
| `Reconnect.MinBackoffMs` / `MaxBackoffMs` / `BackoffMultiplier` | `0` / `30000` / `2.0` | Geometric backoff after a failed request or a dropped `/sample` stream |
@@ -324,6 +344,26 @@ the two need disambiguating, not just OR-ing together). On a changed
`/hosts` page — see [Known limitations](#known-limitations) for what that does
and does not do.
**Lag measurement and the adaptive `count` (review 2026-10-21 T-5).** Throughput is capped at `SampleCount`
observations per chunk, one chunk per `SampleIntervalMs`; a machine producing more used to fall behind the ring
buffer forever, re-baselining every buffer-length (dropping intermediate transitions) with nothing surfacing it.
Each chunk is now measured: `lag = lastSequence + 1 − nextSequence` (unknown when the Agent omits `lastSequence`)
and whether it was **saturated** (carried the full `count`). Three consecutive saturated chunks with a positive (or
unknown) lag reopen the stream at the cursor reached with `count` doubled, up to 16× `SampleCount` (max 65 536). A
raised stream that delivers nothing (an Agent refusing the larger count) steps back to the last count that delivered
and stops raising; at the ceiling a still-behind stream logs one Warning per pump. Every new pump starts from the
configured `SampleCount`. An Agent that refuses the raised count with an `MTConnectError` under HTTP 200 goes through
the ordinary unusable-answer path first — one transient Degraded blip and a backoff — before the step-back. The
driver now implements `IDriverDiagnostics`: `sampleCount` (effective), `sampleLag`,
`sampleSaturatedChunks` and `sampleCountRaises` ride the `/hosts` diagnostics bag.
**A pump carries the session generation it was started against (review 2026-10-21 T-1).** A lifecycle restart
retires the running pump (or abandons it after its bounded teardown wait); if that pump finishes after the new
session is installed, its cursor publish, a late chunk and a late re-baseline are all dropped — checked under the
same lock the new session is installed and the generation bumped under. It used to land the OLD Agent's
`instanceId` + sequence on the new session, so the next pump read its first chunk as an Agent restart (index
cleared, one Bad flicker on every subscribed tag, a needless re-baseline).
## Write-back — observation injection, off by default
> ⚠️ **A write sets what the Agent *reports*, not what the machine does.** The value
@@ -391,8 +431,26 @@ step answers a different operator question:
different `level|text` wire format, and are out of scope for v1.
**The Agent's refusal text is logged at Warning and never stored.** It stays out of
`DriverHealth.LastError`, which is type-only by repo-wide discipline, and a refused write
is not evidence the read path degraded — so a refusal leaves driver health untouched.
`DriverHealth.LastError`, which is type-only by repo-wide discipline. A refused write is not
evidence the read path degraded, but it **is** the Agent answering: any answer — accepted or
refused — advances `LastSuccessfulRead`, with the state following the read / `/sample` flags
(a write cannot clear either of those paths' degradation). A write that got **no** answer
(transport failure, deadline) sets **Degraded** with the stamp preserved and a type-only
`LastError` (review 2026-10-21 I-3, coordinator ruling). One success writer is shared: that
degradation is cleared by the next answered **write**, the next answered **`/current` read**, or
the next **`/sample` chunk** the current session's pump delivers (review r2 N-5: a delivered
chunk proves the Agent is alive — the analogue of OpcUaClient's good keep-alive — so it cannot
stick on a subscription-only driver). A read served from the live pump index does not clear it
by itself; the chunks behind that index do.
**One request per device (review 2026-10-21 T-6).** The slots that survive the refusals above
go out as one `POST {agent}/{device}?id1=v1&id2=v2…` per device (the Agent's PUT surface takes
several pairs) instead of one POST per slot. A batch closes at a slot naming another device or
repeating a DataItem already in it, so two writes to one DataItem still land in request order.
An Agent refusal of a multi-slot request is retried slot by slot, so a refused DataItem costs
only its own slot (`BadCommunicationError`); a transport failure fails that request's slots. An Agent may apply
some pairs of a multi-pair PUT before refusing it, so the per-slot retry can re-send a pair that already landed —
harmless for value-setting (the Agent drops an identical repeat).
**Device resolution needs no extra config.** `TagBinding.WriteTargetsByDataItemId`,
built at bind time from a walk of the cached probe model, remembers which device declared
+10 -4
View File
@@ -138,11 +138,17 @@ numeric, String, and BitInRegister array reads.
**Live-verify** — Mac-verifiable against the simulator at `10.100.0.35:5020` (no fixture
bring-up required; the sim is always-on on the shared Docker host).
**Array writes** (#610) — a whole-array write goes out as one FC15 (coils) or one FC16 (numeric
holding registers); the driver refuses rather than splits an FC16 longer than `maxRegistersPerWrite`
(default 123). The server mirrors that: an array tag is client-writable only when
**Array writes** (#610, #741) — a whole-array write goes out as one FC15 (coils) or one FC16
(holding registers: numeric, `String` at `(stringLength+1)/2` registers per element, and
`BitInRegister` as one FC03 + one FC16 read-modify-write of the span); the driver refuses rather than
splits an FC16 longer than `maxRegistersPerWrite` (default 123; a larger value never lifts FC16's
protocol ceiling of 123, and a BitInRegister array's unchunked FC03 also bounds it by
`maxRegistersPerRead`). The server mirrors that: an array tag
is client-writable only when it is addressed by `region` + `address` keys and
`arrayLength × registers-per-element` fits the cap (and a coil array ≤ 1968), else its node stays
read-only — see [Uns.md §Array writes](../Uns.md#array-writes-610), including the upgrade note.
read-only. A string-only (`addressString`) array stays read-only, because the composer cannot resolve
it under the driver's family — see [Uns.md §Array writes](../Uns.md#array-writes-610) and
[modbus-addressing.md §Array writes](../v2/modbus-addressing.md#array-writes-610-741).
**Deferrals** — multi-dimensional arrays (`arrayLength` is always 1-D); per-element historization.
+26 -6
View File
@@ -1,5 +1,8 @@
# MQTT Driver
> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.
In-process MQTT **broker-subscribe** driver. It holds one MQTTnet 5 client against one broker,
subscribes to the topics its authored tags name, and forwards each PUBLISH straight to
`ISubscribable.OnDataChange`. Ingest is **push, not poll**; writes go back out as ordinary publishes
@@ -117,7 +120,7 @@ driver are pure organisational grouping in the RawPath.
"HostId": "", // Host Application identity — spBv1.0/STATE/{HostId}
"ActAsPrimaryHost": false, // claim the primary host id (Primary-gated; see below)
"RequestRebirthOnGap": true, // a seq gap asks the edge node to re-announce (NCMD)
"BirthObservationWindowSeconds": 15 // how long discovery collects births before calling it stable
"BirthObservationWindowSeconds": 15 // how long the browse picker collects births before calling them stable
}
}
```
@@ -128,7 +131,7 @@ driver are pure organisational grouping in the RawPath.
| `HostId` | `""` | Sparkplug Host Application identity — the `spBv1.0/STATE/{HostId}` topic. Optional for observation; **required** to claim. Set without `ActAsPrimaryHost`, the driver only observes |
| `ActAsPrimaryHost` | `false` | Claim the Sparkplug Primary Host Application identity. **Primary-gated** — only the node holding the redundancy Primary role claims it. See [Primary host](#primary-host-actasprimaryhost) |
| `RequestRebirthOnGap` | `true` | On a detected sequence-number gap, publish a rebirth-request NCMD instead of running on stale metric state |
| `BirthObservationWindowSeconds` | `15` | Discovery's birth-collection window. Must be ≥ 1 |
| `BirthObservationWindowSeconds` | `15` | The Sparkplug browse picker's birth-collection window, clamped to 1–120 s (R06:M-4). The open budget grows by it; the open lingers while NBIRTH/DBIRTH traffic is still arriving (returning after a 1 s quiet spell — an idle plant costs 1 s); and an operator **Request rebirth** waits, bounded by it and capped at 15 s (below the AdminUI's 20 s per-call timeout), for the births it asked for to arrive and go quiet for 500 ms, so the tree re-rendered afterwards carries them. A caller deadline during that wait still reports the publish. The runtime driver does not read it (its `DiscoverAsync` has no production caller — `SupportsOnlineDiscovery` is false) |
`MqttDriverForm` authors all five in Sparkplug mode. The sub-object is **merged, never replaced** (a
key a newer driver adds inside it survives an older AdminUI), and in `Plain` mode it is not touched at
@@ -249,7 +252,10 @@ One subscription (`spBv1.0/{GroupId}/#`) feeds a per-edge-node state machine
[Structured metrics](#structured-metrics-template--dataset--propertyset).
- **Unsupported datatypes are skipped with a warning, never guessed**: `PropertySetList`, `Unknown`,
extension values, and any `Template`/`DataSet` that cannot be flattened honestly. `Int8`→`Int16` and
`UInt8`→`UInt16` widen; `*Array` variants map to the element type with the array bit set.
`UInt8`→`UInt16` widen; `*Array` variants map to the element type with the array bit set — but
**array metrics are not read in v1** (warned once, never fed; → #754), so the browse picker offers them, and any
metric whose type is unsupported, **without an address**: the browse-commit refuses them rather than
committing a tag that waits for initial data forever (R06:M-5).
- **Ingest state is reset at every session/authoring seam** — a reconnect or a `ReinitializeAsync`
must not carry a previous session's alias table into a new one.
@@ -408,9 +414,18 @@ value. `MqttConnection` therefore raises `MqttConnectRejectedException`:
code. (`ServerMoved` is permanent, `UseAnotherServer` is temporary — per the MQTT 5 spec.)
A reconnect that completes without re-subscribing would be a healthy-looking connection that receives
nothing forever, so `OnReconnectedAsync` **throws** when zero filters are granted, tearing the session
down rather than swallowing it. A SUBACK failure degrades just the affected RawPaths to
`BadCommunicationError`.
nothing forever, so `OnReconnectedAsync` **throws** when none of the previously live filters is granted
(or the SUBSCRIBE gets no verdict), tearing the session down rather than swallowing it. A SUBACK failure
degrades just the affected RawPaths to `BadCommunicationError`.
**A failed SUBSCRIBE is retried, never forgotten (R06:M-1).** Every topic whose SUBSCRIBE was refused or got
no verdict joins a *pending* set that the next `SubscribeAsync` and every reconnect re-request; a grant moves
it to the live set, and a redeploy that drops its tags drops it. Pending topics do not count toward the
reconnect's total-failure verdict, so one ACL-denied topic cannot reconnect-loop the driver. A plain-mode
`SubscribeAsync` whose SUBSCRIBE got **no verdict at all** (transport throw, SUBSCRIBE deadline) degrades its
refs and then **throws**, so `DriverInstanceActor` holds no handle and its health-tick reconcile re-subscribes;
a SUBACK *refusal* still returns a handle (the broker answered). A subscribe issued with no transport attached
records its topics as pending, never as established (M-11).
There are **two** kinds of reconnect and only one of them fires that callback. A drop the connection's
own supervisor recovers keeps the same `MqttConnection`, so `Reconnected` → `OnReconnectedAsync`
@@ -479,6 +494,11 @@ with their real type; a birth type the encoder cannot represent (any `*Array`, D
matters and is deliberately not configurable: a retained command would be replayed by the broker to the device on *every* future
subscribe — including the device's own reconnect — so one stale write would be re-applied forever.
**A batch publishes concurrently, one topic at a time (R06:M-8).** Writes to *different* topics in one
`WriteAsync` go out together, at most 8 in flight, so a stalled broker costs a batch about one publish deadline
rather than one per request; writes to the *same* topic stay sequential in request order, so the last one wins.
A PUBACKed (QoS 1) publish advances `LastSuccessfulRead` like an inbound message; a QoS 0 Sparkplug command does not, because its completion only means the bytes left the local socket.
**Plain QoS 1 is at-least-once.** Know the consequences before pointing a Plain `writeTopic` at a non-idempotent
device topic:
+39 -1
View File
@@ -25,7 +25,9 @@ out-of-process isolation.
| Project | Target | Role |
|---------|--------|------|
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/` | net10.0 | In-process driver — session lifetime, read / write / subscribe / alarm / history passthrough |
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser/` | net10.0 | `IDriverBrowser` — live address-picker browse used by the AdminUI |
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser/` | net10.0 | `IDriverBrowser` — live address-picker browse used by the AdminUI. Its side-panel `AttributesAsync` reads a variable's DataType / ValueRank / UserAccessLevel and maps them through the same `UpstreamAttributeMapping` discovery uses; it authenticates Anonymous, Username **or Certificate** through the driver's own `OpcUaUserIdentityFactory`, resolving `secret:` credentials through the host's secret store (fail-closed) — review 2026-10-21 O-5 (it was a stub, and Certificate was refused) |
| | | ⚠️ Browse Certificate auth reads `UserCertificatePath` on the **AdminUI host's** filesystem, which may differ from the driver node's; the browse session disposes the certificate it loaded when it closes. |
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Session/` | net10.0 | The SDK-typed layer the driver AND the browser share: `NamespaceMap`, `UpstreamAttributeMapping`, `OpcUaUserIdentityFactory` |
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/` | net10.0 | Config records + enums bound from `DriverConfig` JSON |
## Minimum deployment
@@ -53,6 +55,8 @@ for every field (security policy/mode, auth type, session timeout, keep-alive,
browse root, node/depth caps). `ReconnectPeriod` was **deleted** (#676 — unused since #670 removed the SDK reconnect
handler, and the AdminUI no longer shows it); stored configs that still carry `reconnectPeriod` keep parsing in the
factory, the Test-Connect probe and the browse picker (unknown keys are skipped), and the form drops it on the next save.
`ProbeTimeoutSeconds` (default 15, AdminUI "Probe timeout") caps the Test-Connect probe's deadline — it can shorten
the budget the host hands the probe, never extend it (review 2026-10-21 O-4; it used to be authored and consumed by nothing).
### Endpoint address: the configured host and port win (#676)
@@ -70,12 +74,40 @@ A single `Session` per driver instance; subscriptions multiplex onto it. Stored
session-relative `ns=N` index) so a remote namespace-table reorder across a
restart doesn't silently re-point references at the wrong namespace.
**Concurrency (review 2026-10-21 O-6).** Reads (including the post-outage recovery / reconcile reads) and acks
share one gate; **writes and the discovery browse take none** (the SDK session is safe for concurrent service
calls, and the host already bounds write concurrency and serialises per tag); subscription management — data and
alarm subscribe/unsubscribe and ConditionRefresh sweeps — has its own gate. One gate used to serialise all of it,
so a write stalled for the operation timeout blocked every read and subscription create, and a discovery browse
would have blocked the data plane for its whole duration.
**Lifecycle generation.** Every `InitializeAsync` / `ShutdownAsync` takes a new generation; the session, namespace
map and every health write land only while their generation is current. An init the host abandoned that finishes
after the next shutdown discards (closes + disposes) its session instead of installing it, and a read whose
response arrives after a shutdown does not write `Healthy` over it. A throw after the session activated closes
that local session (O-2). `ShutdownAsync` spends **one** 4 s budget on wire calls — every subscription delete
concurrently, then the close with what is left — and abandons waits the SDK does not cancel (O-9; it used to be
5 s per subscription, serially, inside the actor's `PostStop`).
### Upstream outage and recovery (Gitea #670)
The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal:
- **Good keep-alive / any data or event notification** advances `DriverHealth.LastSuccessfulRead` — a
subscription-only driver never calls `ReadAsync`, so without this its read stamp froze at connect time.
A good keep-alive with no outage latched also **clears a `Degraded` left by a failed read/write** (it used to move
only the stamp, so a subscription-only driver stayed `Degraded` until its next re-initialise).
- **Read / write / ack responses are replies.** Any service response — per-node refusals such as `BadNotWritable`
included — restores `Healthy` and advances the stamp; only a call that got **no** response (transport failure,
timeout — including an SDK request timeout surfacing as a cancellation the caller did not cause) writes `Degraded`
with the stamp preserved; acks follow the same rule. **Deliberate deviation:** a call the *caller* cancelled touches
no health — including a write the host cancelled at its own write deadline against a hung upstream; the session
keep-alive owns liveness and reports that outage itself. A reply that was in flight across a bad keep-alive advances
the stamp but does not write `Healthy` while the outage is published — the keep-alive recovery ends the outage.
An ack none of whose condition ids resolves sends nothing and logs a Warning — it does **not** throw (a deterministic
per-ack refusal must not count toward the host's AlarmAcknowledge circuit breaker, or five re-acks of one
unresolvable condition would block every genuine ack). Only a call with no reply throws; with no session an ack
fails `BadNotConnected` (before or inside the gate alike).
- **Bad keep-alive** ⇒ health `Degraded` (`LastError = "keep-alive: <status>"`, read stamp preserved), host status
`Stopped`, and — **once per outage**, not per keep-alive — every subscribed ref is published `BadNoCommunication`
with its last value retained and **source + server timestamp = the detection time** (live-gate F1: stamped with the
@@ -231,6 +263,12 @@ initialise (the AdminUI form drops them on the next save). Pinned by
> forgotten when the subscribed set changes (the host rebuilt, recreating the nodes inactive). Index keys are server-stable (`nsu=`), so a namespace reorder across a re-initialise keeps
> matching. docker-dev seeds an opc-plc driver (`MAIN-opcua`); the alarm tag is
> created only when `@OpcPlcAlarmSourceNodeId` is set in `docker-dev/seed/seed-clusters.sql`.
>
> **An ack that never got a reply fails (review 2026-10-21 O-8).** A transport failure on the `Acknowledge` call
> (closed channel, faulted session) — or a session lost while the ack queued — now throws with its status
> (`ServiceResultException`, e.g. `BadConnectionClosed` / `BadNotConnected`), so the host logs the ack as failed
> instead of applied. A *reply* still returns normally, including per-ack refusals such as
> `BadConditionBranchAlreadyAcked` (logged per ack): a refusal is the server answering.
> This driver does **not** implement `IRediscoverable` — there is no
> push-driven rediscovery signal from a remote OPC UA server in this driver.
+50 -8
View File
@@ -88,6 +88,9 @@ After the handshake, Test Connect reads 2 bytes from `MB0`. A CPU with PUT/GET d
default) completes the handshake but denies every read, so the probe fails with an actionable "enable
PUT/GET communication" message instead of a false green tick. A CPU that merely rejects that address (no
`MB0`) still passes — the connection is fine; any other read fault fails the probe with its message.
Both are recognised in the bare-`Exception` shape S7.Net's read path really raises. The handshake and
test read are bounded by the config's `timeoutMs` (the driver's own per-operation timeout), capped by the
Test Connect timeout — before R07:H7 the config value was ignored.
## Address forms
@@ -103,9 +106,18 @@ Addresses use Siemens TIA-Portal / STEP 7 Classic syntax, parsed by
| Timer | `T0` / `T15` | Timer area (read-only — see Wide types section) |
| Counter | `C0` / `C10` | Counter area (read-only — see Wide types section) |
Parsing is strict and runs once at `InitializeAsync` so a config typo fails fast
at load instead of surfacing as `BadInternalError` on every read. Bit offsets
must be 0-7, byte offsets non-negative, DB numbers >= 1.
Parsing is strict. Bit offsets must be 0-7, byte offsets non-negative, DB numbers >= 1.
**Tag shape rules are judged per tag, at every gate** (`S7TagShapeRules`, called by
`S7TagDefinitionFactory.FromTagConfig`): the AdminUI editor refuses the tag, the deploy gate
(`Inspect`) warns, and the driver's init **skips** it (reported through `ISkippedTagReporter`; it reads
`BadNodeIdUnknown`) while every other tag keeps working. Refused: an address that does not parse, a
wide/structured type that is an array or is not byte-addressed (`DBB`/`MB`/`IB`/`QB`), a narrow type at the
wrong access width (`Int32` on `DBW0`, `Bool` on `DBB0`...), a Timer not typed `Float64` or a Counter not typed
`Int32`, a writable or array Timer/Counter, a writable `Bool` array, and a `Bool` array that does not start
at bit 0 (a Bool array reads bit 0 of each byte, so `DB1.DBX0.3` used to silently return bit 0 of bytes
0..n). These used to be init guards that THREW, so one typo faulted every tag on the PLC and the actor
retried the same init every 10 s forever (R07:H3).
## Data types
@@ -277,7 +289,7 @@ AdminUI `/hosts` page (Gitea #640). See `docs/Telemetry.md` ("Driver diagnostics
| `IWritable` | `WriteAsync` → `S7.Net.Plc.WriteAsync` | Read-only tags (`Writable=false`) return `BadNotWritable` |
| `ITagDiscovery` | `DiscoverAsync` | Emits a flat `S7/` folder of the configured tags — no live browse |
| `ISubscribable` | per-tag poll loop with capped exponential backoff | S7 has no push model; floor is 100 ms (the CPU services the comms mailbox once per scan) |
| `IHostConnectivityProbe` | periodic `S7.Net.Plc.ReadStatusAsync` (CPU-status PDU) | `host:port` host key; `Running`/`Stopped` transitions raise `OnHostStatusChanged` |
| `IHostConnectivityProbe` | periodic `S7.Net.Plc.ReadStatusAsync` (CPU-status PDU) | `host:port` host key; `Running`/`Stopped` transitions raise `OnHostStatusChanged`. A tick that finds the gate held past the probe deadline (a long read pass) gives **no verdict** — it no longer reports `Stopped` for a connection that is busy, not lost (R07:H7) |
### Single-connection policy
@@ -302,8 +314,9 @@ Portal under *Protection & Security* for the CPU.
| Tag not in config | `BadNodeIdUnknown` | unchanged |
| Read-only tag written | `BadNotWritable` | unchanged |
| Unimplemented data type | `BadNotSupported` | unchanged |
| PUT/GET denied | `BadNotSupported` | `Faulted` (config alert) |
| CPU / hardware fault | `BadDeviceFailure` | `Degraded` |
| PUT/GET denied | `BadNotSupported` | `Faulted` (config alert) — also counts as *answered*, advances `LastSuccessfulRead` (no watchdog reconnect loop) |
| CPU refusal of the item (object does not exist, address out of range, type inconsistent, hardware fault) | `BadDeviceFailure` | unchanged — counts as *answered*, advances `LastSuccessfulRead`; one Warning per tag |
| Other device-layer fault (`PlcException` that is neither a refusal nor connection-fatal) | `BadDeviceFailure` | `Degraded` |
| Socket / timeout | `BadCommunicationError` | `Degraded` |
| Reconnect suppressed (backoff window) | `BadCommunicationError` | `Degraded` |
| Array write of the wrong shape / length / element type | `BadTypeMismatch` | unchanged |
@@ -314,7 +327,36 @@ accumulate their outcomes and write health once (`Healthy < Degraded < Faulted`)
`Faulted` item is not erased by a later `Degraded` or successful sibling. The write-side `Faulted` is
**sticky against reads**: a clean read pass (reads often work against a write-protected CPU) leaves
`Faulted` in place and only advances `LastSuccessfulRead`; the next write that succeeds clears it. A
transient `Degraded` write failure is still cleared by a clean read, as before.
transient `Degraded` write failure is cleared by a clean read, or by the next write batch the CPU
answers without a failure.
**A refusal is an answer (R07:H4).** S7.Net's read paths raise the CPU's item error as a bare
`Exception` ("Received error from PLC: Object does not exist." — live-probed on the snap7 fixture), its
write paths as a `PlcException(WriteData)`; both are classified over the whole exception chain, and the
access-denied text inside either (S7.Net's exact "Accessing object not allowed", and never when the chain
also carries a transport fault) is the `Faulted` alert — which is ALSO an answer: it advances
`LastSuccessfulRead`, so a factory-default CPU with PUT/GET disabled stays `Faulted` instead of being
reconnected by the comms-loss watchdog every 90 s. A multi-var chunk that falls back per tag memoises the
members refused on their own, like a refused span. A refusal is deterministic, so it neither
degrades health nor reopens the connection, and because it proves the CPU is reachable it advances
`LastSuccessfulRead` and restores `Healthy` after an earlier comms fault. Before this, a driver whose
every tag was refused (a wrong DB number) sat `Degraded` with a frozen timestamp and the comms-loss
watchdog reconnected it every 90 s against a perfectly reachable PLC.
**A dead handle ends the pass's wire traffic (R07:P1).** Once a read pass or write batch marks the
connection dead (a wire op blew its deadline on a frozen-but-established peer, a socket dropped), the
remaining spans / multi-var chunks / tags / writes of that pass are stamped `BadCommunicationError`
without I/O; the next call reopens. A multi-var chunk that dies on a transport loss no longer falls back
to one doomed per-tag read per member. Without this an op that ignores its cancellation token would cost
one full deadline per remaining item under the gate, and `Degraded` (and so the comms-loss watchdog's
clock) only became visible after the whole pass.
**A refused span falls back per member (R07:P2).** A grouped block read covers the union of its members'
ranges, so one member the CPU refuses (running past the DB's real end, an over-declared `stringLength`)
used to make every neighbour in the block Bad on every poll. When the CPU refuses a multi-member span the
driver re-reads its members one at a time for that pass and remembers which tags must not be grouped —
the members refused on their own, or every member when none was — so later passes group only the rest.
The memo lasts until the next initialize. A span failure that is not a refusal still stamps every member.
**Connect backoff.** A dead PLC costs one connect attempt per backoff window (1 s doubling to a 30 s
cap), not one per read / poll tick / probe tick: inside the window `EnsureConnectedAsync` fails fast
@@ -359,7 +401,7 @@ normal dev (the S7 sim is on the shared Docker host but currently offline).
**Array writes (S7-016)** — a writable `Byte` / `Int16` / `UInt16` / `Int32` / `UInt32` / `Float32` array
is written as ONE contiguous `WriteBytesAsync` of `count x element-bytes` (the inverse of the read
decode), and the value must be an array of exactly `arrayLength` elements (else `BadTypeMismatch`, no
health impact). **Writable `Bool` arrays are refused at init** (`NotSupportedException`): the array
health impact). **Writable `Bool` arrays are refused per tag** (see the tag shape rules above): the array
read is byte-granular, so a matching write would overwrite the 7 neighbouring bits of every byte it
touches, and a safe bit-packed write needs read-modify-write. Mark Bool arrays `writable: false`. A `null` array element is rejected (`BadTypeMismatch`, nothing written)
rather than coerced to 0. An array write larger than one PDU is split by S7.Net into several writes and is
+24 -3
View File
@@ -1,5 +1,8 @@
# SQL Poll Driver
> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.
Getting-started guide for the `Sql` driver — a **read-only** Equipment-kind driver that polls rows
out of a relational database and publishes them as OPC UA variables. For the design rationale read
[`docs/plans/2026-07-15-sql-poll-driver-design.md`](../plans/2026-07-15-sql-poll-driver-design.md);
@@ -16,7 +19,7 @@ already writes to. There is no protocol here and nothing to "connect" to a machi
(`Npgsql`) and `Oracle` (ODP.NET managed — **Oracle 12.2 or later**, the floor the
`FETCH FIRST` paging needs). `SqlProvider` also declares `MySql` and `Odbc`, but those remain
**reserved names, not working backends** — authoring one fails the driver rather than silently
falling back.
falling back. → #753
Two provider-specific things are worth knowing before you author against them:
@@ -30,7 +33,7 @@ Two provider-specific things are worth knowing before you author against them:
**Read-only, structurally.** `SqlDriver` does not implement `IWritable`, so no configuration can
produce a write. The `allowWrites` config field exists and is **inert**; the factory *warns* when it
is authored `true` rather than failing, on the grounds that the flag cannot do harm but an operator
who believes writes are enabled can.
who believes writes are enabled can. → #753
## Tag models
@@ -106,7 +109,7 @@ canonical text form. SQL Server behaves the same way.
authored `keyValue` refused the tag while the identically-bound `whereValue` was served — and the
deploy gate reported the key as *missing* when it was right there in the blob.)
### `Query` — deferred to P3
### `Query` — deferred to P3 (→ #753)
Named arbitrary-`SELECT`. **The v1 parser rejects it**, and that refusal is enforced end-to-end —
parser, validator, planner and the AdminUI editor all decline it rather than accepting the value and
@@ -166,6 +169,24 @@ emits a `SqlQueryPlan` per distinct group key. Grouping is deterministic (groups
order of their first member, members keep their input order, parameters keep first-appearance order),
so an unchanged configuration produces byte-identical SQL.
**Every key-value and where-pair query is row-capped (R06:S-1).** A key-value plan reads at most twice as
many rows as it has distinct keys and asks for one more (`TOP (2n+1)` / `LIMIT 2n+1` / `FETCH FIRST 2n+1 ROWS
ONLY`); a where-pair wide-row plan asks for two. The reader stops there even if a provider ignores the limit.
- **A duplicated key is refused per key.** When a key comes back more than once, only the tags on THAT key
publish `BadConfigurationError`; every other key in the group reads normally, whatever else is present or
absent. (Before this, a duplicate was last-row-wins Good.)
- **A gross excess fails the group.** Receiving the extra row past the cap means a history/events table or a
non-unique where column, which cannot be attributed because the rest is unread, so every member of the group
publishes `BadConfigurationError`. For a where-pair any second row is this case.
- Each case logs a per-plan rate-limited warning naming the table and the selector (or an example key).
`topByTimestamp` keeps its single-row limit.
- **Upgrade note:** a deployment whose key-value source carries stray duplicate rows, or whose where column is
not unique, will see those tags go Bad where they used to read an arbitrary row at Good.
Every such refusal, like the other refusals (absent row, NULL cell, uncoercible cell), is the database
answering: it advances `LastSuccessfulRead` and restores `Healthy` after a comms fault.
⚠️ **Group keys are case-sensitive on identifiers.** Two tags authored `dbo.T` and `DBO.T` plan as
**two** groups — one extra round-trip, not a correctness problem, but worth knowing if a group count
looks higher than expected.
+14
View File
@@ -362,6 +362,20 @@ element-typed CLR array (`bool[]`, `sbyte[]`, `byte[]`, `short[]`, `ushort[]`, `
down (found live, below). A bit outside the parent's width, or a bit of a non-integer parent,
is refused with `BadTypeMismatch`.
### Array writes (#739, review 07 U9)
An array node is **writable** (`DriverArrayWriteSupport`) for every element type the codec lays out —
numeric, `Bool` (one byte per element, as TwinCAT stores a BOOL array), `STRING`/`WSTRING` and the temporal
types. The write takes exactly `arrayLength` elements and **converts** each one (`ConvertForWrite`; ScadaBridge
writes `int[]` to an Int16 node); a length mismatch or an out-of-range element is `BadOutOfRange` and a scalar
value `BadTypeMismatch`, before any wire traffic. `WriteArrayAsync` then plans the layout from the resolved PLC
array with the same `CheckArray` the read uses and encodes with `TwinCATArrayCodec.Encode` (the inverse of the
decode); a shorter `arrayLength` writes the leading sub-range, and a multi-dimensional PLC array is written
row-major from the flat 1-D value. A BOOL **bit-in-word** path (`MAIN.Word.3`) stays read-only as an array: it is a
scalar bit write. Under a `DateTime` node only `DATE`/`DT`/`WSTRING` elements are writable, and under a `String`
node only `STRING`/`WSTRING` (other conversions would depend on the text). An array deployed read-only before #739
turns writable on the next deploy. Unit- and XAR-gate-tested; real-hardware proof is pending.
### Notification size cap
ADS documents no notification size limit, and a live TwinCAT 2.11 target registered and
@@ -1,5 +1,8 @@
# QuestDB historian — live gate (T9), 2026-10-11
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
**Verdict: PASSED.** All six legs of spec §7 passed on the docker-dev rig. The rig ran `feat/questdb-historian` at
`2ebe0a42`, with the central pair on image `0701f6a833ea`, rebuilt for this gate. The QuestDB backend pointed at the
overlay `questdb` service (`questdb/questdb:10.0.1`, `docker-dev/docker-compose.questdb.yml`), with default table
@@ -8,7 +11,7 @@ prefix `hist_`.
The gate also produced two findings, neither in the QuestDB backend. Both are described under "Gate findings" below:
- A crash failover (SIGKILL of the recording Primary) leaves a recording gap of about 30 s. This is the #537
Primary-gated capture waiting for Akka downing; it is backend-independent.
- One `libgssapi_krb5.so.2` load error is printed per central boot. It is benign.
- One `libgssapi_krb5.so.2` load error is printed per central boot. It is benign. → #758
## Setup
+117
View File
@@ -0,0 +1,117 @@
# OtOpcUa system review — merged summary
Master @ `d51d35bd`, 2026-10-21/22. 8 parallel Fable reviewers, read-only, every finding cited to file:line in the per-slice reports:
| # | Slice | Report | Findings |
|---|---|---|---|
| 1 | OpcUaServer | `01-opcuaserver.md` | ~10 |
| 2 | Runtime + Cluster | `02-runtime-cluster.md` | 21 |
| 3 | Host / ControlPlane / Security / Config | `03-host-controlplane-security-config.md` | 19 |
| 4 | AdminUI | `04-adminui.md` | 24 |
| 5 | Core libraries | `05-core.md` | 12 |
| 6 | Session drivers (Galaxy, OpcUaClient, MQTT, MTConnect, Sql) | `06-drivers-session.md` | ~25 |
| 7 | PLC drivers (AB, Modbus, S7, TwinCAT, FOCAS, Calculation) | `07-drivers-plc.md` | 46 |
| 8 | Historians + clients + analyzers | `08-historians-clients.md` | 20 |
## Cross-cutting themes (fix the pattern, not N instances)
1. **The host never disposes a driver — only `ShutdownAsync`** (`DriverInstanceActor.cs:1550`, `DiscoveryDriverBrowser.cs:128`). Any driver whose teardown lives in `Dispose[Async]` leaks per respawn / per browse:
Galaxy (whole runtime + mxaccessgw session — **Critical**), Calculation (collectible script ALCs), OpcUaClient (logger factory, gate, CTS). Fix once in the runtime (dispose after shutdown) and audit each driver's `ShutdownAsync`.
2. **Timed-out scripts leak a thread-pool thread per evaluation** — found independently by 3 reviewers: `TimedScriptEvaluator` (scripted alarms + virtual tags, `TimedScriptEvaluator.cs:78`, `RoslynVirtualTagEvaluator.cs:158`) and `CalculationEvaluator.cs:187`. One infinite-loop script trends the node to pool starvation while health stays Healthy. Needs a quarantine/breaker after N timeouts plus a cap on outstanding orphaned evaluations. The sandbox also allows `System.Threading` (blocking primitives).
3. **The health/watchdog contract is wrong in both directions across drivers.**
- *Refusals counted as comms faults, which triggers 90 s reconnect loops on a reachable device:* AbCip H1/F9, AbLegacy F5, Modbus H4, S7 H4, TwinCAT `NotSupportedException` write.
- *Real outages that never degrade:* TwinCAT dead AMS router, TwinCAT native mode (`LastSuccessfulRead` never advances), FOCAS FIXED-only, Galaxy EventPump death, OpcUaClient alarm ack swallowing errors, MQTT failed SUBSCRIBE.
- *Device blackout from a probe-tag typo:* AbCip F1.
4. **A plain `Dictionary` is mutated during re-init while the poll loop reads it:** FOCAS `_devices`, AbCip device tables, AbLegacy `_devices` + `_tagsByRawPath`. The swap-by-reference fix already exists for some tables; apply it to all.
5. **One dead or slow device stalls every device in the driver** because reads are serial with no per-device backoff: AbLegacy (no backoff at all, can trip the watchdog on a healthy device), FOCAS, TwinCAT all-or-nothing native subscribe, S7 span-wide failure.
6. **Array writes are advertised writable but unimplemented** in AbCip, AbLegacy and TwinCAT: scalar encode, stale-element rewrite, or a cast exception reported as `BadTypeMismatch` + `Degraded`.
7. **Config knobs bound or authored but never consumed:** MQTT `birthObservationWindowSeconds`, Galaxy `DiscoverPageSize`, `ProbeTimeoutSeconds` (Galaxy, OpcUaClient), Modbus Family/MELSEC/DisableFC23, `TelemetryDialOptions.CallTimeoutSeconds`, plus `/auth/token` JWTs that nothing validates.
8. **Mailbox-blocking `ReceiveAsync` in singletons and driver actors:** AdminOps (a 60 s Test Connect stalls fleet-wide acks and deploys), DriverInstanceActor writes/subscribes (spurious write timeouts and reverts), and untimed teardown waits in PostStop / alarm-engine Dispose.
## Ranked top findings
### Critical / High
| # | Finding | Where |
|---|---|---|
| 1 | Galaxy `ShutdownAsync` is a no-op, so every respawn/deploy/browse leaks a full runtime + gateway session | `GalaxyDriver.cs:735` |
| 2 | Uncapped raw HistoryRead (`NumValuesPerNode=0`) materialises the whole window on all 3 backends; reachable anonymously with ACL off | `HistoryPaging.cs:116`, `SqlServerHistorianDataSource.cs:99`, QuestDb `:101`, Gateway `:213` |
| 3 | Deploy pipeline wedges permanently (`Dispatching` with 0 node rows → `0==0` deadline branch never clears; every later deploy refused) | `ConfigPublishCoordinator.cs:163-200,355` |
| 4 | OPC UA SDK start failure swallowed; node with 4840 unbound stays Ready, Primary-eligible, and acks Applied | `OtOpcUaServerHostedService.cs:163` |
| 5 | Script timeout thread leak (theme 2) | 3 sites |
| 6 | `ScriptedAlarmHostActor` restart reuses its own disposed engine, so scripted alarms die silently on that node | `ScriptedAlarmHostActor.cs:676` |
| 7 | Driver-dispose gap leaks Calculation script ALCs on every calc edit (theme 1) | `DriverInstanceActor.cs:1550` |
| 8 | Removed tags resurrected as root-parented phantoms by a racing publish, poisoning a later re-add | `OtOpcUaNodeManager.cs:941` |
| 9 | New subscriptions bind to dead `NodeState` after rebuild/remove (SDK `MonitoredNodes` map never cleared): `Good` on create, never data | `OtOpcUaNodeManager.cs` vs SDK `MonitoredNodeMonitoredItemManager.cs:75` |
| 10 | Galaxy spawns a new `DeployWatcher` per re-init without stopping the old one | `GalaxyDriver.cs:631` |
| 11 | Galaxy per-platform ScanState probing structurally dead (handles never registered) | `PerPlatformProbeWatcher.cs:125` |
| 12 | AdminUI script >32 KB silently stops syncing and Save persists stale source (SignalR 32 KB default) | `monaco-init.js:213`, no `AddHubOptions` |
| 13 | Gateway capped events read breaks out of the gRPC stream, so the gateway evicts its pooled session every time | `HistorianGatewayClientAdapter.cs:111` |
| 14 | AbCip probe-tag typo, then permanent device blackout (Test Connect shows green) | `AbCipDriver.cs:666-701` |
| 15 | TwinCAT native subscribe all-or-nothing: 0 data + 2k register/delete per 30 s, health Healthy | `TwinCATDriver.cs:690-721` |
| 16 | TwinCAT native `LastSuccessfulRead` never advances, so one bad write forces a full reconnect; a dead router never degrades | `TwinCATDriver.cs:540,889` |
| 17 | FOCAS `_devices` plain Dictionary cleared under 4 live readers | `FocasDriver.cs:46,412` |
| 18 | FOCAS macro/param `(short)` narrowing: `MACRO:99999` reads var −31073 at Good | `WireFocasClient.cs:408` |
| 19 | AbLegacy strictly serial, no backoff: one dead device stalls all and trips the watchdog on healthy ones | `AbLegacyDriver.cs:375-479` |
| 20 | Modbus transport never self-heals after 2 consecutive socket failures | `ModbusTcpTransport.cs:66` |
| 21 | Modbus deadband swallows Good-after-Bad recovery at the same value, so the node stays Bad | `ModbusDriver.cs:236` |
| 22 | S7 one mis-authored tag faults the whole driver at init (10 s retry loop); no editor or deploy-gate check | S7 H3 |
| 23 | Calculation compile-error script recompiled every trigger under the driver-wide gate | `CompiledScriptCache.cs:71` |
### Medium highlights
- **Performance:**
- OpcUaServer: rebuild O(k²) per folder under `Lock`; equipment-subtree removal O(M·N); full events reconcile on every condition removal.
- Runtime: only 4 drivers implement `IBatchSubscribable`; historization fsync per burst caps throughput; the deploy apply runs on the single pinned OPC UA thread.
- AbCip: `MaxConcurrentReads=8` caps native MSP packing at 8 services per round trip (~12× more round trips than needed).
- QuestDB: write = 3 RTT + a 10 ms WAL poll per group.
- Alarm-history writers: 1 event per round trip on all backends.
- AdminUI: per-keystroke full Roslyn compilation with no cancellation; `/hosts` full re-render per health publish; `/raw` tree not virtualised.
- MTConnect: `/current` fetched and fully parsed per read.
- OpcUaClient: a single `_gate` serialises everything.
- **Leaks:**
- Scripted-alarm `_valueCache` never pruned on reload.
- Deployment artifact blobs and audit tables have no retention.
- OpcUaClient half-created subscription per subscribe retry.
- AdminUI driver-status stores never evict (ghost `/hosts` rows).
- **Correctness:**
- `StartDeployment` reads config twice with no transaction (the validated set ≠ the shipped artifact).
- One duplicate audit event drops the whole 500-event batch.
- API-key pepper not validated at start.
- Pre-auth DoS: distinct unknown key IDs each cost a SQL query under the SessionManager lock.
- Gateway `TimeAverage`/`Total` return wrong values at Good.
- Gateway writer relabels Local timestamps as UTC.
- Value writers swallow OCE (the recorder's shutdown guard is dead).
- FOCAS `String` type publishes numbers at Good.
- AB/TwinCAT bit-in-word RMW hardcodes a 32-bit parent.
- AbCip BOOL-array element read decodes the wrong bit.
- AdminUI single-click deletes without confirmation on the equipment page and ScriptEdit; Monaco offers `ctx.SetVirtualTag`, which the runtime drops.
## Checked and fine (don't re-chase)
- **Monitored items and continuation points:** monitored-item / event-item map lifecycle and continuation-point bounds.
- **Auth:** ACL decision cache; the gRPC auth interceptors; ApiKey constant-time compare.
- **Persistence and SQL:** DbContext lifetimes + `AsNoTracking`; all dynamic SQL identifiers regex-validated (no injection); historization outbox disk-backed and bounded.
- **Script and dependency engine:** collectible-ALC unload (when disposal actually happens); DependencyGraph Tarjan.
- **Transport and wire layers:**
- The whole `Ab.Transport` layer and native lease balance.
- ADS handle lifetimes.
- FOCAS wire parsers.
- Modbus desync recovery.
- MQTT teardown + Sparkplug cache bounds.
- MTConnect stream handling.
- SQL driver disposal/injection hygiene.
- **Actor and polling plumbing:** `PollGroupEngine`; actor timer/DPS subscription cleanup.
## Cross-repo corroboration (mxaccessgw session, 2026-10-22)
The gateway review reports two things that bear on the Galaxy session leak (finding #1):
- **Leaked sessions are held for a long time.** A session that never calls `StreamEvents` is held for the full `DefaultLeaseSeconds` (1800 s). Detach-grace only starts after a subscriber has attached at least once, `GatewaySession.ExtendLease` has no callers, and there is no renew RPC. So each leaked GalaxyDriver session pins an x86 worker for 30 min, and about 64 respawns in that window would use up `MaxSessions`.
- **The open-sessions gauge drifts upward.** Sessions closed by the timed sweep never decrement `mxgateway.sessions.open` (`TryBeginCloseIfExpired` sets `_closeStarted`, so `CloseSessionCoreAsync` skips `SessionClosed()`).
This makes #1 a fleet-availability risk, not just a leak.
## Suggested remediation order
1. Theme 1 (runtime disposes drivers) + Galaxy `ShutdownAsync`/DeployWatcher (#1, #7, #10). Small change, removes the only Critical.
2. HistoryRead server-side cap/paging (#2).
3. Deploy-pipeline wedge + OPC UA start-failure health (#3, #4).
4. Script timeout quarantine (theme 2, #5, #23) + scripted-alarm restart (#6).
5. Node-manager phantom/stale-MonitoredNode pair (#8, #9). Needs a live gate, since unit tests won't show it.
6. Driver health-contract sweep (theme 3) + Dictionary swaps (theme 4) as one batch per driver family.
7. AdminUI 32 KB hub limit (#12). One-liner.
+175
View File
@@ -0,0 +1,175 @@
# Review 01 — `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer` (master @ d51d35bd)
Scope: `OtOpcUaNodeManager` (6,988 lines), `AddressSpaceApplier`, `AddressSpaceComposer`, `AddressSpacePlan`/`Planner`,
`AddressSpaceChangeClassifier`, `SdkAddressSpaceSink`, `OtOpcUaSdkServer`, `OpcUaApplicationHost`, `Security/*`,
`HistoryContinuationStore`, the HistoryRead helpers (`ReverseWalk`, `ReverseRawReader`, `ReverseEventsReader`,
`HistoryBounds`, `HistoryPaging`, `RawPageLimit`), `AdvertisedAggregateFunctions`, `SdkServiceLevelPublisher`.
SDK behaviour claims below were verified against the pinned SDK sources
(`~/Desktop/OpcUaTestServer/external/UA-.NETStandard`, 1.5.378 line): `NodeState.cs`,
`Diagnostics/CustomNodeManager.cs`, `NodeManager/MonitoredItem/MonitoredNodeMonitoredItemManager.cs`,
`NodeManager/MasterNodeManager.cs`. All paths are absolute; line numbers are from the current master.
---
## 1. Memory / resource leaks
### L1. Lazy `CreateVariable` resurrects removed (or never-declared) NodeIds as orphan variables under the shared root — and then poisons a later re-add
- **Severity:** High · **Confidence:** High (mechanism) / Medium (trigger frequency)
- **Location:** `/Users/dohertj2/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs:937-945` (`WriteValueLocked`), `:1059-1071` (`WriteAlarmCondition` fallback), `:6964-6986` (`CreateVariable`), `:2974-2975` (`EnsureVariable` early return)
- **Evidence:**
```csharp
if (!_variables.TryGetValue(key, out var variable))
{
variable = CreateVariable(nodeId, ns); // parented under _root, DataType=BaseDataType, CurrentRead only
_variables[key] = variable;
}
```
and `EnsureVariable`: `if (_variables.ContainsKey(key)) return;`
- **Failure scenario:** `WriteStatuses` was deliberately made non-creating because "the caller's per-driver ref index can legitimately name a node a surgical remove has just dropped" (`:825-828`). `WriteValues`/`WriteValue` still create. So a driver publish (or the UNS fan-out) that races a surgical `RemoveVariableNode` re-creates `s=<RawPath>` (and `s=<Area/Line/Equip/Name>`) as a flat, read-only, `BaseDataType` variable **directly under `OtOpcUa`**, browsable, with no `OnWriteValue`, no historian registration, no `_arrayOnScalarRefused`. It stays until the next *full* rebuild — surgical deploys made that rare. Worse: if the same tag is later re-added surgically, `EnsureVariable` early-returns on the phantom, so the "re-added" tag permanently keeps the phantom's wrong parent/DataType/writability. The planner never names the phantom, so no surgical remove ever reaches it.
- **Suggested fix:** Make `WriteValueLocked` non-creating (mirror `WriteStatusLocked`: unknown key ⇒ skipped, counted, Debug-logged). The create-on-first-write path predates `EnsureVariable` and has no legitimate production producer left. If some test relies on it, gate it behind an explicit `AllowLazyNodeCreation` that defaults off.
### L2. Stale `MonitoredNode2` survives a rebuild / remove+recreate: new subscriptions bind to the dead `NodeState`, and ACL "restore" re-attaches to dead nodes
- **Severity:** High · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:3471-3537` (`RebuildAddressSpace`), `:3573-3607` (`RemoveVariableNode`), `:4801` / `:4889` (revalidation restore). SDK: `MonitoredNodeMonitoredItemManager.cs:75-83`.
- **Evidence (SDK):**
```csharp
if (!MonitoredNodes.TryGetValue(handle.Node.NodeId, out MonitoredNode2 monitoredNode)) { ... new MonitoredNode2(m_nodeManager, cachedNode); }
handle.Node = monitoredNode.Node; // <-- the OLD NodeState
```
`RebuildAddressSpace`/`RemoveVariableNode` detach nodes from `PredefinedNodes` and the parent, but never touch the manager's `MonitoredNodes` map. `DeleteMonitoredItem` only drops the entry when its item count hits zero.
- **Failure scenario:** Client A (e.g. ScadaBridge) holds an item on `Plant/Modbus/dev1/Speed`. A deploy rebuilds (or surgically removes + re-adds) that node. A's item goes silent (known). Now ANY session — A after reconnecting, or a UaExpert operator — creates a new item on the same NodeId: the SDK reuses the existing `MonitoredNode2` and sets `handle.Node` to the **dead** NodeState, so the new item never receives a value either, with `Good` create status. Only when *every* session has deleted its items on that NodeId does a fresh create work. Same class: `RevalidateMonitoredItems` restore does `tracked.Handle.MonitoredNode?.Add(item)` and `RevalidateEventItems` calls `SubscribeToEvents(SystemContext, tracked.Handle.Node, …)` — after a rebuild both re-attach to dead nodes. Memory-wise each live item pins its dead subtree via `LiveItem.Handle` (bounded by item count, but it's the whole detached subtree).
- **Suggested fix:** In `RebuildAddressSpace`, `RemoveVariableNode`, `RemoveAlarmConditionNode`, `RemoveEquipmentSubtree`: for each removed NodeId, `MonitoredNodes.TryGetValue(nodeId, out var mn)` → queue `BadNodeIdUnknown` to each `mn.DataChangeMonitoredItems`, then `MonitoredNodes.Remove(nodeId)` (and drop the matching `_liveItems`/`_eventItems` entries). For a rebuild that recreates the same NodeId you could instead re-point the `MonitoredNode2` to the new `NodeState` — but removing it and letting clients re-create is simpler and matches the documented "rebuild kills subscriptions" contract while making re-create actually work.
### L3. `OnEquipmentTagWrite` continuation never observes a faulted write task
- **Severity:** Low · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:2202-2209`
- **Evidence:** `var outcome = t.IsCompletedSuccessfully ? t.Result : new NodeWriteOutcome(false, "write dispatch faulted");` — `t.Exception` is never read on the faulted branch.
- **Failure scenario:** Every faulted `WriteAsync` (gateway Ask timeout/exception) raises `TaskScheduler.UnobservedTaskException` at finalization; with a handler that logs, each failed device write becomes a late, context-free error line; the real reason is never logged here either (`outcome.Reason` is the constant string).
- **Suggested fix:** On the non-success branch read `t.Exception?.GetBaseException()` into the reason and log it (the pattern `AddressSpaceApplier.ProvisionHistorizedTags:727-733` already uses).
### L4. Legacy `bool[2]` placeholder path is reachable in production and transiently shadows the equipment folder in `PredefinedNodes`
- **Severity:** Medium · **Confidence:** Medium
- **Location:** `OtOpcUaNodeManager.cs:1059-1071`; producer `AddressSpaceApplier.cs:100-103` (`SafeWriteAlarmCondition(eq.EquipmentId, RemovedConditionState, …)`).
- **Evidence:** The applier writes `RemovedConditionState` against `eq.EquipmentId` for every removed equipment; an equipment id is a *folder*, never a condition, so `_alarmConditions` misses and the "legacy two-element placeholder … (and the existing unit tests)" branch runs `CreateVariable(alarmNodeId, ns)` + `AddPredefinedNode`. SDK `AddPredefinedNode` is `PredefinedNodes.AddOrUpdate(...)` (`CustomNodeManager.cs:586`), so the phantom variable **replaces the equipment folder's `PredefinedNodes` entry** (same realm + same `s=` id) until the folder loop in `RemoveEquipmentSubtree` (or the rebuild) removes it.
- **Failure scenario:** During a PureRemove/Rebuild apply a concurrent Read/Browse of the equipment folder resolves to a `bool[2]` variable. The phantom is cleaned up by the same apply on every path I traced (InScope matches `key.Id == equipmentNodeId`; an early `false` ratchets to rebuild), so it is transient — but it is a production path running through a self-described test fallback, and it does one needless `ClearChangeMasks` per removed equipment.
- **Suggested fix:** Make `WriteAlarmCondition` a logged no-op for unknown ids (as `WriteAlarmQuality` already is), and drop the applier's equipment-id condition write (an equipment never has a condition node).
### L5. (bounded, note) `_notifierFolders` / `_eventNotifierSources` not demoted on `RemoveAlarmConditionNode`
- **Severity:** Low · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:3637-3664`
- Documented and bounded by folder count; a raw device/group folder promoted for a native alarm stays a root notifier (and an advertised event-history source) after its last condition is removed until a rebuild. Harmless except `HistoryReadEvents` on it answers from the historian for a folder with no alarms. No action needed beyond awareness.
---
## 2. Unimplemented / stubbed / dormant
### U1. `HistoryReadEvents` projects `EventType`, `SourceNode`, `ConditionName` (and every other field) as `Variant.Null`
- **Severity:** Low · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:5942-5957`
- Only `EventId/SourceName/Time/ReceiveTime/Message/Severity` are populated. A client whose select clause includes `EventType` (most generic alarm viewers) gets nulls; a where-clause on `OfType` cannot be honoured. Documented as "spec-conformant", but it is a functional gap for historical alarm browsing. Fix: `HistoricalEvent` would need to carry `EventType`/`ConditionName`; map `SourceNode` from the source name when resolvable.
### U2. Equipment-tag materialisation + its surgical branch are dead code in v3
- **Severity:** Low · **Confidence:** High
- **Location:** `AddressSpaceComposer.cs:590-592` (`EquipmentTags = Array.Empty<EquipmentTagPlan>()` always), `AddressSpaceApplier.cs:197-215` (surgical `ChangedEquipmentTags` loop), `:1098-1161` (`MaterialiseEquipmentTags`), `AddressSpaceChangeClassifier.cs:162-172`.
- The composer never emits equipment tags, so the applier's `MaterialiseEquipmentTags` pass, the `UnsEquipmentVar`/alarm-under-equipment branch and `TagDeltaIsSurgicalEligible` are unreachable in production (kept "structurally empty"). Not a bug; it is maintained, tested dead weight that still participates in `IsEmpty`/classification. Candidate for removal.
### U3. Historian capability bits are fixed at promotion/materialise time
- **Severity:** Low · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:2831-2848` (`EnsureFolderIsEventNotifier` registers `_eventNotifierSources` only if `_historianDataSource is not NullHistorianDataSource` *now*), `:2987` / `:3084` (`HistoryWrite` bit depends on `_historianDataUpdater` *now*).
- Documented. Consequence: a `HistorianDataSource`/`HistorianDataUpdater` assigned after the first apply leaves existing folders without event history / HistoryUpdate until a full rebuild, and `SetHistorianDataSource(null)` later does **not** remove the HistoryRead bit from already-promoted folders (they then answer `BadHistoryOperationUnsupported`). Acceptable given the Host ordering, but there is no runtime guard that the ordering held.
### U4. Native conditions: Confirm / shelve / Enable / Disable ⇒ `BadNotSupported`; HistoryUpdate events/structure arms not overridden
- **Severity:** Low · **Confidence:** High
- **Location:** `:1496-1540`, base `HistoryUpdate` dispatch (not overridden for `UpdateEventDetails`/`UpdateStructureDataDetails`).
- Both are deliberate and documented; listed so the parent does not re-investigate.
### U5. Test seams present in production type (not dormant DI, just noting)
- `HistoryContinuationStore` internal setter + `InMemoryHistoryContinuationStore` (`HistoryContinuationStore.cs:104-158`), `SessionAclDecisionCache.GetOrAdd` ("production has no caller"), `AllEventsMonitoringConditionVisits`, `TrackLiveMonitoredItem` internal. All default to the production implementation; none alters behaviour. Fine.
---
## 3. Performance limiters
### P1. `RebuildAddressSpace` is O(k²) per folder because `NodeState.RemoveChild` is a linear scan + `RemoveAt`
- **Severity:** Medium · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:3475-3479`, `:3495-3499`, `:3513-3517`; SDK `NodeState.cs:4247-4265`.
- **Evidence:** `foreach (var v in _variables.Values) { v.Parent?.RemoveChild(v); PredefinedNodes?.Remove(v.NodeId); }` and SDK `RemoveChild` walks `m_children` with `ReferenceEquals` then `RemoveAt(ii)` (element shift). Removing all k children of one folder, in map order, costs ~k²/2 compares + shifts, all under `Lock`.
- **Failure scenario:** A device folder with 10k tags ≈ 50M compares under the manager lock on every full rebuild (alarm-bearing delta, container rename, etc.), stalling every Read/Browse/publish for the duration; UNS equipment folders with many references multiply it. `ForgetPredefinedSubtree` also allocates one `List<BaseInstanceState>` per node recursively.
- **Suggested fix:** Detach whole subtrees instead of leaves: remove top-level folders from `_root` once (children go with them), then clear the maps and `PredefinedNodes` keys in bulk; or iterate `folder.GetChildren` and build a new child list. Keep `PredefinedNodes.Remove` per id (dictionary, O(1)).
### P2. `RemoveEquipmentSubtree` scans every map on every call ⇒ O(M·N) for M removed equipment
- **Severity:** Medium · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:3685-3752`
- **Evidence:** `_variables.Keys.Where(InScope).ToList()`, same for `_alarmConditions`, `_notifierFolders`, `_eventNotifierSources`, `_folders`, plus `foreach (var wiring in _alarmNotifierWiring.Values) wiring.Folders.Where(...)` — each a full scan with `StartsWith` per key, under `Lock`.
- **Failure scenario:** A PureRemove deploy that drops 200 equipment on a 50k-node server does 200 × ~6 full-map scans ≈ 60M key comparisons under the lock, on the pinned publish thread, serialising all SDK service calls.
- **Suggested fix:** Walk the equipment `FolderState`'s children (the SDK tree already has them) to enumerate descendants, or maintain a per-equipment child index populated by `EnsureVariable`/`EnsureFolder`/`MaterialiseAlarmCondition`.
### P3. `ReconcileAllEventsMonitoring` runs a full O(C) reconcile on every `RemoveAlarmConditionNode` and every all-events (un)subscribe
- **Severity:** Medium · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:3659`, `:3714`, `:4240`, `:4298-4322`
- **Evidence:** Builds a new `HashSet` over `_alarmConditions.Values` plus `.Where().ToList()` each call. The materialise side was already made incremental ("made a deploy of N alarms O(N²) under Lock") but the removal side was not.
- **Failure scenario:** Surgically removing M alarm-bearing raw tags or scripted alarms costs O(M·C) under `Lock`; a client that creates/modifies many `i=2253` items (e.g. per-subscription) pays O(C) per call.
- **Suggested fix:** Add `LowerAllEventsMonitoring(condition)` (mirror of `RaiseAllEventsMonitoring`) for the two surgical removal sites; keep the full reconcile for rebuild and all-events (un)subscribe.
### P4. HistoryRead/HistoryUpdate arms block SDK request threads (sync-over-async) with a 5 s admission wait
- **Severity:** Low · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:5997-6014` (`RunBounded` → `Task.WhenAll(...).GetAwaiter().GetResult()`), `:6065-6080` (`gate.Wait(≤5 s)`).
- Documented and bounded (16 batches × 4 nodes), and HistoryRead is not under `Lock`. Under a slow historian (deadline 60 s) up to 16 SDK request threads sit blocked for a minute while 16 more block 5 s each before fast-failing; with `MaxRequestThreadCount = 100` this is survivable but eats 1/3 of request capacity. Consider `MaxConcurrentHistoryReadBatches` default lower on small nodes, or an async `HistoryReadAsync` override if the SDK version exposes one (`IAsyncNodeManager` exists in this SDK line).
### P5. `WireAlarmNotifiers` / SDK notifier lists are linear per folder
- **Severity:** Low · **Confidence:** High
- **Location:** `:1723-1760`; SDK `NodeState.AddNotifier` (`NodeState.cs:2385-2410`, linear dedup).
- An equipment folder referencing K native alarms costs O(K²) `ReferenceEquals` on wiring plus per-call `ToHashSet`/`Where().ToList()`/`Any` allocations. Bounded by K (tens–hundreds); only matters on rebuild of alarm-heavy equipment.
### P6. Hard-coded SDK request quotas
- **Severity:** Low · **Confidence:** High
- **Location:** `OpcUaApplicationHost.cs:435-441` — `MinRequestThreadCount = 5, MaxRequestThreadCount = 100, MaxQueuedRequestCount = 200`, `TransportQuotas = new TransportQuotas()` (defaults).
- Not configurable from `appsettings`; a burst above 200 queued requests is refused by the SDK. Fine for current fleet sizes; flagging as the ceiling.
---
## 4. Other correctness / concurrency hazards
### H1. Removed-tag resurrection + re-add poisoning (see L1) — correctness, not just leak
Already covered; the correctness half (a re-added tag silently inherits the phantom's shape) is the more serious consequence.
### H2. `AddPredefinedNode` is AddOrUpdate: any id collision between a variable and a folder in the same realm silently replaces the folder's lookup entry
- **Severity:** Medium · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:2901`, `:3014`, `:1559`, `:6984`; SDK `CustomNodeManager.cs:586`.
- `_variables`, `_folders`, `_alarmConditions` are separate maps keyed by `(realm, id)`, so nothing prevents the same `(realm, id)` existing as both a folder and a variable (L4 shows a production producer). The SDK dictionary keeps whichever was added last. Suggest a cross-map guard in `EnsureFolder`/`EnsureVariable`/`MaterialiseAlarmCondition` (`ResolveNodeState(key) is not null ⇒ log + return`).
### H3. `ConditionRefresh` (ACL path) walks notifier subtrees outside `Lock`
- **Severity:** Low · **Confidence:** High
- **Location:** `OtOpcUaNodeManager.cs:4646-4651`
- Only the `RootNotifiers` copy is taken under `Lock`; `notifier.ConditionRefresh(...)` then walks children while a rebuild can be mutating child lists. **The SDK base does exactly the same** (`CustomNodeManager.cs:3488-3515`), and `NodeState` guards its child list with `m_childrenLock`, so this is parity with the SDK rather than a regression. Listed so it is not re-raised.
### H4. `RemainingShelveMs` and condition timestamps use `DateTime.UtcNow`, not `Clock`
- **Severity:** Low · **Confidence:** High
- **Location:** `:1087`, `:1452`, `:1514`, `:1926`, `:2621-2626`, `:4778`.
- Inconsistent with `HistoryUpdate`/`EventHorizon`, which use the injected `TimeProvider`; purely a testability/consistency nit.
### H5. `OnEquipmentTagWrite` relies on `gateway.WriteAsync` never throwing synchronously
- **Severity:** Low · **Confidence:** Medium
- **Location:** `:2202`
- A synchronous throw escapes the handler while the SDK holds `Lock` in `Write`; the SDK's per-node `try` (`CustomNodeManager.cs` Write, around line 1885) converts it to a Bad result, so the lock is released — no deadlock, but the optimistic/prior bookkeeping is skipped and no audit event is raised. Wrap the call in try/catch → treat as failed outcome.
---
## 5. Checked and fine (do not re-investigate)
- **`_liveItems` / `_eventItems` / `_allEventsItems` lifecycle.** `OnMonitoredItemDeleted` removes data items; `MasterNodeManager` routes every delete/transfer of event items through `SubscribeToEvents(…, unsubscribe:true)` / `SubscribeToAllEvents(…, true)` (`MasterNodeManager.cs:3329-3345`, `:3528-3545`), which the overrides handle. Not session-leaking.
- **`DataPlaneAuthority` event handler** is detached in the setter and in `Dispose` (`:448-454`, `:488-500`). Correct.
- **`SessionAclDecisionCache`** is doubly bounded (20k/shard, 100k global) with sealed-shard counters; sharded by key, cleared on generation advance. No unbounded growth.
- **`HistoryContinuationStore`** production path is the SDK per-session store (capped at `MaxHistoryContinuationPoints`, disposed with the session). `InMemoryHistoryContinuationStore` is test-only and capacity-bounded.
- **`CreateDeadline()` CTS** is `using`-disposed per arm; `RunBounded`'s `SemaphoreSlim` is `using`-disposed; the process-wide `_historyReadBatchLimiter` is never disposed but holds no OS handle.
- **`_rebuildCarryOver` / `_restoredConditionReports`** cleared at `CompleteAddressSpaceApply`, which the publish actor calls in a `finally` (`OpcUaPublishActor.cs:500`). Bounded.
- **`AddRootNotifier` cost**: SDK copies `Server.EventManager.GetMonitoredItems()` per newly-promoted folder and calls `SubscribeToEvents(…, true)` for all-events items; the protected override short-circuits those (`:4257-4261`). O(E) per folder on first promotion only.
- **`WriteValues` chunking** (512 per `Lock` take) is a deliberate, documented trade-off; `ClearChangeMasks` under `Lock` is the SDK pattern.
- **Alarm method handlers under `Lock`** (`ReportAfterMethod`, `IsNativeAlarmNode`) — `Lock` is a reentrant `Monitor`; the router delegates are fire-and-forget by contract.
- **Revalidation lock ordering**: `QueueValue` (item lock) is taken *outside* `Lock`, then `Lock` is taken for attach/detach — no nested inversion with the publish path (`Lock` → item lock).
- **`AdvertisedAggregateFunctions.Apply`** takes the Diagnostics node manager's `Lock` from the `HistorianDataSource` setter — called from host start, never while our `Lock` is held.
- **`OpcUaApplicationHost.DisposeAsync`** detaches `ImpersonateUser` and stops the application; the hosted service stops/disposes the server (`OtOpcUaServerHostedService.cs:357-365`).
- **Impersonation `GetAwaiter().GetResult()`** is irreducible (SDK sync callback under `SessionManager` lock) and the API-key authenticator does no I/O; documented.
- **`AddressSpacePlanner.DiffById`** is dictionary-based O(n); `AddressSpaceComposer` builds all lookups as dictionaries (no quadratic joins found). `BuildEquipmentFolderPaths` is computed twice per compose (once for refs, once for raw) — trivial.
- **`ReverseWalk`** has explicit termination arguments (segment always narrows; `granularity` floor), a `LookAhead`/`Floor` clamp, and the tie-cluster budget throws rather than looping.
@@ -0,0 +1,308 @@
# Review 02 — Runtime + Cluster (Akka actors, deploy apply/acks, redundancy gates, mesh/config/telemetry transports, LocalDb)
Scope: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/**` and `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/**` at master `d51d35bd`.
Production code only. Every finding below was verified by reading the cited lines; drivers / Core / ControlPlane were
read only where a Runtime behaviour depends on them.
Severity scale: Critical / High / Medium / Low. Confidence: High = mechanism verified in code; Medium = mechanism verified,
magnitude or trigger frequency estimated.
---
## 1. Memory / resource leaks
### L1 — `IDriver` instances are never disposed; the runtime only ever calls `ShutdownAsync` (Calculation leaks compiled-script ALCs; OpcUaClient leaks a LoggerFactory + gate + CTS per respawn)
- **Severity:** High (Calculation) / Low (OpcUaClient) · **Confidence:** High (never-disposed), Medium (growth rate)
- **Location:**
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs:1527-1555` (`PostStop`)
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:3759-3775` (`StopChild`), `:3811-3827` (`HandleRestartDriver`)
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriver.cs:11` (no `IDisposable`/`IAsyncDisposable`)
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriver.cs:323-330` vs `:719-727`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs:3326-3367`
- **Evidence:** the only teardown the runtime performs on a stopped child is
```csharp
try { _driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult(); } // PostStop
```
No code path in Runtime/Host/Core.Resilience ever does `(driver as IDisposable)?.Dispose()` (grep over those trees found none).
`CalculationDriver.ShutdownAsync` only stops timers and clears `_pending`; `_evaluator.Dispose()` lives exclusively in
`CalculationDriver.Dispose()` (line 726). `CalculationEvaluator._cache` is a `CompiledScriptCache` that owns "a collectible
`AssemblyLoadContext` per evaluator" and unloads them only from its `Dispose()`/`Clear()` (`CompiledScriptCache.cs:130-135`).
`OpcUaClientDriver.ShutdownAsync` does detach the cert-validation handler, but the per-instance `SdkTelemetry.ForLogger(...)`
`LoggerFactory` (`SdkTelemetry.Release`, line 3367/3395), `_gate` and `_sessionWorkCts` are released only in `Dispose`/`DisposeAsync`.
- **Failure scenario:** `DriverSpawnPlanner` respawns a driver on ANY `DriverConfig` change (`DriverSpawnPlan.cs:62-68`), and a
Calculation driver's merged config folds in its tag/script set, so every edit to a calculation tag or script = one abandoned
`CalculationEvaluator` holding every compiled script assembly it ever loaded. Same on operator Restart and on driver removal.
On a calc-heavy node a day of authoring leaves N loaded script ALCs pinned by an unreachable-but-unloaded context; JIT code +
metadata are never reclaimed. OpcUaClient leaks one `LoggerFactory` (with its `ForwardingLoggerProvider`) + semaphore + CTS per respawn.
- **Fix:** in `DriverInstanceActor.PostStop`, after `ShutdownAsync`, do `if (_driver is IAsyncDisposable ad) await-ish ad.DisposeAsync(); else if (_driver is IDisposable d) d.Dispose();`
— but ONLY when the actor is being stopped for good, not on a supervision restart (the Props closure re-uses the same driver
object; see H2). Override `PreRestart` to skip the dispose, or have the host pass an "owned" flag. Alternatively make
`IDriver : IAsyncDisposable` and dispose in `StopChild`/`HandleRestartDriver` after `Context.Stop` via a watch.
### L2 — `DriverHostActor._lastReprovisionUtc` is never pruned
- **Severity:** Low · **Confidence:** High
- **Location:** `DriverHostActor.cs:531` (declaration), `:3897` (write); `StopChild` (`:3759`) never removes.
- **Evidence:** `_lastReprovisionUtc[msg.DriverInstanceId] = now;` with no `Remove` anywhere. The comment says "bounded by the number
of driver children" but it is bounded by the number of *distinct ids ever reprovisioned*, which grows with every driver
that was removed after crashing. ~16 bytes/entry; cosmetic unless ids are generated per deploy (they are stable GUID-ish ids).
- **Fix:** `_lastReprovisionUtc.Remove(driverInstanceId)` in `StopChild`.
### L3 — `DependencyMuxTagUpstreamSource._cache` retains every RawPath ever pushed, across deploys
- **Severity:** Low · **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/DependencyMuxTagUpstreamSource.cs:45-48`
- **Evidence:** `// Intentionally never pruned` — `_cache[path] = snapshot` with no eviction. The engine re-registers mux interest per
apply, so renamed/removed tags stop being pushed, but their last snapshot (boxed value + two DateTimes) stays forever.
- **Failure scenario:** a site that renames raw paths often (RawTree rename is a surgical remove+add) accumulates one stale snapshot
per retired path for the process lifetime. Bounded by churn, not by live config.
- **Fix:** on `ScriptedAlarmHostActor.OnAlarmsLoaded`, prune `_cache` to the loaded dep-ref union (add a `RetainOnly(IReadOnlySet<string>)`).
### L4 — `TelemetryLocalHub._evicted` grows per removed driver id
- **Severity:** Low · **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/TelemetryLocalHub.cs:47, 77-89, 92-98`
- **Evidence:** `Evict` adds to `_evicted`; only `Readmit` (re-spawn of the SAME id) removes. A driver removed for good keeps its id in
the set forever. Bounded by distinct removed ids; a few bytes each. Noted because the tombstone is load-bearing (#609) and cannot
simply be cleared on a timer without re-opening the late-write resurrection.
- **Fix:** acceptable as is; if desired, stamp eviction time and sweep entries older than the longest plausible in-flight capability
call (minutes), since the tombstone only needs to outlive the stopping child.
---
## 2. Unimplemented / stubbed / dormant
### U1 — `GetDiagnostics` → `NodeDiagnosticsSnapshot` is a stub payload AND the only client has no caller
- **Severity:** Medium · **Confidence:** High
- **Location:** `DriverHostActor.cs:2499-2517`; `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/FleetDiagnosticsClient.cs:35`;
`AdminUI/Clients/ServiceCollectionExtensions.cs:17`
- **Evidence:**
```csharp
new DriverInstanceDiagnostics(DriverInstanceId: Guid.Empty, Name: kv.Key,
State: kv.Value.Stubbed ? "Stubbed" : "Spawned", ConnectedDevices: 0, FaultedDevices: 0, LastChangeUtc: DateTime.UtcNow)
```
Connectivity, device counts and change time are placeholders regardless of the child's actual state. `IFleetDiagnosticsClient` is
registered in AdminUI DI but no `.razor`/`.cs` resolves it (grep: only the registration + the client itself). The message is handled
in all three behaviours so it looks live, but the data is fake and nobody reads it. `RunningFromCache` is the one real field.
- **Fix:** either delete `GetDiagnostics`/`NodeDiagnosticsSnapshot`/`IFleetDiagnosticsClient` (the `/hosts` page already gets real
state over the telemetry channels) or populate it from the child health snapshots. Leaving a plausible-looking stub is the
"dormant DI registration" class the review asked for.
### U2 — `TelemetryDialOptions.CallTimeoutSeconds` is bound and defaulted but consumed nowhere
- **Severity:** Low · **Confidence:** High
- **Location:** `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs:75-76`
- **Evidence:** `public int CallTimeoutSeconds { get; set; } = 30;` — zero references outside the options file (validators included; the
only other `CallTimeout*` hits are the unrelated Galaxy gateway option). The telemetry dialer therefore runs its server-streaming
call with no deadline from this knob, and an operator setting `TelemetryDial:CallTimeoutSeconds` changes nothing.
- **Fix:** wire it into the dialer's `CallOptions`/keep-alive, or delete it and the doc row in `docs/Configuration.md`.
### U3 — `HistorianAdapterActor.Receive<AlarmHistorianEvent>` has no production sender
- **Severity:** Low · **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs:63-66`
- **Evidence:** comment says "kept for a future engine→historian path"; `grep "new AlarmHistorianEvent("` over `src` finds no producer
(the adapter's own `Translate` is target-typed `new(`). The handler carries a WARNING about double-writes if anyone ever uses it.
- **Fix:** delete the receive (and the warning) — a latent double-historize trap with no current benefit.
### U4 — `DependencyMuxActor.OnAttributeValuePublished` is a documented test-only seam kept in the production actor
- **Severity:** Low · **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs:55, 106-123`; `DriverInstanceActor.cs:110-117`
- **Evidence:** "No production sender remains" / "treat it as dead in production". Allocates a per-subscriber single-item batch.
Harmless, but it is a message with a handler and no sender, which the review asked to flag.
- **Fix:** move the per-value shape to a test helper that wraps it in a one-item `AttributeValueBatchPublished`.
### U5 — `DriverHostActor` constructor keeps an unused `ILoggerFactory? loggerFactory` positional parameter
- **Severity:** Low (nit) · **Confidence:** High
- **Location:** `DriverHostActor.cs:639-640, 705, 840`
- **Evidence:** "Retained for constructor-signature stability; unused". Given the file's own warning that the positional Props list is
a mis-binding hazard (`:727-731`), dead positional slots raise that risk for no benefit.
---
## 3. Performance limiters
### P1 — Per-tag drivers pay a 3-hop actor chain and one SDK sink lock per changed tag; only 4 of 12 drivers batch
- **Severity:** Medium · **Confidence:** High
- **Location:** `DriverInstanceActor.cs:975-990` (handler choice), `:1384-1390` (`OnDataChangeForward`),
`DriverHostActor.cs:1602-1644` (`ForwardBatchToMux`), `OpcUaPublishActor.cs:402-423`;
`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs:404-416`
- **Evidence:** `IBatchSubscribable` is implemented by Modbus, FOCAS, S7 and AbCip only (grep). TwinCAT, AbLegacy and Sql use the same
`PollGroupEngine` but expose only `ISubscribable`, so the engine's per-tag `_onChange(...)` fires `self.Tell(new DataChangeForward(..))`
once per changed tag; each becomes a 1-item `AttributeValueBatchPublished` → host → `_dependencyMux.Tell` + 1-item
`AttributeValueBatchUpdate(List)` → `_sink.WriteValues` (one lock acquisition each). Galaxy, MQTT, MTConnect and OpcUaClient are
per-tag by nature too.
- **Failure scenario:** a 2k-tag TwinCAT/Sql device whose values all move per 1 s cycle = ~6k actor messages/s + 2k node-manager lock
round-trips/s on the pinned OPC UA thread, versus 3 messages + 1 lock for a Modbus device of the same size. That thread is also
every client's Publish path.
- **Fix:** implement `IBatchSubscribable` on the three remaining `PollGroupEngine` drivers (the engine already builds the batch;
it is a one-line event forward), and consider a time-boxed coalescer (e.g. 10–20 ms) in `DriverInstanceActor` for the push drivers.
### P2 — `DriverInstanceActor` suspends its mailbox for the whole device round-trip on writes/subscribes (serialized writes, head-of-line blocking of data changes)
- **Severity:** Medium · **Confidence:** High
- **Location:** `DriverInstanceActor.cs:520-523, 542` (`ReceiveAsync<WriteAttribute>`, `<Subscribe>`, `<Unsubscribe>`, `<SubscribeAlarms>`),
`:858-901` (5 s write bound), `:948-1011` (10 s subscribe + 5 s unsubscribe), `:1296-1355` (10 s alarm unsub + 10 s sub)
- **Evidence:** Akka `ReceiveAsync` suspends the actor until the task completes. A write holds the actor up to 5 s; a data
(re)subscribe up to 15 s; an alarm re-subscribe up to 20 s. Meanwhile every `DataChangeForward`/`DataChangeBatchForward` and every
further `WriteAttribute` queues behind it. The host's `Ask` for a write is 8 s (`DriverHostActor.cs:2176-2177`), so the SECOND of two
concurrent operator writes to a slow device can time out at the host ("write timeout") while the device actually accepted it later.
- **Failure scenario:** operators issuing several writes to a Galaxy/OPC UA upstream with ~2 s latency get spurious "write timeout"
reverts (the optimistic-write self-correction then reverts a value the device DID take); value publishes stall for the duration.
- **Fix:** run writes via `PipeTo` (capture `Sender`, don't suspend the mailbox) with a per-driver in-flight cap, keeping
`ReceiveAsync` only for the subscribe/unsubscribe pair that must serialize against `_subscriptionHandle`.
### P3 — `ContinuousHistorizationRecorder` performs one awaited fsync per ingested burst on the actor thread (serial fsync cap; unbounded mailbox backlog)
- **Severity:** Medium · **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:308-309, 463-466`;
`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/Recorder/FasterLogHistorizationOutbox.cs:158-169`
- **Evidence:** `ReceiveAsync<DependencyValueBatch>(OnValueBatchAsync)` → `await _outbox.AppendBatchAsync(entries, ...)`; under the default
`CommitMode = PerEntry` that is `_log.CommitAsync` (fsync) before the handler returns. Ingest throughput is therefore
`1 / fsync_latency` bursts/s regardless of core count, and the mux keeps `Tell`ing into an unbounded mailbox.
- **Failure scenario:** 50 drivers × 1 Hz poll on a disk with 10 ms fsync = 500 ms/s busy; at 100+ bursts/s (many drivers, or MQTT push)
the mailbox grows without bound and heap climbs until the input stops. Nothing meters mailbox depth.
- **Fix:** default `CommitMode` to `Periodic` (100 ms) for production, or move the append off the mailbox (pipe the commit outcome back,
like the drain) with a bounded in-flight count; expose mailbox depth via `GetStatus`.
### P4 — `DriverInstanceActor.PostStop` blocks a dispatcher thread on an unbounded `ShutdownAsync`
- **Severity:** Medium · **Confidence:** High
- **Location:** `DriverInstanceActor.cs:1550`
- **Evidence:** `_driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult();` — no token, no `WaitAsync(timeout)`. Runs on the
default dispatcher for every `Context.Stop(child)` (deploy removal, Restart, respawn-on-config-change) and inside `DriverHostActor`'s
`PreRestart` child-stop cascade.
- **Failure scenario:** a driver whose shutdown waits on a dead socket/COM proxy (OpcUaClient subscription deletes against a half-dead
session, Galaxy gateway stream close, FOCAS handle release) pins a fork-join thread for its whole wait; a few such stops starve the
default dispatcher, and `CoordinatedShutdown` / a `DriverHostActor` restart (which waits for all children to terminate before
`PostRestart`) hangs for the sum of them.
- **Fix:** `ShutdownAsync(cts.Token).WaitAsync(TimeSpan.FromSeconds(5))` with a logged timeout; or stop children via a two-phase
"ShutdownRequested → PipeTo → PoisonPill" protocol so no actor thread blocks.
### P5 — `OpcUaPublishActor` runs the entire deploy apply (parse, plan, materialise) on the single pinned OPC UA thread; legacy path also does a SQL read there
- **Severity:** Medium (by design, worth a note) · **Confidence:** High
- **Location:** `OpcUaPublishActor.cs:483-725` (`HandleRebuild`/`HandleRebuildCore`), `:942-977` (`LoadArtifact`/`LoadLatestArtifact`)
- **Evidence:** the actor is pinned (`opcua-synchronized-dispatcher`) so a rebuild of a 50k-node composition serialises every live
`AttributeValueBatchUpdate` and (because the node manager lock is the SDK's) every client Publish behind it. The surgical-deploy work
bounds the common case, but `MaterialiseHierarchy` still runs in full each apply and a classifier miss is a full rebuild. The
`_dbFactory`-backed `LoadArtifact` branch (only reachable for a `RebuildAddressSpace` without bytes — no production caller today) would
also block that thread on central SQL.
- **Fix:** parse + plan (`DeploymentArtifact.Open`, `ParseComposition`, `AddressSpacePlanner.Compute`) off-thread via `PipeTo`, keeping only
the sink materialisation on the pinned thread; delete the bytes-less `LoadArtifact` branch or route it through the host.
### P6 — `VirtualTagActor` publishes a cluster-wide DPS `ScriptLogEntry` on EVERY failed evaluation (no rate limit)
- **Severity:** Medium · **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs:353-356, 375-399`
- **Evidence:** `OnEvaluationFailed` → `PublishLog(level, reason)` unconditionally ("Always meters + publishes the per-failure
script-log entry"), then `DistributedPubSub...Publish(ScriptLogsTopic, entry)` + `_telemetryHub.Emit`. The Bad-status publish is
deduped once per transition; the log line is not.
- **Failure scenario:** a script that throws on every input (e.g. a null-dereferencing cast after a device started sending nulls)
on a 1 Hz dependency emits 1 DPS message/s per virtual tag across the mesh and into every telemetry subscriber; 200 such tags =
200 msg/s of identical text into the AdminUI `/script-log` bridges and the gRPC telemetry stream (bounded channels drop-oldest,
so real entries are shed).
- **Fix:** rate-limit per vtag (first N per minute, then a summary count), mirroring `SdkTelemetry`'s 20/min window in the OpcUaClient driver.
### P7 — `DbHealthProbeActor` runs a synchronous EF query on its actor thread with no explicit command timeout
- **Severity:** Low · **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Health/DbHealthProbeActor.cs:52-65`
- **Evidence:** `db.Deployments.AsNoTracking().Take(1).ToList()` every 5 s; relies on SqlClient's implicit 15 s connect / 30 s command
timeouts. `OpcUaPublishActor` Asks it with a 1 s timeout and fail-safe-demotes, so the consumer is fine; the cost is a dispatcher
thread held for up to 30 s per tick during a SQL hang, and ticks queue behind each other.
- **Fix:** `Task.Run` + `PipeTo`, or `SetCommandTimeout(2s)` like `DriverHostActor.LoadArtifactBlob` does.
---
## 4. Other hazards (concurrency / lifecycle)
### H1 — A supervision restart of `ScriptedAlarmHostActor` disposes the shared `ScriptedAlarmEngine`; the restarted actor then fails every `LoadAsync` forever (scripted alarms on the node silently die)
- **Severity:** High · **Confidence:** High (mechanism), Medium (trigger frequency)
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs:155-163` (Props captures the engine),
`:676-687` (`PostStop` disposes it), `:406-429` (documented "restarts the actor" policy), `DriverHostActor.cs:1176-1183` (engine built once);
`src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs:239-241, 1153-1158`
- **Evidence:**
```csharp
// PostStop
_cts.Cancel(); _engine.OnEvent -= _onEngineEvent; _engine.Dispose(); _cts.Dispose();
```
Akka's default `PreRestart` calls `PostStop()`, then re-creates the actor from the SAME `Props` — i.e. around the same, now-disposed,
`engine` and `upstream`. `ScriptedAlarmEngine.Dispose` sets `_disposed`, kills the shelving timer and `UnsubscribeFromUpstream()`;
`LoadAsync` then throws `ObjectDisposedException` (`:241`) and every reactive path early-returns on `_disposed`. The host's own doc
(`:406-416`) states that an exception escaping `OnDependencyChanged` "restarts the actor" and calls that acceptable because "the next
delivery re-converges" — it cannot, because the engine is dead. The new incarnation re-subscribes `OnEvent` (ctor) and keeps piping
`Status.Failure` → `OnLoadFailed` ("engine LoadAsync failed — alarms not (re)loaded") on every deploy until `DriverHostActor` itself restarts.
- **Failure scenario:** any throw inside a `Receive` of this actor — `_upstream.Push` argument validation, a `ToSnapshot`/`ToDefinition`
mapping surprise, `ReassertConditionNodes` → `_engine.GetProjections()` — and the node's scripted alarms stop evaluating with only a
Warning per deploy as evidence; condition nodes stay at their last state (no Bad quality, no `/alerts` row), which reads as "no alarms".
- **Fix:** either (a) override `PreRestart` to NOT dispose (detach `OnEvent` only; dispose in a real stop via a flag) and re-subscribe in
`PostRestart`, or (b) make the engine owned per incarnation: have `DriverHostActor` pass a factory `Func<(upstream, engine)>` and build
them in the actor ctor, so a restart gets a fresh engine. Add a test: throw from a `DependencyValueBatch` handler, then assert a later
`ApplyScriptedAlarms` loads cleanly.
### H2 — `DriverInstanceActor.InitializeAsync` lets reconnect attempts overlap on the same driver object (concurrent `ShutdownAsync` + `InitializeAsync`)
- **Severity:** Medium · **Confidence:** High
- **Location:** `DriverInstanceActor.cs:743, 791` (`RetryConnect` every `_reconnectInterval` = 10 s), `:807-856` (`Task.Run`, generation token),
`src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceOptions.cs:88, 100, 112` (Initialize timeouts 5 / 8 / 15 s)
- **Evidence:** the periodic timer fires regardless of whether the previous attempt is still running; `_initGeneration` only discards the
RESULT of an older attempt. Each attempt first runs `driver.ShutdownAsync(CancellationToken.None)` (unbounded, unwrapped) and then the
invoker-bounded `InitializeAsync`. With Tier C's 15 s Initialize timeout (> 10 s cadence), or `NullDriverCapabilityInvoker` (no timeout —
tests/harness graphs), attempt N+1's `ShutdownAsync` runs while attempt N's `InitializeAsync` is still inside the driver.
- **Failure scenario:** exactly the dispose-before-overwrite race the drivers' rule file warns about (`.claude/rules/drivers.md`,
"MUST tear down before re-initialising"): a transport field is overwritten by attempt N while attempt N+1's teardown is disposing the one
it read a moment earlier — a leaked socket or an `ObjectDisposedException` inside the driver per overlap, for as long as the device is down.
- **Fix:** gate the retry on "no attempt in flight" (a `_initInFlight` bool cleared by the piped `InitializeSucceeded/Failed`), and bound
the teardown (`ShutdownAsync(ct).WaitAsync(5 s)`).
### H3 — `ScriptedAlarmHostActor.OnAlarmCommand` and `ContinuousHistorizationRecorder.GetStatus` suspend their mailbox on unbounded awaits
- **Severity:** Low · **Confidence:** High
- **Location:** `ScriptedAlarmHostActor.cs:581-648` (`CancellationToken.None` on every engine op), `ContinuousHistorizationRecorder.cs:313-320`
- **Evidence:** both are `ReceiveAsync`; the engine ops end in `IAlarmStateStore.SaveAsync` (LocalDb/SQLite) and `GetStatus` in
`_outbox.CountAsync`. A hung SQLite file lock (replication snapshot, disk full) stalls dependency-value ingestion / status replies for
the duration with no timeout.
- **Fix:** linked token with a timeout (10 s) and log on cancel.
### H4 — `HistorianAdapterActor` fans every alert into an unbounded number of concurrent SQLite inserts
- **Severity:** Low · **Confidence:** High
- **Location:** `HistorianAdapterActor.cs:66, 79, 159-170`
- **Evidence:** `_ = EnqueueAsync(Translate(t))` per transition — no in-flight cap. Each catches and logs, so nothing is lost silently, but an
alarm storm (a native source re-raising hundreds of conditions on reconnect) spawns hundreds of concurrent `ExecuteAsync` against one
SQLite connection pool, where they serialise on the file lock and hold pool threads.
- **Fix:** a `Channel<AlarmHistorianEvent>` drained by one loop, or a bounded `SemaphoreSlim`.
### H5 — `DriverHostActor` `Stale` behaviour has no receive for driver value/alarm bursts or a late `ArtifactForApply`
- **Severity:** Low · **Confidence:** High (unreachable in practice today)
- **Location:** `DriverHostActor.cs:2448-2497`
- **Evidence:** `Stale()` registers neither `AttributeValueBatchPublished`, `AttributeAlarmPublished`, `ArtifactForApply` nor `ApplyWedged`,
so they would hit `Unhandled` (dead-letter log noise, dropped). It is only entered from `Bootstrap`'s catch before any child is spawned,
so no child exists to send them — but `BeginDirectApply`'s orphan replay in the same try block has already `Become(Applying)` +
started a `Task.Run` pipe-back before the catch could run; nothing between them throws today, which is the same "by accident of
ordering" argument the file itself rejects for the cache pipe-back (`:3159-3166`).
- **Fix:** register the four receives in `Stale` as drops, like the others.
---
## 5. Checked and fine (looked like leaks/hazards, are bounded)
- **`DriverHostActor._pendingAcks` / apply-outcome timers** — every entry is removed by `RebuildCompleted`, `ApplyOutcomeTimedOut` or
`DropPendingAck`; timer keyed per correlation (`:2778-2800`). Bounded by in-flight applies (one at a time).
- **Stash growth in `Applying`** — bounded by distinct deployment ids via `_deferredDispatches` (`:1529-1546`), cleared in `BecomeSteady`.
- **Routing maps (`_nodeIdByDriverRef`, `_driverRefByNodeId`, alarm maps, `_desiredByDriver`, `_valueRefsByDriver`)** — clear-and-rebuild per
apply (`:3485-3565`); `NativeAlarmProjector.RetainOnly` prunes removed conditions (`:3550`); `_commsLatched` prunes to empty (`:1839-1844`).
- **DPS / cluster subscriptions** — the DPS mediator watches subscribers and removes them on `Terminated`; `DriverHostActor.PostStop`
unsubscribes its cluster subscriptions (`:1080-1088`); `ClusterRoleInfo.SubscriberActor.PostStop` unsubscribes (`ClusterRoleInfo.cs:276`).
EventStream subscriptions are keyed by actor ref and dropped on termination.
- **`IDriver` event handlers** — data/batch/alarm/rediscovery/host-status/connection-loss handlers are all detached in `PostStop`
(`DriverInstanceActor.cs:1544-1549`), so a respawn around the same driver object does not accumulate dead-`Self` closures.
- **Child actor lifecycle** — `VirtualTagHostActor` watches + stops children and prunes on `Terminated`; `PeerProbeSupervisor` reconciles
against the snapshot with generation-suffixed names; `DependencyMuxActor` watches subscribers and unregisters on `Terminated`;
`DriverHostActor` uses generation-suffixed child names so stop+respawn never collides.
- **`DriverInstanceActor._faultTimestamps`** — pruned to the 5-minute window on every health publish (`:1425-1431`).
- **gRPC channels** — one `GrpcChannel` per endpoint, cached, disposed by DI at host shutdown (`GrpcApplyAckSender`, `GrpcDeploymentArtifactFetcher`);
per-endpoint deadlines (`AckTimeoutSeconds` 10 s, `FetchTimeoutSeconds` 30 s) are positive by default and validated.
- **LocalDb artifact cache** — newest-2 retention per cluster inside the store transaction, idempotent re-store skip (`LocalDbDeploymentArtifactCache.cs:139-166, 204-235`).
- **`ParsedDeploymentArtifact` / `JsonDocument`** — opened once per apply, disposed in `HandleArtifactForApply`'s `finally` and in
`OpcUaPublishActor.HandleRebuildCore`'s `using`; the cache continuation captures only scalars (`DriverHostActor.cs:3207-3231`).
- **`TelemetryLocalHub` channels** — bounded `DropOldest`; `Detach` removes the subscriber and completes the writer; `Emit` never blocks.
- **`ContinuousHistorizationRecorder` drain** — single-flight (`_draining`), bounded parallelism (`SemaphoreSlim(_drainConcurrency)`),
outbox capacity 500k drop-oldest; `_tagWriteFailures` entries are cleared on the tag's next success and bounded by tag count.
- **`ConfigureAwait(false)` + `Self`/`Sender`** — every such site captures `Self`/`Sender` before the await
(`ContinuousHistorizationRecorder.IngestAsync:410`, `GetStatus:317`, `DriverHostActor.HandleRouteNodeWrite:2175`, `HandleSubscribeAsync:955-956`);
`OpcUaPublishActor.OnHealthTick`'s `ContinueWith` only constructs a record. No regression of the #491 class found.
- **`DriverHostActor` restart** — default `PreRestart` stops all children and Akka waits for their termination before `PostRestart`, so the
re-run `PreStart` (`SpawnVirtualTagHost`/`SpawnScriptedAlarmHost` with fixed names) cannot hit an actor-name collision; a NEW
`ScriptedAlarmEngine` is built, so H1 applies only to the alarm host's own restart.
- **`DriverHostActor.DriverMemberCount()` on the per-write / per-transition gate path** — `Cluster.State` is the cached read-view snapshot
and the LINQ count is over a 2–6 member set; negligible.
- **Unreachable-member set (`_unreachableNodes`)** — removed members deliberately retained (documented #672 reasoning); bounded by
addresses ever seen in a pair.
@@ -0,0 +1,380 @@
# Review 03 — Host / ControlPlane / Security / Configuration
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
Repo: `/Users/dohertj2/Desktop/OtOpcUa` @ `d51d35bd` (master). Read-only audit of production code in
`src/Server/ZB.MOM.WW.OtOpcUa.Host`, `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane`,
`src/Server/ZB.MOM.WW.OtOpcUa.Security`, `src/Core/ZB.MOM.WW.OtOpcUa.Configuration`.
Every finding below was verified by reading the cited code. Line numbers are from the files as checked in.
Severity: Critical / High / Medium / Low. Confidence: High / Medium.
---
## 1. Memory / resource leaks
### 1.1 Runaway virtual-tag script leaks one spinning thread-pool thread per evaluation, with no breaker
- **Severity:** High **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs:155-160`;
`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/TimedScriptEvaluator.cs:17-25`;
`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagHostActor.cs` (no timeout bookkeeping — only line 224 mentions timeouts).
- **Evidence:**
```csharp
var timed = new TimedScriptEvaluator<VirtualTagContext, object?>(evaluator, _runTimeout);
var raw = timed.RunAsync(context).GetAwaiter().GetResult(); // RoslynVirtualTagEvaluator.cs:158-159
```
`TimedScriptEvaluator` remarks: *"when a script times out, the underlying ScriptRunner task continues
running on a thread-pool thread … In the CPU-bound-infinite-loop case that's effectively 'leaked' …
the orphan thread is rare."*
- **Failure scenario:** The "rare" claim only holds per evaluation. A change-triggered or timer-triggered
virtual tag whose script spins (e.g. `while(true)`, or a `ctx.GetTag` loop that never converges)
is re-evaluated on every upstream change; each evaluation times out after 2 s and leaves a thread
burning 100 % CPU forever. `VirtualTagHostActor` does nothing on a `Failure("timed out…")` result
except publish Bad quality — no quarantine, no back-off, no consecutive-timeout cap. A 1 Hz input
produces ~1 orphaned core per second; the node is CPU-starved (Akka dispatchers, OPC UA publish
thread, SQL) within minutes and only a process restart recovers it. A Designer-role user can author
this through the AdminUI.
- **Suggested fix:** Track consecutive timeouts per `virtualTagId` in `VirtualTagHostActor`/evaluator and
quarantine the tag (stop evaluating, publish Bad, log Error, surface on `/uns`) after N (e.g. 3)
timeouts until the script is edited. Longer term: cooperative `CancellationToken` polling injected
into compiled scripts, or an out-of-process runner (→ #750).
### 1.2 `Deployment` / `NodeDeploymentState` / `ConfigEdit` / `ConfigAuditLog` grow without bound — every deploy stores a full artifact blob forever
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs:338-357`;
`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/OtOpcUaConfigDbContext.cs:720` (`ArtifactBlob varbinary(max)`);
no `Deployments.Remove`/`ExecuteDelete`/retention anywhere in `src` (grep verified).
- **Evidence:**
```csharp
db.Deployments.Add(new Deployment { …, ArtifactBlob = artifact.Blob }); // AdminOperationsActor.cs:338-345
```
- **Failure scenario:** Each `StartDeployment` persists the gzip'd snapshot (tens of KB to MBs on a 50k-tag
fleet) plus a `ConfigEdit` marker and one `NodeDeploymentState` row per node. Nothing prunes any of
them. Over a year of routine edits the `Deployment` table alone is GBs of blobs; `Deployments.razor`
projects the blob away (fine), but `ClusterOverview.razor:135` and
`ConfigPublishCoordinator.PreStart` (`ConfigPublishCoordinator.cs:164-167`) each materialise the full
newest row including the blob. The `IX_Deployment_Status` index keeps the in-flight query cheap, so
this is capacity/backup growth rather than an immediate hot-path cost.
- **Suggested fix:** Retention job on the admin singleton (keep last N sealed + anything referenced by a
`NodeDeploymentState` newer than X days; null out `ArtifactBlob` on older rows rather than deleting,
so the audit row survives). Project `ArtifactBlob` away in the two full-row reads.
### 1.3 `MeshGrpcCommandDialer` channel cache never evicts a stale endpoint
- **Severity:** Low **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/MeshGrpcCommandDialer.cs:93, 194-199`.
- **Evidence:** `_channels.GetOrAdd(endpoint, GrpcChannel.ForAddress)` keyed by the `http://{Host}:{GrpcPort}`
string; only `Dispose()` removes entries.
- **Failure scenario:** Every `ClusterNode.Host`/`GrpcPort` edit (host rename, port move, IP→DNS migration)
leaves the old `GrpcChannel` — and its idle HTTP/2 connection/socket — alive for the life of central.
Bounded by the number of distinct endpoints ever configured, so it is a slow leak, not a flood.
- **Suggested fix:** On `ContactsLoaded`, dispose channels whose endpoint is no longer in the target set
(the actor already knows the set), or use `SocketsHttpHandler.PooledConnectionIdleTimeout`.
### 1.4 `OtOpcUaServerHostedService.StopAsync` detaches every sink but leaves the SDK listening until host dispose
- **Severity:** Low **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs:448-476` (StopAsync)
vs `482-490` (DisposeAsync → `_appHost.DisposeAsync()` → `ApplicationInstance.StopAsync()`).
- **Evidence:** `StopAsync` nulls the write gateway, historian, authority and authorizer and returns
`Task.CompletedTask`; the only `ApplicationInstance.StopAsync()` is in `DisposeAsync`.
- **Failure scenario:** Between `IHostedService.StopAsync` and host disposal (Akka coordinated shutdown,
other hosted services' StopAsync) the OPC UA endpoint still accepts and activates sessions and serves
the last-written address-space values with the authority source set to empty — reads/subscribes
succeed on stale data, writes get "unavailable". Clients see a server that is up but inert instead of
a clean `BadServerHalted`/connection refusal that would trigger their redundancy failover sooner.
- **Suggested fix:** Call `_appHost.DisposeAsync()` (or an explicit `StopAsync`) from `StopAsync`
*after* the detaches, making `DisposeAsync` idempotent.
### 1.5 `CookieAuthenticationStateProvider` — dead type with a fire-and-forget ping loop
- **Severity:** Low **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Security/Blazor/CookieAuthenticationStateProvider.cs:40`.
- **Evidence:** `_pingLoop ??= Task.Run(() => PingLoopAsync(_cts.Token));` — and no registration or
reference to the type anywhere else in `src` (grep: only its own file).
- **Failure scenario:** Not reachable today, so no live leak. If someone registers it as the circuit
`AuthenticationStateProvider`, the injected `HttpClient` has no `BaseAddress` server-side (the repo's
own memory note "Blazor Server self-HubConnection" documents this trap), `GetAsync("/auth/ping")`
throws, the loop logs once and dies, and the circuit never learns its cookie expired.
- **Suggested fix:** Delete it (see 2.5), or wire it properly with a `BaseAddress` and register it.
---
## 2. Unimplemented / stubbed / dead features
### 2.1 A failed OPC UA SDK start is swallowed; no health check reflects it; the node stays "Ready", stays Primary-eligible, and acks deploys as Applied
- **Severity:** High **Confidence:** High (swallow + no health probe) / Medium (clean-ack claim)
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs:163-182`;
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs:34-96` (checks: configdb, akka,
cluster-primary, localdb-replication — nothing for the OPC UA endpoint; the only custom
`IHealthCheck` in Host is `LocalDbReplicationHealthCheck`);
`src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs:5-8` (falls back to `NullOpcUaAddressSpaceSink`).
- **Evidence:**
```csharp
catch (Exception ex)
{
_logger.LogError(ex, "OtOpcUaServerHostedService: SDK start failed; OpcUaPublishActor writes will continue to no-op");
// Don't rethrow — the rest of the host (admin UI, driver actors, etc.) can still boot.
return;
}
```
- **Failure scenario:** Port 4840 already bound, PKI directory unwritable, cert validation failure, or a
bad `OpcUa:*` override on a driver-only site node: the process logs one Error and then reports
`/health/ready` 200, joins the Akka mesh, is elected driver Primary by age, publishes ServiceLevel
through the still-`Null` publisher (so the peer is not promoted), and every deploy applies through
the Null sink and acks `Applied` (the Null sink's surgical methods return false → `SafeRebuild()` → Null
`RebuildAddressSpace()` no-op — the ack path was not traced further, hence Medium on that clause).
Clients get connection-refused from the node the fleet says is primary and healthy. The comment's
rationale ("admin UI can still boot") does not apply to a driver-only node, which has no AdminUI.
- **Suggested fix:** Add an `IHealthCheck` (tags Ready+Active) that reports Unhealthy while
`_server?.NodeManager is null` after StartAsync ran; on a driver-only node (no admin role)
rethrow so `Program.cs`'s fatal handler exits non-zero and the supervisor restarts; feed the
"SDK not started" fact into the ServiceLevel calculator so the peer takes Primary.
### 2.2 Deploy pipeline can wedge permanently: a `Deployment` stuck in `Dispatching` with no `NodeDeploymentState` rows blocks every future deploy
- **Severity:** High **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs:163-200`
(PreStart recovery), `202-248` (HandleDispatch), `355-382` (HandleDeadline);
`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs:193-206, 338-359`.
- **Evidence:**
```csharp
// PreStart: recovers Dispatching OR AwaitingApplyAcks rows
var inflight = db.Deployments.Where(d => d.Status == DeploymentStatus.Dispatching || d.Status == DeploymentStatus.AwaitingApplyAcks)…
_expectedAcks = nodeStates.Select(…).ToHashSet(NodeIdComparer); // EMPTY when HandleDispatch never ran
…
// HandleDeadline:
if (_acks.Count == _expectedAcks.Count) { /* Race … Already sealed/failed elsewhere. */ return; } // 0 == 0
```
`AdminOperationsActor` writes `Status = Dispatching` and `SaveChangesAsync()` **before** `_coordinator.Tell(...)`
(lines 338-359) and refuses any new deploy while a `Dispatching`/`AwaitingApplyAcks` row exists (193-206).
- **Failure scenario:** Any of: (a) `HandleDispatch` throws before its `SaveChanges` (transient SQL fault
seeding `NodeDeploymentState`) → actor restarts, message dropped; (b) the process dies between the
AdminOps `SaveChangesAsync` and the coordinator handling the Tell; (c) the singleton proxy's buffered
`DispatchDeployment` is lost during a singleton hand-over. In all three the row stays `Dispatching`
with zero `NodeDeploymentState` rows. `PreStart` recovers it with an empty expected set, the deadline
fires, `HandleDeadline` hits the `0 == 0` "race" branch and returns **without** updating the row or
clearing `_current`. From then on every `StartDeployment` is rejected with
`AnotherDeploymentInFlight` until someone hand-edits SQL. No log line names the condition.
- **Suggested fix:** In `PreStart`, treat a recovered deployment with an empty expected set as a failed
dispatch: re-run `DiscoverDriverNodes` + seed rows + re-dispatch, or mark it `TimedOut`/`Failed` and
`ResetForNext()`. In `HandleDeadline`, only take the "race" branch when `_expectedAcks.Count > 0`.
Consider moving the `Dispatching` row write into the coordinator so the row and the expected set are
always created together.
### 2.3 `Security:ApiKeys:Pepper` is never validated at startup — an unset pepper silently rejects every key
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:164-168` (plain `Configure<ApiKeyOptions>`, no validator);
`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/ApiKeyOpcUaUserAuthenticator.cs:114-127`;
`src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/ApiKeyOptions.cs` (`Pepper = string.Empty` default).
- **Evidence:** Program.cs comment: *"getting this wrong on either side is a silent authentication
failure rather than a startup error"* — acknowledged but not closed. The authenticator discovers it
only when `ApiKeySecret.Verify` throws `ArgumentException` on the first activation, and then only logs.
- **Failure scenario:** A new site node deployed without `Security__ApiKeys__Pepper` (or with a different
pepper from central) boots green, passes health, applies deployments, and denies every machine client
with the same opaque "Invalid API key." until someone reads the log. On a fused central node a wrong
pepper also makes the AdminUI *mint* keys that can never verify anywhere.
- **Suggested fix:** `AddValidatedOptions<ApiKeyOptions, ApiKeyOptionsValidator>` requiring a non-blank
pepper (minimum length) whenever `hasDriver || hasAdmin`; every other security section in this file
already uses `ValidateOnStart`.
### 2.4 JWT issuance is a dead end: `/auth/token` mints tokens nothing validates, and the signing-key check fires only on first use
- **Severity:** Low **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Security/Endpoints/AuthEndpoints.cs:39` (`/auth/token`);
`src/Server/ZB.MOM.WW.OtOpcUa.Security/Jwt/JwtTokenService.cs:66` (ctor throws), `106`, `139`
(`TryValidate`/`BuildValidationParameters` — no callers outside the file; no `AddJwtBearer` anywhere).
- **Evidence:** grep over `src` for `TryValidate|BuildValidationParameters|JwtTokenService` finds only
the registration, `IssueToken`, and the type itself.
- **Failure scenario:** The endpoint is an authenticated-only surface that hands out HS256 bearer tokens
no endpoint in the product accepts. With `Security:Jwt:SigningKey` absent (the Host `appsettings*.json`
carry none; only docker-dev sets it) the singleton's constructor throws `InvalidOperationException`
on the first `/auth/token` call → HTTP 500 instead of a startup failure.
- **Suggested fix:** Either delete `/auth/token` + `JwtTokenService`, or finish the feature
(`AddJwtBearer` using `BuildValidationParameters()`) and validate `JwtOptions` with `ValidateOnStart`.
### 2.5 Registered-but-never-resolved: `IAuditActorAccessor`; never-registered: `CookieAuthenticationStateProvider`
- **Severity:** Low **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:56`
(`TryAddScoped<IAuditActorAccessor, HttpAuditActorAccessor>()`); `src/Server/ZB.MOM.WW.OtOpcUa.Security/Audit/AuditActor.cs`;
`src/Server/ZB.MOM.WW.OtOpcUa.Security/Blazor/CookieAuthenticationStateProvider.cs`.
- **Evidence:** grep over `src` (cs + razor) for `IAuditActorAccessor|AuditActor.Resolve` → only the
registration and the two Security files; `CookieAuthenticationStateProvider` → only its own file.
- **Failure scenario:** Dead surface that reads as if audit rows are attributed to the HTTP principal.
Whoever eventually wires `HttpAuditActorAccessor` into a Blazor circuit will get `HttpContext == null`
after the first render and every audit row stamped `"system"`.
- **Suggested fix:** Delete both, or resolve the actor from `AuthenticationStateProvider` in the AdminUI
and actually consume `IAuditActorAccessor` where `AuditEvent.Actor` is built.
### 2.6 `AuditWriterActor` documents per-row dedup on the DB unique index, but one constraint hit drops the whole 500-event batch
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Audit/AuditWriterActor.cs:17-21` (claim) vs `86-117` (behaviour).
- **Evidence:**
```csharp
// "a retry of an already-flushed batch hits the constraint and we drop the duplicate insert
// without losing the rest of the batch" (lines 19-21)
…
using var db = _dbFactory.CreateDbContext();
foreach (var evt in snapshot) db.ConfigAuditLogs.Add(…);
db.SaveChanges(); // one statement batch
…
catch (Exception ex) { _log.Error(ex, "AuditWriter flush failed; {Count} events dropped", snapshot.Count); }
```
- **Failure scenario:** A single duplicate `EventId` (any at-least-once producer — the audit seam is
`Tell`-based through a singleton proxy that re-delivers across hand-over) makes `SaveChanges` throw
`DbUpdateException` for the whole batch; up to 499 unrelated audit events are discarded and the
Error line blames "flush failed". Audit is best-effort by contract, but the contract the comment
states is not the one implemented.
- **Suggested fix:** Catch `DbUpdateException` with SQL error 2601/2627, remove the offending entry
from the change tracker and retry once; or pre-filter with `WHERE EventId IN (...)`; or fix the comment.
### 2.7 `AuthEndpoints.LoginAsync` swallows the LDAP exception with no log
- **Severity:** Low **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Security/Endpoints/AuthEndpoints.cs:72-79`.
- **Evidence:** `catch (Exception) { return Results.StatusCode(503); }` — no logger call.
- **Failure scenario:** During a directory outage every login returns 503 and the server log contains
nothing from this path; the operator has to infer the cause from the GLAuth/AD side.
- **Suggested fix:** Log at Warning with the exception (the role-mapper branch a few lines down already does).
---
## 3. Performance limiters
### 3.1 The admin singleton's mailbox is suspended for the whole of every `ReceiveAsync` — a 60 s Test-Connect or a large deploy blocks AdminUI alarm acks, restarts and the next deploy
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs:77-82, 387-432`.
- **Evidence:**
```csharp
ReceiveAsync<StartDeployment>(HandleStartDeploymentAsync);
ReceiveAsync<TestDriverConnect>(HandleTestDriverConnectAsync); // awaits probe.ProbeAsync up to Math.Clamp(timeout,1,60) s
Receive<AcknowledgeAlarmCommand>(HandleAcknowledgeAlarm);
```
Akka.NET `ReceiveAsync` suspends the mailbox until the returned Task completes.
- **Failure scenario:** One operator pressing Test Connect against an unreachable PLC (TCP connect
black-hole, default 60 s clamp) stalls every other admin operation fleet-wide — including
`AcknowledgeAlarmCommand`/`ShelveAlarmCommand` from the AdminUI alarm page — for up to a minute.
`HandleStartDeploymentAsync` on a 50k-tag DB (two full snapshots + validation + gzip) has the same
effect for seconds.
- **Suggested fix:** Run the probe on a child/`Task.Run` + `PipeTo` (capture `Sender`), or move
`TestDriverConnect` out of the singleton entirely (it needs no DB and no coordination). Same for the
deploy compose: do the DB work off-actor and only serialize the "is one in flight" check.
### 3.2 `StartDeployment` reads the entire config DB twice with no transaction — validation and the composed artifact can diverge
- **Severity:** Medium **Confidence:** High
- **Location:** `AdminOperationsActor.cs:218` (`DraftSnapshotFactory.FromConfigDbAsync` — 15 tables),
`:231-239` (clusters + nodes again), `:313` (scripts again), `:334` (`ConfigComposer.SnapshotAndFlattenAsync` — 17 tables);
`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs:39-64`;
`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Validation/DraftSnapshotFactory.cs:7-37`.
- **Evidence:** Both loaders are sequential `AsNoTracking().ToListAsync()` per table on the same
context with no `BeginTransaction(IsolationLevel.Snapshot)`; the artifact is composed from a second
read taken after validation returned.
- **Failure scenario:** (a) Cost: ~32 sequential round trips and double materialisation of every row
per deploy — on the admin singleton's thread (see 3.1). (b) Correctness: a `/uns` or `/raw` save that
lands between the validator read and the composer read ships an artifact the `DraftValidator` never
saw (e.g. an effective-name collision or a cross-cluster ACL that the gate exists to stop).
- **Suggested fix:** Load once (`DraftSnapshot` already holds nearly every table the composer needs —
add `ServerClusters`/`ClusterNodes`/`ApiKeys` to it and compose from the snapshot), inside a
`SNAPSHOT`/`SERIALIZABLE` read transaction. Project `Script.SourceCode` once instead of three times.
### 3.3 Pre-authentication SQL round trip per *distinct* unknown key id, serialized under the SDK `SessionManager` lock
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/BoundedApiKeyStore.cs:46-50, 104-122, 180-184`;
`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/ConfigDbApiKeyStore.cs:101-119`;
`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs:184-190` (sync bridge under the SDK lock).
- **Evidence:** The negative cache is keyed by `apiKeyId`; its own remark says the ceiling makes *"a flood
of unknown ids … not become a SQL flood"*, but `Store` only clears wholesale at 1024 entries and a
flood of **distinct** ids misses every time. Each miss = one `SingleOrDefaultAsync` bounded by
`StoreTimeoutMs` (2 s) while the SDK holds its SessionManager-wide lock.
- **Failure scenario:** Any client that can reach 4840 on an admin/fused node — no credential needed,
only the `apikey` username and a well-formed `otua_<random>_<x>` password — can hold the global
activation lock for one indexed query per attempt, ~1-2 ms normally and up to 2 s during SQL
slowness, starving every legitimate activation (anonymous and X.509 included). The ACL path on a
driver-only node (`ArtifactApiKeyStore`) is immune.
- **Suggested fix:** Rate-limit/fail-fast per remote endpoint before the store call, or move the ConfigDb
key set into memory (the artifact snapshot already carries it on every node — the fused node could
read `IDataPlaneAuthoritySource` first and only fall through to SQL on a miss that the snapshot
cannot answer), or prefix-check `keyId` against an in-memory bloom/set refreshed on `StoreCacheSeconds`.
### 3.4 Central's inbound-ack gate blocks a gRPC thread on a synchronous secret-store read after every contact refresh
- **Severity:** Low **Confidence:** High
- **Location:** `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshPskProvider.cs:316-353`
(`GetAwaiter().GetResult()` under a `Lazy` ExecutionAndPublication lock, 3 s bound);
`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs:782-783`
(`InvalidateAll()` on **every** refresh, default every 60 s).
- **Evidence:** The provider's remarks call the block acceptable because it *"happens at most once per
cluster per process"*; the actor's remarks say the cache is dropped *"on every contact refresh"*. Both
are true; together they mean once per cluster per minute, and the first ack/dial per cluster after
each refresh pays it on a gRPC request thread (or, for dials, a thread-pool thread).
- **Failure scenario:** With a slow secrets store (SQLite on a busy disk, or a future networked KEK)
every deploy's first ack per cluster waits up to 3 s and, past the bound, is denied `PermissionDenied`
— a transient, confusing authentication failure on an otherwise correct key.
- **Suggested fix:** Refresh the per-cluster keys asynchronously from the actor's `ContactsLoaded`
handler and swap the cache atomically, instead of invalidating and re-reading on the request path.
### 3.5 `ConfigPublishCoordinator.PreStart` materialises the full newest `Deployment` row including `ArtifactBlob`
- **Severity:** Low **Confidence:** High
- **Location:** `ConfigPublishCoordinator.cs:164-167` — `.OrderByDescending(d => d.CreatedAtUtc).FirstOrDefault()` with no `Select`.
- **Failure scenario:** One blob read on every singleton start/hand-over; harmless today, grows with 1.2.
- **Suggested fix:** `.Select(d => new { d.DeploymentId, d.CreatedAtUtc })`.
---
## 4. Other hazards
### 4.1 Admin operations echo raw exception text to the UI
- **Severity:** Low **Confidence:** High
- **Location:** `AdminOperationsActor.cs:375-384, 423-431, 459-462, 489-492` (`Message: ex.Message`).
- **Failure scenario:** A `SqlException`/`UriFormatException` message (connection string fragments, host
names) reaches any `ConfigEditor`-role browser session. Low because the audience is already
authenticated admin staff.
- **Suggested fix:** Return a fixed message + correlation id; keep the detail in the log.
### 4.2 Fail-open window on the mesh known-cluster gate at central boot (documented, bounded)
- **Severity:** Low **Confidence:** High
- **Location:** `CentralCommunicationActor.cs:216-230, 286-293, 755-793`; `IMeshPskProvider.cs:63-69, 199-201`.
- **Evidence:** Until the first successful `ClusterNode` load, `_knownClusters` is null and `TryGetPsk`
consults the secret store for **any** `x-otopcua-cluster` value. A 15 × 2 s bootstrap retry and the
60 s periodic refresh heal it once SQL answers.
- **Failure scenario:** During a central boot with ConfigDb down, an unauthenticated prober can drive
one bounded secret-store read + warning log per request. Keys are still required; this is only the
cost/oracle the gate exists to remove. Acceptable, but worth knowing it is "fail-open until SQL".
### 4.3 `ActiveNodeHealthCheck`/Primary election and the OPC UA endpoint are decoupled (see 2.1)
Covered under 2.1 — listed here because the consequence is an availability hazard rather than a stub.
---
## 5. Checked and fine
- **DbContext lifetimes.** Every consumer in the slice takes `IDbContextFactory<OtOpcUaConfigDbContext>`
and creates a short-lived (pooled) context per call; read paths use `AsNoTracking()` (47 sites);
the only scoped `OtOpcUaConfigDbContext` registration (`Configuration/ServiceCollectionExtensions.cs:41`)
exists for DataProtection's key repository and returns the context to the pool at scope end.
- **`BoundedApiKeyStore` deadline race.** Abandoned lookups are observed (`ObserveAbandoned`), never cached,
CTSs disposed on both arms; faults/timeouts never cached. The negative-cache amplification (3.3) is the
only gap.
- **`ApiKeyOpcUaUserAuthenticator`.** Uniform deny message, anonymous-principal key refused, catch-all deny;
one HMAC per activation — no per-request crypto beyond that.
- **gRPC auth interceptors (5).** All fail closed on an unset key, use `FixedTimeEquals`, scope by method
prefix, single public ctor; `CentralCommandAuthInterceptor` returns one status text for all denials.
- **`MeshPskProvider.TryGetPsk` Lazy-exception handling.** A throwing logger cannot poison the cache for the
process lifetime (entry evicted, fail-closed).
- **`CentralCommunicationActor` lifecycle.** `_lifecycle` CTS cancelled + disposed in `PostStop`, token reads
guarded by `ObjectDisposedException`; per-dial results piped as messages; nothing captured off-thread.
- **`AclRevalidationTimer`** — created last in `StartAsync`, disposed first in `StopAsync` and in
`DisposeAsync`, dispose-before-overwrite on restart, tick never throws.
- **`ActorSystemTerminationWatchdog`, `DriverResilienceStatusPublisherService`, `SecretReplicationStarter`,
`ClusterNodeAddressReconcilerActor`, `RedundancyStateActor`** — timers/subscriptions are cleaned up in
`PostStop`/`StopAsync`; periodic loops catch and continue; nothing resolves the `ActorSystem` at
construction.
- **`TelemetryStreamGrpcService`** — static `_activeStreams` cap is paired on every path; stream lifetime
bounded at 4 h; subscription disposed by `using`.
- **`TelemetryStreamClient` / `MeshGrpcCommandDialer`** — owned `GrpcChannel`s disposed; keep-alive pings set.
- **`Program.cs`** — `IHistorianBackendProvider` registered via factory (so the container disposes it);
Kestrel re-bind computed exactly once; HTTPS surface refuses takeover rather than silently dropping
TLS; startup failures exit non-zero; `HostAbortedException` passes through.
- **`DraftValidator`** — purely in-memory over the snapshot; `GroupBy/First` guards against duplicate ids
throwing mid-pass; ACL scope rules mirror the picker (`NodeAclScopeIdUnexpected` etc.).
- **`LdapGroupRoleMappingService`, `ClusterAuditQuery`** — context per call, `AsNoTracking`, parameterised `Contains`.
- **`AlarmSfLegacyMigrator`** — read-only legacy open, single transaction, rename only after commit,
`INSERT OR IGNORE` with payload-derived ids makes a re-run idempotent.
- **`OtOpcUaLdapAuthService`** — singleton over a stateless library client; `Enabled` wins over `DevStubMode`;
plaintext transport refused unless `AllowInsecure`.
- **`AuthEndpoints`** — open-redirect guard on `returnUrl`, role-mapper fault fails closed to zero roles
(documented and tested).
+198
View File
@@ -0,0 +1,198 @@
# AdminUI review — `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` (master @ d51d35bd)
Scope: Blazor Server AdminUI — `/raw` tree, `/uns`, equipment page, live-status panels, SignalR bridges + in-process stores, ScriptAnalysis (Roslyn) endpoints + Monaco glue, driver browsers / address pickers, CSV import/export. Production code only. Every finding below was verified by reading the cited code; read-only review, nothing edited.
Overall: this slice is in good shape. The hard Blazor-Server lifecycle problems (timer drain, event unsubscribe, superseded-modal-open races, N+1 queries, Razor string slicing) have clearly been worked through already. What remains is mostly **scale limiters** (whole-tree renders, per-keystroke Roslyn compiles, unbounded fan-out) and a few **stubs still wired into the live UI**. One finding is a real correctness hazard (#P1).
---
## Memory / Resource leaks
### L1. Driver-status / resilience snapshot stores never evict — ghost drivers on `/hosts` until process restart
- **Severity:** Medium · **Confidence:** High
- **Location:** `Hubs/InMemoryDriverStatusSnapshotStore.cs:6-17`, `Hubs/InMemoryDriverResilienceStatusStore.cs:6-18`, consumer `Hosts/HostsDriverView.cs` (`Build` iterates `snapList`, i.e. `DriverStore.GetAll()`), `Components/Pages/Hosts.razor:452-453`.
- **Evidence:**
```csharp
private readonly ConcurrentDictionary<string, DriverHealthChanged> _byInstance = new();
public void Upsert(DriverHealthChanged snapshot) { _byInstance[snapshot.DriverInstanceId] = snapshot; ... }
public IReadOnlyCollection<DriverHealthChanged> GetAll() => _byInstance.Values.ToArray();
```
There is no `Remove`/TTL; `HostsDriverView.Build` derives the driver rows from the snapshot list, not from `DriverInstances`.
- **Failure scenario:** delete (or delete+recreate under a new id) a driver instance; its last `DriverHealthChanged` stays in the singleton forever. `/hosts` keeps rendering it as a frozen "stale" row (name/type fall back to id since the config join misses), and `HostsDriverView.Build` cost grows with every id ever seen since the admin node started. Long-lived admin nodes on a churny fleet accumulate.
- **Fix:** evict entries whose `DriverInstanceId` is absent from the config load Hosts already performs every 5 s (`_instances`), or add a `Remove` driven by the driver-removal / deploy message, or expire by `PublishedUtc` (e.g. > 10 min).
### L2. `DriverTestConnectButton` — `System.Timers.Timer` with an `async void` Elapsed handler, never drained
- **Severity:** Low · **Confidence:** Medium
- **Location:** `Components/Shared/Drivers/DriverTestConnectButton.razor:66-74, 82`
- **Evidence:**
```csharp
_clearTimer = new System.Timers.Timer(30_000) { AutoReset = false };
_clearTimer.Elapsed += async (_, _) => { _result = null; await InvokeAsync(StateHasChanged); };
...
public void Dispose() => _clearTimer?.Dispose();
```
- **Failure scenario:** `Timers.Timer.Dispose` does not wait for an in-flight callback, and an `async void` lambda's post-await exception (component/circuit already gone) cannot be observed by the timer's exception suppression — it surfaces as an unhandled thread-pool exception. The sibling components (`DriverStatusPanel`, `Alerts`) explicitly moved to `System.Threading.Timer` + `DisposeAsync` for this reason; this one was left behind.
- **Fix:** mirror `DriverStatusPanel.ShowOpResult` (`System.Threading.Timer`, `IAsyncDisposable`, token check for stale fires).
### L3. `AlarmsHistorian` page — sync timer dispose, no overlap guard, blanket `catch {}`
- **Severity:** Low · **Confidence:** High
- **Location:** `Components/Pages/AlarmsHistorian.razor:58-74, 87`
- **Evidence:** `_timer = new Timer(_ => _ = InvokeAsync(RefreshAsync), ...)` … `public void Dispose() => _timer?.Dispose();` and `RefreshAsync` wraps the `Ask` in a bare `catch { }`.
- **Failure scenario:** after dispose a queued tick still issues an actor `Ask` (2 s timeout) against a gone component; every failure (including "this role has no historian actor") is swallowed and the page sits on "Loading…" forever — the comment admits the role-gated message "lands when we add role gating to the UI".
- **Fix:** copy the `Fleet.razor` pattern (`IAsyncDisposable`, `_disposed`, overlap guard, cancel CTS) and surface the "no historian on this role" state.
### L4. Browse sessions can be orphaned when a picker is disposed mid-`OpenAsync` (bounded by the reaper)
- **Severity:** Low · **Confidence:** High
- **Location:** all `Components/Shared/Drivers/Pickers/*AddressPickerBody.razor` `OpenBrowseAsync` (e.g. `GalaxyAddressPickerBody.razor:146-157`) and `RawBrowseModal.razor` `OpenBrowseAsync`; disposal at e.g. `GalaxyAddressPickerBody.razor:217-225`.
- **Evidence:** `var result = await BrowserService.OpenAsync(...); if (result.Ok) _token = result.Token;` — if the hosting modal is closed/navigated away while the open is in flight, `DisposeAsync` has already run with `_token == Guid.Empty`, then the token lands on a dead component and nobody calls `CloseAsync`.
- **Failure scenario:** an OPC UA client session / MQTT connection / SQL connection stays open against the plant device for up to `IdleTtl` (2 min) + one reaper tick (30 s). Harmless at low volume; a flapping operator can hold several at once.
- **Fix:** in `OpenBrowseAsync`, after the await check a `_disposed` flag and immediately `CloseAsync(result.Token)`.
### L5. In-process stores invoke subscriber delegates synchronously on the bridge actor thread; one throwing subscriber aborts delivery and (for two bridges) restarts the actor
- **Severity:** Low · **Confidence:** High
- **Location:** `Hubs/InMemoryDriverStatusSnapshotStore.cs:13`, `Hubs/IInProcessBroadcaster.cs` (`Publish => Received?.Invoke(item)`), `Hubs/AlertSignalRBridge.cs` `ForwardAsync` (the `_broadcaster.Publish(msg)` sits **outside** the `try`), `Hubs/DriverResilienceStatusBridge.cs` (`Receive<…>(msg => _store.Upsert(msg))`, no catch).
- **Failure scenario:** every open circuit's handler runs inline on the actor thread (`InvokeAsync` is fire-and-forget so it is cheap, but any synchronous throw — e.g. a handler on a circuit whose renderer is mid-teardown — stops the multicast at that subscriber; remaining panels miss the update, and the alert/resilience bridge actor restarts and re-subscribes to DPS). It also means the DPS bridge's throughput is coupled to the number of open admin browser tabs.
- **Fix:** iterate `GetInvocationList()` with per-subscriber try/catch (or post to a channel), and move `Publish` inside the `try` in `AlertSignalRBridge`.
---
## Unimplemented / stubbed
### S1. `ctx.SetVirtualTag` is advertised by the editor but documented (by the editor itself) as inert at runtime
- **Severity:** Medium · **Confidence:** High
- **Location:** `ScriptAnalysis/ScriptAnalysisService.cs:349-353` (hover text), `TryGetTagPathLiteral` offers path completion for `SetVirtualTag` (`:295-300`).
- **Evidence:** `"⚠ Cross-tag SetVirtualTag writes are currently dropped in single-tag mode (the cascade engine is not wired into the host); this write will not take effect at runtime."`
- **Failure scenario:** an author gets completions + green diagnostics for a call that silently does nothing when deployed. Only a hover reveals it.
- **Fix:** either wire the cascade path or emit an `OTSCRIPT_*` warning marker from `Diagnose` for `ctx.SetVirtualTag(...)` so the Problems list shows it, not just hover.
### S2. Inlay-hints endpoint is a no-op, but the JS still POSTs the whole script to it
- **Severity:** Low · **Confidence:** High
- **Location:** `ScriptAnalysis/ScriptAnalysisService.cs:468-472` (`InlayHints(...) => new(Array.Empty<InlayHint>())`), `ScriptAnalysisEndpoints.cs:24`, `wwwroot/js/monaco-init.js:104-121` (`registerInlayHintsProvider` → `fetch("/api/script-analysis/inlay-hints", { body: model.getValue() })`).
- **Failure scenario:** Monaco re-requests inlay hints on content/viewport changes, so every edit costs an extra authenticated POST carrying the full source, for an always-empty answer.
- **Fix:** drop the provider registration (and endpoint) until the feature exists.
### S3. `MonacoEditor.MarkersChanged` / `RevealLineAsync` ("problems panel") have no consumer; marker array is still round-tripped JS→.NET after every diagnostics pass
- **Severity:** Low · **Confidence:** High
- **Location:** `Components/Shared/MonacoEditor.razor:39-44, 129-131, 134-138` ("The marker DTO is not modelled yet — typed as object[] until a later task…"); `monaco-init.js:209` (`dotNetRef.invokeMethodAsync("OnMarkersChanged", markers)`); `grep MarkersChanged` finds no consumer in `ScriptEdit.razor` / `VirtualTagModal.razor`.
- **Fix:** either build the problems panel or remove the callback + JS invoke.
### S4. Dead service surface: `IUnsTreeService.LoadTagsForEquipmentAsync` / `EquipmentTagRow` / `tagCounts`
- **Severity:** Low · **Confidence:** High
- **Location:** `Uns/UnsTreeService.cs` (`LoadTagsForEquipmentAsync` returns `Array.Empty<EquipmentTagRow>()`; `LoadStructureAsync` builds an always-empty `tagCounts`), `Uns/EquipmentChildRows.cs:7`. No production caller.
- **Fix:** delete the member, record and the zero-count plumbing (v3 retired the binding).
### S5. Deploy API `202 Accepted` Location points at a route that does not exist
- **Severity:** Low · **Confidence:** High
- **Location:** `Api/DeployApiEndpoints.cs:86-88` — `Results.Accepted($"/api/deployments/{r.DeploymentId?.Value}", …)`; the only route mapped is `MapPost("/api/deployments")`; no `MapGet` anywhere in `src/Server`.
- **Failure scenario:** a REST client that follows the `Location` for status polling gets a 404.
- **Fix:** add a GET that reads `Deployments`/`NodeDeploymentStates` for the id, or return `Accepted(value)` without a location.
### S6. Equipment CSV import is a naive comma split (no RFC-4180), unlike the raw-tag CSV import
- **Severity:** Low · **Confidence:** High
- **Location:** `Components/Shared/Uns/ImportEquipmentModal.razor` `ParseCsv` — `// NOTE: simple comma split — no RFC4180 quoting; values with commas are not supported.`
- **Failure scenario:** a quoted `Manufacturer` containing a comma silently shifts every later column (Model/SerialNumber…) with no error. The raw-tag importer (`CsvParser` + `CsvColumnMap`) already does this correctly.
- **Fix:** reuse the raw CSV parser.
### S7. Dev-only demo page ships in the production assembly
- **Severity:** Low · **Confidence:** High
- **Location:** `Components/Pages/Dev/ContextMenuDemo.razor:1-3` — `@page "/dev/context-menu-demo"`, `[Authorize(Policy = AuthenticatedRead)]`.
- **Fix:** `#if DEBUG` the route or move it to a test host.
---
## Performance limiters
### P1. Monaco ships the full editor text JS→.NET on every keystroke; Blazor Server's 32 KB `MaximumReceiveMessageSize` is never raised
- **Severity:** High · **Confidence:** Medium (limit + code path verified; failure mode on exceed is per ASP.NET Core SignalR behaviour)
- **Location:** `wwwroot/js/monaco-init.js:213-214` (`editor.onDidChangeModelContent(() => dotNetRef.invokeMethodAsync("OnValueChanged", editor.getValue()).catch(function () {}))`), `Components/Shared/MonacoEditor.razor:119-127`, `EndpointRouteBuilderExtensions.cs:56` (`AddRazorComponents().AddInteractiveServerComponents()` — no `AddHubOptions`; `grep MaximumReceiveMessageSize src` → nothing).
- **Failure scenario:** (a) cost: each keystroke transfers O(script size) over the circuit — a 20 KB script typed at 5 keys/s is 100 KB/s per author, before the separate `fetch` calls below. (b) correctness: once the script exceeds 32 KB (paste a large script), the JS→.NET invoke exceeds the hub's receive limit; the failure is swallowed by `.catch(function () {})`, so `_lastSentValue`/`Value` silently stop tracking the editor and **Save persists stale source** (ScriptEdit `SubmitAsync` reads `_form.SourceCode`, which only `ValueChanged` updates). SignalR may also tear the circuit down on the oversize frame.
- **Fix:** read the text back on demand (`MonacoBlazor.getValue` already exists) at Save / blur instead of per keystroke, or send `onDidChangeModelContent` *changes* and apply them server-side; and raise `HubOptions.MaximumReceiveMessageSize` (or document the cap).
### P2. Roslyn: a full `CSharpCompilation` + semantic model per request, per keystroke, with no cancellation anywhere
- **Severity:** Medium · **Confidence:** High
- **Location:** `ScriptAnalysis/ScriptAnalysisService.cs:60-67` (`Analyze` → `CSharpCompilation.Create(...)` every call), callers `CompleteAsync:220`, `Hover:323`, `SignatureHelp:422`, `Diagnose:97`; `ScriptAnalysisEndpoints.cs:19-24` (handlers do not take `CancellationToken`/`RequestAborted`); `monaco-init.js:32-72` (`quickSuggestions.strings = true`, providers ignore Monaco's cancellation token, no `AbortController`).
- **Evidence:** `var compilation = CSharpCompilation.Create("ScriptAnalysis", new[] { tree }, Sandbox.References, CompileOptions); return (tree, compilation, compilation.GetSemanticModel(tree), …)`.
- **Failure scenario:** typing inside `ctx.GetTag("…")` fires a completion request per character plus hover plus signature help plus the 500 ms diagnostics; each is a fresh compile against the full sandbox reference set on the admin node's thread pool, and an abandoned request runs to completion. Two or three concurrent authors on a big script make the admin node (which also hosts the control-plane actors) visibly busy.
- **Fix:** keep one `CSharpCompilation` per scoped service and use `compilation.ReplaceSyntaxTree` (cheap incremental re-bind); plumb `HttpContext.RequestAborted` through; honour Monaco's `CancellationToken` + `AbortController` in the providers; consider server-side debounce on completions.
### P3. `ScriptTagCatalog.GetPathsAsync` rebuilds a HashSet of every fleet tag path and sorts on every completion keystroke
- **Severity:** Medium · **Confidence:** High
- **Location:** `ScriptAnalysis/IScriptTagCatalog.cs` `GetPathsAsync` (`new HashSet<string>(entries.Select(e => e.Path), OrdinalIgnoreCase)` → `Where(StartsWith)` → `OrderBy` → `Take(200)`), `GetTagInfoAsync` (linear `FirstOrDefault` per hover).
- **Failure scenario:** the 5 s snapshot cache fixed the DB side, but the in-memory side is still O(n log n) allocation per keystroke; on a 50k-tag fleet that is tens of MB of garbage per typed character.
- **Fix:** materialise a sorted, de-duplicated `string[]` (and a path→info dictionary) inside the `Snapshot`; prefix-search with `Array.BinarySearch`.
### P4. `/hosts` rebuilds every cluster group and re-renders the whole page on every driver health publish
- **Severity:** Medium · **Confidence:** High
- **Location:** `Components/Pages/Hosts.razor:478-479` (`OnSnapshotChanged(DriverHealthChanged _) => InvokeAsync(() => { RebuildDriverGroups(); StateHasChanged(); })`), `Hosts/HostsDriverView.Build` (full re-projection: `ToList` + per-cluster `Where` scans), publish cadence documented at `Hosts.razor:301-302` ("5 s publish interval").
- **Failure scenario:** N enabled drivers ⇒ N full rebuilds + N full-page diffs every 5 s per open `/hosts` tab (200 drivers ≈ 40/s), on top of the page's own 5 s tick. Each `HostsDriverRow` ctor also re-sorts `Diagnostics` and `HostStatuses`.
- **Fix:** set a dirty flag in `OnSnapshotChanged` and let the existing 5 s `TickAsync` do the rebuild (or throttle to ≤1 rebuild/s), and rebuild only the affected cluster.
### P5. `RawTree` re-creates every node's context-menu list per render and re-renders a `ContextMenu` child per row; no virtualization
- **Severity:** Medium · **Confidence:** High
- **Location:** `Components/Shared/Raw/RawTree.razor:107-137` (`RenderNode` → `var items = MenuFor(node)` → `new List<ContextMenuItem> { new() { … OnClick = () => … } }` per node per render), `:129` (`<ContextMenu Items="items" …>` — `Items` is a reference-type parameter, so Blazor treats it as changed on every render), `grep Virtualize` → unused anywhere in AdminUI; `Uns/RawTreeService.cs` `LoadDeviceChildrenAsync` / `LoadTagGroupChildrenAsync` return every tag of the device/group.
- **Failure scenario:** expand a device with a few thousand root tags: every spinner toggle, checkbox click (picker mode `ToggleTagSelection` calls `StateHasChanged`), or modal open/close re-renders thousands of rows, allocates thousands of menu lists + closures, and re-renders thousands of `ContextMenu` children. Same shape in `DriverBrowseTree` (`RenderNode` per browse node) and `UnsTree`.
- **Fix:** cache the per-node menu on `RawNode` (invalidate on `Enabled` change), add `ShouldRender`/`@key` discipline, or wrap each expanded level in `<Virtualize>`; page tags from the service (e.g. 500 per level with a "show more").
### P6. Whole CSVs are rendered into the circuit: import review grid has no paging, export embeds the file twice in the DOM
- **Severity:** Medium · **Confidence:** High
- **Location:** `Components/Shared/Raw/RawCsvImportModal.razor:85` (`@foreach (var row in _result.Rows)` — every parsed row of a file up to the 16 MB cap at `:261`), `Components/Shared/Raw/RawCsvExportModal.razor:46` (`<textarea …>@_csv</textarea>`) and `:54` (`href="@_dataUri"` = base64 of the same text) with both strings retained in `_csv`/`_dataUri` while the modal is open.
- **Failure scenario:** exporting a 50k-tag device produces a multi-MB CSV; the render batch pushed over SignalR is ≈2.3× that size (textarea + base64) and the DOM holds it; importing a large file renders 100k+ `<tr>`s in one go and recomputes `OkCount/ErrorCount/RemainingCount` (three full scans) on every render.
- **Fix:** `<Virtualize>` the review grid; stream the export via `IJSRuntime` + `DotNetStreamReference` or a download endpoint; show a bounded preview (first N lines) in the textarea.
### P7. Alerts / ScriptLog: one `StateHasChanged` per received event; list re-filtered per render
- **Severity:** Low · **Confidence:** High
- **Location:** `Components/Pages/ScriptLog.razor:128-134` (`OnEntry` → `InvokeAsync(() => { _rows.Insert(0, entry); …; StateHasChanged(); })`), `:95-110` (`VisibleRows` filters + `ToList()` on every render), `Components/Pages/Alerts.razor:279-286`.
- **Failure scenario:** a chatty script (`log` per evaluation per tag) drives one full-page render per line on every open `/script-log` tab; `Insert(0)` on a `List<T>` of 500 is a memmove per line.
- **Fix:** batch via a 100–250 ms coalescing timer; use a ring buffer / `LinkedList`.
### P8. `/scripts` list loads every script *with its full `SourceCode`*
- **Severity:** Low · **Confidence:** High
- **Location:** `Components/Pages/Scripts.razor:117-123` (`db.Scripts.AsNoTracking().OrderBy(s => s.Name).ToListAsync()` — full entity), source only needed for the on-demand "view source" dialog (`FindSourceScript`).
- **Fix:** project `{ScriptId, Name, Language, SourceHash, RowVersion}` for the list; fetch `SourceCode` when the dialog opens.
### P9. Diagnostics responses are not sequenced — a slow older response can overwrite newer markers
- **Severity:** Low · **Confidence:** High
- **Location:** `wwwroot/js/monaco-init.js:204-212` (`scheduleDiagnostics` debounces the *start* but `fetchDiagnostics` results are applied whenever they arrive).
- **Failure scenario:** type, pause 500 ms (request A, slow — the admin node is busy per P2), type again, pause (request B, fast): B's markers land, then A's stale markers overwrite them; the editor shows errors for code that no longer exists until the next edit.
- **Fix:** generation counter captured before the fetch, ignore results whose generation != current; add `AbortController`.
---
## Other hazards
### H1. Destructive actions with no confirmation step
- **Severity:** Medium · **Confidence:** High
- **Location:** `Components/Pages/Uns/EquipmentPage.razor:243, 298` (per-row Delete buttons bound directly to `DeleteVirtualTag` / `DeleteAlarm`, which delete immediately at `:447-456`, `:492-501`); `Components/Pages/ScriptEdit.razor:72, 145-160` (`DeleteAsync` deletes the script on click). Contrast: `/uns` and `/raw` route every delete through a confirm modal (`GlobalUns.razor` `_confirmNode`, `RawTree.razor` `ShowConfirm`).
- **Failure scenario:** a mis-click on a dense row deletes a scripted alarm / virtual tag (and, after the next deploy, its OPC UA node and any ScadaBridge subscription to it) with no undo.
- **Fix:** reuse `RawConfirmDialog`.
### H2. Modal parameter dictionaries and `EventCallback`s are re-created on every render
- **Severity:** Low · **Confidence:** High
- **Location:** `Components/Shared/Drivers/DriverConfigModal.razor` (`_formParameters => new() { … EventCallback.Factory.Create(...) }`), `DeviceModal.razor` (`_deviceFormParameters`), `RawTagModal.razor` `BuildEditorParameters()`.
- **Failure scenario:** `DynamicComponent` sees new parameter values every render, so the hosted driver/device/tag-config form runs `OnParametersSet` on every parent render. Correctness is protected only by each form's `_lastParsed` guard (verified present in all 12 driver forms + 9 device forms); any future form that forgets the guard will clobber in-progress edits on the first live-status push. This is the exact bug class the `MqttDriverForm` note describes.
- **Fix:** build the dictionary once per open (in `LoadAsync`) and cache the callbacks in fields.
### H3. `monaco.editor.setTheme` is global
- **Severity:** Low · **Confidence:** High
- **Location:** `monaco-init.js:218-221` (comment acknowledges it) + `MonacoEditor.razor:158-162`.
- **Failure scenario:** `VirtualTagModal` and the host page can each mount an editor; toggling theme on one flips the other, and the per-component `_dark` flags drift from reality. Cosmetic.
---
## Checked-and-fine (looked suspicious, verified OK)
- **Timer/event lifecycle on the live pages** — `Fleet`, `Hosts`, `DriverStatusPanel`, `Alerts`, `ScriptLog`, `PageErrorBoundary`, `Certificates` all unsubscribe singleton events, drain `System.Threading.Timer` via `DisposeAsync`, cancel a page CTS, and guard `StateHasChanged` after teardown (`_disposed`, `_tickRunning`). `ToastNotification` binds auto-dismiss delays to a disposal CTS.
- **Browse pipeline** — `BrowseSessionRegistry` + `BrowseSessionReaper` (2 min idle TTL, 30 s tick, drain on shutdown), 20 s per-call timeout, "Browse" buttons hidden while `_token != Guid.Empty` (no double-open leak), `RawBrowseModal.CloseAsync` always closes the session, `RefreshTree` re-keys the tree without opening a second session, rebirth is authorized server-side and re-probed before publish.
- **EF query shapes** — `RawTreeService` / `UnsTreeService` / `ScriptTagCatalog` / `AclScopeService` use `AsNoTracking` + projections and batch child counts with `GroupBy`/`Contains` (no N+1 found); rename script scan runs server-side in batches; `ImportTagsAsync` / `ImportEquipmentAsync` pre-load lookup sets and `SaveChanges` once; `Deployments` pages `Take(50)`, `ClusterAudit` pages 200.
- **Razor string slicing (#504 class)** — every `[..N]` is either guarded by a `Length > N` check or applied to a fixed-width `Guid.ToString("N")`; DB values go through `DisplayText.Abbreviate`.
- **Superseded-open races** — `RawBrowseModal`, `RawCsvImportModal`, `RawCsvExportModal`, `DeviceModal`, `DriverConfigModal`, `RawTagModal`, `VirtualTagModal` all carry an open-generation / loaded-key guard so a late await cannot install another target's state.
- **Caches** — `ScriptTagCatalogSnapshotCache` (5 s TTL, single-flight gate, keeps old snapshot on failure) and `DeploymentDriftProbe` (watermark-keyed, single-flight) are bounded single-entry caches, not growing maps. No static mutable state in components (only `JsonSerializerOptions`).
- **Roslyn references** — `Sandbox.References` / `Preamble` are process-static and shared; no `AdhocWorkspace`/`MetadataReference` is created per request (the per-request cost is the compilation itself, see P2).
- **JS interop** — the single `DotNetObjectReference` (`MonacoEditor`) is disposed; `MonacoBlazor.dispose` is called under `JSDisconnectedException` handling; prerender `InvalidOperationException` is handled.
- **TelemetryDialSupervisor** — the `Task.Run` dial loop only `Tell`s `Self`; `Dialer.Cancel()` cancels **and disposes** the CTS; reconnect timer per node is replaced, not stacked.
- **Auth on the push/analysis surfaces** — `AlertHub`/`ScriptLogHub`/`DriverStatusHub` are `[Authorize]`; `/api/script-analysis/*` requires `ConfigEditor`; `/api/deployments` is API-key gated with constant-time compare.
- **Driver-type coverage** — all 12 `DriverTypeNames` have a driver form (`DriverFormMaps`), tag-config editor + validator; Sql/MTConnect/Calculation deliberately have no device form (`SingleConnectionDriverTypes`), so nothing is "registered but unauthorable" in this slice.
- **`GlobalUns` "Delete for this node kind is not yet available"** — unreachable; `UnsTree.RenderActions` never renders Delete for Tag/VirtualTag.
- **`ClusterNamespaces.razor` placeholder** — intentional redirect stub so old links don't 404.
- **`InputFile`** — streamed with a 16 MB cap (not subject to the 32 KB hub limit in P1).
- **`TwinCATDriverForm` "PLC diagnostics… not yet live-gated"** — persisted and consumed by the driver; the note is about test coverage, not a dangling option (driver slice).
+127
View File
@@ -0,0 +1,127 @@
# Core slice review (Commons, Core, Core.Abstractions, Core.Scripting[.Abstractions], Core.ScriptedAlarms, Core.VirtualTags, Core.AlarmHistorian)
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
Reviewer 05. Read-only audit of production code. master @ d51d35bd.
Scope note: this slice is unusually mature — collectible-ALC script unloading, per-generation
resilience-pipeline keys, retire/reinstate tombstoning, and the scripted-alarm drain machinery have
all clearly been through multiple hardening rounds, and most of the "obvious" leaks are already
closed and commented. The findings below are the residue.
---
## Memory / Resource leaks
### 1. `ScriptedAlarmEngine._valueCache` is never pruned across reloads — unbounded growth per deploy
- Severity: Medium · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs:163` (field), `:260-266` (LoadAsync clears), `:547-549` (OnUpstreamChange write)
- Evidence: `LoadAsync` clears `_alarms`, `_alarmsReferencing`, `_scratchByAlarmId`, `_lastQualityBucketByAlarmId`, `_compileCache` — but **not** `_valueCache`. It then re-seeds only the *currently* referenced paths (`foreach (var path in _alarmsReferencing.Keys) _valueCache[path] = ...`). `OnUpstreamChange` also writes `_valueCache[path] = value;` unconditionally (even before the `_disposed` check).
- Failure scenario: each deploy/reload whose alarm set drops a referenced tag leaves that tag's entry in `_valueCache` forever. Over many deploys with churning tag sets, `_valueCache` accumulates stale keys that nothing ever reads or evicts — a slow leak on a long-lived process. It also carries dead entries into `TryLookup` (message-template resolution) and `WorstInputStatus` snapshots, though those only read the referenced subset so the effect is memory, not correctness.
- Suggested fix: clear `_valueCache` in `LoadAsync` alongside the other maps (it's re-seeded immediately after), or intersect it with `_alarmsReferencing.Keys` at the end of load.
### 2. A timed-out script leaks one thread-pool thread *per evaluation*, and alarm re-eval makes that unbounded
- Severity: Medium (High impact if a script loops) · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/TimedScriptEvaluator.cs:78-97`; consumed at `ScriptedAlarmEngine.cs:770`
- Evidence: `RunAsync` does `var runTask = Task.Run(() => _inner.RunAsync(context, ct)); await runTask.WaitAsync(Timeout, ct);`. The class XML doc states the orphaned ScriptRunner thread "is tied up until the runtime decides to return, which it may never do" and calls the orphan "rare".
- Failure scenario: a CPU-bound (or blocking — see Finding 10) predicate times out; the engine logs and returns `seed`, releasing `_evalGate`. The *next* upstream change re-evaluates the same alarm, spawning *another* `Task.Run` that also hangs. A single accidental `while(true){}` in one authored predicate therefore leaks one pool thread per change event, not one total — trending toward thread-pool starvation across the whole server, not just that alarm. The "rare orphan" framing understates the per-evaluation accumulation.
- Suggested fix: out of scope to fully fix in-process, but at minimum gate re-evaluation of an alarm whose predicate has already timed out (circuit-break the alarm after N consecutive timeouts) so one bad script cannot spawn unbounded orphan threads. Document the accumulation in the class remarks.
### 3. `ScriptedAlarmEngine.Dispose` drains in-flight tasks with no timeout — can hang shutdown
- Severity: Medium · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs:1168-1180`
- Evidence: `Task[] toAwait; ... Task.WhenAll(toAwait).GetAwaiter().GetResult();` — a synchronous, **un-timed** block. The drained tasks (`DrainDirtyAsync`, `ShelvingCheckAsync`) run with `CancellationToken.None`, and each contains `await _store.SaveAsync(..., ct)`.
- Failure scenario: if the injected `IAlarmStateStore.SaveAsync` (LocalDb/SQL-backed in production) blocks — connection stall, lock, replication hiccup — the background task never completes, so `Dispose` (and the host shutdown that calls it) hangs indefinitely. Contrast `PollGroupEngine.DisposeAsync` (`:449`) which bounds the identical pattern with `.WaitAsync(TimeSpan.FromSeconds(5))`.
- Suggested fix: bound the drain with `.WaitAsync(someTimeout)` like PollGroupEngine does, and/or pass a cancelable token to the background tasks that Dispose can trip.
### 4. Persisted alarm `Comments` list grows unbounded and is re-serialized whole on every transition
- Severity: Low · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/Part9StateMachine.cs:334-336`; persisted via `ScriptedAlarmEngine` `SaveAsync` of the full `AlarmConditionState`
- Evidence: `AppendComment` only ever `existing.Add(...)`; nothing trims. Every ack/confirm/shelve/unshelve/AddComment/AutoUnshelve appends, and each transition persists the entire `AlarmConditionState` (comments included) and replicates it to the pair peer.
- Failure scenario: a long-lived, frequently-operated alarm accumulates an ever-growing audit list; each `SaveAsync` re-serializes the whole list (O(n) per transition, O(n²) over the alarm's life) and replicates it. GxP retention means the data must be kept, but keeping it *inline in the condition row* makes every write cost grow with history.
- Suggested fix: this is a design trade-off, not a bug — note it. If it bites, move the append-only audit trail to its own table keyed by alarm id so the condition row stays fixed-size.
---
## Unimplemented / stubbed
### 5. No true stubs found in the slice
- Severity: — · Confidence: High
- Every `NotSupportedException` / `NotImplementedException` hit is a *deliberate, contractual* default or guard, not a gap:
- `IHistorianDataSource.ReadModifiedAsync` / reverse-events defaults throw `NotSupportedException` by design (backend opt-in), documented at `Core.Abstractions/Historian/IHistorianDataSource.cs:205,275`.
- `GenericDriverNodeManager.cs:68` throws `NotSupportedException` when a driver lacks `ITagDiscovery` — correct.
- `HistoryAggregateNotSupportedException` (`HistoryReadContracts.cs:74`) is a typed signal, not a stub.
- `AlarmPredicateContext.SetVirtualTag` (`AlarmPredicateContext.cs:55`) throws by design (predicates must be pure).
- No TODO/FIXME/HACK in slice production code. No dead options I could find unconsumed.
### 6. Empty catch blocks are all justified, with one worth a glance
- Severity: Low · Confidence: Medium
- Location: `ScriptedAlarmSource.cs:101` `try { OnAlarmEvent?.Invoke(...); } catch { }`
- Evidence: subscriber exceptions are swallowed silently (no log) in the `IAlarmSource` fan-out, unlike the engine's own `FireEvent` (`ScriptedAlarmEngine.cs:975`) which logs. The other empty catches in the slice (PollGroupEngine CTS cancel/drain, DriverHost best-effort shutdown, `s.Dispose()` on unsubscribe) are standard best-effort teardown and fine.
- Failure scenario: a throwing alarm subscriber (e.g. the OPC UA alarm-source adapter) is silently dropped with no diagnostic — a lost alarm event with no trace. Low because the inner engine path does log.
- Suggested fix: log at warning in the `ScriptedAlarmSource` catch, matching `FireEvent`.
---
## Performance limiters
### 7. `LocalDbStoreAndForwardSink.GetStatus()` runs a synchronous `COUNT(*)` on every call
- Severity: Low · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs:520-532`
- Evidence: `GetStatus` opens a fresh connection and runs `SELECT COUNT(*) ... WHERE dead_lettered = 1` synchronously (`ExecuteScalar`) each time. The queue depth itself is the cached `_queuedRowCount`, but dead-letter depth is always a live scan.
- Failure scenario: `GetStatus` feeds the Admin UI `/alarms/historian` page and `/healthz`. A health probe or dashboard polling it on a short interval runs a table scan each tick on the shared LocalDb file. Minor at normal queue sizes; grows with dead-letter backlog.
- Suggested fix: maintain a cached dead-letter counter the way the queued counter is cached, resyncing on the same periodic cadence, or index-scan only.
### 8. Sync-over-async blocking teardown in `PollGroupEngine.Unsubscribe`
- Severity: Low · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs:225` (`task.Wait(DrainTimeout)`)
- Evidence: the sync `Unsubscribe` blocks the caller up to 5 s draining the loop. The code itself documents this and offers `UnsubscribeAsync` as the preferred path ("prefer `UnsubscribeAsync` on any path that runs inside an actor mailbox turn").
- Failure scenario: a caller that uses the sync overload from an actor turn or request pipeline stalls that thread for up to 5 s. Already called out in the XML doc; flagging so consumers are audited to prefer the async overload.
- Suggested fix: none in this file — the async path exists; ensure callers use it (out-of-slice).
### 9. `ScriptLogTopicSink` / `ScriptLogCompanionSink` call `RenderMessage()` per event on the hot path
- Severity: Low · Confidence: Medium
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptLogTopicSink.cs:64`, `ScriptLogCompanionSink.cs:57`
- Evidence: every forwarded script log event is eagerly rendered to a string (`logEvent.RenderMessage()`) and, in the topic sink, published onto the Akka DPS bus. The topic sink's floor is `Information` by default.
- Failure scenario: a script that logs at Information in its evaluation body (which runs per upstream change) fans a rendered string onto the cluster bus on every evaluation. The min-level discard protects below the floor, but Information+ is forwarded verbatim.
- Suggested fix: fine as-is given the min-level gate; consider a higher default floor for the topic sink, or rate-limiting, if a chatty script is observed saturating the bus.
---
## Other hazards
### 10. Script sandbox allows `System.Threading` blocking/synchronization primitives and `Type.GetType`/`RuntimeHelpers`
- Severity: Medium · Confidence: Medium
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ForbiddenTypeAnalyzer.cs:60-80` (namespace deny-list), `:116-151` (type deny-list)
- Evidence: the namespace deny-list covers `System.Threading.Tasks` but **not** `System.Threading` itself (only `Thread`/`ThreadPool`/`Timer` are denied type-granularly). So `SemaphoreSlim`, `Monitor`/`lock`, `ManualResetEventSlim`, `SpinWait`, `Mutex`, and `EventWaitHandle` are all reachable. Separately, `System.Type.GetType(string)` (on the allowed `System.Type`) and `System.Runtime.CompilerServices.RuntimeHelpers` (namespace not denied; only `...CompilerServices.Unsafe` is type-denied) resolve.
- Failure scenario: (a) a blocking primitive (`new SemaphoreSlim(0).Wait()`, `SpinWait.SpinUntil(() => false)`, a named `Mutex`) blocks the script's pool thread past the timeout — same orphan-thread leak as Finding 2, and a named `Mutex`/`EventWaitHandle` touches a *system-global* kernel object. (b) `Type.GetType("System.IO.File")` / `RuntimeHelpers.GetUninitializedObject(...)` obtain a forbidden type/object without a `typeof`/cast the pass-2 `TypeSyntax` walk would catch. I could **not** demonstrate a full escape — pass-2 catches the obvious follow-ups (declaring the reflection result via `var` resolves to a `System.Reflection` type, and `MethodInfo.Invoke` / casting to a forbidden type are denied) — so this is a hardening gap, not a proven break.
- Suggested fix: add `System.Threading` to `ForbiddenNamespacePrefixes` with a type-granular allow-list for the genuinely-needed sync-free primitives (`CancellationToken` is the only one scripts legitimately touch, and scripts are synchronous so even that is marginal); add `System.Type` member `GetType`/`GetMethod`/… handling and `System.Runtime.CompilerServices.RuntimeHelpers` to the type deny-list.
### 11. Scripts can allocate unbounded memory / the sandbox has no memory budget
- Severity: Low · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptSandbox.cs:30-35` (class remarks acknowledge it)
- Evidence: the sandbox comment states "cannot prevent a script from allocating unbounded memory or spinning in a tight loop — those are budget concerns, handled by the per-evaluation timeout". The timeout bounds wall-clock but not allocation; a `new byte[int.MaxValue]`-style allocation or a growing collection in a loop can OOM the whole in-process server before the 250 ms timeout fires.
- Failure scenario: an authored script (buggy or hostile) allocates aggressively and OOMs the server. Only mitigated by operators authoring sane scripts and the preview harness.
- Suggested fix: documented trade-off; a true fix needs an out-of-process runner (noted as a v3 concern). Flagging for awareness. → #750
### 12. `ScriptedAlarmEngine.OnUpstreamChange` writes `_valueCache` before the disposed check
- Severity: Low · Confidence: High
- Location: `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs:549-550`
- Evidence: `_valueCache[path] = value;` executes on line 549; `if (_disposed) return;` is line 550. So a late push after Dispose still mutates `_valueCache` (which Dispose never clears).
- Failure scenario: benign in practice — subscriptions are torn down before Dispose returns, and the dict is dropped with the engine. Combines with Finding 1 only as another un-cleared-cache note.
- Suggested fix: move the write below the `_disposed` guard; or accept it (harmless).
---
## Checked and fine (notable)
- **Collectible-ALC script lifecycle** (`ScriptEvaluator.cs`, `CompiledScriptCache.cs`): every compile loads into a per-evaluator collectible `AssemblyLoadContext`; `Clear`/`Dispose` unload them; failed loads/binds `alc.Unload()` before rethrow; `ScriptedAlarmEngine.LoadAsync` and `Dispose` both route through the cache's `Clear`/`Dispose` and the engine explicitly clears `_alarms` so the evaluator delegates aren't rooted. The "defeated the fix by compiling directly" regression is documented and closed. Genuinely well handled.
- **`ScriptSandbox` reference caches** (`PinnedReferences`, `BclReferences`): static, process-constant, bounded by the BCL assembly set — intended perma-cache, not a leak.
- **`DependencyGraph`** (`Core.VirtualTags`): cycle detection is iterative Tarjan (no stack blow-up), topo-rank is cached and invalidated on mutation (no per-event O(V+E)), shared `EmptySet` avoids per-call allocation. Not thread-safe (plain `Dictionary`), but it's a publish/compose-time single-threaded structure — correct by its usage contract.
- **Resilience pipeline cache** (`DriverResiliencePipelineBuilder`, `CapabilityInvoker`, `DriverResilienceStatusTracker`, `DriverCapabilityInvokerFactory`): lock-free `ConcurrentDictionary` hot path, generation-keyed to prevent stale-child re-cache, retire/reinstate tombstoning to stop removed-instance snapshot resurrection, clamp-before-build so a pipeline build never throws. Thorough.
- **`CompiledScriptCache`**: `ConcurrentDictionary<Lazy<>>` with `ExecutionAndPublication` (compile-once), value-scoped `TryRemove` so a concurrent re-add isn't evicted. No capacity bound, but documented as operator-/config-bounded; fine for now.
- **`LocalDbStoreAndForwardSink` capacity**: default 1M cap with drop-oldest eviction + evicted-count telemetry, periodic counter resync against replication drift, idempotent `ON CONFLICT DO NOTHING` inserts, poison-row dead-lettering with retention purge. Store-and-forward growth is capped. Good.
- **`ApiKeySecret`**: fails closed on empty pepper, constant-time compare, CSPRNG secret, hex key-id to dodge the base64 `/`→split hazard. Solid.
- **`DataPlaneAuthorityState`**: monotonic generation counter (not hash-derived — the comment explains why that would silently serve stale ACLs), trie built-and-installed before snapshot publish, Changed handlers invoked outside the lock and per-handler try/caught. Careful.
- **`PollGroupEngine`**: fixed-rate pacing, capped-exponential backoff on failure, fatal-exception classification that tears down vs. swallow-and-retry, deterministic teardown that awaits the loop before disposing the CTS, structural array comparison to avoid spurious change events. Mature.
@@ -0,0 +1,302 @@
# Review 06 — Drivers (session slice): Galaxy, OpcUaClient, Mqtt, MTConnect, Sql
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
Repo: `/Users/dohertj2/Desktop/OtOpcUa` @ `d51d35bd` (master). Production code only (`src/Drivers/...`). Read-only audit; every finding was verified by reading the cited lines (fork-sourced MQTT/MTConnect/SQL findings were spot-checked by the lead against the cited lines). Line numbers are as of this commit. Prefix key: **G** Galaxy, **O** OpcUaClient, **M** MQTT, **T** MTConnect, **S** Sql.
Host context that shapes several findings (verified outside the slice):
- **The runtime never disposes a driver instance.** `DriverInstanceActor.PostStop` calls only `_driver.ShutdownAsync()` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs:1550`); `DriverHostActor.StopChild` only stops the actor (`DriverHostActor.cs:3759-3775`); `grep -rn "is IAsyncDisposable\|as IDisposable" src/Server/ZB.MOM.WW.OtOpcUa.Runtime` is empty. The `/raw` browse picker (`src/Core/ZB.MOM.WW.OtOpcUa.Commons/Browsing/DiscoveryDriverBrowser.cs:128-150`) builds a throwaway driver per capture and also only calls `ShutdownAsync`. **`ShutdownAsync` is therefore the only teardown a driver ever gets.**
- The comms-loss watchdog's recovery is `ShutdownAsync` → `InitializeAsync` on the SAME driver object, retried every 10 s while `Reconnecting` (`DriverInstanceActor.cs:808-855`).
---
## 1. Memory / resource leaks
### G-1. `GalaxyDriver.ShutdownAsync` is a no-op — the whole gRPC runtime outlives the driver on every respawn, removal and `/raw` browse
- **Severity: Critical** · **Confidence: High**
- Location: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriver.cs:735-741` (ShutdownAsync); `:1517-1568` (DisposeAsync, where teardown actually lives).
- Evidence:
```csharp
public Task ShutdownAsync(CancellationToken cancellationToken)
{
if (_disposed) return Task.CompletedTask;
_logger.LogInformation("GalaxyDriver {InstanceId} shutting down", _driverInstanceId);
_health = new DriverHealth(DriverState.Unknown, _health.LastSuccessfulRead, null);
return Task.CompletedTask;
}
```
Everything `BuildProductionRuntimeAsync` (`:390-437`) creates — `_ownedMxClient` (gRPC channel), `_ownedMxSession` (a SECOND `MxGatewayClient` + MX session + `Register` handle, `Runtime/GalaxyMxSession.cs:85-87`), the `EventPump` (two background tasks draining `StreamEvents`), `GatewayGalaxyAlarmFeed` (background `StreamAlarms` loop that reconnects forever, `Runtime/GatewayGalaxyAlarmFeed.cs:102-148`), `ReconnectSupervisor`, `PerPlatformProbeWatcher`, `DeployWatcher` (`WatchDeployEvents` stream loop) and `_ownedRepositoryClient` — is released only in `DisposeAsync`, which nothing in the host calls.
- Failure scenario: (a) a config change is a respawn (new `GalaxyDriver` from the factory; the old child only gets `ShutdownAsync`) → the old instance keeps its MX session registered on mxaccessgw, keeps draining `StreamEvents`/`StreamAlarms`, keeps the deploy-watch stream open and its supervisor loop alive — permanently, one set per deploy. (b) Every AdminUI `/raw` browse of a Galaxy device (`DiscoveryDriverBrowser.CaptureAsync` → `InitializeAsync` → `DiscoverAsync` → `ShutdownAsync`) leaks a full runtime: two gRPC channels, an MX session, both streams, a 50k-slot bounded channel. The gateway accumulates live sessions and STA-pumped advises; OtOpcUa accumulates threads and channels.
- Suggested fix: make `ShutdownAsync` do the device-state half of `DisposeAsync` (stop pump/alarm feed/deploy watcher/probe watcher/supervisor, dispose session + clients, null the seams so a later `InitializeAsync` rebuilds) and keep `DisposeAsync` as the idempotent final teardown. The `#630` contract already puts device state under `InitializeAsync`/`ShutdownAsync` for the other eleven drivers.
### G-2. Every `InitializeAsync` on a live `GalaxyDriver` creates a new `DeployWatcher` without stopping the previous one
- **Severity: High** · **Confidence: High**
- Location: `GalaxyDriver.cs:631-654` (`StartDeployWatcherAsync`), called unconditionally from `InitializeAsync` `:377`.
- Evidence:
```csharp
var source = new GatewayGalaxyDeployWatchSource(_ownedRepositoryClient);
_deployWatcher = new DeployWatcher(source, _logger); // previous watcher dropped, still running
_deployWatcher.OnRediscoveryNeeded += (_, args) => OnRediscoveryNeeded?.Invoke(this, args);
var startTask = _deployWatcher.StartAsync(CancellationToken.None);
```
`InitializeAsync` (`:357-382`) skips `BuildProductionRuntimeAsync` on a second call (seams non-null) but always runs this; `DeployWatcher.RunLoopAsync` (`Browse/DeployWatcher.cs:133-235`) runs until cancelled.
- Failure scenario: the watchdog's recovery is `ShutdownAsync` (no-op, G-1) + `InitializeAsync`. While mxaccessgw is down the supervisor reports `Degraded` with a frozen `LastSuccessfulRead` (G-11), so the host trips every ~90 s and each trip spawns another watch-stream loop with its own 1–30 s backoff. When the gateway returns, N watchers each raise `OnRediscoveryNeeded` per real deploy and N streams stay open for the process lifetime.
- Suggested fix: stop/dispose the existing watcher at the top of `StartDeployWatcherAsync` (or early-return when one exists), and stop it in `ShutdownAsync` (G-1).
### O-1. OpcUaClient `SubscribeAsync` leaves a half-created subscription on the session when `CreateItemsAsync` (or `CreateAsync`) throws — one per host retry
- **Severity: Medium** · **Confidence: High**
- Location: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs:1781-1826`.
- Evidence:
```csharp
session.AddSubscription(subscription);
await subscription.CreateAsync(cancellationToken).ConfigureAwait(false);
...
item.Notification += notifHandler;
...
await subscription.CreateItemsAsync(cancellationToken).ConfigureAwait(false);
_subscriptions[id] = new RemoteSubscription(subscription, handle, itemHandlers);
```
No `catch`: on a throw the SDK `Subscription` stays in `session.Subscriptions` (created server-side if `CreateAsync` succeeded) with every `MonitoredItem.Notification` closure still attached, untracked by `_subscriptions`, so neither `UnsubscribeAsync` nor `ShutdownAsync` deletes it. The alarm path has exactly this cleanup (`:2204-2226`: "a half-built subscription left on the session would accumulate one per retry"); the data path does not.
- Failure scenario: a server that accepts `CreateSubscription` but rejects `CreateMonitoredItems` (e.g. `BadTooManyMonitoredItems`, transient timeout) makes `DriverInstanceActor`'s health-tick reconcile retry every tick; each retry adds a live, publishing subscription on the upstream server until the next re-initialise disposes the session — and servers with per-session caps then refuse the real subscribe.
- Suggested fix: wrap `:1789-1824` in `try/catch` mirroring the alarm path — detach handlers, bounded `session.RemoveSubscriptionAsync(subscription)`, rethrow.
### S-1. SQL key-value and where-pair plans have no row cap; an ambiguous selector materialises every matching row on every poll
- **Severity: Medium** · **Confidence: High**
- Location: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlGroupPlanner.cs:233-238` (key-value `SELECT … WHERE key IN (…)`, no limit), `:271-281` (where-pair `… WHERE col = @w`, no limit); `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlPollReader.cs:445-452`:
```csharp
var rows = new List<object?[]>();
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) { ... rows.Add(row); }
```
`SelectWideRow` (`:552-560`) explicitly accepts that "an ambiguous selector materialises every matching row for one poll".
- Failure scenario: a tag authored against a log/history table (`keyColumn=TagName` on a 50M-row samples table, or `whereColumn=Line` on an events table) pulls every matching row into memory **every poll** (default 5 s). Memory spikes to the match-set size, GC churn, and `commandTimeout` is the only brake; nothing refuses this at deploy time and the only signal is a once-per-minute contract warning.
- Suggested fix: cap the read loop (stop after N rows, publish a configuration-error status and warn when the cap is hit), or emit dialect `TOP/LIMIT N+1` so ambiguity is still observable without unbounded materialisation.
### M-1. A failed MQTT SUBSCRIBE (no connection drop) returns a live handle and drops the topic from the re-subscribe set — tags stay dark behind a Healthy connection
- **Severity: Medium** · **Confidence: High** (state-kept-wrong after failure; same family as a leak)
- Location: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttSubscriptionManager.cs:667-684` (transport throw → `DegradeTopic` → `return 0`), `:719-721` (`_subscribedTopics.TryRemove(topic, out _)`), `:417-418` (`SubscribeAsync` returns the handle regardless), `:489-499` (`OnReconnectedAsync` re-subscribes `_subscribedTopics` only).
- Failure scenario (plain mode): the SUBSCRIBE overruns its deadline or the broker transiently refuses a filter. The topic leaves `_subscribedTopics`, its tags go `BadCommunicationError`, but `SubscribeAsync` still returns a handle so `DriverInstanceActor` considers the subscription live and never re-subscribes; the next reconnect re-SUBSCRIBEs only `_subscribedTopics`, which no longer holds it. Connection state stays `Connected`/Healthy and `LastMessageUtc` advances from other topics, so the watchdog never trips. Dark until a redeploy.
- Suggested fix: keep failed filters in a pending set that `OnReconnectedAsync`/the next `SubscribeAsync` retry, or throw from `SubscribeAsync` on a transport failure so the actor's reconcile re-subscribes.
### T-1. MTConnect: a late (superseded/abandoned) pump publishes its stale cursor onto the NEW session — the doc says it is dropped, the code has no generation guard
- **Severity: Low** · **Confidence: Medium**
- Location: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs:2011-2020` (pump `finally { PublishAgentCursor(instanceId, cursor); }`, comment "dropped if a lifecycle change has already installed a newer session"), `:2491-2508` (`PublishAgentCursor` only early-returns on `session is null` or an identical `(InstanceId, NextSequence)`; it CAS-writes onto whatever session is installed — nothing checks `_sessionGeneration`), `:1789-1811`, `:2914-2922`.
- Failure scenario: a pump retired into `_supersededPumps` or abandoned after `TeardownWait` finishes after `StartCoreAsync` installed the new session; the next pump start reads the stale `InstanceId`, treats the first chunk as an Agent restart → `ObservationIndex.Clear()` + Bad flicker on every subscribed tag + a needless re-baseline (`:2061-2081`). Bounded (one flicker), no leak.
- Suggested fix: stamp the pump with `_sessionGeneration` and have `PublishAgentCursor` refuse when it differs (or compare `ReferenceEquals(session.Client, client)`).
### Low-severity leak nits (all High confidence)
- **G-3** `ReconnectSupervisor.ReportTransportFailure` (`Runtime/ReconnectSupervisor.cs:127-141`) allocates a new `_loopCts` per recovery cycle and disposes only the last one in `Dispose()`. Bounded by outage count.
- **O-2** `OpcUaClientDriver.InitializeAsync` (`:521-568`): the catch closes `Session` (the property, still null) rather than the local `session`, so a throw in `:544-556` (`AttachDiagnostics`/`NamespaceMap.FromSession`/`RecordSkippedTags`) orphans an activated session with its keep-alive handler until GC. Rare.
- **O-3** `SdkTelemetry.ForLogger` (`OpcUaClientDriver.cs:85`, `SdkTelemetry.cs:28-40`) creates an `ILoggerFactory` per driver construction, released only in `Dispose`/`DisposeAsync` (`:3365`, `:3395`) — which the host never calls; `_gate` and `_sessionWorkCts` likewise. One small object set per respawn, unbounded count.
- **M-2** `LastValueCache` (`MqttSubscriptionManager.cs:398`, `Sparkplug/SparkplugIngestor.cs:553`, `LastValueCache.cs:42-46`) gains an entry per non-authored reference ever subscribed and `Register` never prunes `Values`; bounded by the union of every RawPath authored across redeploys.
- **M-3** A failed initial connect leaves the new `MqttConnection`/`IMqttClient` undisposed (`MqttDriver.cs:967-986`, catch at `:1029` rethrows without disposing; contrast `EstablishAsync` at `:1011-1019`). Only one stale connection is held at a time (`_transport` is replaced each attempt), no socket.
- **S-2** A timed-out SQL group keeps its connection + `maxConcurrentGroups` slot until the provider returns (`SqlPollReader.cs:275-335`, `:345-357`); documented and bounded (default 4), but the bound is per driver, not per poll — pool sizing should know.
- **S-3** `SqlPollReader._gate` is never disposed and a new reader is built per `ApplyBinding` (`SqlPollReader.cs:53-59`, `SqlDriver.cs:264-273`); managed-only today.
---
## 2. Unimplemented / stubbed
### G-4. Galaxy per-platform `ScanState` host probing is structurally dead: the probe watcher's handles are never registered in `SubscriptionRegistry`, so `EventPump` drops every probe event
- **Severity: High** · **Confidence: High**
- Location: `Health/PerPlatformProbeWatcher.cs:125-150` (subscribes via `_subscriber.SubscribeBulkAsync`, stores handles only in its own `_itemHandlesByPlatform`), `Runtime/EventPump.cs:314-317`, `GalaxyDriver.cs:1465-1489`.
- Evidence:
```csharp
// EventPump.DispatchDataChange
var subscribers = _registry.ResolveSubscribers(ev.ItemHandle);
if (subscribers.Count == 0) return; // stale event after unsubscribe — drop quietly
```
`SubscriptionRegistry.Register` is called only from `ReadViaSubscribeOnceAsync` (`:959`) and `SubscribeAsync` (`:1110`) — never for the watcher's handles. `OnPumpDataChange` (`:1480-1488`), the only caller of `OnProbeValueChanged`, is reached only via the pump's fan-out, so a `*.ScanState` event never gets there. The only test (`tests/.../Health/PerPlatformProbeWatcherTests.cs`) drives `OnProbeValueChanged` directly.
- Also: after a reconnect (`ReopenAsync` → `GalaxyMxSession.RecreateAsync`) the watcher's handles are dead and nothing re-subscribes them — `SyncPlatformsAsync` is called only from `DiscoverAsync` (`:800`); `ReplayAsync` rebinds only `_subscriptions`.
- Failure scenario: `IHostConnectivityProbe.GetHostStatuses()` for Galaxy only ever carries the synthetic transport entry; per-$WinPlatform/$AppEngine Running/Stopped on `/hosts` never updates. Each discovery also leaves N `ScanState` items advised on the gateway for nothing (released only by `Dispose`, G-1).
- Suggested fix: register the probe handles in the registry under a reserved subscription id (and `Rebind` them in `ResubscribeAllAsync`), or route raw `MxEvent`s to the watcher by item handle before the registry lookup.
### M-4. `sparkplug.birthObservationWindowSeconds` is authored, range-checked and persisted by the AdminUI but consumed by nothing
- **Severity: Medium** · **Confidence: High**
- Location: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/MqttDriverOptions.cs:222-227` (`BirthObservationWindowSeconds = 15`, documented as the browse/discovery birth-collection window). `grep -rn BirthObservationWindowSeconds src` finds no reader outside Contracts and the AdminUI form (`MqttDriverForm.razor:222`, `MqttDriverFormModel.cs:168,274,376`); `MqttDriverBrowser.OpenBudget` uses `ConnectTimeoutSeconds` clamped 5–30 s (`MqttDriverBrowser.cs:188-189`).
- Failure scenario: an operator raising the window for slow births sees nothing change; discovery stability is governed by the host's `UntilStable` loop and the browse window by `ConnectTimeoutSeconds`.
- Suggested fix: feed it into `MqttDriverBrowser.OpenBudget`/the UntilStable cadence, or remove it from the contract and form.
### G-5. `Repository.DiscoverPageSize` has an AdminUI field and is parsed by the factory, but nothing consumes it
- **Severity: Low** · **Confidence: High**
- Location: `GalaxyDriverFactoryExtensions.cs:94`, `Contracts/GalaxyDriverOptions.cs:114`, `Browse/GatewayGalaxyHierarchySource.cs:24-25` (`_client.DiscoverHierarchyAsync(cancellationToken)` — no page size), AdminUI `GalaxyDriverForm.razor:94`.
- Suggested fix: thread it into `DiscoverHierarchyAsync` or drop the field.
### G-6 / O-4. `ProbeTimeoutSeconds` exists on both `GalaxyDriverOptions` and `OpcUaClientDriverOptions` and is consumed nowhere
- **Severity: Low** · **Confidence: High**
- Location: `GalaxyDriver.cs:732` (only use: equality compare); `OpcUaClientDriverOptions.ProbeTimeoutSeconds` has 0 references in driver, browser, session or any host project. Both probes take the caller's `timeout` argument instead.
### O-5. OpcUaClient AdminUI browse session's `AttributesAsync` is a stub; Certificate auth is refused "in v1"
- **Severity: Low** · **Confidence: High**
- Location: `OpcUaClient.Browser/OpcUaClientBrowseSession.cs:56-61` (`Array.Empty<AttributeInfo>()`), `OpcUaClientDriverBrowser.cs:61-63`.
- Impact: no DataType/access info in the live picker side-panel (the `/raw` browse-commit via `DiscoveryDriverBrowser` does enrich); a certificate-authenticated upstream cannot be browsed from the AdminUI.
### M-5. Sparkplug array metrics are refused ("not supported in v1") yet the browser still offers them as bindable
- **Severity: Low** · **Confidence: High** (documented decision)
- Location: `SparkplugIngestor.cs:1274-1285` (`if (binding.DataType.IsSparkplugArray()) { WarnOnce(...); return; }`; DataSet/Template/PropertySet at `:1287-1302`); `MqttBrowseSession.cs:387-392` lists the leaf with `IsArray: true`.
- Impact: an operator can commit a tag the ingestor never feeds — warned once, then `BadWaitingForInitialData` forever.
- Suggested fix: mark array leaves not-bindable in `DescribeMetric` as `mapped is null` already is.
- Tracking: the array-read capability itself → #754.
### T-2. MTConnect Agent-restart detection lives only on the `/sample` pump; `ReadAsync` discards `instanceId`
- **Severity: Low** · **Confidence: High**
- Location: `MTConnectDriver.cs:716-738` (`ReadAsync` ignores `current.InstanceId`), `:2061-2081` (only `chunk.InstanceId != instanceId` check), `:1687-1711` (`AnnounceAgentRestart`).
- Impact: a driver with no active subscription (browse-only) keeps a cached `/probe` model across an Agent restart and `OnRediscoveryNeeded` never fires. Production always subscribes, so browse-only edge.
### S-4. SQL: `SqlTagModel.Query` is on the enum but unsupported end-to-end; `MySql`/`Odbc` providers are reserved; `allowWrites` is parsed and ignored
- **Severity: Low** · **Confidence: High** (all documented deferrals that fail loudly or warn once)
- Location: `Driver.Sql.Contracts/SqlEquipmentTagParser.cs:75-76`, `SqlGroupPlanner.cs:336-339`; `SqlDriverFactoryExtensions.cs:214-225`, `SqlDriverProbe.cs:196-211`, `SqlDriverBrowser.cs:135-141`; `SqlDriverFactoryExtensions.cs:151-161` + `SqlDriver.cs:30-31` (no `IWritable`).
- Note: a `model: "Query"` tag is silently skipped (ledger + `BadNodeIdUnknown`) rather than refused at the deploy gate.
- Tracking: the Query/MySql/Odbc/allowWrites features → #753.
### Dead parsed surface (Low, High confidence)
- **T-3** `MTConnectObservation.SampleRate` parsed per observation (`MTConnectStreamsParser.cs:424-428`) and never read; `MTConnectTagDefinition.MtType/MtSubType/Units` carried but only `MtCategory` is read (`MTConnectDriver.cs:3493`).
- **S-5** `IS_NULLABLE` projected by every dialect's `ListColumnsSql` (`SqlServerDialect.cs:107-109`, `PostgresDialect.cs:141-144`, `OracleDialect.cs:184-187`) and never read by `SqlBrowseSession.cs:359-371` / `SqlCatalogLoader.cs:117-124`.
---
## 3. Performance limiters
### T-4. Every MTConnect `ReadAsync` batch fetches + parses the whole device `/current` and builds a fresh FrozenDictionary index, even while the pump holds a live one
- **Severity: Medium** · **Confidence: High**
- Location: `MTConnectDriver.cs:718-732`; `MTConnectObservationIndex.cs:149-164` (`ToFrozenDictionary` per construction).
- Evidence: `var current = await BoundedAsync(client.CurrentAsync, …); var snapshot = new MTConnectObservationIndex(binding.IndexTags); snapshot.Apply(current, baseline: true);`
- Failure scenario: OPC UA clients polling with Read (no subscription) against a 2–5k DataItem Agent cost one full XML transfer + parse + index build per Read call regardless of batch size; a few such clients make the Agent the bottleneck. The single-writer rationale (`:654-671`) forbids writing the shared index, not reading it.
- Suggested fix: read through `ObservationIndex` while `_pumpTask` is live and `!_streamDegraded`, keep `/current` as the fallback; or prebuild `binding.IndexTags` as a FrozenDictionary on `TagBinding`.
### T-5. MTConnect subscription throughput is hard-capped at `SampleCount / SampleIntervalMs` (default 1000 obs/s) with no lag detection or metric
- **Severity: Medium** · **Confidence: Medium**
- Location: `MTConnectAgentClient.cs:154-156` (`&interval=…&count=…`), `MTConnectDriver.cs:2083-2091` (gap → re-baseline), `:1961-1978`.
- Failure scenario: a machine producing more observations/s than `count/interval` makes the cursor fall steadily behind the Agent ring buffer; every buffer/rate seconds the driver warns, re-baselines via `/current` (losing intermediate transitions) and reopens — forever, every delivered value backlog-stale. Nothing surfaces the lag and nothing adapts `count`.
- Suggested fix: counter/Warning when chunks repeatedly carry exactly `SampleCount` observations or `expected - chunk.FirstSequence` shrinks toward zero; document the cap next to `SampleCount`.
### G-7. `GalaxyDriver.DiscoverAsync` downloads the full Galaxy hierarchy twice per discovery
- **Severity: Medium** · **Confidence: High**
- Location: `GalaxyDriver.cs:787-800`: `discoverer.DiscoverAsync(...)` (which calls `_source.GetHierarchyAsync`, `Browse/GalaxyDiscoverer.cs:68`) then `await source.GetHierarchyAsync(cancellationToken)` again just to select platforms.
- Failure scenario: two full `DiscoverHierarchyAsync` sweeps (every object + attribute) per browse/deploy on a large galaxy — and the second feeds the dead probe path (G-4).
- Suggested fix: have `GalaxyDiscoverer` expose the hierarchy it fetched.
### O-6. One `SemaphoreSlim _gate` serialises OpcUaClient reads, writes, subscribes, acks AND the full recursive discovery browse
- **Severity: Medium** · **Confidence: High**
- Location: `OpcUaClientDriver.cs:1213` (Read), `:1302` (Write), `:1436-1466` (DiscoverAsync holds the gate across `BrowseRecursiveAsync` + enrichment), `:1781`, `:2195`, `:2301`.
- Failure scenario: a stalled write (up to `_options.Timeout`) blocks every read and subscription create on that driver; recovery/reconcile reads queue behind it. Discovery on the runtime instance would block the data plane for the browse duration (latent today — discovery runs on the throwaway `DiscoveryDriverBrowser` instance).
- Suggested fix: separate read/write gating from subscription management; don't hold the gate across the whole recursive browse.
### S-6. SQL per-poll warning spam on a persistently failing or frozen group is not rate-limited
- **Severity: Low** · **Confidence: High**
- Location: `SqlPollReader.cs:345-357` (`TimedOut` warns every poll per group), `:411-419` (`QueryAsync` warns with the full `plan.SqlText` every poll per failed group); only contract warnings go through `ShouldWarnAboutContract()` (`:714-721`).
- Failure scenario: a dropped table or paused server with 20 groups at 1 s = 20 warnings/s with SQL text, indefinitely.
- Suggested fix: reuse the rate-limiter per plan, or log once per state transition.
### S-7. SQL query plans are re-planned and the slot dictionary re-allocated on every poll
- **Severity: Low** · **Confidence: Medium**
- Location: `SqlPollReader.cs:167-182` (`Dictionary<SqlTagDefinition, List<int>>` keyed by a 12-field positional record, rebuilt per `ReadAsync`), `:206` (`SqlGroupPlanner.Plan` per poll) vs `SqlQueryPlan.cs:68-70` ("safe to cache and reuse across polls" — nothing caches).
- Suggested fix: cache plans per subscription reference list, invalidated on `ApplyCatalogGateAsync` swap.
### Serial write paths (Low, High confidence)
- **G-8** Galaxy writes are strictly serial, three round-trips on first touch (`Runtime/GatewayGalaxyDataWriter.cs:124-131`, `:152-183`: AddItem → AdviseSupervisory → Write); acks serial by design (`GalaxyDriver.cs:1364-1381`).
- **M-8** MQTT `WriteAsync` publishes serially, one awaited QoS 1 round-trip per request bounded by `ConnectTimeoutSeconds` (`MqttDriver.cs:695-749`, `MqttConnection.cs:1172`) — 50 writes × up to 15 s on a stalled broker.
- **T-6** MTConnect writes are one POST per slot (`MTConnectDriver.cs:804-878`) though the Agent PUT surface accepts several `id=value` pairs.
### Other Low perf items (High confidence)
- **M-6** Wildcard-authored MQTT topics are matched linearly against every inbound message on MQTTnet's dispatcher thread (`MqttSubscriptionManager.cs:586-592`); concrete topics are a dictionary lookup. Fine for hundreds of wildcard tags, not thousands.
- **M-7** `TryDecodeUtf8` allocates `new UTF8Encoding(false, true)` per Raw/Scalar delivery (`MqttSubscriptionManager.cs:1374`); `MqttBrowseSession.cs:150` already uses a static.
- **T-7** The MTConnect connectivity probe parses the full `/probe` device model every 5 s and discards it (`MTConnectDriver.cs:1496-1504`, `MTConnectAgentClient.cs:169-180`); documented trade-off (`:1456-1462`). `ResponseHeadersRead` + status check would do.
- **O-7** `OnDataChange` is invoked under the driver-wide `_deliveryLock` (`OpcUaClientDriver.cs:368-380`, `:1888-1912`) — deliberate; acceptable only while every subscriber just `Tell`s.
---
## 4. Other hazards
### G-9. The comms-loss watchdog's re-init does nothing for Galaxy (no teardown, no rebuild); recovery rests entirely on the in-driver supervisor
- **Severity: Medium** · **Confidence: High**
- Location: `GalaxyDriver.cs:366-369` (`if (_subscriber is null && _dataWriter is null && _hierarchySource is null) BuildProductionRuntimeAsync`), `:735-741`.
- Evidence: after the first init the seams are non-null, so a watchdog-driven `ShutdownAsync`+`InitializeAsync` neither tears down nor reconnects — it sets `_health = Healthy` (`:381`, masked by `GetHealth()` preferring `_supervisor.IsDegraded`) and spawns another DeployWatcher (G-2). If the supervisor loop ends with a dead session (e.g. `ReopenAsync` returning early on `_ownedMxSession is null`, `:448`, then `Replaying`→`Healthy`), the host has no lever short of a respawn.
- Suggested fix: as G-1 — make `ShutdownAsync` release the runtime so the host's re-init genuinely rebuilds.
### O-8. OpcUaClient `AcknowledgeAsync` swallows non-cancellation transport failures, so the host reports the Part 9 ack as applied
- **Severity: Medium** · **Confidence: High**
- Location: `OpcUaClientDriver.cs:2384-2391`:
```csharp
catch (Exception ex)
{
_logger.LogWarning(ex, "...AcknowledgeAsync failed; acknowledgements may not have been applied.");
}
```
A `ServiceResultException` from `session.CallAsync` (faulted session, closed channel) returns normally; the local condition is acked while upstream still shows it unacknowledged. The next `ConditionRefresh` dedup will re-report it as unacked, but the operator received success.
- Suggested fix: rethrow (or return per-ack failures) so the OPC UA `Acknowledge` call fails with the transport status.
### G-11. Galaxy `LastSuccessfulRead` never advances on data — stamped only at init and on supervisor `Healthy`
- **Severity: Low** · **Confidence: High**
- Location: `GalaxyDriver.cs:381, 593-597, 661, 739`; no write from `OnPumpDataChange`/`ReadViaSubscribeOnceAsync`.
- Failure scenario: the watchdog's conjunction degenerates to "non-Healthy for 90 s" for Galaxy, so any supervisor `Degraded` lasting 90 s trips the host re-init even while values still flow on the old stream; with G-2 that is one leaked DeployWatcher per 90 s.
- Suggested fix: stamp (throttled) from `OnPumpDataChange`, as OpcUaClient's `RecordLivenessEvidence` does.
### G-10. `EventPump` dispatch loop dying leaves the read loop alive and health `Healthy` — clean termination evading the watchdog
- **Severity: Low** · **Confidence: Medium**
- Location: `Runtime/EventPump.cs:276-295`.
- Evidence: `DispatchLoopAsync` catches any exception, logs "events past this point will be lost until restart" and exits; `RunAsync` keeps draining into the bounded channel until it fills, then only counts drops. Nothing calls `NotifyStreamFault`; `GetHealth()` stays `Healthy` and (G-11) the stamp never moves anyway. `Dispatch` isolates handler exceptions, so the remaining throw sites are `StatusCodeMap.FromMxStatus`/`ToSnapshot` — hence Medium confidence on reachability.
- Suggested fix: treat a dispatch-loop exit while not cancelled as a stream fault, as the read loop does for a server-completed stream (`:173-188`).
### T-8. MTConnect logs and surfaces the raw `AgentUri` (which may carry `user:pass@`) while its own probe deliberately redacts it
- **Severity: Low** · **Confidence: Medium**
- Location: `MTConnectDriver.cs:1396` (`HostName => _options.AgentUri`), log lines at `:852, 1215, 2064, 2086, 2136, 2757, 2790`; contrast `MTConnectDriverProbe.cs:227-243, 251-256` (`RedactedDisplay`, "ex.Message is an open door for a credential leak").
- Suggested fix: compute a redacted display URI once at `StartCoreAsync` and log/expose that.
### S-8. SQL factory quotes the JSON parser message on a malformed config; the probe and browser withhold it for credential hygiene
- **Severity: Low** · **Confidence: Medium**
- Location: `SqlDriverFactoryExtensions.cs:1002-1006` (`$"... is not valid JSON: {ex.Message}"`) vs `SqlDriverProbe.cs:84-95`, `SqlDriverBrowser.cs:292-303` (message withheld "because the malformed text may carry a connection string"). The deployed config should carry only `connectionStringRef`, so this bites only on a pasted literal — but the message then reaches `InitializeFailed` → health `LastError` → `/hosts`.
### O-9. OpcUaClient `ShutdownAsync` can take N×5 s against a half-dead upstream and blocks the actor's `PostStop`
- **Severity: Low** · **Confidence: High**
- Location: `OpcUaClientDriver.cs:1121-1135` (serial best-effort `DeleteAsync` per subscription, each bounded by 5 s, skipped only when `_sessionLost == 1`), `DriverInstanceActor.cs:1550` (`ShutdownAsync(...).GetAwaiter().GetResult()`).
- Suggested fix: one bounded budget for the whole teardown, or delete concurrently.
### Low hazards, documented / bounded (High confidence)
- **M-10** `MqttDriver.ShutdownAsync`/`DisposeAsync` fall back to an ungated `TeardownAsync` after `ConnectTimeoutSeconds` (`MqttDriver.cs:402-409`, `:929-935`); the post-connect disposed re-check (`:993-997`) covers `DisposeAsync` only, so a wedged `InitializeCoreAsync` completing after a timed-out `ShutdownAsync` can publish one stray live connection — the actor's own Shutdown→Initialize sequence tears it down next time (bounded to one).
- **M-11** `MqttSubscriptionManager.EstablishAsync` with no transport records topics as subscribed and returns success (`:646-665`); unreachable today (`AttachTo` always precedes connect, `MqttDriver.cs:972-979`) but a silent "subscribed, receives nothing" if the ordering ever changes.
- **O-10** `ConnectionLost` raised before the host processed `InitializeSucceeded` is swallowed — documented known gap (`OpcUaClientDriver.cs:140-145`).
- **S-9** A stale SQL poll outliving the 5 s dispose wait (operationTimeout 15 s) can call `ObservePollOutcome` during the next `InitializeAsync`; `UpdateHealthUnlessFaulted` guards only `Faulted`, so a late Good read can briefly flip `Initializing`→`Healthy` (cosmetic; next `WriteHealth` overrides).
- **S-10** The SQL catalog gate rejects rather than reloads on schema drift (`SqlDriver.cs:929-972`, `:898-901`): a column added after init stays `BadNodeIdUnknown` until a reinit. Deliberate (#523 no-I/O-on-healthy rule); listed so operators know the surface is the skip ledger.
---
## 5. Checked and fine
**Galaxy**
- `ReadViaSubscribeOnceAsync` (`GalaxyDriver.cs:856-1009`): subscribe-once reads are the only MxAccess read primitive; handles unsubscribed in `finally`; internal 30 s deadline covers `CancellationToken.None`; a `SubscribeBulk` completing after the deadline is unsubscribed by continuation.
- `SubscriptionRegistry` prunes the reverse map on `Remove`/`Rebind`; nothing grows past the live subscription set.
- `EventPump` channel is bounded (50 000 default, configurable) with explicit `TryWrite` + `EventsDropped`; a server-completed stream is treated as a fault (`EventPump.cs:173-188`). Replay-gap handling is unreachable by design (`afterWorkerSequence: 0`, `GatewayGalaxySubscriber.cs:161`) and documented.
- `SubscribeAlarmsAsync` collapses to a single handle (`:1310-1322`); the per-reconnect handle growth it describes is closed.
- `GatewayGalaxyDataWriter` handle/advise caches are cleared on reconnect (`InvalidateHandleCaches`, `GalaxyDriver.cs:453`) and bounded by the written tag set; the fail-closed advise path forgets a failed handle.
- `GalaxyBrowseSession`/`GalaxyDriverBrowser`, `GalaxyDriverProbe`: client ownership transfers into the session and is disposed on `DisposeAsync`/`finally`; open path disposes on failure.
**OpcUaClient**
- `ReleaseSessionAsync` (`:1055-1095`) detaches keep-alive + diagnostics handlers, bounds `CloseAsync` by 5 s, skips the wire close when the session is known lost, and re-arms every outage latch; `InitializeAsync` calls it first, so a re-init cannot orphan the previous session.
- Comms-loss contract (#527/#670/#676): bad keep-alive → `Degraded` with the stamp preserved; good keep-alives/notifications advance it only while no subscription is latched lost; definitive statuses raise `ConnectionLost` once per session; `PublishCommsLoss` fans `BadNoCommunication` with value retained; recovery dedup is `(value, status)` (`IsReconcileRepair`).
- `_lastValues`/`_lastGoodSourceTicks`/`_deliverySeq`/`_awaitingRecovery`/`_replayGuard` are keyed by authored RawPath and cleared in `ShutdownAsync`; bounded even though `UnsubscribeAsync` does not prune them.
- `CancelSessionWork` deliberately does not dispose the old CTS (documented race with the keep-alive thread); no timer attached.
- Discovery releases continuation points on cancel (`:1533-1550`); `MaxDiscoveredNodes`/`MaxBrowseDepth` bound the browse. Browser session/`OpcUaClientDriverBrowser`/probe dispose on every path.
**MQTT**
- Teardown-before-rebuild still holds: `InitializeCoreAsync` calls the idempotent `TeardownAsync()` first (`MqttDriver.cs:955`), which reaches all three disposable fields — `_primaryHost`, `_hostProbeCts`, `_connection` (`:1381-1405`); no new disposable field escaped it.
- `MqttConnection` reconnect loop does not leak per attempt: one `IMqttClient` created once (`:718-724`), per-attempt CTSs `using`-scoped (`:689-693`), single supervisor task (`??=`, `:909`) awaited bounded on dispose (`:498-509`).
- Sparkplug caches are bounded by configuration, not wire: unauthored edge nodes dropped before decode (`SparkplugIngestor.cs:736-745`); `Births`, `_trackers`, `_lastRebirthTicks`, `_nodeBirthSeen`, `_metadataRebirths` pruned to authored nodes on `Register` (`:421-454`) and cleared on session reset; NDEATH/DDEATH evict (`:1076`, `:1091`); `_birthSignatures` pruned to `_authoredScopes` (`MqttDriver.cs:1221-1227`); `_warned` capped at 1024 (`:1912-1922`); `_pendingRebirths` self-prunes (`:1803-1810`). Rebirth storms gated (10 s per-node debounce, 3-attempt metadata cap, signature gate on rediscovery).
- Comms-loss contract: `GetHealth()` derives state from `MqttConnection.State` and `LastSuccessfulRead` from `LastMessageUtc` (`MqttDriver.cs:421-440`); `ConnectionLost` fires on the `Connected→Reconnecting` edge (`MqttConnection.cs:976-992`); both ingest paths stale to `BadNoCommunication` (`MqttSubscriptionManager.cs:269-282`, `SparkplugIngestor.cs:366-379`); plain-mode suppression keyed on `(status, value)` (`:824-828`); every (re)subscribe re-pushes cached snapshots.
- Inbound handling runs synchronously on MQTTnet's dispatcher with one JSON parse per message shared across matched tags; back-pressure is MQTTnet's per-client serialisation; the only handler `Tell`s.
- Browse session: atomic node-cap reservation, bounded payload inspection (512 B / 1 MiB birth), client disposed on `DisposeAsync` and on open failure. Primary host: separate client id, Will + explicit OFFLINE, claim backoff 2→30 s, connection disposed when ONLINE publish fails.
**MTConnect**
- Comms-loss contract: stream failure → `Degrade(error)` preserving `LastSuccessfulRead` (`:3152-3159`); every chunk incl. heartbeats → `PublishHealthy(now)` (`:2102-2103`); a quiet enumeration end under a live token is reported as `MTConnectStreamEndedException`/Transient, never idle (`:2106-2117`, `MTConnectAgentClient.cs:237-243`) — no clean-termination evasion; `/current` and `/sample` own separate degraded flags.
- UNAVAILABLE → `BadNoCommunication`, value null (`MTConnectObservationIndex.cs:421-425`); conditions stay active with quality-only Bad (`MTConnectConditionAlarmMapper.cs:90-97`), as the rules file specifies.
- instanceId/sequence: restart checked before gap (`:2055-2081`); gap and OUT_OF_RANGE both re-baseline (`:2127-2141`); backoff resets only when the stream delivered (`:1975-1990`).
- Re-init without teardown (#630): `InitializeAsync` stops pump + probe and disposes the old client before building (`:565-574`); a client built in a failed attempt is disposed (`:2783`). `MTConnectAgentClient` disposed on every teardown path; `HttpResponseMessage`/streams `using`/`await using` (`MTConnectAgentClient.cs:175-191, 208-214`); heartbeat watchdog bounds every body read, buffer capped at 32 MiB, DTD prohibited.
- `_supersededPumps` is pruned (`:1742-1754`); index/tracker/told-record maps are bounded by the Agent's DataItem count. Streams parser is forward-only `XmlReader` with a single-string fast path; no JSON on the hot path.
**SQL**
- Every `DbConnection`/`DbCommand`/`DbDataReader` path uses `await using` (`SqlPollReader.cs:371-405`, `SqlDriver.cs:982-989`, `:1086-1098`, `SqlCatalogLoader.cs:153-203`, `SqlBrowseSession.cs:396-404`); the browser disposes on open failure and hands ownership to the session otherwise.
- CTS/slot ownership on timeout paths is transferred to the work task and disposed in its `finally` (`SqlPollReader.cs:282-316`, `SqlDriver.cs:1042-1060`); `AbandonedTask.Detach` observes faults.
- Connection string never logged: only `Endpoint` is rendered (`SqlDriver.cs:1114-1128`); health/host messages use exception type names; browser redacts and drops inner exceptions.
- SQL injection surface closed: values bound everywhere; identifiers rewritten to catalog spellings by the allow-list gate before quoting (`SqlCatalogGate.cs:142-151`) then `QuoteIdentifier` with doubling + control-char rejection in all three dialects; Oracle `BindByName` set at every site; >2-part names rejected.
- Comms-loss contract: `ObservePollOutcome` (`SqlDriver.cs:606-675`) advances the stamp on any "database answered" code and degrades on timeouts; all-`BadTimeout` freezes it → watchdog → `ShutdownAsync` (`_poll.DisposeAsync`, 5 s cap) → `InitializeAsync` → re-subscribe; a query-failure-only outage is `Degraded` with the clock advancing, so no reconnect loop. Groups run in parallel under `Task.WhenAll` bounded by `maxConcurrentGroups`; key-value groups chunk at dialect bind caps; `CommandTimeout` floored at 1 s; factory enforces `operationTimeout > commandTimeout`.
+280
View File
@@ -0,0 +1,280 @@
# Review 07 — PLC drivers (AbCip, AbLegacy, Ab.Transport, Ab.LibplctagShared, Modbus, S7, TwinCAT, FOCAS, Calculation, Cli)
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
Repo: `/Users/dohertj2/Desktop/OtOpcUa` @ `d51d35bd` (master). Production code only. Every finding below was verified by reading the cited lines; sub-reviewer candidates that could not be confirmed were dropped. Paths are relative to `src/Drivers/` unless absolute.
Legend: **Sev** Critical/High/Medium/Low · **Conf** High/Medium.
---
## 1. Memory / resource leaks
### L1. Calculation: a script that overruns `RunTimeout` leaks one thread-pool thread per trigger, unbounded
**Sev High · Conf High** — `ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationEvaluator.cs:186-187`, `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/TimedScriptEvaluator.cs:78-81`, `ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationTimerScheduler.cs:66-80`
```csharp
var timed = new TimedScriptEvaluator<VirtualTagContext, object?>(evaluator, _runTimeout);
var raw = timed.RunAsync(context).GetAwaiter().GetResult(); // no token passed
// TimedScriptEvaluator: var runTask = Task.Run(() => _inner.RunAsync(context, ct), ct); await runTask.WaitAsync(Timeout, ct)
```
The timeout *abandons* the `Task.Run` thread (nothing can interrupt a synchronous Roslyn delegate). The timer group's `InFlight` flag is cleared in `finally` once `_evaluate` returns — i.e. as soon as the 2 s timeout fires — so a CPU-bound script (`while(true)`, a large array loop) on a 1 s timer leaks one pool thread per tick, forever; change-triggered tags do the same per input change. Health stays `Healthy` (`CalculationDriver.cs:341`), so nothing alerts until the pool is exhausted and unrelated work (gRPC, other drivers' `Task.Run`) stalls.
**Fix:** track the in-flight runner per tag and refuse to start another while it runs (publish `BadInternalError` "previous evaluation still running" + a counter); pass a `CancellationToken` so awaiting scripts can be cancelled; longer-term, out-of-process runner.
### L2. Calculation: `ShutdownAsync` never clears compiled scripts — collectible ALCs of every previous generation survive each respawn
**Sev Medium · Conf High** — `CalculationDriver.cs:323-330` (ShutdownAsync), `:719+` (Dispose is the only caller of `ClearCompiledScripts` besides AdoptConfig `:257`), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs:1550` (PostStop calls only `ShutdownAsync`)
Nothing in the runtime disposes an `IDisposable` driver; every config change is a respawn (#665), so each edit leaves the old generation's ALCs + delegates alive until GC finalises the driver graph — the "accreting ALCs" the `IScriptCacheOwner` contract exists to prevent, just deferred to GC timing. The same applies to the sync `Dispose()` on AbCip/Modbus/FOCAS (`AbCipDriver.cs:1900`, `ModbusDriver.cs:1761`, `FocasDriver.cs:1873`) — dead code in production.
**Fix:** call `_evaluator.ClearCompiledScripts()` from `ShutdownAsync` (idempotent), or dispose `IDisposable` drivers from `PostStop`.
### L3. Modbus: `ConnectAsync` overwrites a failed `TcpClient` without disposing it; the reconnect backoff loop runs *inside* the transport gate
**Sev Medium · Conf High** — `ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusSocketLifecycle.cs:84-92, 102-131`, `ModbusTcpTransport.cs:68-101`
```csharp
_client = new TcpClient(target.AddressFamily); // previous failed _client dropped, never Dispose()d
...
if (delay > TimeSpan.Zero) await Task.Delay(delay, ct); // inside SendAsync's _gate; up to 10 attempts, 30 s cap
```
A connection-refused/reset device makes one poll's `SendAsync` hold `_gate` for ≈ 81 s (0 + 0.2 + 0.4 … 30 s × 10), leaking up to 9 `TcpClient`s per loop (finalizer-reclaimed only). Every other read, every `WriteAsync` (the actor's 5 s write CTS cancels the gate wait → `BadInternalError`) and the probe loop (2 s CTS → host "Stopped") queue behind the one dead socket. `Dns.GetHostAddressesAsync` (`:79`) is outside the per-op `CancelAfter`.
**Fix:** `_client?.Dispose()` before reassignment; cap the in-gate reconnect to one attempt and leave multi-attempt backoff to the driver/watchdog; put DNS under the linked timeout CTS.
### L4. AbCip native: a deterministic controller refusal other than Unsupported/TypeMismatch re-creates the tag runtime on every poll (lease acquire/release churn; Forward Open/Close churn when it is the device's only tag)
**Sev Medium · Conf High** — `ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs:1009-1021` (generic catch → `EvictRuntime` + `Degraded`), `:1475-1488` (cold create catch), `AbCipDriverProbe.cs:189` (`IsReachableStatus => Ok or NotFound`), `NativeAbCipTagRuntime.cs:394-401` (Dispose → `ReleaseInBackground`), `ZB.MOM.WW.OtOpcUa.Driver.Ab.Transport/DeviceSessionPool.cs:106-138` (last lease closes the connection)
A tag the controller answers `NotFound` (0x04/0x05 — a renamed tag) never gets cached: each poll allocates a runtime, acquires a lease, reads, releases. If it is the only tag on that `DeviceKey`, each release is the last lease → Forward Close + UnRegisterSession, and the next poll TCP-connects + RegisterSession + Forward Opens again (throttled to ≥ 1 s by `ForwardOpenThrottle`). See H1 for the health/backoff side.
**Fix:** cache the runtime with its refusal status (the `TypeMismatch` branch at `:1457-1467` already does exactly this) for `NotFound`/`OutOfBounds`/`NotAllowed`, and treat them as per-tag Bad with no eviction.
### L6. FOCAS: handle-recycle loop disposes the device client outside `ConnectGate` — orphans a fresh connection or kills a poll mid-read
**Sev Medium · Conf High** (opt-in feature, default off) — `ZB.MOM.WW.OtOpcUa.Driver.FOCAS/FocasDriver.cs:1665` (`state.DisposeClient()` with no gate), `:2024-2028` (`Client?.Dispose(); Client = null;` non-atomic), `:1849-1850` (`EnsureConnectedAsync` installs under the gate), `:1814` (fast path reads `Client` without it)
Interleaving (a): recycle reads the old client, `EnsureConnected` installs a fresh connected one, recycle nulls it — the fresh client (two TCP sockets) is never disposed. (b): recycle disposes the client a poll is mid-read on → `ObjectDisposedException` → `BadCommunicationError` + Degraded for the rest of that poll, every recycle interval. The driver's own comment says the gate exists because unserialised dispose/install "disposed each other's client".
**Fix:** take `device.ConnectGate` in the recycle loop and dispose/null under it.
### L7. AbLegacy: per-key `SemaphoreSlim` tables grow with every distinct resolved indirect address (bounded)
**Sev Low · Conf Medium** — `ZB.MOM.WW.OtOpcUa.Driver.AbLegacy/AbLegacyDriver.cs:907-917` (`resolvedName = $"{def.Name}=>{concreteText}"`), `:1109`. Each index value an indirect `N7:[N7:0]` tag sweeps through leaves one semaphore in `_runtimeLocks` and one in `_creationLocks` until `DisposeRuntimes`. Ceiling = file size (≤ 256 words × ≤ 1000 files).
### L8. TwinCAT lifecycle nits
**Sev Low · Conf High** — `ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATDriver.cs:234` (`ShutdownAsync` awaits the probe task unbounded while `:197`/`:1445` bound it — a probe wedged in the synchronous `AdsClient.Connect`, `AdsTwinCATClient.cs:80`, hangs the actor's teardown-first reconnect); `AdsTwinCATClient.cs:36-41` (notification channel drops silently at 50k, no counter); `:646-651` (`NotificationRegistration.Dispose` removes from `_notifications` but not the queue, so a callback can fire after unsubscribe); in-place `InitializeAsync` never clears `_tagsByRawPath` (`:147-163`) and `_nativeSubs` keep references to disposed `DeviceState`s (only reachable via a direct re-init with changed config — the runtime respawns, #665).
### L5. Ab.Transport: per-frame allocations on the hot path (low)
**Sev Low · Conf High** — `Ab.Transport/Eip/EncapsulationClient.cs:283-292` allocates `head` + `frame` per reply; `Wire/CipWriter.cs` allocates a 256-byte buffer + `ToArray()` per request; `Logix/LogixTagClient.cs:62-63` builds a `List<byte>` then `ToArray()`. Modbus transports likewise allocate PDU/ADU/response arrays per request (`ModbusTcpTransport.cs`, `ModbusRtuOverTcpTransport.cs`), and `NormalizeWordOrder` copies even for BigEndian. At PLC poll rates this is Gen0 noise; only matters at thousands of PDUs/s. Not a defect.
---
## 2. Unimplemented / stubbed
### U1. Modbus: `Family`, `MelsecSubFamily`, `DisableFC23` are authored in the AdminUI but nothing consumes them; the whole `Driver.Modbus.Addressing` project has no production caller
**Sev Medium · Conf High** — `ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriverFactoryExtensions.cs:87-93`, `ModbusDriverOptions.cs:91-97` (DisableFC23 documented no-op), `ModbusTagDefinitionFactory.cs:34-95` (reads numeric `address` + `region` only); `grep` finds zero references to `ModbusAddressParser`/`ParsedModbusAddress`/`DirectLogicAddress`/`MelsecAddress` outside the Addressing assembly.
The "Family" dropdown (DL205 octal V-memory, MELSEC hex X/Y) changes nothing at runtime; an operator who picks MELSEC and authors `D100` gets `BadNodeIdUnknown`. The only region/type cross-check (`ModbusAddressParser.cs:176-180`) is therefore never run (see H5).
**Fix:** route the editor's address field through `ModbusAddressParser.Parse(text, Family, SubFamily)` and persist the resolved region/offset, or remove the three fields until they do something.
### U2. Modbus: block-read coalescing is OFF by default and never covers coils, discrete inputs, `BitInRegister` or `String`
**Sev Medium · Conf High** — `ModbusDriver.cs:476-478`, `:904-913`; `ModbusDriverOptions.cs:117` (`MaxReadGap = 0` = disabled)
```csharp
var coalesced = _options.MaxReadGap > 0 ? await ReadCoalescedAsync(...) : new HashSet<int>();
if (tag.Region is not (HoldingRegisters or InputRegisters)) continue; // coils/DI never coalesce
if (tag.DataType is String or BitInRegister) continue;
```
Default-configured device with 200 holding registers = 200 FC03 round-trips per poll; 16 `BitInRegister` tags on one status word = 16 reads of the *same* register; 64 adjacent coils = 64 FC01s. `0` means "disabled" rather than "adjacent only".
**Fix:** treat `MaxReadGap >= 0` as enabled with gap 0 = adjacent, default ~8; add FC01/FC02 bit-block coalescing; fold same-register `BitInRegister` tags into one read.
### U3. Modbus: `String[]` and `BitInRegister[]` tags read fine and are advertised Operate-writable, but every write fails `BadInternalError`
**Sev Low · Conf High** — `ModbusDriver.cs:655-673` (decode supports both), `:1177-1182, 1295-1301, 1314` (`ElementByteCount` throws for String; `Convert.ToBoolean(bool[])` → `InvalidCastException`). **Fix:** refuse at `FromTagConfig` or return `BadNotSupported`.
### U4. S7: Bool arrays ignore the parsed bit offset
**Sev Low · Conf High** — `ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs:1340-1356, 1643-1649`. `ReadArrayAsync` reads `count` bytes at `ByteOffset`; `DecodeArrayBlock` takes `block[i] & 0x01`. `DB1.DBX0.3` with `arrayLength: 8` returns bit 0 of bytes 0..7, silently. **Fix:** reject non-zero bit offsets on Bool arrays at authoring/init, or decode bit-packed.
### U5. Calculation: a change-triggered tag with zero literal `ctx.GetTag` reads never evaluates and nothing warns
**Sev Medium · Conf High** — `CalculationDriver.cs:178-186` (`_changeDependents` from `DependencyRefs` only), `:354-359`; `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Validation/DraftValidator.cs:395-456` (only Missing/NotFound/Cycle). `ctx.GetTag("Plant/" + line + "/Temp")` or `return ctx.Now.Hour;` with the default `changeTriggered:true` stays `BadWaitingForInitialData` forever while the driver reports Healthy. **Fix:** deploy-gate `CalculationNoEffectiveTrigger` when change-triggered && deps.Count == 0 && no timer; or evaluate once at init.
### U6. AbLegacy native: PLC-5 writes are not shipped, PLC-5 reads are N-file only, PLC-5 sub-elements refused (documented, surfaced correctly)
**Sev Low · Conf High** — `Ab.Transport/Pccc/PcccServices.cs:16-19` (`Plc5WriteVerified = false`), `:112-114`; `ZB.MOM.WW.OtOpcUa.Driver.AbLegacy/NativeAbLegacyTagRuntime.cs:150, 261-263`. All surface as `AbStatus.Unsupported` → `BadNotSupported` with no health impact, and `Native` is opt-in for AbLegacy. Noted as a capability gap, not a defect. → #752
### U8. FOCAS: `FocasDataType.String` is authorable but every read path publishes a number at Good
**Sev Medium · Conf High** — editor lists every enum member (`AdminUI/Components/Shared/Uns/TagEditors/FocasTagConfigEditor.razor:18`); `ZB.MOM.WW.OtOpcUa.Driver.FOCAS/Wire/FocasConstants.cs:116` (`_ => FocasPmcDataType.Byte`), `FocasPmcCoalescer.cs:135` (`_ => (object)raw`, a `long`), `Wire/WireFocasClient.cs:392-393` (PARAM default reads Int32), `:456-458` (MACRO returns `double`). A String-typed OPC UA node receives a boxed `long`/`int`/`double` at Good. **Fix:** reject `String` in `FocasTagDefinitionFactory.FromTagConfig`/capability matrix and drop it from the editor.
### U9. TwinCAT: array writes unimplemented but advertised writable; string writes bypass the codec
**Sev Medium · Conf High** — `TwinCATDriver.cs:521-524` (`def.ArrayLength` ignored), `AdsTwinCATClient.cs:908-925` (`ConvertForWrite` has no array case → `Convert.ToInt32(object)` → `InvalidCastException` → reported `BadTypeMismatch` at `:532-535` not `BadNotSupported`), `:586-590` (`DiscoverAsync` advertises arrays `Operate`); strings go to `_client.WriteValueAsync(symbolPath, string)` (`:472-473`) with no length check against `STRING(n)` and host-ANSI encoding the read side avoids (`TwinCATStringCodec.cs:7-10`). **Fix:** implement raw-bytes array/string writes mirroring the codecs, or refuse `BadNotSupported` and advertise `ViewOnly`.
### U10. AbCip/AbLegacy: array writes — config/UI allow them, both drivers encode a scalar
**Sev Medium · Conf Medium** — `AbCipDriver.cs:1191-1255` (no array branch; `WriteWidthRefusal` skips arrays `:1739`), `NativeAbCipTagRuntime.cs:319-384` (`Convert.ToInt32(value)`), `:236` (`client.WriteAsync(path, type, null, 1, buffer)`); `LibplctagTagRuntime.cs:228-290`; `NativeAbLegacyTagRuntime.cs:287-302, 418-449`. An `int[]` → `InvalidCastException` → `BadTypeMismatch` **and** Degraded (`:1281-1288`); a scalar to an array tag writes element 0 natively, while libplctag `SetInt32(0,…)` resends the whole last-read buffer (stale elements 1..n-1). Medium confidence only because the server-side array-write gate was not traced end to end. **Fix:** implement array encode (elements = count) or refuse `IsArrayTag(def)` writes with `BadNotSupported` before any runtime exists.
### U11. AbCip: BOOL-array element reads decode "low byte of the DWORD ≠ 0", not the addressed bit
**Sev Medium · Conf Medium** — `NativeAbCipTagRuntime.cs:440-442` (`IsTypeMismatch` exempts authored Bool answered `0xD3`), `:279` (`Bool => bitIndex is int bit ? GetBit(bit) : Span(offset, 1)[0] != 0`), `LibplctagTagRuntime.cs:121-123` (`GetInt8(offset) != 0`); `AbCipDriver.cs:956-958` (width check skipped for Bool). `BoolArr[n]` reads Good with "any of bits 0-7 set", subscript ignored; the write side already refuses this shape (`WriteWidthRefusal` → `BadNotSupported`). No capture/test covers which DWORD the controller returns. **Fix:** decode bit `n % 32` of element `n / 32`, or return `BadTypeMismatch` as the write path does.
### U12. TwinCAT native subscribe silently drops unresolvable references
**Sev Low · Conf High** — `TwinCATDriver.cs:669-670` (`continue` on resolve/device miss) vs the poll path's `BadNodeIdUnknown` (`:282-290`). The node keeps whatever status the server seeded. **Fix:** push a `BadNodeIdUnknown` snapshot before `continue`.
### U7. CLI gaps (low)
**Sev Low · Conf High** — `Cli/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Cli/Commands/SubscribeCommand.cs:60-66` writes from the poll thread with no lock/try-catch (every other CLI does); AbCip/AbLegacy `probe`/`read`/`write` print a stack trace on Ctrl-C (no `OperationCanceledException` arm); `Driver.Cli.Common/SnapshotFormatter.cs:107-114` renders arrays as `System.Int32[]`.
---
## 3. Performance limiters
### P1. S7: a read pass keeps issuing wire ops on a handle already marked dead; each pays the full 5 s deadline, and health is written only at the end of the pass
**Sev High · Conf High** — `S7Driver.cs:707-716` (span loop), `:741-765` (per-tag loop), `:758, 1037, 1156` (`_plcDead = true` set), `:2310-2313` (`MarkConnectionDeadIfFatal`), `:787-800` (single health write in `finally`)
No loop checks `_plcDead` between items. On a frozen-but-established peer (cable pull, firewall DROP — the case `S7OperationDeadline` exists for) the first span times out after `Timeout` and marks the handle dead; the remaining N spans/tags each still go to the wire and each wait another 5 s. A 20-span pass ≈ 100 s under `_gate`, and `Degraded` is not visible until the pass ends — so the #527 watchdog's 90 s clock starts only then: detection ≈ pass length + 90 s. The probe loop cannot help (same gate).
**Fix:** once `_plcDead` flips inside a pass, stamp the remaining items `BadCommunicationError` without I/O, record the failure, exit the pass.
### P2. S7: one bad member fails its entire span every poll — no per-tag fallback (the planner's stated invariant is false)
**Sev High · Conf High** — `S7Driver.cs:1138-1165`; `S7ReadSpanPlanner.cs:11-16`
```csharp
catch (Exception ex) { var status = RecordReadFailure(ex, outcome);
foreach (var member in span.Members) results[member.OriginalIndex] = new DataValueSnapshot(null, status, null, now); return; }
```
A span's block is the union of its members' ranges; if one member runs past the DB's actual length or over-declares `stringLength` (width = `StringLength + 2`, `:1430`), the CPU refuses the whole block and every neighbour within the 220-byte span is `BadDeviceFailure` + Degraded on every poll. The multi-var path (`:1040-1047`) and Modbus's coalescer (`ModbusDriver.cs:980-984`, auto-prohibit + per-tag fallback) both fall back; the span path does not.
**Fix:** on a non-connection-fatal span `PlcException`, re-read members individually for that pass and remember the span as "do not group".
### P3. Modbus: transport never self-heals after a second consecutive socket failure — every later call fails fast until the 90 s watchdog tears down all subscriptions
**Sev High · Conf High** — `ModbusTcpTransport.cs:66, 87-97`; `ModbusRtuOverTcpTransport.cs:94, 116-127`; `ModbusSocketLifecycle.cs:147-154`
```csharp
if (_life.Stream is null) throw new InvalidOperationException("Transport not connected"); // :66, BEFORE the gate and the retry arm
catch (Exception ex) when (_life.AutoReconnect && ModbusSocketLifecycle.IsSocketLevelFailure(ex)) // :87, single retry
```
A per-op timeout tears the socket down; the one retry reconnects and resends; if that also fails, `_stream` stays null and `:66` throws before `:87` can ever reconnect. Recovery is exclusively the comms-loss watchdog (Degraded + 90 s frozen) → `ShutdownAsync` → `_poll` disposed (all subscriptions) → re-init → re-subscribe. Two lost replies in a row from a slow device cost ≥ 90 s of Bad data plus full subscription churn. S7 reopens lazily on the next call.
**Fix:** when `_life.Stream is null && AutoReconnect`, attempt a backoff-throttled connect inside the gate instead of throwing.
### P4. Calculation: a script with a compile error is re-fed to Roslyn on every trigger, under the driver-wide `_evalGate`, and floods both logs
**Sev High · Conf High** — `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/CompiledScriptCache.cs:71-80` (faulted `Lazy` evicted), `CalculationEvaluator.cs:158-161`, `CalculationDriver.cs:518` (`lock (_evalGate)` around the run), `:596` (`PublishScriptLog` on every failure)
Source never changes between triggers (fixed in the artifact), so each trigger of a broken tag is a full `CSharpCompilation.Create` + diagnostics serialised against every other calc tag on the driver; one broken tag on a 250 ms input starves the rest. The node publish is correctly once-per-transition, but the host Warning and script-log entry fire every time.
**Fix:** negative-cache compile failures by source hash (`ClearCompiledScripts` already drops it on redeploy); gate the Warning/script-log on the Good→Bad transition.
### P5. Calculation: sync-over-async on a pool thread with a nested `Task.Run` manufactures false timeouts under load
**Sev Medium · Conf High** — `CalculationDriver.cs:494` (pump is `Task.Run`), `CalculationTimerScheduler.cs:53` (Timer callbacks are pool threads), `CalculationEvaluator.cs:187`. The `WaitAsync` clock starts before the inner item is scheduled, so pool starvation (which L1 causes) is charged to the script: "script timed out after 2.0s" → `BadInternalError` for a script that never ran. **Fix:** make the pump async and `await timed.RunAsync`, or run inline with a watchdog.
### P7. AbLegacy: reads are strictly serial with no per-device backoff — one dead device stalls every device in the driver and can trip the watchdog against a healthy one
**Sev High · Conf High** — `ZB.MOM.WW.OtOpcUa.Driver.AbLegacy/AbLegacyDriver.cs:375-479` (plain `for` + `await runtime.ReadAsync` at `:425`), `:443, 474` (failed read evicts the handle), `:1101-1128` (`EnsureTagRuntimeAsync` re-inits with a full `Timeout` each), no `ConnectionBackoff`/`ConnectGate` (grep: none); `NativeAbLegacyTagRuntime.cs:127` (`AllowPacking = false`)
The host subscribes all of a driver's refs in one `PollGroupEngine` loop (`DriverHostActor.cs:3631`). With device B down and 50 tags, each poll pays 50 × `Timeout` (2 s) serially ≈ 100 s; device A's tags are read once per ~100 s. Per-tag health writes (`:446/462/477`) flip Degraded/Healthy and `LastSuccessfulRead` only advances on an A success, so a cycle > 90 s trips the comms-loss watchdog against *reachable* device A, and the re-init restarts the same cycle. Native transport does not help (no MSP packing; adjacent N/F/B words never coalesced although the array path proves the transport can). AbCip has a milder form: cold waiters hold a window slot while queued on `ConnectGate` (`AbCipDriver.cs:797 → 1431`), so a dead device's backoff-expiry attempt can occupy all 8 slots for `Timeout` once per ≤ 30 s.
**Fix:** per-device `ConnectionBackoff` + single-flight connect gate (copy AbCip), bounded-parallel window across devices, adjacent-element coalescing per device.
### P8. AbCip native: `MaxConcurrentReads` (default 8) caps Multiple Service Packet coalescing at 8 services per round trip
**Sev Medium · Conf High** — `AbCipDriver.cs:779-784, 797` (window slot taken before `runtime.ReadAsync` → `scheduler.SendAsync`), `AbCipDriverOptions.cs:121` (`= 8`, documented for libplctag per-handle concurrency); `Ab.Transport/Cip/RequestScheduler.cs` `TakeBatch`/`PumpAsync` pack only what is queued when the pump wakes. A 4002-byte Large Forward Open can carry ~100+ DINT reads per packet (the scheduler budgets request + reply bytes), so a 1000-tag CompactLogix pays ~125 round trips per poll instead of ~10.
**Fix:** on `AbCipTransport.Native` make the window effectively unbounded (≥ 64) — the scheduler's reply budget already prevents over-packing.
### P9. FOCAS: one slow CNC stalls every device in the subscription; the per-device gate is shared by three loops
**Sev Medium · Conf High** — `FocasDriver.cs:565-651` (sequential walk of the whole batch), `SynchronizedFocasClient.cs:143-156` (gate shared with the fixed-tree loop — 9-block `cnc_rddynamic2` per axis every 250 ms — and the alarm loop). A dead device is cheap (backoff → `ConnectionThrottledException`), but a *slow* device (TCP-accept-then-stall, the live-observed 31i-B case) costs `Timeout` (2 s) per tag per poll during which no other device is read. Per-poll nit: `TryReadFixedTree` builds `HostAddress + "/"` for every device for every reference (`:1515`), run twice per poll (`:681`, `:577`).
**Fix:** bucket by device and read devices concurrently (`Task.WhenAll`, each serialised by its own gate); precompute the prefix on `DeviceState`.
### P10. TwinCAT: a Warning per bad tag per poll with no once-per-tag dedup
**Sev Medium · Conf High** — `TwinCATDriver.cs:353-362` (`CompleteRead` logs `LogWarning` for every non-Good status every poll). 50 typo'd symbols on a 500 ms poll = 100 warnings/s indefinitely (AbCip and FOCAS warn once). **Fix:** a "warned" set keyed by reference, cleared on Good or re-init.
### P11. AbCip: UDT read planner recomputes member layout per member per poll
**Sev Low · Conf High** — `AbCipUdtReadPlanner.cs:51-83` (`AbCipUdtMemberLayout.TryBuild(parent.Members)` at `:70` for every member reference every poll — O(members²) per parent per tick, allocating a dictionary each time); `AbCipDriver.cs:780-784` allocates a `SemaphoreSlim`, `Task[]`, `ReadOutcome[]` + closures per poll. Only material with `EnableDeclarationOnlyUdtGrouping` on and large groups. **Fix:** cache the layout per parent definition.
### P6. Modbus: region/type compatibility is not enforced — with coalescing ON, one `HoldingRegisters`+`Bool` tag kills the whole subscription's poll cycle every tick
**Sev Medium · Conf High** — `ModbusDriver.cs:931` (`RegisterCount(tag)` outside any try), `:1429-1436` (`throw new InvalidOperationException($"Non-register data type …")`), `ModbusTagDefinitionFactory.cs:52-65` (no cross-check). `RegisterCount` throws out of `ReadCoalescedAsync` → out of the reader → `PollGroupEngine` returns no snapshots for any tag and backs off to 30 s, forever. With coalescing off the same tag reads `BadCommunicationError` + Degraded. Conversely `Coils` + `Int16` silently returns a `bool` into an Int32 node.
**Fix:** reject region/type mismatches in `FromTagConfig`; wrap the per-tag planner math so a bad tag goes to the per-tag path.
---
## 4. Other hazards
### H1. AbCip: deterministic controller refusals (NotFound / OutOfBounds / NotAllowed / unmapped status) latch `Degraded`, and the non-NotFound ones open the device-wide 1→30 s backoff window that fails every sibling cold create
**Sev Medium · Conf High** — `AbCipDriver.cs:1009-1021` (generic catch → `ReadOutcome.Degraded`), `:1442-1444` (`ConnectionThrottledException` for siblings), `:1475-1488` (`RecordFailure` unless `IsReachableStatus`), `AbCipDriverProbe.cs:189` (`Ok or NotFound` only), `:1937` (`new ConnectionBackoff(1 s, 30 s)`); `Ab.Transport/AbStatusCodes.cs:9-23` (0x1C/0x20/0xFF-0x2104… → `BadReply` → `BadCommunicationError`)
`drivers.md` says only Unsupported/TypeMismatch are "refusals, not comms faults". A tag authored `Arr[999]` (0xFF/0x2105 → `OutOfBounds`) is answered by a working session but records a *Forward-Open failure*, so any sibling needing a cold create (new subscription, or evicted after a transient) fails fast for up to 30 s with `BadCommunicationError`. Health latches Degraded while the tag exists; if it is the device's only tag, `LastSuccessfulRead` freezes and the #527 watchdog re-inits the driver every 90 s against a reachable controller.
**Fix:** widen `IsReachableStatus` to every status that proves the controller answered (`NotFound, OutOfBounds, NotAllowed, TooLarge/TooSmall, TypeMismatch, Unsupported`); treat them as per-tag Bad with no health change (as the TypeMismatch branch does); map 0x1C/0x20/0xFF-0x2104 explicitly instead of to `BadReply`.
### H2. Modbus: deadband swallows the Good-after-Bad recovery publish (value-only dedup — the exact #527 trap `drivers.md` warns about)
**Sev High · Conf High** — `ModbusDriver.cs:236-259`
```csharp
if (snapshot.Value is null) return true; // Bad passes through, baseline NOT touched
...
if (Math.Abs(newD - oldD) < tag.Deadband.Value) return false; // status never consulted
```
Tag publishes 10.0 (baseline 10.0) → comms drop → Bad published, baseline kept → device returns holding 10.0 → engine fires onChange (status differs) → `|10−10| < deadband` → **false**: the Good recovery is never published and the node stays Bad until the value drifts past the deadband. Deadband is only authorable via raw TagConfig JSON today (`ModbusTagDefinitionFactory.cs:80-82`), which narrows exposure, not the defect.
**Fix:** drop the baseline (`_lastPublishedByRef.TryRemove`) when returning true for a null/Bad snapshot, or baseline on `(value, status)`.
### H3. S7: one mis-authored tag faults the whole driver at init and the actor retries the same init every 10 s forever; neither the editor nor the deploy gate catches the throwing cases
**Sev High · Conf High** — `S7Driver.cs:270` (`RejectUnsupportedTagConfigs()` throws), `:313-316` (`S7AddressParser.Parse` throws `FormatException`), `:336-348` (catch → `Faulted` → rethrow); `S7TagDefinitionFactory.cs:59-64` accepts any non-blank address; `DriverInstanceActor.cs:743` retries. The comment at `:245-246` says "one bad tag must not fail the whole driver init". A typo'd `DB1.DBZ0`, `DB1.DBW0` typed `Float64`, or `T5` typed `Int32` darkens every tag on that PLC. Modbus skips the tag (`ModbusDriver.cs:337-345`). CSV import / API blobs bypass the editor entirely.
**Fix:** convert init guards + parse pass into per-tag skips (Modbus pattern) and run `S7AddressParser.TryParse` + guards inside `FromTagConfig`/`Inspect` so the deploy gate refuses them.
### H4. Modbus and S7: protocol refusals degrade driver health; all-tags-refused produces a 90 s reconnect/resubscribe loop
**Sev Medium · Conf High** — `ModbusDriver.cs:507-511` (`ModbusException` 01/02/03 → `Degraded`), `:492` vs `:510` (health is last-writer-wins per tag, so it flaps with tag order — `drivers.md`'s "latches Degraded forever" is wrong); `S7Driver.cs:1278-1285` (`PlcException` incl. `ReadData` bad DB/address → `Degraded`). When every tag of a driver refuses (wrong DB number, wrong register map, gateway 0x0B for an offline RS-485 slave), `LastSuccessfulRead` freezes AND health is Degraded → the #527 conjunction trips every 90 s against a perfectly reachable device.
**Fix:** map Modbus 01/02/03 and S7 non-fatal `ReadData`/`WriteData` to per-tag Bad + once-per-tag Warning with no health write (the AbCip rule); keep 04/05/06/0A/0B and connection-fatal codes on Degraded.
### H5. S7: no driver-side lifecycle gate — overlapping `InitializeAsync` calls can leak an opened PLC connection and orphan a probe loop
**Sev Medium · Conf Medium** — `S7Driver.cs:226-349` (no gate around the body; only `_gate` at `:150` for I/O), `:304` (`Plc = plc`), `:328-333` (`_probeCts/_probeTask` assigned); `DriverInstanceActor.cs:743, 807-855` (`RetryConnect` every 10 s with `_initGeneration` dropping only the *result*). `ShutdownAsync` drains (≤ 5 s poll + ≤ 5 s probe + ≤ 5 s gate) + connect (≤ 5 s) can exceed 10 s, so attempt B starts while A is in `OpenAsync`; A's `Plc` and probe loop are then never released (only the latest `_probeCts` is cancelled). Modbus has `_lifecycleGate` (`ModbusDriver.cs:91-101`) for exactly this.
**Fix:** add the Modbus `_lifecycleGate` pattern around Initialize/Shutdown/Reinitialize.
### H6. Calculation: a driver Restart leaves every calc tag `BadWaitingForInitialData` until each input happens to change
**Sev Medium · Conf High** — `CalculationDriver.cs:87-88, 354, 475`; `DriverHostActor.cs:3811-3826` (Restart = new instance, only the restarted child re-subscribes); `DependencyMuxActor.cs:43-88` (no last-value replay). Deploy is fine (poll drivers `forceRaise` on re-subscribe); Restart has no re-push, and values arriving before `_running` are dropped (`:354`). **Fix:** mux retains last `(value,status,ts)` per ref and replays on `RegisterInterest`; record inputs regardless of `_running`.
### H10. AbCip: a missing/typo'd probe tag turns into a permanent device blackout — the probe loop records every exception as a Forward-Open failure
**Sev High · Conf High** — `AbCipDriver.cs:666-679` (`backoffVerdict = ex is AbCipTagConfigurationException ? null : false;` — every other exception incl. `AbTransportException(NotFound)`), `:700-701` (`RecordFailure` every `Probe.Interval` 5 s), `:1442-1444` (data path fails fast while the window is open), `:1482-1485` (the data path *does* classify `NotFound` as reachable — the probe does not), `AbCipDriverProbe.cs:146-151, 164-167` (Test Connect reports the same situation green: "controller reachable; probe tag not found")
With `RecordFailure` every 5 s, `ComputeDelay` saturates at 30 s so `_nextAttemptUtc` is always ≥ 25 s ahead and `ShouldAttempt` is **never** true: every cold runtime create throws `ConnectionThrottledException` → `BadCommunicationError` + Degraded for every tag not already hot, and after any single eviction a tag never comes back. Scenario: `Probe.ProbeTagPath` typo, or the probe tag deleted by a PLC download. Host `Stopped`, driver Degraded, all new/evicted tags dead indefinitely; a node restart does not help.
**Fix:** classify `ate.Status` with `IsReachableStatus` in the probe catch — `NotFound` ⇒ verdict `true` (the session answered), one config Warning, and decide `HostState` separately from the backoff verdict.
### H11. TwinCAT native subscribe is all-or-nothing: one bad symbol or one unreachable device yields zero data for the whole driver, with register/delete churn every 30 s and health still `Healthy`
**Sev High · Conf High** — `TwinCATDriver.cs:690-706` (`TwinCATNotificationRefusedException` is the *only* per-tag catch), `:710-721` (any other exception: dispose every registration + rethrow); `AdsTwinCATClient.cs:540-542` (unknown symbol → `InvalidOperationException`), `:584-586`, `:1214-1216`/`:1240-1248` (device offline / backoff → connect exception); `TwinCATDriverOptions.cs:43` (`UseNativeNotifications = true` default)
The host bulk-subscribes every tag in one call, so one typo'd `symbolPath` or one dead PLC in a multi-device driver tears down every notification already registered and fails the subscription; the health-tick reconcile retries every 30 s — a 2,000-tag driver issues 2,000 `AddDeviceNotification` + 2,000 `DeleteDeviceNotification` per 30 s forever, serving nothing, with `_health` untouched so the watchdog never fires. The poll fallback handles the same inputs per tag.
**Fix:** catch `Exception` around `RegisterNotificationAsync`: one-off `BadNodeIdUnknown`/`BadCommunicationError` snapshot, store the intent as a dead registration so `RetryDeadRegistrationsAsync` picks it up; short-circuit a device-level connect failure for the rest of that device's tags.
### H12. TwinCAT native mode: `LastSuccessfulRead` never advances, so one Degraded write latches a full driver reconnect; a router-level failure never degrades at all
**Sev High · Conf High** — `TwinCATDriver.cs:681-683` (notification callback publishes Good without touching `_health`), `:889-943` (probe only moves `HostState`), `:176` (`LastSuccessfulRead` = init stamp); debits: `:540-544` (any write exception incl. throttled connect → Degraded), `:527-531` (`NotSupportedException` write → Degraded — the #611 "refusal is not a comms fault" case), `:1294-1310` (replay failure, incl. a correctly-isolated type-mismatch tag re-attempted on every reconnect); `DriverInstanceActor.cs:637-646` (silence measured from `max(LastSuccessfulRead, _connectedSinceUtc)`)
On a connection older than 90 s, the first health tick after one Degraded write self-sends `DisconnectObserved` → `ShutdownAsync` + re-init + resubscribe: every ADS notification on every device deleted and re-registered because one write timed out. Inverse: an unreachable AMS router (`client.ConnectAsync` throws `:1234-1248`) is caught by the probe (`:920-928`) and never writes `_health` — values stay stale-Good, driver `Healthy` indefinitely.
**Fix:** stamp `LastSuccessfulRead` (and restore Healthy) from the notification dispatch path (throttled `Volatile.Write` per batch) and from a successful probe; write Degraded with `LastSuccessfulRead` preserved after N consecutive probe/connect failures; drop the health write from the `NotSupportedException` branch.
### H13. FOCAS: `_devices` is a plain `Dictionary` cleared and refilled in place under a live poll loop, the fixed-tree loop, the alarm loop and the actor health tick
**Sev High · Conf High** — `FocasDriver.cs:46` (`Dictionary<string, DeviceState>`), `:412` (`_devices.Clear()` in `ReleaseDeviceStatesAsync`), `:225` (refill in `InitializeAsync`); readers `:595` (`ReadAsync` TryGetValue), `:1513` (`foreach (var state in _devices.Values)` — "Collection was modified"), `:1145` (`GetHostStatuses` from the actor), `:1779` (alarm loop). The driver explicitly keeps the poll engine alive across re-init (`:49-50`) and build-and-swaps the *tag* tables for that reason, but not `_devices`. TwinCAT uses `ConcurrentDictionary` (`TwinCATDriver.cs:31-34`). Same class, lower exposure: AbCip `_devices`/`_devicesByRoutingKey` (`AbCipDriver.cs:101, 106`, cleared `:529-530` vs read `:901` outside the try) and AbLegacy `_devices` + `_tagsByRawPath` (`AbLegacyDriver.cs:31, 35`, `.Clear()` at `:259, 343` while the resolver reads it) — AbCip already swaps `_tagsByRawPath` by reference (`:532`) for exactly this reason.
**Fix:** `ConcurrentDictionary`, or build-and-`Volatile.Write`-swap like `_tagsByRawPath`.
### H14. FOCAS: macro/parameter numbers are narrowed to `short` after the capability matrix admitted up to 99999
**Sev High · Conf High** — `Wire/WireFocasClient.cs:408` (`_wire.ReadMacroAsync((short)address.Number, …)`), `:377-393` (parameters likewise); `FocasCapabilityMatrix.cs:59-65` (30i/31i/32i macro range `(0, 99999)`; `Unknown` → `int.MaxValue`); the wire format already carries `Int32` (`FocasWireProtocol.cs:285-288`). `MACRO:99999` → `(short)99999 == -31073`: the CNC is asked for a different variable and the result is published Good under the authored tag.
**Fix:** widen the adapter signatures to `int`, or reject `Number > short.MaxValue` at init into the skipped-tag ledger.
### H15. FOCAS: a driver serving only `FIXED:` tags never leaves `Healthy` when its CNC disappears
**Sev Medium · Conf High** — `FocasDriver.cs:1243-1250, 1385-1391` (fixed-tree loop swallows every failure at Debug, never touches `_health`), `:1181-1185` (probe likewise), `:1544-1546` (deliberately does not credit), `:611-621` (`ReadAsync` returns the fixed-tree snapshot without touching health), `:1529-1530` (values go Bad after `FixedTreeStaleAfter` ≥ 30 s). `/hosts` shows a healthy driver with every node Bad; the #527 watchdog can never fire (the loops do self-reconnect, so recovery works — the health surface lies). Identity/Axes/Program/Status are the typical FOCAS deployment shape.
**Fix:** after a successful axis pass write Healthy + `LastSuccessfulRead`; after a failed tick past `FixedTreeStaleAfter` write Degraded preserving it.
### H16. AbLegacy: a deterministic refusal (array of String/Timer/Counter/Control) latches Degraded and churns handles every poll
**Sev Medium · Conf High** — `AbLegacyDriver.cs:471-478` (generic catch → `EvictRuntime` + `BadCommunicationError` + Degraded), `:597-601` (write path same), `NativeAbLegacyTagRuntime.cs:411-414` / `LibplctagLegacyTagRuntime.cs:200-206` (`DecodeArray` throws `NotSupportedException`; comment "the driver rejects them before reaching here" has no corresponding check); `AbLegacyTagDefinitionFactory.cs:73-79` accepts `isArray:true` with any `dataType`. Every poll: create handle (one init read), decode throws, dispose, Bad, Degraded. **Fix:** catch `NotSupportedException` → `BadNotSupported`, keep the handle, no health write; reject the combination in `FromTagConfig`/`Validate`.
### H17. AbCip: bit-in-word write hardcodes a DINT parent; INT/SINT hosts fail as comms faults
**Sev Medium · Conf High** — `AbCipDriver.cs:1358, 1363` (`DecodeValue(AbCipDataType.DInt)` / `EncodeValue(DInt)` on the parent), `:1385` (parent runtime created with no `AuthoredType`). Logix allows `.n` on SINT/INT/DINT; reads of `Flags.3` on an INT host work, but the write's `DecodeValue(DInt)` on a 2-byte buffer throws `OutOfBounds` → generic catch `:1322-1331` → `EvictRuntime` (the bit tag's, not the parent's), `BadCommunicationError`, Degraded. **Fix:** pick the parent width from `GetSize()`/CIP type code; map mismatch to `BadTypeMismatch` with no health write.
### H18. TwinCAT: BOOL-within-word write is an RMW at a hardcoded 32-bit width with no parent check
**Sev Medium · Conf Medium** — `TwinCATDriver.cs:492-511` (reads/writes the parent as `UDInt`); the notification path validates the bit against the real width (`AdsTwinCATClient.cs:522-531`, `TwinCATBitAccess.CheckParent`) but the write path does neither. A bit beyond a 16-bit parent is written into neighbouring bytes or rejected; a signed parent with the sign bit set reads back ≥ 2³¹ and a checked conversion → `OverflowException` → `BadOutOfRange` for *any* bit write. Writes were outside the TwinCAT live gate. **Fix:** resolve the parent via `DescribeSymbolAsync`, `CheckParent`, RMW on raw bytes of `parent.ByteSize`.
### H19. AbCip write-side health: client conversion errors latch Degraded; a good write overwrites Degraded with Healthy
**Sev Medium · Conf High** — `AbCipDriver.cs:1272-1296` (`FormatException`/`InvalidCastException`/`OverflowException` → Degraded; nothing touched the wire), `:1186-1187, 1254` (`Healthy` on success, no worst-wins — undoing a Degraded the poll loop just recorded; the batch read path's `ApplyBatchHealth :858-882` exists to avoid exactly this). AbLegacy maps the same three to `BadTypeMismatch`/`BadOutOfRange` with no health write (`:602-609`). Operator surface only (`LastSuccessfulRead` still advances on reads). **Fix:** drop the health writes from those catches and the success branches.
### H20. FOCAS wire: `ParseAlarms` layout heuristic is ambiguous at 19 legacy records
**Sev Low · Conf High** — `Wire/FocasWireClient.cs:1114-1121` picks the 76-byte layout whenever `Length % 76 == 0`; a legacy 80-byte reply of exactly 19 records (1,520 B) also satisfies it and decodes as 20 garbage vendor records.
### H21. AbCip `_health` is not `volatile` (sibling says it is load-bearing)
**Sev Low · Conf High** — `AbCipDriver.cs:136` vs `AbLegacyDriver.cs:41-46` (whose comment explains why the actor-thread reader needs the barrier). Reference assignment is atomic and the reader crosses method calls, so fine in practice.
### H7. Modbus: `WriteAsync` maps transport failures to `BadInternalError`; S7 probe reports the host `Stopped` on gate contention
**Sev Low · Conf High** — `ModbusDriver.cs:1135-1146` (only `InvalidDataException` → `BadCommunicationError`; desync `IOException`, `SocketException`, timeout OCE → `BadInternalError`; encode `OverflowException` → `BadInternalError` not `BadOutOfRange`); `S7Driver.cs:2499-2546` (2 s `gateCts` around `_gate.WaitAsync` → `TransitionTo(Stopped)` on a long read pass), `S7DriverProbe.cs:74-76` ignores config `TimeoutMs`.
### H8. Calculation: array-valued results never dedup
**Sev Low · Conf High** — `CalculationDriver.cs:615` `Equals(st.LastValue, value)` is reference equality for arrays: a `double[]`-returning timer script republishes every tick through the dual-namespace fan-out and the historian. **Fix:** `StructuralComparisons.StructuralEqualityComparer` for `Array`.
### H9. Modbus: `SendOnceAsync` indexes the response PDU without a length check
**Sev Low · Conf High** — `ModbusTcpTransport.cs:157-171`. `respLen == 1` → `respPdu[0]` throws `IndexOutOfRangeException` (not an `IOException`, so no teardown/retry) → `BadCommunicationError` + Degraded for a malformed reply. **Fix:** treat `respLen < 2` (`< 3` for exception frames) as a `TruncatedFrame` desync.
---
## 5. Checked and fine (do not re-chase)
**Ab.Transport (read in full by me):**
- `EncapsulationClient`: single-flight gate; caller cancel never faults the socket; every timeout finite (`ValidateTimeout`); `Fault()` closes the socket; `DisposeAsync` UnRegisters with a 500 ms budget; `_gate` deliberately never disposed (abandoned exchange may release late).
- `CipConnection`: one shared open with `OpenBudget`; predecessor-closed chaining; one-shot keep-alive timer re-armed after each tick; Forward Close on dispose (1 s budget); `RetireClosedByTargetAsync` on 0x0107/0x0203 so a target-dropped class-3 connection reopens.
- `RequestScheduler`: one pump per device (no cross-device stall); waiter tokens never reach the wire; MSP packing budgeted on request **and** expected-reply bytes; cancellation registrations disposed exactly once.
- `DeviceSessionPool`: ref-counted leases, close chain prevents overlapping connections; `ForwardOpenThrottle` ≥ 1 s per device, never pruned (bounded by device count, documented).
- `LogixTagClient`: fragmented read capped at 2 MiB; struct-handle change mid-read detected; fragmented write chunk alignment handles wide elements.
- Decoders are bounds-checked: `CipReader.Take`, `MultipleServicePacket.Parse` (offset range), `EncapsulationCodec.ParseReply` (length/items), `PcccServices.ParseReply` (TNS before STS).
- `CipPath.EncodeRoute` rejects malformed links/odd pairs; `DeviceKey` canonicalises the route.
**Modbus:** #527 dispose-before-overwrite in `InitializeCoreAsync` (`:307-319`) intact under `_lifecycleGate`; desync recovery tears the socket down before throwing (TCP `:176-191`, RTU `ModbusRtuFraming.cs:125-165`) and a coherent exception PDU does not; `BitInRegister` RMW serialised per register via `_rmwLocks` (bounded 65 536, no OS handle); `NormalizeWordOrder` correct for 8-byte types; block reads validate byte count before slicing; auto-prohibition bisection bounded and cleared on teardown.
**S7:** `EnsureConnectedAsync` under `_gate` with `ConnectionBackoff` (one attempt per window), dead handle disposed once, failed `OpenAsync` handle disposed; worst-state-wins health with `AnySuccess` advancing `LastSuccessfulRead`; access-denied → sticky Faulted + `BadNotSupported` without reconnect churn; value-range/encode errors → `BadOutOfRange`/`BadTypeMismatch` with no health impact; multi-var chunk 12 safe for 240-byte PDU with per-tag fallback; `S7OperationDeadline` bounds every async op; Bool/wide arrays and Timer/Counter writes refused up front; shutdown drains bounded; TSAP/CPU map/codecs refuse rather than clamp.
**Calculation:** timer `Timer`s + CTS disposed in scheduler `Dispose`, `StopTimers()` at the top of every lifecycle entry (no stacking on re-init); lock order `_evalGate → _lock` honoured, publishes fire outside both; `(value,status)` dedup on the recovery path correct (`:615`, `MarkInputsBad :639` per sub-code); `ClearCompiledScripts` vs in-flight run handled by the single retry on `ObjectDisposedException`; `AdoptConfig` clears `_pending` and drops removed tags' state; all input-cache access under `_lock`.
**Cli / Contracts:** all five subscribe commands unsubscribe in `finally` and reach `ShutdownAsync`; CLI flags land (`--transport`, `--poll-only`, `--disable-reconnect`); the only Options property no driver reads is `ProbeTimeoutSeconds` (consumed by AdminUI Test Connect); all six factories use `TryReadEnumStrict`; S7 `NotSupportedException`s at `S7Driver.cs:1826-1830/2208-2212` are unreachable guards on the narrow path.
**AbCip / AbLegacy:** native lease balance — every acquire has a release on every path (`NativeAbCipTagRuntime` init `:131-134` / `Dispose :398`; `EnsureTagRuntimeAsync` disposes on every init failure `:1465, 1472, 1477, 1496`; probe loop `:674, 715`; `AbCipDriverProbe` `finally :178`; `NativeAbLegacyTagRuntime` ctor acquire / `InitializeOrKeepRefusedAsync :1069-1073`; enumerator + template reader `finally`); `ConnectionLease.DisposeAsync` idempotent. `LibplctagOrphanRelease` wraps Initialize/Read/Write on both runtimes and the listing/template readers. AbCip `_tagsByRawPath`/`_declaredTags` swap by reference with `volatile` (`:109, 112, 461-462, 532`); `DisposeHandles` disposes runtimes, parent runtimes and both semaphore tables (`:2057-2069`); `ConnectGate` intentionally not disposed (`:1951-1958`). Alarm projection subscriptions retired out-of-band with CTSs disposed; told-records pruned by `RetainOnly` per subscribe; warned-once sets bounded by the authored set. `AbCipTemplateCache` never pruned — bounded by the controller's UDT count, staleness documented. AbCip read/write refusal classification (TypeMismatch, Unsupported, config exception, `NotSupportedException` on write) keeps the handle and leaves health alone; batch read health is worst-wins with `LastSuccessfulRead` advancing on any success. AbLegacy `WriteRefusal` before runtime creation (`:546-550`) + `LibplctagLegacyName` consistent across transports. Native STRING decode clamps LEN and handles the 88-byte padding; bit access and `Span` bounds-checked. `GetHostStatuses()` from the actor is try/catch-guarded (`CurrentHostStatuses`). `IRediscoverable` absent on AbCip is a documented gap.
**TwinCAT / FOCAS:** ADS notification/variable handle lifetimes are balanced — `NativeRegistration.Dispose` deletes the AMS notification (`TwinCATDriver.cs:875-880` → `AdsTwinCATClient.cs:646-651`), replay swaps and disposes the old handle (`:1281-1282`), replay and probe-retry retract an orphan created by a racing unsubscribe (`:1286-1291, 1348-1353`), `ShutdownAsync`/`Dispose` dispose registrations before clients, `AdsTwinCATClient.Dispose` detaches the event, stops the dispatch loop and disposes the `AdsClient` (`:821-835`); `ReadRawAsync` variable handles deleted in `finally` (`:421-434`); failed connect disposes its client (`:1240-1248`); stale client disposed before replacement (`:1219-1224`); `RecycleClientAsync` runs under the gate. Sum-read classification matches #645. Codecs: `TwinCATStringCodec` NUL-terminates with 1252/UTF-8 by target; `TwinCATArrayCodec` throws on a short buffer and the notification path drops rather than publishes (`:612-615`); `TwinCATTemporalCodec.Encode` range-checks. FOCAS re-init/shutdown: all three loops cancelled + bounded-joined (5 s), CTSs disposed, client disposed; alarm projection paused not re-created; tag tables built fresh and swapped (`:238-318`); connect serialised per device with a `ReferenceEquals` guard (`:1855-1865`). FOCAS alarm health contract (#647) implemented as documented (`:485-499`). FOCAS wire parsers bounds-check via `RequireLength`; a mid-request failure closes both sockets; `ReadExactlyAsync` disposes the stream on cancellation; `SynchronizedFocasClient` pairs the gate with a per-call timeout. PMC coalescing memo bounded by the tag set. `AdsTwinCATClient.ConnectAsync` ignoring the token for the blocking SDK `Connect` is documented as the un-fixed root. Unverified assumption worth a wire capture: `DescribeSymbolAsync` per read relies on the Beckhoff symbol cache making `ReadSymbolAsync` local after the first call (`AdsTwinCATClient.cs:368-372`).
**PollGroupEngine (shared):** one task per subscription; only the loop's own token exits; cancel + join (≤ 5 s) before CTS dispose; `LastValues` per subscription; reader-contract violation throws loudly.
**Runtime seam:** `DriverInstanceActor.PostStop` calls `ShutdownAsync` only — the sync `IDisposable.Dispose()` overloads (`GetAwaiter().GetResult()`) on AbCip/Modbus/FOCAS have no production caller.
@@ -0,0 +1,226 @@
# Review 08 — Historian backends (Gateway / SqlServer / QuestDb / Sql.Common), Client.*, Analyzers
Repo: `/Users/dohertj2/Desktop/OtOpcUa` @ `d51d35bd` (master). Read-only review; every finding below was verified by reading the cited code. Line numbers are from the files as of this commit.
Severity: Critical / High / Medium / Low. Confidence: High = the code path is unambiguous; Medium = depends on runtime/SDK/gateway behaviour I could not execute here but which the repo's own comments or the package docs assert.
---
## 1. Memory / resource leaks
### 1.1 Uncapped raw HistoryRead (`NumValuesPerNode = 0`) materialises a tag's whole window in memory on every backend
- **Severity:** High **Confidence:** High
- **Location:**
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs:99-126`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs:101-129`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs:213-216, 285-305` and `Mapping/SampleMapper.cs:69-75`
- Node-manager side (outside slice, confirms there is no paging): `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs:116` ("`NumValuesPerNode == 0` … never page; return everything in one shot"); SqlServer/QuestDb report `MaxRawValuesPerRead => 0` (SqlServerHistorianDataSource.cs:324, QuestDbHistorianDataSource.cs:365).
- **Evidence:**
```csharp
// SqlServer
var top = maxValuesPerNode == 0 ? int.MaxValue : (int)Math.Min(maxValuesPerNode, int.MaxValue);
...
$"SELECT TOP (@top) {SampleColumns} FROM {_conn.Q("Sample")} WHERE TagId = @id AND {where} ORDER BY SourceTimeUtc {order}"
while (await reader.ReadAsync(...)) samples.Add(new DataValueSnapshot(...)); // List<DataValueSnapshot>, unbounded
// Gateway
if (cap == 0) { var all = await PassAsync(int.MaxValue) ... return all.Result.Samples; } // List<HistorianSample>
var snapshots = SampleMapper.ToSnapshots(samples); // second full copy
```
- **Failure scenario:** Any admitted session (reads are ungated for Anonymous with `Security:Acl:Enabled=false`, the default) issues HistoryRead-Raw with `NumValuesPerNode = 0` over a wide window on a dense tag (a year of 1 Hz = 31.5M rows). SqlServer streams the clustered-key seek into a `List<DataValueSnapshot>` (boxed value + two DateTimes + status ≈ 100+ B each ⇒ several GB), the Gateway path builds the protobuf list *and* the snapshot list, then the SDK encodes the whole response. `MaxTimeWeightedRawRows` caps only processed reads; `HistoryReadDeadline` (60 s) is time, not rows, and a key seek fills GBs in well under 60 s. The server's `TransportQuotas` are SDK defaults (`OpcUaApplicationHost.cs:472`), so the oversized response is rejected *after* the allocation. With `MaxConcurrentHistoryReadBatches = 16` this is a trivial memory-exhaustion vector, and it also works on the Gateway backend when the gateway is in streaming mode (the default `AllowStreamingBeyondCeiling`).
- **Suggested fix:** Treat `0` as "server-chosen page size" (Part 11 allows a server to return fewer values plus a continuation point): add a `ServerHistorian:MaxRawValuesPerRead`-style hard cap (advertise it as `HistoryServerCapabilities/MaxReturnDataValues`) and route `N = 0` through the existing `RawPageLimit` paging. Independently, bound the backends' `TOP/LIMIT`/`max_values` by that cap so a misbehaving caller cannot bypass the node manager.
### 1.2 Gateway events read drops `maxEvents` on the wire and abandons the gRPC stream at the cap
- **Severity:** High **Confidence:** Medium (the eviction behaviour is asserted by the repo's own comments and the gateway's `HistorianRpcFaults`, not re-verified here)
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs:111-124`; `GatewayHistorianDataSource.cs:576-598`; contrast `RawAggregateReader.cs:18-24` and `GatewayHistorianDataSource.cs:202-204`.
- **Evidence:**
```csharp
// Adapter: maxEvents is accepted and never forwarded (the 0.3.0 package API has no max parameter)
public IAsyncEnumerable<HistorianEvent> ReadEventsAsync(string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken ct)
{ ... return _inner.ReadEventsAsync(startUtc, endUtc, filter, ct); }
// Data source: breaks out of the server stream once the cap is full
if (hasCap && collected.Count == maxEvents) { truncated = true; break; }
```
while the raw path documents: "Every stream is consumed to completion (never broken out of — a cancelled gateway stream evicts its pooled historian session)" and `RawAggregateReader` says "the gateway currently classifies that cancellation as a connection fault and evicts its shared historian session (HistorianGateway `HistorianRpcFaults.IsConnectionFault`: any `RpcException` ⇒ true)".
- **Failure scenario:** Every capped HistoryReadEvents that has more than `NumValuesPerNode` events in its window (the normal case for a consumer paging alarm history, and for the #691.4 reverse walk-back which issues many small capped reads) cancels the server stream mid-way ⇒ the gateway evicts its historian session ⇒ the next read of *any* kind pays a reconnect, and concurrent raw reads sharing that session fail. Uncapped reads (`int.MaxValue`) additionally stream the entire window regardless of what the client asked for.
- **Suggested fix:** Drain the stream to completion with a bounded budget like the raw path (count past the cap without collecting), and add a `max_events` field to the gateway RPC so the gateway stops streaming at the cap; until then at least set `EventReadHorizon` guidance in docs.
### 1.3 `FasterLogHistorizationOutbox.PeekBatchAsync` holds a monitor across FasterLog disk reads, blocking the recorder actor's appends
- **Severity:** Low **Confidence:** High
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/Recorder/FasterLogHistorizationOutbox.cs:215-252` (peek) and `:147-156` (append index under the same lock); drain runs off-actor via `DrainOnceAsync(...).PipeTo` at `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:696`.
- **Evidence:**
```csharp
lock (_state)
{
...
using FasterLogScanIterator iter = _log.Scan(_nextScanAddress, _log.TailAddress, recover: false);
while (batch.Count < max && iter.GetNext(out byte[] bytes, out int len, out long address, out long next))
```
- **Failure scenario:** `FasterLog` is built with default settings (`new FasterLogSettings { LogDevice = _device }`, line 76), i.e. an ~8 MB in-memory window; `Capacity` defaults to 500 000 entries (≈35 MB on disk). Once the backlog's head has aged out of memory (any multi-minute gateway outage), `GetNext` reads pages from the device synchronously while `_state` is held; the actor thread's `AppendBatchAsync` then blocks on `lock (_state)` for the duration of the read — a sync block on an Akka dispatcher thread, once per 2 s drain pass, exactly during outage recovery.
- **Suggested fix:** Scan outside the lock (snapshot `_nextScanAddress` + the live-list cursor under the lock, iterate unlocked, re-take the lock only to flag/drop unreadable head records), or size `MemorySizeBits` to the configured capacity.
### 1.4 Client raw history read leaves the server continuation point unreleased when stopped by the client cap
- **Severity:** Low **Confidence:** High
- **Location:** `src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs:175-206` vs the modified-read path `:263-272`.
- **Evidence:** `while (continuationPoint != null && continuationPoint.Length > 0 && allValues.Count < maxValues);` then `return allValues;` — no `releaseContinuationPoints` call; `HistoryReadModifiedAsync` explicitly releases ("A read stopped by the client's own cap leaves a server cursor behind; release it.").
- **Failure scenario:** A page that returns exactly `maxValues` (the common case on a dense tag) carries a continuation point the client never resumes or releases. Bounded in practice: the server's store is per-session and capped at `MaxHistoryContinuationPoints` (100, oldest-evicted, `OpcUaServer/HistoryContinuationStore.cs:50-55`), so the Client.UI's long-lived session just cycles 100 slots; the CLI disconnects immediately.
- **Suggested fix:** Mirror the modified path's release call.
### 1.5 Client subscription adapters delete server-side but never remove the `Subscription` from the SDK `Session`
- **Severity:** Low **Confidence:** Medium
- **Location:** `src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs:321-333` (`_session.AddSubscription(subscription)`), `DefaultSubscriptionAdapter.cs:104-117, 122-133` (`_subscription.DeleteAsync(true, ct)` only).
- **Failure scenario:** `OpcUaClientService.SubscribeAlarmsAsync/UnsubscribeAlarmsAsync` and `MainWindowViewModel.MonitorAlarmsForSelectedNodeAsync` create a fresh SDK `Subscription` per cycle; each is deleted on the server but stays in `Session.Subscriptions` until the session is disposed, so a UI session that re-targets alarms repeatedly accumulates dead `Subscription` objects (and anything the SDK does over `Session.Subscriptions`, e.g. transfer/recreate on its own reconnect, iterates them).
- **Suggested fix:** Call `_session.RemoveSubscription(subscription)` after a successful delete (the adapter would need the session reference), or create subscriptions through `Session.RemoveSubscription`-aware helpers.
---
## 2. Unimplemented / stubbed / silently wrong
### 2.1 Gateway native `TimeAverage` / `Total` are served with **Good** status although the repo documents the values are wrong until `historiangw#27`
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/NativeAggregatePlan.cs:39-43`; served via `GatewayHistorianDataSource.cs:338-343, 350-387`; advertised in `ServedAggregates` (`:40-43`).
- **Evidence:**
```
⚠️ Today historiangw#27 (the interval reaches AVEVA in ticks where it reads milliseconds) makes AVEVA answer ONE
Good row per request — the whole-intervals read lands on its earliest interval with a whole-span value and the rest
are absent — in both directions.
```
(`.claude/rules/historian.md` agrees: "until historiangw#27 lands both directions return Good-status wrong values".)
- **Failure scenario:** A client asks for 24 hourly `TimeAverage` intervals and receives one Good, whole-day value stamped on the first interval and 23 absent intervals; nothing on the wire says the answer is degraded. ScadaBridge or a report will store it as truth.
- **Suggested fix:** Until the gateway fix ships, either drop `TimeAverage`/`Total` from `ServedAggregates` (⇒ `BadAggregateNotSupported`, consistent with the other five time-weighted aggregates on this backend) or mark every row `UncertainDataSubNormal` + log once. Gate the native path behind a config flag that defaults off until #27 is live-verified.
### 2.2 Gateway backend capability gaps vs. SqlServer/QuestDb (documented, but a consumer-visible matrix)
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianProvider.cs:54` (`CreateUpdater => null`), `GatewayHistorianDataSource.cs:40-43` (12 of 17 aggregates), `:125-126` and `:556-562` (reverse raw/events `NotSupportedException`), `ReadModifiedAsync` not overridden (interface default ⇒ unsupported), no `CreateRetention`, `GatewayHistorianValueWriter.cs:28-38` (quality discarded by the gateway — "C-002"), `Mapping/HistorianTypeMapper.cs` (String/DateTime not provisioned).
- **What a client sees on Gateway only:** HistoryUpdate ⇒ `BadHistoryOperationUnsupported`; `IsReadModified` ⇒ unsupported; `TimeAverage2/Total2/Interpolative/PercentGood/PercentBad` ⇒ `BadAggregateNotSupported`; every stored sample reads back Good (comms-loss markers are dropped, driver Bad statuses are lost); reverse/end-only raw and events reads are emulated by forward walks (`ReverseRawReader`/`ReverseEventsReader`) that cost many round trips on sparse tags.
- **Suggested fix:** Nothing to "implement" in this repo for most of it (gateway-side), but publish the matrix in `docs/Historian.md` and have the AdminUI backend picker show it; the quality loss in particular deserves a startup Warning when `ContinuousHistorization` is enabled on Gateway.
### 2.3 Client CLI/UI offer `StandardDeviation`, which no backend serves, and omit six aggregates that are served
- **Severity:** Low **Confidence:** High
- **Location:** `src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/AggregateType.cs` (12 members incl. `StandardDeviation`); `Client.CLI/Commands/HistoryReadCommand.cs:50, 202-221`; `Client.UI/ViewModels/HistoryViewModel.cs:47-62`. Served set: `Sql.Common/AggregateFamily.cs:50-51` (`FamilyOf != null` ⇒ 17: Average/Min/Max/Count/Range/Start/End/Delta/Min-MaxActualTime/TimeAverage/TimeAverage2/Total/Total2/Interpolative/PercentGood/PercentBad).
- **Failure scenario:** `historyread --aggregate stddev` always returns `BadAggregateNotSupported`; `Range`, `Delta`, `MinimumActualTime`, `MaximumActualTime`, `PercentGood`, `PercentBad` cannot be requested from the project's own client at all, so the gate recipes in `docs/Historian.md` cannot be run with Client.CLI.
- **Suggested fix:** Align `AggregateType` + `AggregateTypeMapper` with `AggregateFamilies.Served`.
### 2.4 `ProbeAsync` / `GetConnectionStatusAsync` exist on the gateway seam but health is derived from read outcomes only
- **Severity:** Low **Confidence:** High
- **Location:** `IHistorianGatewayClient.cs:106, 111`; `GatewayHistorianDataSource.cs:616-638` (`ProcessConnectionOpen: _totalSuccesses > 0 && _consecutiveFailures == 0`).
- **Failure scenario:** A freshly booted node with a healthy gateway reports the historian as closed until some client performs a HistoryRead; a node that only *writes* (continuous historization) never flips it. Operators reading `/hosts` get a false negative.
- **Suggested fix:** Use `ProbeAsync` on a slow timer (or on `GetHealthSnapshot` with a cached result) now that the seam carries it.
### 2.5 Two Gateway read arms record a caller cancellation as a backend failure
- **Severity:** Low **Confidence:** High
- **Location:** `GatewayHistorianDataSource.cs:607-612` (`ReadEventsAsync`) and `:519-524` (`ReadAtTimeNativeAsync`) lack the `catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) throw;` that `ReadRawAsync`/`ReadProcessedAsync`/`ReadAtTimeAsync` have (`:141-144, 377-380, 484-487`).
- **Failure scenario:** A `HistoryReadDeadline` expiry or client abort during an events read bumps `TotalFailures`/`ConsecutiveFailures` and flips `ProcessConnectionOpen` to false ⇒ health alarms on a healthy gateway.
---
## 3. Performance limiters
### 3.1 QuestDB live-write path: three round trips plus a 10 ms-poll visibility wait per tag group per drain pass
- **Severity:** Medium **Confidence:** High
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs:91-102, 128-143`; `QuestDbWalVisibility.cs:49-91`.
- **Evidence:**
```csharp
await _catalog.EnsureAsync(tag, dataType, ct); // cached after first
var present = await ExistingTicksAsync(tag, rows, ct); // SELECT ts ... per cluster
for (...) await InsertAsync(...); // multi-row INSERT
await _wal.WaitAsync(_conn.SampleTable, ct); // SELECT sequencerTxn, then poll writerTxn every 10 ms ≤ 5 s
```
- **Failure scenario:** With `DrainConcurrency = 8` and WAL apply lagging under load, up to eight pollers each issue ~100 `wal_tables()` queries/s (≈800/s) against the same database that is already behind — the wait amplifies load exactly when the DB is slowest, and every tag group pays ≥3 round trips, so a 5 000-tag deployment at 1 s cadence spends `⌈5000/8⌉ × (3 RTT + apply latency)` per pass and falls behind the 2 s cadence. The existence pre-read is only needed for first-wins idempotency across redeliveries, and the visibility wait only so that an *immediate* redelivery sees the rows.
- **Suggested fix:** Wait once per drain pass (expose a `FlushAsync`/pass-end hook on `IHistorianValueWriter`, or let the recorder call `WaitAsync` once after `Task.WhenAll`), back the poll off exponentially (10→50→200 ms), and consider a per-pass existence read keyed by the outbox batch rather than per tag.
### 3.2 All three alarm-history writers insert one event per round trip, serially
- **Severity:** Medium **Confidence:** High
- **Location:** `Gateway/GatewayAlarmHistorianWriter.cs:66-78` (one `SendEventAsync` per event); `SqlServer/SqlServerAlarmHistorianWriter.cs:55-59, 76-89` (new pooled connection + single-row `INSERT … WHERE NOT EXISTS` per event); `QuestDb/QuestDbAlarmHistorianWriter.cs:64-69, 89-105`.
- **Failure scenario:** The `LocalDbStoreAndForwardSink` drains `BatchSize` events per tick; an alarm storm of 10 000 events drains at `10 000 × RTT` (≥ 10–20 s on SQL, minutes over a WAN to the gateway), and the Primary-only drain means the backlog sits in LocalDb meanwhile. The per-event design exists to isolate poison events, but SqlServer already has the TVP + `OUTPUT` machinery to get per-row outcomes from one statement.
- **Suggested fix:** SqlServer: one `INSERT … SELECT FROM @events WHERE NOT EXISTS …` with `OUTPUT inserted.EventId` and map outcomes by id (fall back to per-row only on a data error). QuestDb: multi-row INSERT (DEDUP key already converges duplicates). Gateway: needs a batch `SendEvents` RPC.
### 3.3 SQL Server retention issues O(tags) `DELETE TOP` statements per purge even when nothing is eligible
- **Severity:** Low **Confidence:** High
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs:89-107, 134-149`.
- **Evidence:** `SELECT TagId FROM Tag` then `foreach (var id in tagIds) { DeleteLoopAsync(Sample…); if (trail) DeleteLoopAsync(SampleModified…); }` — the loop always runs at least one statement per table per tag, and the applock is held for the whole pass.
- **Failure scenario:** 50 000 historized tags ⇒ ≥100 000 statements per hourly purge (plus the full `Event` scan) on a database that has nothing to delete, serialised on one connection while other nodes' purges skip.
- **Suggested fix:** Pre-filter: `SELECT TagId FROM Tag t WHERE EXISTS (SELECT 1 FROM Sample s WHERE s.TagId = t.TagId AND s.SourceTimeUtc < @cutoff)` (a key seek per tag inside one statement), or keep a per-tag `OldestSampleUtc` high-water mark.
### 3.4 QuestDB `event.source` and `sample_modified.tag` symbols carry no index
- **Severity:** Low **Confidence:** High
- **Location:** `QuestDb/QuestDbTables.cs:73-77` (`source SYMBOL`), `:95-99` (`tag SYMBOL`), vs the sample table's `tag SYMBOL INDEX` added for #683 (`:64-67`, `:116`).
- **Failure scenario:** `ReadEventsAsync` with a `sourceName` filter (`QuestDbHistorianDataSource.cs:402-407`) and every `ReadModifiedAsync` (`:183-185`) scan all partitions in the window rather than seeking; the #683 gate measured the same shape costing seconds on the sample table.
- **Suggested fix:** Add `INDEX` on both symbols in the DDL plus the same idempotent `ALTER … ADD INDEX` migration the bootstrapper already runs for `sample.tag`.
### 3.5 `GatewayTagPresenceCache` adds a catalog-browse RPC before HistoryReads, keyed by a glob filter
- **Severity:** Low **Confidence:** Medium
- **Location:** `Gateway/GatewayTagPresenceCache.cs:16-32` (60 s positive / 5 s negative TTL, 4 096 entries); `HistorianGatewayClientAdapter.cs:85-93` (`BrowseTagNamesAsync(tag, ct)` drained fully). Package doc (0.3.0 XML): `BrowseTagNamesAsync(filter)` — "Tag name filter pattern accepted by the historian (e.g. `Sys*`)".
- **Failure scenario:** A consumer sweeping history for N tags pays N extra streaming RPCs per minute of sweep; with more than 4 096 distinct tags the LRU-less eviction (`_entries.Keys.First()`) thrashes. A historian tag name containing `*`/`?` (legal in AVEVA) streams every matching catalog entry on each check.
- **Suggested fix:** Use the package's `GetTagMetadataAsync` (exact lookup) now that the seam exists, lengthen the positive TTL, or drop the pre-check and treat an empty raw read as "no data" (which it already does).
### 3.6 Client: N+1 browse in `BrowseAsync`, and O(n) scan per notification in the UI
- **Severity:** Low **Confidence:** High
- **Location:** `Client.Shared/OpcUaClientService.cs:223-242` (`HasChildrenAsync` ⇒ one Browse per Object child); `Client.UI/ViewModels/SubscriptionsViewModel.cs:61-73` (`foreach (var item in ActiveSubscriptions) if (item.NodeId == e.NodeId)` on the UI thread per notification).
- **Failure scenario:** `subscribe --recursive` / the UI's recursive subscribe over a 50 000-node raw tree issues ~50 000 Browse calls to find children; 2 000 subscribed items at 1 Hz = 4 M string compares/s on the Avalonia UI thread.
- **Suggested fix:** Browse with `includeSubtypes`/`resultMask` carrying `NodeClass` only and infer `HasChildren` lazily on expand; index `ActiveSubscriptions` by NodeId in a dictionary.
---
## 4. Other hazards
### 4.1 Every production `IHistorianValueWriter` swallows `OperationCanceledException`, so the recorder's shutdown guard is dead code
- **Severity:** Medium **Confidence:** High
- **Location:** `Gateway/Recorder/GatewayHistorianValueWriter.cs:112-120`; `SqlServer/SqlServerHistorianValueWriter.cs:93-97`; `QuestDb/QuestDbHistorianValueWriter.cs:104-108` — all `catch (Exception) { …; return false; }`. The consumer expects the opposite: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:744-755`:
```csharp
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
// Shutdown cut this write short. ... swallowed as `ok = false` it becomes a consecutive failure, and when a
// sibling group already wrote this pass the fold books it — so a routine node stop can tip a perfectly
// HEALTHY tag over its attempt budget and discard its queued samples on the way out. Rethrow ...
throw;
}
```
- **Failure scenario:** At shutdown mid-pass the cancelled groups return `false`; if any sibling already wrote, `_tagWriteFailures[tag]++` for each cancelled tag (recorder `:826-834`) — precisely the outcome the guard was written to prevent. Today the damage is bounded (+1 per process stop, counters are in-memory, `_lifetime` cancels only in `PostStop`, recorder `:340`), so no data is lost, but the contract between writer ("never throws") and recorder ("rethrow cancellation") is contradictory and the next refactor that cancels drains without restarting (a capture-gate close, a backend switch) will discard healthy tags' backlog after `MaxWriteAttempts` such events.
- **Suggested fix:** `catch (OperationCanceledException) when (ct.IsCancellationRequested) { throw; }` ahead of the catch-all in all three writers (the alarm writers already special-case it), and state it on `IHistorianValueWriter`.
### 4.2 Gateway value writer relabels a Local-kind timestamp as UTC instead of converting it
- **Severity:** Medium **Confidence:** Medium (depends on a driver ever producing `DateTimeKind.Local`)
- **Location:** `Gateway/Recorder/GatewayHistorianValueWriter.cs:93-98`:
```csharp
live.Timestamp = Timestamp.FromDateTime(DateTime.SpecifyKind(timestampUtc, DateTimeKind.Utc));
```
vs the SQL backends' shared normaliser `Sql.Common/LiveWritePreparer.cs:31`: `t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc);`.
- **Failure scenario:** A driver that stamps `DateTime.Now`-derived source times (or an OpcUaClient upstream server returning Local) historizes correctly on SqlServer/QuestDb but shifted by the node's UTC offset on Gateway — and the HistoryUpdate path on SQL backends keeps the kind (`HistoryUpdatePlanner` ⇒ `LiveWritePreparer.Truncate` only), so the same value can land at two different times on two backends.
- **Suggested fix:** Run gateway writes through `LiveWritePreparer.Prepare` (it already does the non-finite ⇒ Empty and dedup work the gateway path lacks), or at minimum `ToUniversalTime()` for `Local`.
### 4.3 SqlServer `DeleteRaw` removes an unbounded range in a single statement/transaction with `OUTPUT INTO`
- **Severity:** Low **Confidence:** High
- **Location:** `SqlServer/SqlServerHistorianDataUpdater.cs:99-113`.
- **Failure scenario:** `DELETE FROM Sample OUTPUT … INTO SampleModified WHERE TagId=@id AND SourceTimeUtc >= @s AND < @e` over years of a 1 Hz tag deletes tens of millions of rows and writes as many audit rows in one transaction: lock escalation on `Sample` (blocking the live writer), transaction-log growth, and a 30 s `CommandTimeout` that aborts and rolls back the whole thing — the retention service deliberately batches `DELETE TOP (n)` for this reason. QuestDB's updater already pages (`QuestDbHistorianDataUpdater.cs:135-166`).
- **Suggested fix:** Page the delete (`DELETE TOP (@n) … OUTPUT …` in a loop under one transaction or per-batch commits; the OPC UA result is a single status either way).
### 4.4 QuestDB HistoryUpdate: non-atomic trail + per-instance locks + duplicating retries (documented residuals)
- **Severity:** Low **Confidence:** High
- **Location:** `QuestDb/QuestDbHistorianDataUpdater.cs:18-21, 28-34, 155-160, 235-240`; `.claude/rules/historian.md` ("a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key)").
- **Failure scenario:** A crash or failed sample INSERT after `WriteTrailAsync` leaves orphan `sample_modified` rows describing changes that never landed; a second node (or the live writer) racing the same key bypasses the 64 striped `SemaphoreSlim`s; a Npgsql timeout retry duplicates trail rows. All documented; listed so the matrix is complete.
- **Suggested fix:** Record a `applied` marker (a second append with the same `modified_id`) after the sample write so `ReadModifiedAsync` can drop orphans; use a DEDUP key `(ts, tag, modified_id)` on the trail.
### 4.5 Client: failover nulls subscription state outside the lock the rest of the class relies on
- **Severity:** Low **Confidence:** High
- **Location:** `Client.Shared/OpcUaClientService.cs:734-735` (`_dataSubscription = null; _alarmSubscription = null;` in `RunFailoverAsync`, no lock) vs `:260-263` (`SubscribeAsync` reads/creates `_dataSubscription` unguarded) and the class comment `:18-22` that promises all bookkeeping is under `_subscriptionLock`.
- **Failure scenario:** A `SubscribeAsync` racing a keep-alive failover can add a monitored item to the dying session's subscription (silently lost, not replayed) or observe a null and create a second subscription on the new session that `ReplaySubscriptionsAsync` then duplicates.
- **Suggested fix:** Take `_subscriptionLock` around the null-outs and guard `SubscribeAsync` with the same snapshot pattern `DisconnectAsync` uses; also `ConnectAsync` (`:97-128`) should refuse (or disconnect) when already connected instead of overwriting `_session`.
---
## 5. Checked and fine (looked like problems, are not)
- **SQL injection:** every dynamic identifier is regex-validated before splicing — SqlServer `Schema` `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (`SqlServerHistorianOptions.cs:49-59`, `SqlHistorianConnection.Q` brackets it, `$(Schema)` substituted only after validation), QuestDb `TablePrefix` `^[A-Za-z0-9_]+$` (`QuestDbHistorianOptions.cs:32-38`, re-checked in `QuestDbTables.Name`), `table_columns('{table}')` uses that validated name, QuestDB `AtTimeSql` inlines only generated ints, TTL days is `int`. All values (tag names, times, principals, event fields) are parameters; the SqlServer alarm writer deliberately uses `NVARCHAR(MAX)` params so truncation is a server error, not a silent clip.
- **Continuous-historization buffer bound:** the FasterLog outbox is disk-backed with `Capacity` (500 000 default) drop-oldest and `DroppedCount`/`UnreadableCount` metrics; drop-oldest truncates + commits, corrupt head records cannot pin the head (`FasterLogHistorizationOutbox.cs:172-200, 243-250`). Not unbounded.
- **Flush on shutdown:** `PerEntry` commit mode fsyncs before `AppendBatchAsync` returns; `Dispose` cancels the periodic loop, awaits it, then `Commit(spinWait: true)` (`:365-406`). Appends in flight at process kill are the documented `Periodic` trade-off only.
- **Time-weighted aggregates stream, they do not materialise:** both SQL backends feed `TimeWeightedAggregator` row by row with `TOP/LIMIT (cap + 1)` and `CheckWindowRows` fails the node one row past `MaxTimeWeightedRawRows`; memory is O(buckets) (`SqlServerHistorianDataSource.cs:528-550`, `QuestDbHistorianDataSource.cs:545-579`, `TimeWeightedAggregator.cs:132-154`). Gateway per-sample aggregates likewise accumulate into `RawBucketAccumulator` (O(buckets)).
- **`BucketPlan.MaxBuckets = 100 000`** bounds every processed read's result list and the aggregator arrays before any I/O.
- **Tag catalogs** (`SqlServerTagCatalog._cache`, `QuestDbTagCatalog._known`) are positive-only and never pruned, but are bounded by the number of distinct historian tag names ever written on this node (≤ a few hundred KB); a renamed tag leaves one stale int entry. Not a leak worth fixing.
- **Gateway channels:** four `HistorianGatewayClientAdapter`s (data source, alarm writer, provisioner, value writer) each own a `GrpcChannel`, each disposed by its owner (`DisposeAsync` on all four). Documented as deliberate; a co-located sidecar tolerates four HTTP/2 connections.
- **`GatewayRowCeiling`** is a single `int` + ticks with `Volatile`/`Interlocked`; a race costs one extra refusal, never a wrong answer. `GatewayAtTimeReader` is capped at `64 + 16 × times` reads and `1 024` rows per read, so a pathological tag fails `BadTooManyOperations` rather than looping.
- **Schema bootstrap:** `SqlServerSchemaBootstrapper`/`QuestDbSchemaBootstrapper` use `SemaphoreSlim(1,1)` + a 5 s negative cache (`BootstrapFailureCache`), so an outage does not serialise every call behind one round trip; the SqlServer applock is session-owned and the pool is cleared when release fails (otherwise the lock would ride a pooled connection forever) — same pattern in retention.
- **Timestamps on the SQL backends:** `datetime2(7)` with explicit UTC `SpecifyKind` on read/write; QuestDB rounds to µs and nudges `Min/MaxValue` so Npgsql never binds ±infinity; pre-1970 is refused/dropped with counters rather than failing a batch. `IHistorianDataSource.TimestampResolutionTicks = 10` lets the paging step correctly on QuestDB.
- **`GatewayHistorianDataSource.Dispose()`** is sync-over-async (`DisposeAsync().AsTask().GetAwaiter().GetResult()`) — shutdown-only, channel shutdown is quick; acceptable.
- **Analyzer (`UnwrappedCapabilityCallAnalyzer`)**: registers nothing when the guarded interfaces are absent from the compilation, resolves symbols once per compilation, and walks ancestors only up to the enclosing wrapper invocation; the one acknowledged limitation (any lambda argument to `ExecuteAsync` suppresses) is documented and currently unreachable. No production hazard.
- **Client `DisposeBridge`** bounds every sync-over-async teardown (`CloseAsync`/`DeleteAsync`) at 5 s and cancels the token when the wait loses, so a dead server cannot hang `Dispose`.
- **`HistorizationOutboxEntrySerializer`** is bounds-checked for both layouts (negative length prefixes, short records, invalid DateTime binary, non-0/1 booleans all return `false`), so a corrupt record is counted and skipped, never thrown.
+56
View File
@@ -0,0 +1,56 @@
# Baseline 2026-10-22 @ ad2b5c40 — Release, filter Category!=E2E&Category!=LiveIntegration
# Result: all assemblies passed (0 failed). Run concurrently with 6 Wave-1 worktree builds.
Passed! - Failed: 0, Passed: 126, Skipped: 0, Total: 126, Duration: 251 ms - ZB.MOM.WW.OtOpcUa.Client.UI.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 202, Skipped: 0, Total: 202, Duration: 1 s - ZB.MOM.WW.OtOpcUa.Client.Shared.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 16, Skipped: 0, Total: 16, Duration: 91 ms - ZB.MOM.WW.OtOpcUa.Core.VirtualTags.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 138, Skipped: 0, Total: 138, Duration: 2 s - ZB.MOM.WW.OtOpcUa.Core.Scripting.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 51, Skipped: 0, Total: 51, Duration: 5 s - ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 359, Skipped: 0, Total: 359, Duration: 7 s - ZB.MOM.WW.OtOpcUa.Commons.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 95, Skipped: 0, Total: 95, Duration: 8 s - ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 455, Skipped: 0, Total: 455, Duration: 6 s - ZB.MOM.WW.OtOpcUa.Driver.Ab.Transport.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 437, Skipped: 0, Total: 437, Duration: 8 s - ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 359, Skipped: 1, Total: 360, Duration: 7 s - ZB.MOM.WW.OtOpcUa.Core.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 21, Skipped: 0, Total: 21, Duration: 196 ms - ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 66, Skipped: 0, Total: 66, Duration: 4 s - ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 1, Skipped: 217, Total: 218, Duration: 249 ms[xUnit.net 00:00:10.55] ZB.MOM.WW.OtOpcUa.Driver.AbLegacy.IntegrationTests.AbLegacyIndirectAddressingTests.Slc500_indirect_word_read_and_write_resolve_through_the_index [SKIP]
Passed! - Failed: 0, Passed: 353, Skipped: 8, Total: 361, Duration: 780 ms - ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 752, Skipped: 0, Total: 752, Duration: 9 s - ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 194, Skipped: 0, Total: 194, Duration: 11 s - ZB.MOM.WW.OtOpcUa.Configuration.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 139, Skipped: 0, Total: 139, Duration: 12 s - ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 597, Skipped: 0, Total: 597, Duration: 9 s - ZB.MOM.WW.OtOpcUa.Driver.AbLegacy.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 81, Skipped: 0, Total: 81, Duration: 346 ms - ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 14, Skipped: 0, Total: 14, Duration: 4 s - ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests.dll (net10.0)
Passed! - Failed: 0, Passed: 269, Skipped: 0, Total: 269, Duration: 667 ms - ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 16, Skipped: 10, Total: 26, Duration: 10 s - ZB.MOM.WW.OtOpcUa.Driver.AbCip.IntegrationTests.dll (net10.0)
Passed! - Failed: 0, Passed: 511, Skipped: 0, Total: 511, Duration: 5 s - ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 1, Skipped: 39, Total: 40, Duration: 96 ms - ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests.dll (net10.0)
Passed! - Failed: 0, Passed: 161, Skipped: 0, Total: 161, Duration: 193 ms - ZB.MOM.WW.OtOpcUa.Driver.Modbus.Addressing.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 42, Skipped: 0, Total: 42, Duration: 4 s - ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 10, Skipped: 0, Total: 10, Duration: 1 s - ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 465, Skipped: 0, Total: 465, Duration: 3 s - ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 16, Skipped: 0, Total: 16, Duration: 5 s - ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests.dll (net10.0)
Passed! - Failed: 0, Passed: 254, Skipped: 0, Total: 254, Duration: 3 s - ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 47, Skipped: 0, Total: 47, Duration: 149 ms - ZB.MOM.WW.OtOpcUa.Driver.AbCip.Cli.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 48, Skipped: 0, Total: 48, Duration: 106 ms - ZB.MOM.WW.OtOpcUa.Driver.AbLegacy.Cli.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 84, Skipped: 0, Total: 84, Duration: 3 s - ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 363, Skipped: 0, Total: 363, Duration: 1 s - ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 49, Skipped: 0, Total: 49, Duration: 449 ms - ZB.MOM.WW.OtOpcUa.Driver.Cli.Common.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 52, Skipped: 0, Total: 52, Duration: 394 ms - ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Cli.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 563, Skipped: 0, Total: 563, Duration: 6 s - ZB.MOM.WW.OtOpcUa.Driver.S7.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 72, Skipped: 0, Total: 72, Duration: 245 ms - ZB.MOM.WW.OtOpcUa.Driver.Modbus.Cli.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 59, Skipped: 0, Total: 59, Duration: 88 ms - ZB.MOM.WW.OtOpcUa.Driver.S7.Cli.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 70, Skipped: 0, Total: 70, Duration: 263 ms - ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Cli.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 635, Skipped: 0, Total: 635, Duration: 21 s - ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 89, Skipped: 0, Total: 89, Duration: 6 s - ZB.MOM.WW.OtOpcUa.Security.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 420, Skipped: 0, Total: 420, Duration: 20 s - ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 22, Skipped: 0, Total: 22, Duration: 7 s - ZB.MOM.WW.OtOpcUa.Analyzers.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 250, Skipped: 0, Total: 250, Duration: 47 s - ZB.MOM.WW.OtOpcUa.Cluster.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 199, Skipped: 0, Total: 199, Duration: 16 s - ZB.MOM.WW.OtOpcUa.Host.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 1674, Skipped: 0, Total: 1674, Duration: 25 s - ZB.MOM.WW.OtOpcUa.AdminUI.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 825, Skipped: 0, Total: 825, Duration: 1 m 20 s - ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 453, Skipped: 0, Total: 453, Duration: 1 m 30 s - ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 837, Skipped: 3, Total: 840, Duration: 1 m 31 s - ZB.MOM.WW.OtOpcUa.Runtime.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 278, Skipped: 0, Total: 278, Duration: 1 m 37 s - ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 267, Skipped: 14, Total: 281, Duration: 3 m 39 s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
Passed! - Failed: 0, Passed: 992, Skipped: 1, Total: 993, Duration: 4 m 17 s - ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.dll (net10.0)
Passed! - Failed: 0, Passed: 124, Skipped: 0, Total: 124, Duration: 4 m 28 s - ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.dll (net10.0)
+63 -5
View File
@@ -1,16 +1,22 @@
# Security
> **Status lives in Gitea.** Open follow-ups and known gaps noted in this doc are tracked as issues on
> gitea.dohertylan.com/dohertj2/lmxopcua (label `migrated`, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.
> **v2 status (2026-05-26).** The four security concerns below are unchanged in v2.
> Paths + project names moved: `OtOpcUa.Server/Security/` → `OtOpcUa.Security/`
> (`Ldap/`, `Jwt/`, `Endpoints/AuthEndpoints.cs`), `OtOpcUa.Admin` is gone (its
> auth + role-grant pages live in `OtOpcUa.AdminUI`), and Admin auth policies
> register from `OtOpcUa.Host/Program.cs` via `AddOtOpcUaAuth`
> (`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs`) rather
> than in a separate Admin process. The Admin UI uses a **single Cookie
> authentication scheme** — there is no `AddJwtBearer` pipeline. The
> `Security:Jwt` section configures `JwtTokenService`, which mints a JWT at the
> `/auth/token` endpoint for **external** consumers (OPC UA clients / automation
> scripts); the cookie itself stores the `ClaimsPrincipal` directly. DataProtection
> than in a separate Admin process. The Admin UI session is a **Cookie**
> scheme; the cookie stores the `ClaimsPrincipal` directly. JWT is **opt-in by
> `Security:Jwt:SigningKey`** (validated at host start, ≥32 bytes): with a key,
> `/auth/token` mints an HS256 token for a **cookie session only** (a bearer
> cannot renew itself), and a first-party `AddJwtBearer` scheme (`Bearer`,
> `ValidAlgorithms = HS256`, parameters from `JwtTokenService.BuildValidationParameters`)
> accepts it on `/api/*` and every `RequireAuthorization` endpoint through the
> `OtOpcUa.CookieOrBearer` default policy scheme. Without a key neither exists. DataProtection
> keys persist to the shared Config DB (`PersistKeysToDbContext<OtOpcUaConfigDbContext>`)
> so cookies survive failover between admin-role nodes.
>
@@ -245,6 +251,25 @@ with two `Security:ApiKeys` knobs:
a completed lookup (a hit or an authoritative miss) is stored, and a cached hit whose `ExpiresUtc` has
since passed is re-read rather than served, so the TTL can never resurrect an expired key.
#### Pre-authentication guards (`KnownIdSet` / `MissBurst` / `MissRefillPerSecond`)
The negative cache above is keyed by **key id**, so a flood of *distinct*, well-formed `otua_<random>_x`
credentials missed it every time and cost one indexed SQL query per attempt — under the SDK's
`SessionManager`-wide lock, with no credential required (review 03 §3.3). `BoundedApiKeyStore` now puts two
guards in front of the store, bound from the same `Security:ApiKeys` section
(`ApiKeyStoreGuardOptions`, Host-owned):
| Key | Default | Meaning |
|---|---|---|
| `KnownIdSet` | `true` | Keep an in-memory set of every `ApiKeyId` the ConfigDb holds, refreshed every `StoreCacheSeconds` (single-flight; a failed refresh keeps the previous set). An id outside it is refused in memory — `otopcua.opcua.apikey_store.bounded{outcome=unknown_id}` — and the store is never asked. A key minted less than `StoreCacheSeconds` ago is refused until the next refresh (the same window `Enabled` revocation already has). `StoreCacheSeconds = 0` disables the set with the cache. |
| `MissBurst` | `100` | Token bucket on lookups that **reach** the store and **miss**, for when the set is unavailable (not loaded yet, or the store cannot list ids). Exhausted ⇒ unknown ids are refused without a store read (`{outcome=miss_budget}`, a Warning-free deny through the authenticator's catch-all). A hit refunds its token, so legitimate clients are never charged. `0` disables the budget. |
| `MissRefillPerSecond` | `20` | Refill rate of the miss bucket. |
The budget is **process-wide, not per remote endpoint**: the OPC UA SDK's impersonation callback (the
`IOpcUaUserAuthenticator` seam) does not carry the client's address, so per-endpoint accounting needs that
seam widened in the OpcUaServer project (a Wave 3 follow-up → #751). Neither guard applies to a driver-only node:
its `ArtifactApiKeyStore` is two in-memory dictionary probes and is never wrapped.
### LDAP bind flow (`OtOpcUaLdapAuthService`)
LDAP is configured under the `Security:Ldap` section (bound to `LdapOptions`, `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs`, `SectionName = "Security:Ldap"`). The app authenticator is `OtOpcUaLdapAuthService` — a thin wrapper around the shared `ZB.MOM.WW.Auth.Ldap` directory client that adds two app-only concerns the shared library deliberately does not model: the `Enabled` master switch and `DevStubMode`. Since #520 it serves the **Admin UI cookie login (`/auth/login`) only** — `LdapOpcUaUserAuthenticator` is deleted and the OPC UA UserName path authenticates API keys instead (see [API-key identity](#api-key-identity-opc-ua-data-plane)).
@@ -320,6 +345,14 @@ of a credential was wrong.
ConfigDb, so keys reach them inside the deployment artifact exactly as `NodeAcl` rows do. Clearing
`Enabled` takes effect on a node only once a new artifact reaches it.
**Galaxy alarm-ack identity per key (lmxopcua#749).** mxaccessgw acknowledges Galaxy alarms as a real
Galaxy account, so a key can carry an optional Galaxy ack identity (AdminUI `/apikeys` → "Ack
identity"). It is held only in the encrypted Secrets store
(`otopcua/apikeys/<keyId>/galaxy-ack-{user,password}`), never in the ConfigDb or the artifact; keys
without one and anonymous sessions use the Galaxy driver's `AlarmAck` fallback account, and with
neither the ack is refused before it reaches the gateway. Deleting a key on `/apikeys` removes its
identity. See [docs/drivers/Galaxy.md § Alarm acknowledge credential](drivers/Galaxy.md#alarm-acknowledge-credential).
### Anonymous is a normal principal
Anonymous sessions are **admitted by default** (`OpcUa:AllowAnonymous`, default `true` — flipping the
@@ -1116,6 +1149,31 @@ See `archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md` (R3-01) for the fu
record, and `docs/Configuration.md` §`MeshTransport` / §`ConfigSource`-`ConfigServe` and `docs/Telemetry.md`
for the affected listener ports.
### The known-cluster gate is fail-OPEN until central's first ClusterNode load (review 03 §4.2)
`CentralCommandAuthInterceptor` verifies an inbound deploy ack against the PSK of the cluster named in
`x-otopcua-cluster`. `MeshPskProvider` consults a **known-cluster gate** first — the set of cluster ids
central read from `ClusterNode` on its last contact refresh — and refuses anything outside it in O(1),
touching neither the config map nor the secret store. That gate is **inactive until the first successful
load** (`CentralCommunicationActor` publishes it from `ContactsLoaded`; an empty publication also
deactivates it, deliberately, so "the DB told me nothing" never becomes a fleet-wide authentication
outage). During a central boot with the ConfigDb down, therefore:
- an unauthenticated caller can name *any* `x-otopcua-cluster` value and make central perform one bounded
(3 s) secret-store read plus one Warning line per request — the cost/oracle the gate exists to remove;
- **keys are still required**: nothing is admitted without the matching PSK, so this is a cost exposure,
not an authentication bypass;
- the window is bounded by the bootstrap retry (15 × 2 s) and the `ContactRefreshSeconds` (60 s) periodic
refresh, and closes on the first load that succeeds. The actor logs the condition on every failed load
(*"The known-cluster gate was NOT refreshed either … it is INACTIVE"*).
Since review 03 §3.4 the per-cluster keys are **refreshed asynchronously** from the same handler
(`IMeshPskProvider.RefreshAsync`: concurrent bounded reads, atomic snapshot swap, a failed read keeps the
previous key) instead of being invalidated on every refresh, so the first ack per cluster per minute no
longer pays a synchronous store read on a gRPC thread. Accepted as documented, bounded behaviour; the
alternative — refusing every cluster until SQL answers — would turn a ConfigDb outage into a deploy-ack
outage on top of it.
---
## OTOPCUA0001 Analyzer — Compile-Time Guard
@@ -0,0 +1,568 @@
# System Review Remediation Implementation Plan
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development (recommended) or superpowers-extended-cc:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Resolve every finding from the 2026-10-21 8-slice Fable system review of OtOpcUa (master `d51d35bd`): 1 Critical, ~25 High, ~60 Medium, ~85 Low. These are memory/resource leaks, unimplemented features, performance limiters, and concurrency/health-contract hazards.
**Architecture:** The work is split into **file-disjoint lanes**. Each lane is executed by a subagent in its own git worktree off master, using TDD inside each task. Each lane gets an independent adversarial review by a model at least as strong as its implementer, then merges to local master in a fixed order. Lanes run in 4 waves. A lane only touches the files it owns. A finding whose fix needs a file another lane owns is sequenced into a later wave, not shared.
**Tech Stack:** .NET 10, Akka.NET, OPC Foundation UA-.NETStandard 1.5.378, EF Core/SQL Server, Roslyn scripting, Blazor Server, gRPC (mxaccessgw / HistorianGateway), xUnit + Shouldly.
**Spec:** the review reports. These are copied into the repo by Task 0 because the scratchpad is session-scoped:
`docs/reviews/2026-10-21/00-SUMMARY.md` … `08-historians-clients.md`.
Finding refs below use `R<report>:<id>`. For example, `R07:H10` is report 07, finding H10. Each finding in a report carries its location, evidence, failure scenario and a fix recipe, and that recipe is the starting design for its task.
## Context
The user asked for a parallel review hunting memory leaks, unimplemented features and performance limiters. Eight Fable reviewers produced verified findings, and the user now wants all of them resolved. The review found 8 cross-cutting patterns (summary §"Cross-cutting themes"). The plan fixes each pattern once, then sweeps the per-driver instances:
1. The runtime never disposes drivers. This is the root of the Critical Galaxy leak: each leaked gateway session pins an x86 worker for 30 min, so about 64 respawns exhaust `MaxSessions` (mxaccessgw cross-check).
2. Timed-out scripts orphan a thread per evaluation.
3. The health and watchdog contract is wrong in both directions across drivers.
4. A plain `Dictionary` is mutated under live poll loops.
5. One device can stall every device in its driver.
6. Array writes are advertised but unimplemented.
7. Config fields are authored but never consumed.
8. Mailbox-blocking `ReceiveAsync` calls.
## Global Constraints
- **Worktree hygiene:** every lane dispatch runs `git reset --hard <current master SHA>` in its worktree and verifies it with `git merge-base --is-ancestor <SHA> HEAD`, because worktrees start on stale commits. Each lane rebases on master before review.
- **File ownership is binding:** a lane edits only the paths in its "Owns" list. If a lane needs a change elsewhere, it creates an `F<N>` follow-up task (memory: track deviations as follow-ups, don't ask) instead of editing.
- **Fix the root, then the instance:** where a pattern has a shared fix (Lane A disposes drivers, Lane E quarantines scripts), driver lanes must still make their own `ShutdownAsync` a complete teardown. The runtime fix is a safety net, not a substitute.
- **Health contract (from `.claude/rules/drivers.md` + #527/#611):**
- A deterministic device refusal is per-tag Bad, with a once-per-tag Warning and no health write.
- Comms faults write `Degraded` and preserve `LastSuccessfulRead`.
- Any data proving the device answered advances `LastSuccessfulRead`.
- Recovery dedup is on `(value, status)`, never value-only.
Every driver lane is reviewed against this checklist.
- **Re-init safety:** device tables that are read by retained poll loops are build-and-swapped (`Volatile.Write`) or made `ConcurrentDictionary`, never `.Clear()`ed in place.
- **Array writes (user decision):** encode `count` elements at the resolved element width, refuse a length mismatch with `BadOutOfRange`, and keep the ScadaBridge wire shape. ScadaBridge writes CLR lists such as `int[]` to Int16 nodes, so the element type converts and is not exact-matched.
- **Dead config fields (user decision): wire every one.** Persisted JSON with unknown keys must keep loading.
- **Absence assertions need a positive control** (memory). Any "no longer leaks / no longer publishes" test must also prove the positive path delivers.
- **AdminUI has no bUnit:** Razor changes are live-verified on the docker-dev rig (`http://localhost:9200`, login disabled) with Chrome automation, not unit tests alone.
- **Historian paging changes require a paged-vs-unpaged fuzz test** against a wire-faithful fake (memory, #692), re-run on every revision.
- **No remote commands on wonder boxes** (no psbridge or winrm). Galaxy and AVEVA gates use network service access only, and any remote action is a userGate task for the user to run.
- **`.claude/` is untracked**, so worktree agents cannot edit `.claude/rules/*`. They put suggested rule text in their final report, and the coordinator applies it in the main checkout.
- **Subagents cannot write to the coordinator scratchpad.** They return reports in their final reply.
- **Never commit secrets.** Commits end with the session attribution line.
**User decisions (already made):**
- "Implement array encode now" for AbCip, AbLegacy and TwinCAT. Real-hardware proof is a userGate.
- "Wire them all" for every authored-but-unconsumed field: MQTT `birthObservationWindowSeconds`, Galaxy `DiscoverPageSize`, Galaxy and OpcUaClient `ProbeTimeoutSeconds`, Modbus `Family`/`MelsecSubFamily`/`DisableFC23`, `TelemetryDial:CallTimeoutSeconds`, `/auth/token` JWT (finish it with `AddJwtBearer`), and the `GetDiagnostics` snapshot (populate it with real data and give it a consumer).
- "Gitea issues + local merge, push on OK": one issue per lane with findings as a checklist, merge to local master, and push/close only after the user OKs. The scadaproj index is updated on its `main`.
---
## Model assignment
| Tier | Model | Used for |
|---|---|---|
| frontier | **fable** | Actor lifecycle (A), OPC UA node manager/SDK (B), deploy coordinator (C1), script runtime + alarm actor restart (E1/E4), Galaxy (F), HistoryRead cap (H1). Fable reviews these lanes. |
| standard | **opus** | All driver lanes, control-plane remainder, Roslyn incremental analysis, sandbox, historian remainder. Opus adversarial review of every lane, including sonnet ones. |
| mechanical | **sonnet** | AdminUI UI fixes, CLI fixes, client library fixes, transport buffer pooling, docs/index edits, Gitea issue filing |
| — | haiku | not used; every task here needs code judgement |
## Waves and parallelism
```
Wave 0 (serial, coordinator) : T0 setup, T1 baseline suite
Wave 1 (6 lanes parallel) : A Runtime · B AddressSpace · C ControlPlane · E Scripting · F Galaxy · H1 HistoryRead cap
Wave 2 (8 lanes parallel) : AbCip · AbLegacy · TwinCAT · FOCAS · Modbus · S7 · G1 OpcUaClient+MTConnect · G2 MQTT+Sql
Wave 3 (4 lanes parallel) : D AdminUI · H2 Historian remainder (+B2 last) · K Clients/CLI/Transport · E5 Alarm comment table
Wave 4 (serial) : T-INT full suite + fixture gates + docker-dev gates + hw userGates + docs/index + push-on-OK
```
The wave order follows file contention:
- Wave 2 driver lanes depend on A's dispose contract only for review context, not for code.
- D waits for the Modbus/FOCAS editor and form edits in Wave 2.
- B2 waits for B (`OtOpcUaNodeManager.cs`).
- H2 waits for H1 (`HistoryPaging.cs`) and owns the recorder.
Merge order inside a wave is alphabetical by lane, rebasing as needed. Two lanes never share a function.
---
## Wave 0
### Task 0: Setup — persist the spec, file the issues, materialize the plan
**Goal:** The reports and plan live in the repo, there is one Gitea issue per lane, and native tasks exist.
**Files:** Create `docs/reviews/2026-10-21/*.md` (copy all 9 from the session scratchpad `…/scratchpad/review/`), `docs/superpowers/plans/2026-10-22-review-remediation.md` (this plan), and `….md.tasks.json`.
**Acceptance Criteria:**
- [ ] `ls docs/reviews/2026-10-21 | wc -l` = 9
- [ ] One Gitea issue per lane (A, B, B2, C, D, E, E5, F, G1, G2, H1, H2, K, AbCip, AbLegacy, TwinCAT, FOCAS, Modbus, S7). Each issue carries its findings as `- [ ] R0x:ID` lines, and the issue numbers are recorded in tasks.json.
- [ ] Cross-repo issues are filed:
- mxaccessgw: lease never extended without `StreamEvents` + no renew RPC; `mxgateway.sessions.open` not decremented on sweep close.
- HistorianGateway: `max_events` on `ReadEvents`; a batch `SendEvents` RPC; historiangw#27 tracking reference.
- [ ] Commit `docs: add 2026-10-21 system review + remediation plan`.
**Verify:** `git show --stat HEAD` lists the reports and the plan.
**Model:** sonnet (issue filing), coordinator commits.
### Task 1: Baseline full suite
**Goal:** Record the pre-change green or flake baseline, so regressions are attributable.
**Acceptance Criteria:**
- [ ] `dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release` succeeds.
- [ ] `dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "Category!=E2E&Category!=LiveIntegration"` has its pass/fail/flaky list saved to `docs/reviews/2026-10-21/baseline.txt`.
**Verify:** `tail -5 docs/reviews/2026-10-21/baseline.txt`.
**Model:** coordinator.
---
## Wave 1 — high-severity core (6 parallel lanes)
### Lane A — Runtime driver lifecycle (fable; review fable)
**Owns:**
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/**`
- `Runtime/Health/**`
- `Runtime/Telemetry/**`
- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs` + the telemetry dialer
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriver.cs`
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Browsing/DiscoveryDriverBrowser.cs`
- the `IFleetDiagnosticsClient` consumer page (one new Razor file) + `AdminUI/Clients/FleetDiagnosticsClient.cs`
**Tests:** `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests`, `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests`
- **A1 Dispose drivers on final stop** (`R02:L1`, `R02:P4`, `R06` host context, `R07:L2`).
- In `DriverInstanceActor.PostStop`: `ShutdownAsync(cts.Token).WaitAsync(5 s)`, with a logged timeout. Then `DisposeAsync`/`Dispose`, but only when the actor is stopping for good. Override `PreRestart` to skip disposal, because the Props closure reuses the driver.
- `DiscoveryDriverBrowser` disposes its throwaway driver in `finally`.
- Tests:
- A fake `IAsyncDisposable` driver is disposed exactly once on `Context.Stop`.
- It is NOT disposed on a supervision restart (positive control: it still serves after restart).
- A hung `ShutdownAsync` doesn't pin a thread beyond 5 s.
- **A2 Single-flight reconnect** (`R02:H2`). Add an `_initInFlight` gate cleared by the piped `InitializeSucceeded`/`Failed`, and bound the teardown. Test: with a 15 s init and a 10 s cadence, `ShutdownAsync` and `InitializeAsync` never overlap (record interleavings in the fake).
- **A3 Writes off the mailbox** (`R02:P2`). Run `WriteAttribute` via `PipeTo`, capturing `Sender`, with a per-driver in-flight cap (default 4). Subscribe/unsubscribe stay `ReceiveAsync`. Tests:
- Two concurrent 3 s writes both complete under the host's 8 s Ask.
- Data changes are delivered during a pending write.
- **A4 Housekeeping:**
- `_lastReprovisionUtc.Remove` in `StopChild` (`R02:L2`).
- `Stale()` drop-receives (`R02:H5`).
- Remove the unused `ILoggerFactory` ctor slot (`R02:U5`).
- Move the `OnAttributeValuePublished` seam to a test helper (`R02:U4`).
- `DbHealthProbeActor` off-thread + 2 s command timeout (`R02:P7`).
- Audit all `PollGroupEngine.Unsubscribe` sync callers and switch them to `UnsubscribeAsync` (`R05:8`).
- `TelemetryLocalHub._evicted` timestamped sweep with TTL > max in-flight call (`R02:L4`).
- **A5 Wire `TelemetryDial:CallTimeoutSeconds`** into the dialer's `CallOptions` deadline (`R02:U2`). Test: a stalled stream is cancelled at the configured deadline.
- **A6 Real `GetDiagnostics`** (`R02:U1`). Populate it from child health snapshots (real id, device counts from `GetHostStatuses`, last-change time), and give `IFleetDiagnosticsClient` a consumer: a per-node diagnostics panel on `/fleet`. That Razor file is owned here and live-verified in Wave 4.
**Lane gate:** Runtime.Tests green. The docker-dev gate (Wave 4) shows a driver config change disposing the old instance: the `ObjectDisposed`/dispose log appears, and process thread/handle counts stay flat across 20 respawns (`dotnet-counters`).
### Lane B — OPC UA address space (fable; review fable)
**Owns:**
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/**`, except `HistoryPaging.cs`/`RawPageLimit.cs` (H1) and the HistoryReadEvents projection (B2)
- `Runtime/OpcUa/OpcUaPublishActor.cs`
**Tests:** `OpcUaServer.Tests`, `OpcUaServer.IntegrationTests`
- **B1 No phantom nodes** (`R01:L1`, `L4`, `H2`):
- `WriteValueLocked` is non-creating (unknown key ⇒ skipped, counted, Debug-logged).
- `WriteAlarmCondition` unknown id ⇒ logged no-op.
- Delete the applier's equipment-id `RemovedConditionState` write.
- Add a cross-map `(realm,id)` collision guard in `EnsureFolder`/`EnsureVariable`/`MaterialiseAlarmCondition`.
- Tests:
- A publish racing a surgical remove leaves no node.
- A re-add after that gets the authored DataType, parent and writability (positive control).
- **B2a Stale MonitoredNode2** (`R01:L2`).
- On every removal path (rebuild, `RemoveVariableNode`, `RemoveAlarmConditionNode`, `RemoveEquipmentSubtree`): queue `BadNodeIdUnknown` to the node's items, `MonitoredNodes.Remove(nodeId)`, and drop `_liveItems`/`_eventItems`.
- Revalidation restore re-resolves the node and does not reuse `Handle.Node`.
- Integration test (real SDK): subscribe, rebuild, create a new item on the same NodeId. The new item receives values (positive control) and the old one gets `BadNodeIdUnknown`.
- **B3 Lock-time performance:**
- Rebuild detaches top-level subtrees, not leaves (`R01:P1`).
- Per-equipment descendant index for `RemoveEquipmentSubtree` (`P2`).
- `LowerAllEventsMonitoring` for the surgical removal sites (`P3`).
- Notifier wiring de-quadratic (`P5`).
- Benchmark test: rebuild a 10k-tag folder and remove 200 equipment from a 50k-node space. Assert completion < 1 s on CI hardware and record the before/after in the commit message.
- **B4 Write-path robustness:**
- `OnEquipmentTagWrite` observes `t.Exception` and wraps a synchronous throw (`R01:L3`, `H5`).
- Shelve/condition timestamps use the injected `TimeProvider` (`H4`).
- SDK request quotas + `TransportQuotas` configurable via `OpcUa:Quotas:*` with the current defaults (`P6`).
- HistoryRead arms: lower the default `MaxConcurrentHistoryReadBatches` on small nodes, and use an `IAsyncNodeManager` override if the SDK line exposes `HistoryReadAsync`; otherwise document it (`P4`).
- **B5 Delete dead v3 equipment-tag materialisation** (`R01:U2`): the composer's `EquipmentTags`, applier `MaterialiseEquipmentTags`, the surgical `ChangedEquipmentTags` loop, and the classifier `TagDeltaIsSurgicalEligible`. Keep `IsEmpty` semantics; the full OpcUaServer suite stays green.
- **B6 Off-thread deploy parse/plan** (`R02:P5`).
- `DeploymentArtifact.Open` + `ParseComposition` + `AddressSpacePlanner.Compute` run via `PipeTo`; only sink materialisation runs on the pinned thread.
- Delete the bytes-less `LoadArtifact` SQL branch.
- Test: value batches keep flowing while a 50k-node plan computes.
- **B7 Capability bits refresh** (`R01:U3`). Re-evaluate event-notifier/HistoryWrite bits when `HistorianDataSource`/`Updater` is set after the first apply, so a late assignment no longer needs a full rebuild.
- Documented, no change: `R01:U4` (native Confirm/shelve, by design) and `R01:L5` (bounded). Note both in the issue.
**Lane gate:** both suites green. Wave 4 docker-dev gate: a ScadaBridge-style subscription survives a surgical add, and a re-created item after a rebuild receives data (Client.CLI `subscribe`).
### Lane C — Control plane / host / security (fable for C1–C2, opus for C3–C5; review fable)
**Owns:** `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/**`, `src/Server/ZB.MOM.WW.OtOpcUa.Host/**` (except `Host/Engines/**`, which belongs to E), `src/Server/ZB.MOM.WW.OtOpcUa.Security/**`, `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/**` (except `Validation/DraftValidator.cs`, which belongs to E).
**Tests:** `ControlPlane.Tests`, `Host.Tests`, `Security.Tests`, `Configuration.Tests` (central SQL)
- **C1 Deploy wedge + consistent snapshot** (`R03:2.2`, `3.2`, `3.5`):
- In PreStart, a recovered deployment with an empty expected set is re-seeded and re-dispatched (or marked `Failed` + `ResetForNext`).
- `HandleDeadline` takes the race branch only when `_expectedAcks.Count > 0`.
- The `Dispatching` row and the expected set are created together in the coordinator.
- Validate + compose from ONE snapshot (extend `DraftSnapshot` with clusters/nodes/ApiKeys) inside a `SNAPSHOT` read transaction.
- Project `ArtifactBlob` away in full-row reads.
- Tests:
- Inject a `HandleDispatch` throw; the next `StartDeployment` succeeds (positive control: a normal deploy seals).
- An edit between validate and compose cannot ship unvalidated.
- **C2 Admin singleton mailbox** (`R03:3.1`). Move `TestDriverConnect` to a per-request child actor (or `Task.Run` + `PipeTo`) and run deploy compose off-actor; only the in-flight check is serialized. Test: an alarm ack completes in < 1 s while a 60 s Test Connect is pending.
- **C3 OPC UA start failure is visible** (`R03:2.1`, `1.4`):
- Add an `OpcUaEndpointHealthCheck` (Ready + Active tags), Unhealthy while the SDK failed to start.
- A driver-only node rethrows (process exits non-zero).
- The "SDK not started" fact feeds the ServiceLevel calculator, so the peer takes Primary. If the calculator lives in a Lane B-owned file (`OpcUaServer/**`), C only exposes the fact (`IOpcUaEndpointState`) and files an `F<N>` follow-up for Wave 3 to consume it.
- `StopAsync` stops the SDK; `DisposeAsync` is idempotent.
- Tests: port-in-use ⇒ `/health/ready` 503, and the ServiceLevel published drops.
- **C4 Security:**
- `ApiKeyOptionsValidator` + `ValidateOnStart` (non-blank pepper, minimum length, whenever driver or admin) (`R03:2.3`).
- **Finish JWT** (`R03:2.4`, user decision): `AddJwtBearer` using `BuildValidationParameters()`; `JwtOptions` `ValidateOnStart` when `/auth/token` is mapped; the `/api/*` machine endpoints accept a Bearer token as well as API keys.
- Wire `IAuditActorAccessor` through `AuthenticationStateProvider`, so audit rows carry the circuit principal (`R03:2.5`).
- Delete the unregistered `CookieAuthenticationStateProvider`.
- LDAP login exception logged at Warning (`2.7`).
- Admin ops return a fixed message + correlation id (`4.1`).
- **C5 Retention + throughput:**
- `DeploymentRetentionActor` on the admin singleton: keep the last N=50 sealed rows per cluster plus anything referenced in the last 30 days; null `ArtifactBlob` on older rows; configurable `Deployments:Retention:*` (`R03:1.2`).
- `AuditWriterActor` handles duplicate keys per row (2601/2627 ⇒ detach and retry) (`2.6`).
- `MeshGrpcCommandDialer` evicts endpoints absent from `ContactsLoaded` (`1.3`).
- Async PSK refresh with an atomic swap instead of invalidate-on-refresh (`3.4`).
- Pre-auth API-key DoS: an in-memory key-id set from the artifact or `IDataPlaneAuthoritySource`, refreshed every `StoreCacheSeconds`; an unknown id is rejected without SQL; plus a per-endpoint token bucket (`3.3`).
- Document the `4.2` fail-open window in `docs/security.md`.
**Lane gate:** suites green. Wave 4 docker-dev gate: kill central-1 mid-dispatch, then the next deploy succeeds. Port 4840 taken ⇒ `/health/ready` 503.
### Lane E — Scripting runtime safety + scripted alarms (fable for E1/E4, opus for E2/E3; review fable)
**Owns:**
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/**`, `Core.ScriptedAlarms/**`, `Core.AlarmHistorian/**`
- `Runtime/ScriptedAlarms/**`, `Runtime/VirtualTags/**`
- `Host/Engines/**`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/**`
- `Configuration/Validation/DraftValidator.cs`
**Tests:** `Core.Scripting.Tests`, `Core.ScriptedAlarms.Tests`, `Core.AlarmHistorian.Tests`, `Driver.Calculation.Tests`, `Runtime.Tests` (VirtualTags/ScriptedAlarms folders)
- **E1 Script quarantine** (`R03:1.1`, `R05:2`, `R07:L1`, `R07:P5`):
- A shared `ScriptRunGuard` in Core.Scripting, keyed per script instance.
- It refuses to start a run while that script's previous orphan is still executing. The result is Bad `"previous evaluation still running"` plus a counter.
- It quarantines after 3 consecutive timeouts: stop evaluating, publish Bad, log Error, metric `otopcua_script_quarantined`, cleared on redeploy or source change.
- Pass a `CancellationToken` into scripts.
- Make the Calculation pump async, so pool starvation is no longer charged to scripts.
- Wired into the scripted-alarm engine, `RoslynVirtualTagEvaluator`/`VirtualTagHostActor`, and `CalculationEvaluator`.
- Tests: a `while(true)` script on a 1 Hz trigger leaves ≤ 1 orphaned thread after 30 triggers (`ThreadPool.ThreadCount` delta). Positive control: a healthy sibling keeps evaluating.
- **E2 Sandbox hardening** (`R05:10`, `R05:11`):
- Deny `System.Threading` with an allow-list (`CancellationToken`, `Interlocked`, `Volatile`).
- Deny `System.Type.GetType`/`GetMethod*`/`InvokeMember` and `System.Runtime.CompilerServices.RuntimeHelpers`.
- Add an allocation budget: sample `GC.GetAllocatedBytesForCurrentThread` around each run and quarantine on breach (`Scripting:MaxAllocBytesPerRun`, default 64 MB). Document that a pre-emptive OOM still needs an out-of-process runner, and file a follow-up issue for that (→ #750).
- Tests: each forbidden form gets an `OTSCRIPT` diagnostic, and the Monaco editor parity test still matches.
- **E3 Scripted-alarm engine** (`R05:1`, `3`, `6`, `7`, `9`, `12`):
- Clear `_valueCache` in `LoadAsync`; move its write below the `_disposed` guard.
- Bounded `Dispose` drain (5 s, cancelable token).
- `ScriptedAlarmSource` catch logs at Warning.
- `LocalDbStoreAndForwardSink` cached dead-letter counter, resynced with the queue counter.
- Script-log sinks rate-limited per source (20/min + summary).
- **E4 Alarm host restart + virtual-tag log storms** (`R02:H1`, `H3`, `P6`, `L3`; `R07:P4`, `H6`, `H8`, `U5`):
- `ScriptedAlarmHostActor` receives a `Func<(upstream, engine)>` factory, building its engine per incarnation.
- Linked 10 s timeouts on `OnAlarmCommand`.
- `DependencyMuxTagUpstreamSource.RetainOnly(loaded deps)`.
- `VirtualTagActor` per-tag failure-log rate limit.
- Calculation:
- negative-cache compile failures by source hash, with the Warning/script-log only on the Good→Bad transition;
- the mux retains the last `(value,status,ts)` and replays it on `RegisterInterest`, with inputs recorded regardless of `_running`;
- structural array dedup;
- `ClearCompiledScripts` from `ShutdownAsync`;
- a DraftValidator rule `CalculationNoEffectiveTrigger`.
- Tests:
- Throw inside a `DependencyValueBatch` handler, then a later `ApplyScriptedAlarms` loads cleanly.
- A calc Restart republishes the current inputs.
**Lane gate:** suites green. Wave 4 docker-dev gate: author a `while(true)` virtual tag. The node stays responsive, the tag shows quarantined, and the thread count stays flat over 2 min.
### Lane F — Galaxy driver (fable; review fable)
**Owns:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy*/**`, the AdminUI `GalaxyDriverForm.razor`.
**Tests:** `Driver.Galaxy.Tests`, `Driver.Galaxy.Browser.Tests`
- **F1 Real teardown** (`R06:G-1`, `G-2`, `G-9`, `G-3`):
- `ShutdownAsync` stops the pump, alarm feed, deploy watcher, probe watcher and supervisor, disposes the session + clients, and nulls the seams so `InitializeAsync` rebuilds. `DisposeAsync` stays the idempotent final teardown.
- `StartDeployWatcherAsync` stops the previous watcher.
- `ReconnectSupervisor` disposes each cycle's CTS.
- Tests with fake gRPC clients:
- After `ShutdownAsync`, every stream is cancelled and both channels are disposed.
- Five watchdog cycles leave exactly one DeployWatcher (positive control: rediscovery still fires once per deploy).
- **F2 Probe watcher + liveness:**
- Register probe handles in `SubscriptionRegistry` under a reserved id, and `Rebind` them in `ResubscribeAllAsync` (`G-4`).
- `GalaxyDiscoverer` returns its fetched hierarchy, so discovery fetches once (`G-7`).
- A throttled `LastSuccessfulRead` stamp from `OnPumpDataChange` (`G-11`).
- An `EventPump` dispatch-loop exit ⇒ `NotifyStreamFault` (`G-10`).
- Test: a `ScanState` event updates `GetHostStatuses` and survives a reconnect.
- **F3 Wire knobs + writes:**
- `DiscoverPageSize` → `DiscoverHierarchyAsync`. If the mxaccessgw RPC lacks a page parameter, add a cross-repo task in mxaccessgw and page client-side meanwhile.
- `ProbeTimeoutSeconds` → probe deadline (`G-5`, `G-6`).
- First-touch write pipeline: concurrent AddItem/Advise across distinct tags, bounded at 8 (`G-8`; acks stay serial by design).
**Lane gate:** suites green. Wave 4 userGate: with a reachable mxaccessgw, 20 config deploys keep `mxgateway.sessions.open` flat, read from the gw metrics endpoint.
### Lane H1 — HistoryRead raw cap (fable; review fable + paging fuzz)
**Owns:** `OpcUaServer/HistoryPaging.cs`, `RawPageLimit.cs`, the `ServerHistorian` options type, and the `ReadRawAsync` methods of the three `*HistorianDataSource.cs` files.
**Tests:** `OpcUaServer.Tests` (paging), `Historian.Conformance`, the backend `*.Tests`
- `NumValuesPerNode = 0` ⇒ server page size `ServerHistorian:MaxRawValuesPerRead` (default 10 000), with a continuation point via the existing `RawPageLimit` paging. The cap is advertised as `HistoryServerCapabilities/MaxReturnDataValues`.
- Backend `TOP`/`LIMIT`/`max_values` are bounded by the same cap (`R08:1.1`).
- A paged-vs-unpaged fuzz test: random windows/caps against a wire-faithful fake for each backend, where the concatenated pages equal the unpaged truth, with boundary ties.
**Lane gate:** the fuzz runs 10k iterations green. The Wave 4 SqlServer/QuestDb fixture gate reads a year window at N=0 and gets pages of ≤ 10k with a CP, and heap stays < 500 MB.
---
## Wave 2 — driver lanes (8 parallel; opus implementer, opus review)
Every driver lane also:
- (a) makes `ShutdownAsync` a complete teardown;
- (b) applies the Global health-contract checklist;
- (c) build-and-swaps device tables;
- (d) adds `IBatchSubscribable` where it runs `PollGroupEngine` (`R02:P1`).
Each lane owns `src/Drivers/<Driver>*/**` + its `tests/Drivers/<Driver>*`, its AdminUI tag-config editor/driver form files, and its `Cli/<Driver>.Cli`.
### Lane AbCip
- Probe classification: `NotFound` ⇒ reachable, with a config warning (`R07:H10`).
- Widen `IsReachableStatus`, cache the runtime with its refusal status, map 0x1C/0x20/0xFF-0x2104 explicitly (`H1`, `L4`).
- Native window ≥ 64 (`P8`); cold waiters don't hold slots (`P7` addendum).
- Bit-in-word parent width from `GetSize()` (`H17`).
- Write-side health no longer latches or overwrites (`H19`).
- BOOL-array element decodes bit n%32 of DWORD n/32 (`U11`).
- **Array write encode** (`U10`).
- UDT layout cache (`P11`).
- Device-table swap (`H13`); `volatile _health` (`H21`).
- `DisposeAsync` instead of sync-over-async (`F10`).
### Lane AbLegacy
- Per-device `ConnectionBackoff` + single-flight connect + bounded parallel devices + adjacent-element coalescing (`R07:P7`).
- `NotSupportedException` ⇒ `BadNotSupported`, no health, reject at `FromTagConfig` (`H16`).
- Table swap (`H13`); prune semaphores on index change (`L7`).
- **Array write encode** (`U10`); `IBatchSubscribable`.
- Document the PLC-5 write capability gap with a hardware follow-up issue (`U6`) (→ #752).
### Lane TwinCAT
- Per-registration isolation with dead-registration retry (`R07:H11`).
- Stamp `LastSuccessfulRead` from notifications + probe, and degrade on N consecutive router failures (`H12`).
- Warn once per tag (`P10`).
- Parent-width bit RMW (`H18`).
- **Array + string writes via the codecs** (`U9`).
- `BadNodeIdUnknown` for unresolved native refs (`U12`).
- Bounded probe join, drop counter, queue purge on dispose, re-init table swap (`L8`); `IBatchSubscribable`.
### Lane FOCAS
- `ConcurrentDictionary`/swap `_devices` (`R07:H13`).
- Widen macro/param numbers to int on the wire client (`H14`).
- Fixed-tree loop writes health (`H15`).
- Reject `String` in the factory + editor (`U8`).
- Concurrent per-device reads (`P9`).
- Recycle under `ConnectGate` (`L6`).
- Alarm-layout heuristic uses the record count from the header (`H20`).
### Lane Modbus
- Deadband baselines on `(value,status)` (`R07:H2`).
- Transport self-heals with a throttled in-gate connect (`P3`).
- Dispose the failed `TcpClient`, one in-gate attempt, DNS under the timeout (`L3`).
- **Wire Family/MelsecSubFamily/DisableFC23** (`U1`): the editor + factory route the address through `ModbusAddressParser` and persist the resolved region/offset; `DisableFC23` gates multi-write.
- Coalescing on by default (gap 8), plus coils/DI/BitInRegister folding (`U2`).
- String[]/BitInRegister[] writes (`U3`).
- Region/type validation at the factory (`P6`).
- Exception codes 01/02/03 are refusals (`H4`).
- Write status mapping (`H7`); PDU length check (`H9`).
### Lane S7
- Dead-handle short-circuit inside a pass (`R07:P1`).
- Per-member span fallback + "do not group" memo (`P2`).
- Per-tag skip + `TryParse` in `FromTagConfig`/Inspect, so the deploy gate refuses bad tags (`H3`).
- Non-fatal `ReadData`/`WriteData` are refusals (`H4`).
- `_lifecycleGate` (Modbus pattern) (`H5`).
- Bool-array bit-offset reject at authoring, or bit-packed decode (`U4`).
- Probe gate contention ≠ Stopped; honour `TimeoutMs` (`H7`).
### Lane G1 — OpcUaClient + MTConnect
**OpcUaClient:**
- Data-subscribe cleanup on throw (`R06:O-1`).
- Ack rethrows the transport status (`O-8`).
- Split the read/write gate from subscription management; no gate across the recursive browse (`O-6`).
- Close the local session on an init throw (`O-2`).
- One teardown budget (`O-9`).
- Wire `ProbeTimeoutSeconds` (`O-4`).
- Browse `AttributesAsync` real + certificate-auth browse (`O-5`).
**MTConnect:**
- Generation guard on cursor publish (`T-1`).
- Read through the live index when the pump is healthy (`T-4`).
- Lag metric + adaptive `count` (`T-5`).
- Instance-id check in `ReadAsync` (`T-2`).
- Delete dead parsed fields (`T-3`).
- Batched PUT (`T-6`).
- Headers-only probe (`T-7`).
- Redacted URI everywhere (`T-8`).
### Lane G2 — MQTT + Sql
**MQTT:**
- Failed SUBSCRIBE ⇒ pending set retried on reconnect/reconcile, `SubscribeAsync` throws on a transport failure (`R06:M-1`).
- **Wire `birthObservationWindowSeconds`** into `MqttDriverBrowser.OpenBudget` + discovery UntilStable (`M-4`).
- Array leaves not bindable, plus a Sparkplug-array-read feature issue (`M-5`) (→ #754).
- Prune `LastValueCache` on Register (`M-2`).
- Dispose the failed connection (`M-3`).
- Wildcard topic trie (`M-6`); static UTF-8 encoder (`M-7`).
- Bounded concurrent publishes (`M-8`); guards (`M-10`, `M-11`).
**Sql:**
- Row cap with a `TOP/LIMIT N+1` dialect + config-error status (`S-1`).
- Rate-limited warnings (`S-6`); plan cache (`S-7`).
- Dispose `_gate` (`S-3`).
- `Query` model refused at the deploy gate via the parser + feature issues for MySql/Odbc/allowWrites (`S-4`) (→ #753).
- `IS_NULLABLE` in browse attributes (`S-5`).
- Withhold JSON parser text (`S-8`); `IBatchSubscribable`.
**Driver lane gates:**
- Wave 4 fixture runs on `10.100.0.35`: modbus `standard`, s7 `s7_1500`, abcip `controllogix`, opcuaclient opc-plc, mqtt + sparkplug, sql. Each lane adds or extends an IntegrationTests case per High finding.
- Hardware userGates for real controllers: AbCip/AbLegacy/TwinCAT array writes, FOCAS macro > 32767, AbCip probe-typo recovery.
---
## Wave 3
### Lane D — AdminUI (opus for D1–D2, sonnet for D3–D4; review opus)
**Owns:** `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/**`, except the per-driver editors/forms owned in Wave 2 and the A6 diagnostics panel.
- **D1 Monaco sync correctness** (`R04:P1`, `P9`, `S2`, `S3`):
- Fetch the editor text on demand at Save/blur via `MonacoBlazor.getValue` instead of per keystroke.
- `HubOptions.MaximumReceiveMessageSize` = 1 MB.
- Diagnostics generation counter + `AbortController`.
- Drop the inlay-hints provider/endpoint.
- Build a Problems list using `MarkersChanged` + `RevealLineAsync`.
- **D2 Roslyn analysis** (`P2`, `P3`, `S1`):
- One compilation per scoped service with `ReplaceSyntaxTree`.
- `RequestAborted` plumbing + Monaco token honouring.
- Sorted `string[]` + dictionary in the catalog snapshot.
- An `OTSCRIPT_SETVTAG_INERT` warning marker.
- **D3 Rendering** (`P4`–`P8`, `H2`):
- `/hosts` dirty flag on tick.
- `RawTree` cached menus + `Virtualize` + 500-per-level paging.
- CSV import grid `Virtualize` + export via a download endpoint.
- ScriptLog/Alerts 200 ms coalescing + ring buffer.
- `/scripts` projection.
- Cached modal parameter dictionaries.
- **D4 Leaks/hazards/stubs** (`L1`–`L5`, `H1`, `H3`, `S4`–`S7`):
- Snapshot-store eviction against the config load.
- `Threading.Timer` in `DriverTestConnectButton`.
- `AlarmsHistorian` page `Fleet.razor` pattern + "no historian on role" state.
- Close an orphaned browse token after dispose.
- Per-subscriber try/catch in broadcasters.
- Confirm dialogs on equipment-page and ScriptEdit deletes.
- Document the theme (`H3`).
- Delete dead `UnsTreeService` members.
- `GET /api/deployments/{id}`.
- Equipment CSV via `CsvParser`.
- Gate `/dev/*` behind `#if DEBUG`.
**Lane gate (userGate-free, coordinator-run):** docker-dev Chrome pass.
- Paste a 100 KB script, Save, reload: the source persists.
- Two-tab `/hosts` with 50 drivers: CPU stays flat.
- Expand a 5k-tag device: renders in < 2 s.
- Each delete confirms.
- The A6 diagnostics panel shows real counts.
### Lane H2 — Historian remainder (opus; review opus)
**Owns:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.*/**` (minus H1's files), `Runtime/Historian/**` (recorder + adapter), `docs/Historian.md`, the AdminUI historian backend picker.
- Gateway events: drain to completion with a budget, cancellation arms (`R08:1.2`, `2.5`).
- Drop native `TimeAverage`/`Total` behind `ServerHistorian:Gateway:EnableNativeTimeWeighted=false` until historiangw#27 (`2.1`).
- Capability matrix in docs + picker + a startup warning for continuous historization on Gateway (`2.2`).
- `ProbeAsync` health timer (`2.4`).
- Writers:
- OCE rethrow in all three (`4.1`).
- Gateway via `LiveWritePreparer` (`4.2`).
- `IHistorianValueWriter.FlushAsync` pass-end hook, QuestDB WAL wait once per pass with exponential poll (`3.1`).
- Batched alarm-history writers: SqlServer TVP + `OUTPUT`; QuestDb multi-row; Gateway bounded pipelining until a batch RPC (`3.2`).
- Retention pre-filter (`3.3`).
- QuestDB symbol indexes + migration (`3.4`).
- Presence cache exact lookup (`3.5`).
- Paged `DeleteRaw` (`4.3`).
- QuestDB trail `applied` marker + DEDUP key (`4.4`).
- Recorder: `CommitMode=Periodic(100 ms)` default + off-mailbox append with an in-flight cap + mailbox-depth metric (`R02:P3`); timeouts on `GetStatus` (`R02:H3`).
- `HistorianAdapterActor` bounded channel + delete the senderless receive (`R02:H4`, `U3`).
- Outbox scan outside the lock (`R08:1.3`).
**Lane gate:** `Historian.Conformance` green on all backends. The Wave 4 AVEVA live gate (userGate, existing recipe in `.claude/rules/historian.md`) checks events paging and that `TimeAverage` is refused.
### Lane K — Clients, CLI, transport (sonnet; review opus)
**Owns:** `src/Client/**`, `src/Drivers/Cli/Driver.Cli.Common` + the AbCip/AbLegacy CLI command files (after the Wave 2 merge), `Driver.Ab.Transport/**`.
- Client:
- release the raw CP (`R08:1.4`);
- `RemoveSubscription` (`1.5`);
- align `AggregateType` with `AggregateFamilies.Served` (`2.3`);
- lazy HasChildren (`3.6`);
- NodeId dictionary in the UI (`3.6`);
- failover under `_subscriptionLock`, and `ConnectAsync` refusing when already connected (`4.5`).
- CLI fixes (`R07:U7`).
- Pooled buffers in `EncapsulationClient`/`CipWriter`/`LogixTagClient` + Modbus transports via `ArrayPool` (`R07:L5`). Benchmark-gated: no regression in `Ab.Transport.Tests` timings.
### Lane B2 — HistoryReadEvents full projection — folded into Lane H2 as its LAST task (opus; review fable)
This runs inside H2 because it shares the backends' event read paths with H2's Gateway events-drain fix. It runs after H2's other tasks, and `OtOpcUaNodeManager.cs` is free by then because B merged in Wave 1.
**Adds to H2's ownership:** the `HistoricalEvent` contract in Core.Abstractions and the HistoryReadEvents projection in `OtOpcUaNodeManager.cs`.
- Carry `EventType`/`ConditionName`, map `SourceNode` when resolvable, and honour an `OfType` where-clause (`R01:U1`).
- Test over the wire: a Client.CLI `historyread --events` with a select clause including EventType returns non-null values.
### Lane E5 — Alarm comment audit table (opus; review fable)
**Owns:** `Core.ScriptedAlarms/Part9StateMachine.cs`, the alarm state store + its LocalDb schema/migration.
- Move the append-only comments to their own table keyed by alarm id, keeping replication to the pair peer. Each condition row stays a fixed size.
- Migrate existing inline comments idempotently (`R05:4`).
- Test: 10k comments, then a transition's `SaveAsync` payload size is constant. A pair-replication test shows the comments on the peer.
---
## Wave 4 — integration (serial, coordinator)
### Task INT-1: Full suite + flake triage
Run the full suite again. Any new failure versus `baseline.txt` is either fixed or filed. Loop any suspect test under `yes` contention.
### Task INT-2: Fixture gates (sims on 10.100.0.35)
For each driver lane, sync and bring up its fixture, then run `*.IntegrationTests` with the env endpoints. Record the results in `docs/reviews/2026-10-21/gates.md`.
### Task INT-3: docker-dev live gates (coordinator, Chrome automation + Client.CLI)
This runs the lane gates for A, B, C, D and E listed above, plus the A6 `/fleet` panel and the H1 N=0 paging read through Client.CLI `historyread`.
### Task INT-4 (userGate): hardware and AVEVA gates
> **USER-ORDERED GATE — NON-SKIPPABLE.** This task was requested by the user in the current conversation (array encode on real controllers). It MUST NOT be closed by walking around it, by declaring it "verified inline", or by substituting a cheaper check. Close only after every item in `acceptanceCriteria` has been re-validated independently, with output captured.
Acceptance criteria:
- An AbCip array write round-trips on a real ControlLogix.
- An AbLegacy N-file array write round-trips on a real SLC/MicroLogix.
- A TwinCAT array + STRING write round-trips on real hardware.
- A FOCAS `MACRO:` > 32767 reads the correct variable.
- Galaxy: 20 deploys keep `mxgateway.sessions.open` flat.
- AVEVA: the events paging + `TimeAverage` refusal gate.
`requiresUserSpecification: true` for the bench endpoints.
### Task INT-5: Docs, rules, index, push-on-OK
- Apply each lane's suggested `.claude/rules/*.md` text in the main checkout.
- Update `CLAUDE.md` where behaviour summaries changed: the health contract, surgical/rebuild notes, the historian N=0 rule.
- Update `../scadaproj/.claude/skills/sister-project-history/otopcua.md` (+ the index row only if the summary changes) on scadaproj `main`, following the CLAUDE.md recipe, and push scadaproj.
- Draft the per-issue closing comments, then ask the user for OK before pushing OtOpcUa master and closing the issues.
- Save project memory for lessons learned.
---
## Verification (end to end)
1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release` + the full unit suite match the baseline or better.
2. Fixture IntegrationTests green per driver (INT-2).
3. docker-dev gates (INT-3):
- Respawn churn leaves threads/handles flat (`dotnet-counters monitor -p <pid> System.Runtime`).
- A re-subscribe after a rebuild receives data.
- The deploy wedge recovers.
- A spin script is quarantined.
- A 100 KB script saves.
- An N=0 HistoryRead pages.
4. Hardware/AVEVA userGates (INT-4).
5. Each Gitea issue checklist has every `R0x:ID` ticked with a commit SHA, or explicitly re-filed as a follow-up.
## Self-review notes
- Coverage: every finding ID in reports 01–08 maps to a lane above. The two documented-no-change items (`R01:U4`, `R01:L5`) are recorded in the B issue, and `R08:2.2` is gateway-side and handled through cross-repo issues + docs.
- Conflict check: the files shared across lanes are `OtOpcUaNodeManager.cs` (B in Wave 1, then H2's last task in Wave 3), `HistoryPaging.cs` (H1 only), the three `*HistorianDataSource.cs` (H1's `ReadRawAsync` in Wave 1, H2 for the rest in Wave 3), the recorder (H2 only), `DraftValidator.cs` (E only), and AdminUI driver forms (driver lanes in Wave 2, then D in Wave 3). No two lanes in the same wave share a file.
- Concurrency budget: up to 8 worktree builds in Wave 2. If the machine thrashes, split Wave 2 into 2×4: AbCip/AbLegacy/TwinCAT/FOCAS, then Modbus/S7/G1/G2.
+93 -4
View File
@@ -89,6 +89,13 @@ For 8-byte values (Int64 / Float64) the same labels apply pairwise.
`40001:F:5` = `Float32[5]` (consumes HR[0..9]). Array + bit suffix is
rejected. Strings are not arrays.
An array written through the grammar string alone (a **string-only** tag —
`addressString` with no `region`/`address` keys) is served and written by the
driver, but its OPC UA node stays **read-only**: whether a write fits is
decided at compose time, where the driver's family is not known. Pin the keys
(the editor's "Resolve with driver family") to make the array writable — see
"Array writes" below.
### Composition
The 3-field shorthand `40001:F:5` is parsed as `(type=F, count=5)` because
@@ -149,7 +156,11 @@ Beyond per-tag addressing, `ModbusDriverOptions` exposes (#139–#143):
- `UseFC15ForSingleCoilWrites` — force FC15 (write multiple coils
qty=1) for single-coil writes. Safety/audit PLCs may require this.
- `UseFC16ForSingleRegisterWrites` — same for FC16 vs FC06.
- `DisableFC23` — kill switch for FC23 (currently unused; reserved).
- `UseFC23ForMultiRegisterWrites` (default `false`) — opt-in: a 2–121-register
holding write is sent as one FC23 (write + read-back). A unit answering
exception 01 moves to FC16 until the next re-init. Without the key the
wire stays FC16 (#741).
- `DisableFC23` — hard kill switch: never send FC23, even when opted in.
### Subscribe (#141)
- Per-tag `Deadband` — suppress sub-threshold publishes on numeric tags.
@@ -165,9 +176,13 @@ Beyond per-tag addressing, `ModbusDriverOptions` exposes (#139–#143):
this tag in isolation regardless of `MaxReadGap`).
### Block-read coalescing (#143)
- `MaxReadGap` (default 0 = off) — gap budget the planner is willing to
bridge between adjacent register tags. With `MaxReadGap=10`, three tags
at HR 100/102/110 collapse into one FC03 of quantity 11.
- `MaxReadGap` — gap budget the planner is willing to bridge between
adjacent tags of one (unit, region). With `MaxReadGap=10`, three tags at
HR 100/102/110 collapse into one FC03 of quantity 11. An explicit `0` =
coalescing off (its historical meaning); the default `8` applies only when
the key is absent (#741). Coils / DiscreteInputs (gap in bits), `String`
scalars and same-word `BitInRegister` tags coalesce too.
- `DisableReadCoalescing` — turn coalescing off regardless of the gap.
### Coalescing auto-recovery (#148 / #150 / #151 / #152)
- A coalesced read that fails with a Modbus exception (write-only or
@@ -231,3 +246,77 @@ the current Kepware "Modbus Ethernet Driver Help" PDF and Ignition's
"Modbus Addressing" user-manual page — if a critical tool's mnemonics
have shifted, add aliases in `ModbusAddressParser.TryParseType` rather
than asking users to rewrite spreadsheets.
## Address strings, families and region/type rules (#741)
- **One authority: `ModbusTagAddressResolution` (Driver.Modbus.Contracts).** The
driver's tag reader, the deploy gate (`Inspect`), the AdminUI editor model
(load / save / `Validate` / the hint text) and the CSV import/export all ask
it which part of a blob is the address; none restates the rule. Its
agreement is pinned by one table-driven test that runs every persisted shape
through every consumer (`ModbusAddressAuthorityAgreementTests`).
- **Whenever a tag carries both `region` and `address`, those keys are its
address** — a classic tag, an editor-pinned tag, and a pre-#741 row that
carried an `addressString` beside its keys (the old driver never read the
string) all load from the keys, exactly as before, and a family change never
remaps them. Only a **string-only** blob (an `addressString` with NO
`region`/`address` — CSV / API authoring) is resolved through
`ModbusAddressParser` with the **driver's** `Family` / `MelsecSubFamily`; a
family that cannot parse it refuses the tag (BadNodeIdUnknown). It is never
served as HR[0].
- `"addressResolved": true` is editor display / CSV state only: it never makes
keys authoritative, and a marker on a blob WITHOUT keys is ignored (the blob
is string-only) and dropped on the next editor save.
- The AdminUI never writes default keys it did not resolve: the editor model
stamps `region`/`address`/type only when the loaded blob had the keys, or when
a field is set (hand edit, CSV address cell, Resolve). A string-only,
non-string-`addressString` or address-less blob keeps its keys unwritten, and
its CSV export leaves the address cells blank.
- The editor's "Resolve with driver family" persists the resolved
`region`/`address`/type fields plus `"addressResolved": true` (the pinned
state). Editing a resolved field by hand in the editor drops the string and
the marker. A typed CSV address cell keeps them only while the string still
describes the cells under some family (so an unchanged export → import of a
pinned row is lossless); a contradicting cell drops them, and the import
review grid reports that column as an override.
- Coils / DiscreteInputs carry only Bool. A bit-region tag persisted with a
register `dataType` (the pre-#741 editor saved `Int16`) is coerced to Bool,
never refused. A register region with `Bool` is refused (use
`BitInRegister`).
- Exception 01/02/03 are device refusals: per-tag Bad, one Warning per tag,
no driver Degraded; reads and writes share one comms-fault health slot.
## Array writes (#610, #741)
Whether an array node is client-writable is decided by
`DriverArrayWriteSupport` (Core.Abstractions) at compose time, and it says yes
only for what `ModbusDriver.WriteAsync` encodes end to end
(`ModbusArrayWriteSupportParityTests` pins the two against each other):
- **Keys only.** The tag must carry `region` + `address` (read as strictly as
the driver reads them). A string-only tag stays read-only — the composer
cannot resolve the grammar under the driver's family.
- **Coils:** one FC15 of at most 1968 coils (the spec ceiling; the driver does
not chunk). Any `dataType` — a bit region is coerced to Bool.
- **HoldingRegisters:** one FC16 whose register count fits
`maxRegistersPerWrite` (default 123; the smallest value on the driver or the
tag's device wins), and never more than FC16's protocol ceiling of 123 — a
larger configured cap does not lift it, because a conformant device refuses
the longer FC16. Registers per element: 1 / 2 / 4 for the numeric types,
`(stringLength+1)/2` for `String` (each element nul-padded to that width), 1
for `BitInRegister` (the span is read with one FC03, the addressed bit of each
register is set or cleared, and the span is written back with one FC16 —
the other 15 bits are preserved; `bitIndex` 0–15). Because that FC03 is one
unchunked read of the whole span, a BitInRegister array must also fit
`maxRegistersPerRead` (default and protocol ceiling 125).
- **Every span** — registers or coils — must end at or before address 65535
(a conformant device answers exception 02 past it). FC23 (`useFC23ForMultiRegisterWrites`) is opt-in,
capped at 121 registers and falls back to FC16 above that, so it never
changes which arrays are writable.
- **Never writable:** DiscreteInputs / InputRegisters, Bool in a register
region, a `String` with no `stringLength`, a `bitIndex` above 15, a missing
or zero `arrayLength`, and a `String` / `DateTime` node over a numeric or bit
element (a DateTime never converts; a String converts only when its text
happens to parse, which the server cannot promise). Elements otherwise convert, so a
ScadaBridge `int[]` to an `Int16` node is writable.
+9 -6
View File
@@ -1,5 +1,8 @@
# TwinCAT driver — v3 backlog
> **Status lives in Gitea.** Open items from this file are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status here; file new work as an issue.
The v2 TwinCAT driver is considered solid: 28 integration tests (14 `[TwinCATFact]` +
16-case `[TwinCATTheory]`) running live against the TCBSD fixture, 110 unit tests,
three latent driver bugs shaken out (notification cycle units, `STRING(N)` mapper,
@@ -27,7 +30,7 @@ user's approval, an `oee-tagdump-mac` route for the Mac's current VPN IP (`10.18
to **Z35474** (TC3) and **Z34834** (TC2, `5.69.178.220.1.1`, 10.201.45.197) by UDP-48899 AddRoute with the
factory login. ⚠️ A same-name AddRoute does **not** replace on either runtime: both presses now carry TWO
`oee-tagdump-mac` entries — the stale `10.181.198.189.1.1` and the new one (route table read back over ADS,
port 10000 / IG 803). Removing the stale one needs an ADS write and was not done. Every other press still
port 10000 / IG 803). Removing the stale one needs an ADS write and was not done (→ #755). Every other press still
has only the stale route, so it refuses the Mac; the VPN IP changes again ⇒ the routes go stale again.
## Status of the original items
@@ -38,14 +41,14 @@ has only the stale route, so it refuses the Mac; the VPN IP changes again ⇒ th
2. **Notification coalescing under load** — run the subscribe test while the PLC cycle is saturated.
Needs the TCBSD fixture (loading a production press is out of the question) and the `GVL_Perf` /
`FB_PerfChurn` fixture project from the stranded `auto/driver-gaps` branch. Fixture-gated.
`FB_PerfChurn` fixture project from the stranded `auto/driver-gaps` branch. Fixture-gated. → #755
3. **Multi-hop AMS route** — a test behind an IPC gateway with a chained route entry. No hardware.
3. **Multi-hop AMS route** — a test behind an IPC gateway with a chained route entry. No hardware. → #755
4. **License-rotation automation** — XAR's 7-day trial expires on schedule. Ops item, not code.
4. **License-rotation automation** — XAR's 7-day trial expires on schedule. Ops item, not code. → #755
5. **Lab rig** — cheapest IPC on a dedicated network; the only route that covers real EtherCAT I/O
timing and cycle jitter under CPU load. Blocked on hardware + budget.
timing and cycle jitter under CPU load. Blocked on hardware + budget. → #755
## Added by #612 / #645
@@ -69,4 +72,4 @@ has only the stale route, so it refuses the Mac; the VPN IP changes again ⇒ th
returned 10,628 variables. No `SymbolsLoadMode` option is needed.
9. **Discovered struct/FB member writes (#612)** — deferred: writes, never on a press; fixture-only.
Likewise **sum-write** was deliberately NOT ported from the stranded branch.
Likewise **sum-write** was deliberately NOT ported from the stranded branch. → #755
@@ -47,7 +47,7 @@ public class HistoryReadCommand : CommandBase
/// <summary>
/// Gets the optional aggregate name used when the operator wants processed history instead of raw samples.
/// </summary>
[CommandOption("aggregate", Description = "Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative")]
[CommandOption("aggregate", Description = "Aggregate function: Average, Minimum, Maximum, Count, Start, End, TimeAverage, TimeAverage2, Total, Total2, Interpolative, Range, Delta, MinimumActualTime, MaximumActualTime, PercentGood, PercentBad")]
public string? Aggregate { get; init; }
/// <summary>
@@ -199,24 +199,33 @@ public class HistoryReadCommand : CommandBase
}
}
private static readonly string SupportedAggregates = string.Join(", ", Enum.GetNames<AggregateType>());
private static AggregateType ParseAggregateType(string name)
{
return name.Trim().ToLowerInvariant() switch
var trimmed = name.Trim();
var alias = trimmed.ToLowerInvariant() switch
{
"average" or "avg" => AggregateType.Average,
"minimum" or "min" => AggregateType.Minimum,
"maximum" or "max" => AggregateType.Maximum,
"count" => AggregateType.Count,
"start" or "first" => AggregateType.Start,
"end" or "last" => AggregateType.End,
"standarddeviation" or "stddev" or "stdev" => AggregateType.StandardDeviation,
"timeaverage" or "timeavg" => AggregateType.TimeAverage,
"timeaverage2" or "timeavg2" => AggregateType.TimeAverage2,
"total" => AggregateType.Total,
"total2" => AggregateType.Total2,
"interpolative" or "interp" => AggregateType.Interpolative,
_ => throw new ArgumentException(
$"Unknown aggregate: '{name}'. Supported: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative")
"avg" => AggregateType.Average,
"min" => AggregateType.Minimum,
"max" => AggregateType.Maximum,
"first" => AggregateType.Start,
"last" => AggregateType.End,
"timeavg" => AggregateType.TimeAverage,
"timeavg2" => AggregateType.TimeAverage2,
"interp" => AggregateType.Interpolative,
"mintime" => AggregateType.MinimumActualTime,
"maxtime" => AggregateType.MaximumActualTime,
_ => (AggregateType?)null
};
if (alias is { } a) return a;
// Every enum name (the served set; a parity test pins it), case-insensitive. Numeric strings are refused:
// Enum.TryParse would accept "6" or "99" as an undefined value.
if (!int.TryParse(trimmed, out _) &&
Enum.TryParse<AggregateType>(trimmed, ignoreCase: true, out var parsed) && Enum.IsDefined(parsed))
return parsed;
throw new ArgumentException($"Unknown aggregate: '{name}'. Supported: {SupportedAggregates}");
}
}
@@ -56,7 +56,7 @@ public class ReadCommand : CommandBase
{
// The SDK's Session.ReadValue surfaces a Bad node status (BadNodeIdUnknown,
// BadUserAccessDenied, ...) as an exception. A read command's job is to REPORT that status,
// not crash with a stack trace (#577) — session-level failures still propagate.
// not crash with a stack trace — session-level failures still propagate.
await console.Output.WriteLineAsync($"Node: {NodeId}");
await console.Output.WriteLineAsync($"Value: ");
await console.Output.WriteLineAsync($"Status: {(StatusCode)ex.StatusCode} ({StatusCodes.GetBrowseName(ex.StatusCode)})");
@@ -24,6 +24,7 @@ internal static class DisposeBridge
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(5);
/// <summary>Runs <paramref name="work" /> to completion (or the timeout), swallowing every failure.</summary>
/// <param name="work">The asynchronous work to run; it receives a token that is cancelled when the timeout elapses.</param>
public static void Run(Func<CancellationToken, Task> work)
{
using var cts = new CancellationTokenSource(Timeout);
@@ -115,22 +115,6 @@ internal sealed class DefaultSessionAdapter : ISessionAdapter
return (nextCp, nextRefs ?? []);
}
/// <inheritdoc />
public async Task<bool> HasChildrenAsync(NodeId nodeId, CancellationToken ct)
{
var (_, _, references) = await _session.BrowseAsync(
null,
null,
nodeId,
1u,
BrowseDirection.Forward,
ReferenceTypeIds.HierarchicalReferences,
true,
0u);
return references != null && references.Count > 0;
}
/// <inheritdoc />
public async Task<Models.HistoryUpdateOutcome> HistoryUpdateAsync(HistoryUpdateDetails details, CancellationToken ct)
{
@@ -164,46 +148,16 @@ internal sealed class DefaultSessionAdapter : ISessionAdapter
ReturnBounds = false
};
var nodesToRead = new HistoryReadValueIdCollection
{
new HistoryReadValueId { NodeId = nodeId }
};
var allValues = new List<DataValue>();
byte[]? continuationPoint = null;
do
{
if (continuationPoint != null)
nodesToRead[0].ContinuationPoint = continuationPoint;
// Use the async overload so this method is genuinely asynchronous,
// honors the cancellation token, and does not block the caller's thread
// (which would block the UI dispatcher for client.ui consumers).
var response = await _session.HistoryReadAsync(
null,
new ExtensionObject(details),
TimestampsToReturn.Source,
continuationPoint != null,
nodesToRead,
ct).ConfigureAwait(false);
var results = response.Results;
if (results == null || results.Count == 0)
break;
var result = results[0];
// A refused read (BadHistoryOperationUnsupported, BadUserAccessDenied, ...) must surface; only
// BadNoData is an empty result. Swallowing it printed "0 values returned" for a refusal.
HistoryReadStatus.ThrowIfRefused(result.StatusCode, nodeId);
if (result.HistoryData is ExtensionObject ext && ext.Body is HistoryData historyData)
allValues.AddRange(historyData.DataValues);
continuationPoint = result.ContinuationPoint;
} while (continuationPoint != null && continuationPoint.Length > 0 && allValues.Count < maxValues);
return allValues;
// Continuation pages are READ, never released, until the server finishes or the cap is reached; a read stopped
// by the cap releases the server's cursor (HistoryReadPaging). This path used to pass
// `releaseContinuationPoints: continuationPoint != null`, which RELEASED the second page instead of reading it,
// silently truncating every raw read to its first server page.
return await HistoryReadPaging.ReadAllAsync(
(cp, release, token) => ReadHistoryPageAsync(details, nodeId, cp, release, TimestampsToReturn.Source, token),
result => result.HistoryData is ExtensionObject { Body: HistoryData historyData }
? historyData.DataValues
: [],
maxValues, nodeId, ct).ConfigureAwait(false);
}
/// <inheritdoc />
@@ -219,59 +173,41 @@ internal sealed class DefaultSessionAdapter : ISessionAdapter
ReturnBounds = false
};
return await HistoryReadPaging.ReadAllAsync(
(cp, release, token) => ReadHistoryPageAsync(details, nodeId, cp, release, TimestampsToReturn.Both, token),
ToModifiedEntries, maxValues, nodeId, ct).ConfigureAwait(false);
}
private static IEnumerable<Models.HistoryModifiedEntry> ToModifiedEntries(HistoryReadResult result)
{
if (result.HistoryData is not ExtensionObject { Body: HistoryModifiedData modified }) yield break;
for (var i = 0; i < modified.DataValues.Count; i++)
{
var info = i < modified.ModificationInfos.Count ? modified.ModificationInfos[i] : null;
yield return new Models.HistoryModifiedEntry(
modified.DataValues[i],
info?.ModificationTime ?? DateTime.MinValue,
info?.UpdateType ?? default,
info?.UserName);
}
}
/// <summary>Issues one single-node HistoryRead (or a release of its continuation point).</summary>
private async Task<HistoryReadResult?> ReadHistoryPageAsync(
ReadRawModifiedDetails details, NodeId nodeId, byte[]? continuationPoint, bool release,
TimestampsToReturn timestamps, CancellationToken ct)
{
var nodesToRead = new HistoryReadValueIdCollection
{
new HistoryReadValueId { NodeId = nodeId }
new HistoryReadValueId { NodeId = nodeId, ContinuationPoint = continuationPoint }
};
var all = new List<Models.HistoryModifiedEntry>();
byte[]? continuationPoint = null;
// The async overload honours the token and does not block the caller's thread (the UI dispatcher for
// client.ui consumers).
var response = await _session.HistoryReadAsync(
null, new ExtensionObject(details), timestamps, release, nodesToRead, ct).ConfigureAwait(false);
do
{
if (continuationPoint != null)
nodesToRead[0].ContinuationPoint = continuationPoint;
var response = await _session.HistoryReadAsync(
null,
new ExtensionObject(details),
TimestampsToReturn.Both,
false,
nodesToRead,
ct).ConfigureAwait(false);
var results = response.Results;
if (results == null || results.Count == 0)
break;
var result = results[0];
HistoryReadStatus.ThrowIfRefused(result.StatusCode, nodeId);
if (result.HistoryData is ExtensionObject ext && ext.Body is HistoryModifiedData modified)
{
for (var i = 0; i < modified.DataValues.Count; i++)
{
var info = i < modified.ModificationInfos.Count ? modified.ModificationInfos[i] : null;
all.Add(new Models.HistoryModifiedEntry(
modified.DataValues[i],
info?.ModificationTime ?? DateTime.MinValue,
info?.UpdateType ?? default,
info?.UserName));
}
}
continuationPoint = result.ContinuationPoint;
} while (continuationPoint != null && continuationPoint.Length > 0 && all.Count < maxValues);
// A read stopped by the client's own cap leaves a server cursor behind; release it.
if (continuationPoint is { Length: > 0 })
{
nodesToRead[0].ContinuationPoint = continuationPoint;
await _session.HistoryReadAsync(null, new ExtensionObject(details), TimestampsToReturn.Both, true,
nodesToRead, ct).ConfigureAwait(false);
}
return all;
return response.Results is { Count: > 0 } results ? results[0] : null;
}
/// <inheritdoc />
@@ -327,9 +263,22 @@ internal sealed class DefaultSessionAdapter : ISessionAdapter
};
_session.AddSubscription(subscription);
await subscription.CreateAsync(ct);
try
{
await subscription.CreateAsync(ct);
}
catch
{
// A subscription the server never created must not linger in Session.Subscriptions either.
try { await _session.RemoveSubscriptionAsync(subscription, CancellationToken.None); }
catch (Exception ex) { Logger.Warning(ex, "Error removing a subscription that failed to create"); }
throw;
}
return new DefaultSubscriptionAdapter(subscription);
// Deleting the adapter also removes the subscription from the SDK session, so a re-targeted alarm/data
// subscription does not accumulate dead Subscription objects in Session.Subscriptions.
return new DefaultSubscriptionAdapter(subscription,
async (s, token) => await _session.RemoveSubscriptionAsync(s, token).ConfigureAwait(false));
}
/// <inheritdoc />
@@ -12,14 +12,23 @@ internal sealed class DefaultSubscriptionAdapter : ISubscriptionAdapter
private static readonly ILogger Logger = Log.ForContext<DefaultSubscriptionAdapter>();
private readonly Dictionary<uint, MonitoredItem> _monitoredItems = new();
private readonly Subscription _subscription;
private readonly Func<Subscription, CancellationToken, Task>? _detach;
private int _detached;
/// <summary>
/// Wraps a live OPC UA subscription so client code can manage monitored items through a testable abstraction.
/// </summary>
/// <param name="subscription">The underlying OPC UA subscription that owns monitored items for this client workflow.</param>
public DefaultSubscriptionAdapter(Subscription subscription)
/// <param name="detach">
/// Removes the subscription from its owning SDK <c>Session</c> once it is deleted. Without it a deleted
/// subscription stays in <c>Session.Subscriptions</c> until the session is disposed, so a UI that re-targets
/// alarms repeatedly accumulates dead subscriptions the SDK still iterates (transfer / recreate on reconnect).
/// </param>
public DefaultSubscriptionAdapter(Subscription subscription,
Func<Subscription, CancellationToken, Task>? detach = null)
{
_subscription = subscription;
_detach = detach;
}
/// <inheritdoc />
@@ -113,17 +122,39 @@ internal sealed class DefaultSubscriptionAdapter : ISubscriptionAdapter
Logger.Warning(ex, "Error deleting subscription");
}
// Detach whether or not the server-side delete succeeded: the adapter is finished with it either way.
await DetachAsync(ct);
_monitoredItems.Clear();
}
private async Task DetachAsync(CancellationToken ct)
{
if (_detach == null || Interlocked.Exchange(ref _detached, 1) != 0) return;
try
{
await _detach(_subscription, ct);
}
catch (Exception ex)
{
Logger.Warning(ex, "Error removing subscription from its session");
}
}
/// <summary>
/// Releases the wrapped OPC UA subscription and clears tracked monitored items held by the adapter.
/// </summary>
public void Dispose()
{
// One bounded bridge (5 s) for delete + detach, so a dispose never blocks longer than before (review m6).
// DetachAsync swallows its own failures; a failed delete must not skip the detach.
try
{
DisposeBridge.Run(ct => _subscription.DeleteAsync(true, ct));
DisposeBridge.Run(async ct =>
{
try { await _subscription.DeleteAsync(true, ct); }
catch (Exception ex) { Logger.Warning(ex, "Error deleting subscription"); }
await DetachAsync(ct);
});
}
catch
{
@@ -90,14 +90,6 @@ internal interface ISessionAdapter : IDisposable
Task<(byte[]? ContinuationPoint, ReferenceDescriptionCollection References)> BrowseNextAsync(
byte[] continuationPoint, CancellationToken ct = default);
/// <summary>
/// Checks whether a node has any forward hierarchical child references.
/// </summary>
/// <param name="nodeId">The node to inspect for child objects or variables.</param>
/// <param name="ct">The cancellation token that aborts the child lookup.</param>
/// <returns><c>true</c> when the node has at least one forward hierarchical child reference; otherwise <c>false</c>.</returns>
Task<bool> HasChildrenAsync(NodeId nodeId, CancellationToken ct = default);
/// <summary>
/// Reads raw historical data.
/// </summary>
@@ -23,12 +23,17 @@ public static class AggregateTypeMapper
AggregateType.Count => ObjectIds.AggregateFunction_Count,
AggregateType.Start => ObjectIds.AggregateFunction_Start,
AggregateType.End => ObjectIds.AggregateFunction_End,
AggregateType.StandardDeviation => ObjectIds.AggregateFunction_StandardDeviationPopulation,
AggregateType.TimeAverage => ObjectIds.AggregateFunction_TimeAverage,
AggregateType.TimeAverage2 => ObjectIds.AggregateFunction_TimeAverage2,
AggregateType.Total => ObjectIds.AggregateFunction_Total,
AggregateType.Total2 => ObjectIds.AggregateFunction_Total2,
AggregateType.Interpolative => ObjectIds.AggregateFunction_Interpolative,
AggregateType.Range => ObjectIds.AggregateFunction_Range,
AggregateType.Delta => ObjectIds.AggregateFunction_Delta,
AggregateType.MinimumActualTime => ObjectIds.AggregateFunction_MinimumActualTime,
AggregateType.MaximumActualTime => ObjectIds.AggregateFunction_MaximumActualTime,
AggregateType.PercentGood => ObjectIds.AggregateFunction_PercentGood,
AggregateType.PercentBad => ObjectIds.AggregateFunction_PercentBad,
_ => throw new ArgumentOutOfRangeException(nameof(aggregate), aggregate, "Unknown AggregateType value.")
};
}
@@ -0,0 +1,96 @@
using Opc.Ua;
using Serilog;
namespace ZB.MOM.WW.OtOpcUa.Client.Shared.Helpers;
/// <summary>
/// The client-side HistoryRead paging loop shared by the raw and modified reads. Continuation pages are
/// READ (<c>releaseContinuationPoints = false</c>) while the caller still wants values; a read stopped by the
/// caller's own cap while the server still holds a cursor releases that cursor so it does not occupy one of the
/// session's continuation-point slots until eviction.
/// </summary>
internal static class HistoryReadPaging
{
private static readonly ILogger Logger = Log.ForContext(typeof(HistoryReadPaging));
/// <summary>Reads (or, with <paramref name="releaseContinuationPoints"/>, releases) one page.</summary>
/// <param name="continuationPoint">The cursor to resume, or <see langword="null"/> for the first page.</param>
/// <param name="releaseContinuationPoints"><see langword="true"/> to release the cursor instead of reading.</param>
/// <param name="ct">The caller's cancellation token.</param>
/// <returns>The per-node result, or <see langword="null"/> when the response carried none.</returns>
public delegate Task<HistoryReadResult?> PageReader(
byte[]? continuationPoint, bool releaseContinuationPoints, CancellationToken ct);
/// <summary>The bound on a best-effort release that does not ride the caller's (possibly cancelled) token.</summary>
internal static readonly TimeSpan ReleaseTimeout = TimeSpan.FromSeconds(5);
/// <summary>Reads pages until the server finishes or the caller's cap is reached.</summary>
/// <typeparam name="T">The element type extracted from each page.</typeparam>
/// <param name="readPage">Issues one HistoryRead for the single node.</param>
/// <param name="extract">Extracts the page's elements from a result.</param>
/// <param name="maxValues">The caller's cap; must be positive. The result is trimmed to it.</param>
/// <param name="nodeId">The node being read, for refusal messages.</param>
/// <param name="ct">The caller's cancellation token.</param>
/// <returns>At most <paramref name="maxValues"/> elements.</returns>
/// <exception cref="ArgumentOutOfRangeException"><paramref name="maxValues"/> is not positive: an uncapped client
/// read would materialise a tag's whole window (the client side of review 08 1.1).</exception>
/// <exception cref="ServiceResultException">A page was refused (see <see cref="HistoryReadStatus"/>).</exception>
/// <exception cref="OperationCanceledException"><paramref name="ct"/> was cancelled mid-read; the outstanding
/// cursor is released (best effort) first.</exception>
public static async Task<List<T>> ReadAllAsync<T>(
PageReader readPage, Func<HistoryReadResult, IEnumerable<T>> extract, int maxValues, NodeId nodeId,
CancellationToken ct)
{
ArgumentOutOfRangeException.ThrowIfNegativeOrZero(maxValues);
var all = new List<T>();
byte[]? continuationPoint = null;
try
{
while (true)
{
var result = await readPage(continuationPoint, false, ct).ConfigureAwait(false);
if (result == null) return Trim(all, maxValues);
HistoryReadStatus.ThrowIfRefused(result.StatusCode, nodeId);
all.AddRange(extract(result));
continuationPoint = result.ContinuationPoint;
if (continuationPoint is not { Length: > 0 }) return Trim(all, maxValues);
if (all.Count >= maxValues) break;
}
}
catch (OperationCanceledException)
{
// Cancelled between pages (or during page N+1) with a cursor outstanding: release it rather than leave it
// occupying one of the session's continuation-point slots, then surface the cancellation.
if (continuationPoint is { Length: > 0 }) await ReleaseAsync(readPage, continuationPoint, nodeId);
throw;
}
// Stopped by the client's own cap: the server still holds a cursor for this session. Release it — on its own
// bounded token, so a cancellation racing the release cannot lose the values already in hand.
await ReleaseAsync(readPage, continuationPoint, nodeId);
return Trim(all, maxValues);
}
private static List<T> Trim<T>(List<T> all, int maxValues)
{
if (all.Count > maxValues) all.RemoveRange(maxValues, all.Count - maxValues);
return all;
}
private static async Task ReleaseAsync(PageReader readPage, byte[] continuationPoint, NodeId nodeId)
{
try
{
using var cts = new CancellationTokenSource(ReleaseTimeout);
await readPage(continuationPoint, true, cts.Token).ConfigureAwait(false);
}
catch (Exception ex)
{
Logger.Warning(ex, "Releasing the HistoryRead continuation point for {NodeId} failed", nodeId);
}
}
}
@@ -1,43 +1,64 @@
namespace ZB.MOM.WW.OtOpcUa.Client.Shared.Models;
/// <summary>
/// Aggregate functions for processed history reads.
/// Aggregate functions for processed history reads — exactly the seventeen the server's SQL historian backends
/// serve (<c>AggregateFamilies.Served</c>; a parity test pins it). The Gateway backend serves a subset and refuses
/// the rest with <c>BadAggregateNotSupported</c>.
/// </summary>
/// <remarks>
/// Values are explicit so the pre-existing members keep their numbers. 6 (the former <c>StandardDeviation</c>,
/// which no backend serves) is retired; do not reuse it.
/// </remarks>
public enum AggregateType
{
/// <summary>Average of values in the interval.</summary>
Average,
Average = 0,
/// <summary>Minimum value in the interval.</summary>
Minimum,
Minimum = 1,
/// <summary>Maximum value in the interval.</summary>
Maximum,
Maximum = 2,
/// <summary>Count of values in the interval.</summary>
Count,
Count = 3,
/// <summary>First value in the interval.</summary>
Start,
Start = 4,
/// <summary>Last value in the interval.</summary>
End,
/// <summary>Population standard deviation of values in the interval.</summary>
StandardDeviation,
End = 5,
/// <summary>Time-weighted average, interpolated bounds (Part 13 TimeAverage).</summary>
TimeAverage,
TimeAverage = 7,
/// <summary>Time-weighted average, stepped bounds (Part 13 TimeAverage2).</summary>
TimeAverage2,
TimeAverage2 = 8,
/// <summary>Time-weighted integral over the interval (Part 13 Total).</summary>
Total,
Total = 9,
/// <summary>Time-weighted integral, stepped bounds (Part 13 Total2).</summary>
Total2,
Total2 = 10,
/// <summary>Interpolated value at the interval start (Part 13 Interpolative).</summary>
Interpolative
}
Interpolative = 11,
/// <summary>Maximum minus minimum in the interval (Part 13 Range).</summary>
Range = 12,
/// <summary>Last minus first value in the interval (Part 13 Delta).</summary>
Delta = 13,
/// <summary>Minimum value with its own timestamp (Part 13 MinimumActualTime).</summary>
MinimumActualTime = 14,
/// <summary>Maximum value with its own timestamp (Part 13 MaximumActualTime).</summary>
MaximumActualTime = 15,
/// <summary>Percentage of the interval with Good data (Part 13 PercentGood).</summary>
PercentGood = 16,
/// <summary>Percentage of the interval with Bad data (Part 13 PercentBad).</summary>
PercentBad = 17
}
@@ -34,7 +34,9 @@ public sealed class BrowseResult
public string NodeClass { get; }
/// <summary>
/// Whether the node has child references.
/// Whether the node MAY have child references: <see langword="true"/> for every Object, confirmed lazily when the
/// caller browses it (an expanded empty folder simply yields no children). Variables and Methods report
/// <see langword="false"/>.
/// </summary>
public bool HasChildren { get; }
}
@@ -15,16 +15,17 @@ public sealed class OpcUaClientService : IOpcUaClientService
{
private static readonly ILogger Logger = Log.ForContext<OpcUaClientService>();
// Guards all access to the subscription-bookkeeping state below
// (_activeDataSubscriptions and _activeAlarmSubscription). The dictionary
// and tuple are mutated from the caller thread, the keep-alive failover
// path, and DisconnectAsync, so every read/write must be inside this lock.
// Guards all access to the subscription state below (_activeDataSubscriptions, _activeAlarmSubscription,
// _dataSubscription, _alarmSubscription). They are touched from the caller thread, the keep-alive failover path
// and DisconnectAsync, so every read/write is inside this lock — including failover's null-outs.
private readonly object _subscriptionLock = new();
// Serialises SubscribeAlarmsAsync / UnsubscribeAlarmsAsync so concurrent callers
// cannot both pass the _alarmSubscription == null check and create duplicate event
// subscriptions. Capacity 1 = at most one waiter; async-friendly (no thread-block).
private readonly SemaphoreSlim _alarmSubscribeSemaphore = new(1, 1);
// Serialises every subscription operation that awaits I/O (Subscribe/Unsubscribe, SubscribeAlarms/
// UnsubscribeAlarms) AND the failover's teardown + reconnect + replay. A subscribe is therefore either
// wholly before a failover (recorded, then replayed onto the new session) or wholly after it (on the new
// session) — never half on the dying session. Async-friendly (no thread block). Deliberately never disposed:
// async-only waits hold no kernel handle, and a failover still running at Dispose releases it afterwards.
private readonly SemaphoreSlim _subscriptionGate = new(1, 1);
// Track active data subscriptions for replay after failover
private readonly Dictionary<string, (NodeId NodeId, int IntervalMs, uint Handle)> _activeDataSubscriptions = new();
@@ -34,6 +35,15 @@ public sealed class OpcUaClientService : IOpcUaClientService
// run at a time. 0 = idle, 1 = failover in progress (Interlocked-managed).
private int _failoverInProgress;
// The connection epoch (review I1): ConnectAsync and DisconnectAsync each start a new one (under
// _subscriptionLock). A failover runs under the epoch it started in and abandons itself — disposing any session it
// opened, making no state transition — the moment the epoch moves, so a user Disconnect + Connect during a
// failover is never undone by it. A keep-alive from a previous connection's session cannot start one either.
private long _connectionEpoch;
/// <summary>The most recent keep-alive-triggered failover task (test seam: lets a test await it).</summary>
internal Task? LastFailover { get; private set; }
private readonly IApplicationConfigurationFactory _configFactory;
private readonly IEndpointDiscovery _endpointDiscovery;
@@ -99,6 +109,15 @@ public sealed class OpcUaClientService : IOpcUaClientService
ThrowIfDisposed();
settings.Validate();
// Overwriting a live (or reconnecting) session would leak it and orphan its subscriptions and keep-alive
// handler. The caller must disconnect first.
if (_state != ConnectionState.Disconnected || _session != null)
throw new InvalidOperationException(
$"Already {_state.ToString().ToLowerInvariant()}; call DisconnectAsync before connecting again.");
long epoch;
lock (_subscriptionLock) epoch = ++_connectionEpoch;
_settings = settings;
_allEndpointUrls = FailoverUrlParser.Parse(settings.EndpointUrl, settings.FailoverUrls);
_currentEndpointIndex = 0;
@@ -108,11 +127,11 @@ public sealed class OpcUaClientService : IOpcUaClientService
try
{
var session = await ConnectToEndpointAsync(settings, _allEndpointUrls[0], ct);
_session = session;
lock (_subscriptionLock) _session = session;
session.RegisterKeepAliveHandler(isGood =>
{
if (!isGood) _ = HandleKeepAliveFailureAsync();
if (!isGood) OnKeepAliveFailed(epoch);
});
CurrentConnectionInfo = BuildConnectionInfo(session);
@@ -133,6 +152,9 @@ public sealed class OpcUaClientService : IOpcUaClientService
if (_state == ConnectionState.Disconnected)
return;
// A new epoch first: a failover still connecting abandons itself instead of resurrecting the connection.
lock (_subscriptionLock) _connectionEpoch++;
var endpointUrl = _session?.EndpointUrl ?? _settings?.EndpointUrl ?? string.Empty;
// Snapshot the subscription adapter references under the lock so a concurrent
@@ -224,18 +246,15 @@ public sealed class OpcUaClientService : IOpcUaClientService
while (references.Count > 0)
{
// Child detection is lazy: an Object MAY have children and is confirmed when the caller expands or
// recurses into it. Probing each Object child with its own Browse made one level of N folders cost N + 1
// round trips, and a recursive subscribe over a 50k-node tree ~50k extra Browse calls.
foreach (var reference in references)
{
var childNodeId = ExpandedNodeId.ToNodeId(reference.NodeId, _session.NamespaceUris);
var hasChildren = reference.NodeClass == NodeClass.Object &&
await _session.HasChildrenAsync(childNodeId, ct);
results.Add(new BrowseResult(
reference.NodeId.ToString(),
reference.DisplayName?.Text ?? string.Empty,
reference.NodeClass.ToString(),
hasChildren));
}
reference.NodeClass == NodeClass.Object));
if (continuationPoint is not { Length: > 0 } pending) break;
(continuationPoint, references) = await _session.BrowseNextAsync(pending, ct);
@@ -251,20 +270,39 @@ public sealed class OpcUaClientService : IOpcUaClientService
ThrowIfNotConnected();
var nodeIdStr = nodeId.ToString();
lock (_subscriptionLock)
await _subscriptionGate.WaitAsync(ct);
try
{
if (_activeDataSubscriptions.ContainsKey(nodeIdStr))
return; // Already subscribed
// A failover may have run (and failed) while this call waited for the gate.
ThrowIfNotConnected();
ISubscriptionAdapter? subscription;
ISessionAdapter session;
lock (_subscriptionLock)
{
if (_activeDataSubscriptions.ContainsKey(nodeIdStr))
return; // Already subscribed
subscription = _dataSubscription;
session = _session!;
}
if (subscription == null)
{
subscription = await session.CreateSubscriptionAsync(intervalMs, ct);
lock (_subscriptionLock) _dataSubscription = subscription;
}
var handle = await subscription.AddDataChangeMonitoredItemAsync(
nodeId, intervalMs, OnDataChangeNotification, ct);
lock (_subscriptionLock)
{
_activeDataSubscriptions[nodeIdStr] = (nodeId, intervalMs, handle);
}
}
if (_dataSubscription == null) _dataSubscription = await _session!.CreateSubscriptionAsync(intervalMs, ct);
var handle = await _dataSubscription.AddDataChangeMonitoredItemAsync(
nodeId, intervalMs, OnDataChangeNotification, ct);
lock (_subscriptionLock)
finally
{
_activeDataSubscriptions[nodeIdStr] = (nodeId, intervalMs, handle);
_subscriptionGate.Release();
}
Logger.Debug("Subscribed to data changes on {NodeId}", nodeId);
@@ -276,18 +314,35 @@ public sealed class OpcUaClientService : IOpcUaClientService
ThrowIfDisposed();
var nodeIdStr = nodeId.ToString();
(NodeId NodeId, int IntervalMs, uint Handle) sub;
lock (_subscriptionLock)
await _subscriptionGate.WaitAsync(ct);
try
{
if (!_activeDataSubscriptions.TryGetValue(nodeIdStr, out sub))
return; // Not subscribed, safe to ignore
(NodeId NodeId, int IntervalMs, uint Handle) sub;
ISubscriptionAdapter? subscription;
lock (_subscriptionLock)
{
if (!_activeDataSubscriptions.TryGetValue(nodeIdStr, out sub))
return; // Not subscribed, safe to ignore
subscription = _dataSubscription;
}
try
{
if (subscription != null) await subscription.RemoveMonitoredItemAsync(sub.Handle, ct);
}
finally
{
// The caller asked to stop: drop the bookkeeping even when the server-side remove failed (a session
// that died under us), or a later failover would replay the item onto the new session (review m5).
lock (_subscriptionLock)
{
_activeDataSubscriptions.Remove(nodeIdStr);
}
}
}
if (_dataSubscription != null) await _dataSubscription.RemoveMonitoredItemAsync(sub.Handle, ct);
lock (_subscriptionLock)
finally
{
_activeDataSubscriptions.Remove(nodeIdStr);
_subscriptionGate.Release();
}
Logger.Debug("Unsubscribed from data changes on {NodeId}", nodeId);
@@ -303,17 +358,25 @@ public sealed class OpcUaClientService : IOpcUaClientService
// Serialise check-and-create under the semaphore: concurrent
// callers cannot both pass the null-check and create duplicate alarm subscriptions.
// The semaphore is async-friendly; the subscription I/O runs inside the critical section.
await _alarmSubscribeSemaphore.WaitAsync(ct);
await _subscriptionGate.WaitAsync(ct);
try
{
if (_alarmSubscription != null)
return; // Already subscribed to alarms
ThrowIfNotConnected();
ISessionAdapter session;
lock (_subscriptionLock)
{
if (_alarmSubscription != null)
return; // Already subscribed to alarms
session = _session!;
}
var monitorNode = sourceNodeId ?? ObjectIds.Server;
_alarmSubscription = await _session!.CreateSubscriptionAsync(intervalMs, ct);
var subscription = await session.CreateSubscriptionAsync(intervalMs, ct);
lock (_subscriptionLock) _alarmSubscription = subscription;
var filter = CreateAlarmEventFilter();
await _alarmSubscription.AddEventMonitoredItemAsync(
await subscription.AddEventMonitoredItemAsync(
monitorNode, intervalMs, filter, OnAlarmEventNotification, ct);
lock (_subscriptionLock)
@@ -325,7 +388,7 @@ public sealed class OpcUaClientService : IOpcUaClientService
}
finally
{
_alarmSubscribeSemaphore.Release();
_subscriptionGate.Release();
}
}
@@ -334,15 +397,27 @@ public sealed class OpcUaClientService : IOpcUaClientService
{
ThrowIfDisposed();
if (_alarmSubscription == null)
return;
await _alarmSubscription.DeleteAsync(ct);
_alarmSubscription = null;
lock (_subscriptionLock)
await _subscriptionGate.WaitAsync(ct);
try
{
_activeAlarmSubscription = null;
ISubscriptionAdapter? subscription;
lock (_subscriptionLock)
{
subscription = _alarmSubscription;
if (subscription == null) return;
}
await subscription.DeleteAsync(ct);
lock (_subscriptionLock)
{
_alarmSubscription = null;
_activeAlarmSubscription = null;
}
}
finally
{
_subscriptionGate.Release();
}
Logger.Debug("Unsubscribed from alarm events");
@@ -354,10 +429,12 @@ public sealed class OpcUaClientService : IOpcUaClientService
ThrowIfDisposed();
ThrowIfNotConnected();
if (_alarmSubscription == null)
ISubscriptionAdapter? subscription;
lock (_subscriptionLock) subscription = _alarmSubscription;
if (subscription == null)
throw new InvalidOperationException("No alarm subscription is active.");
await _alarmSubscription.ConditionRefreshAsync(ct);
await subscription.ConditionRefreshAsync(ct);
Logger.Debug("Condition refresh requested");
}
@@ -462,7 +539,7 @@ public sealed class OpcUaClientService : IOpcUaClientService
ThrowIfDisposed();
ThrowIfNotConnected();
// The server's ShelvingState child is SDK-created with a numeric namespace-0 id, so a synthesized
// The server's ShelvingState child is SDK-created with a numeric id in namespace 0, so a synthesized
// "<condition>.ShelvingState" node id never exists. The shelve methods are called on the condition
// object itself with the AlarmConditionType_ShelvingState_* method ids, which the server dispatches
// to the condition's ShelvingState methods.
@@ -640,13 +717,16 @@ public sealed class OpcUaClientService : IOpcUaClientService
/// </summary>
public void Dispose()
{
if (_disposed) return;
_disposed = true;
lock (_subscriptionLock)
{
if (_disposed) return;
_disposed = true;
_connectionEpoch++; // a failover still running abandons itself
}
_dataSubscription?.Dispose();
_alarmSubscription?.Dispose();
_session?.Dispose();
_alarmSubscribeSemaphore.Dispose();
lock (_subscriptionLock)
{
@@ -688,19 +768,27 @@ public sealed class OpcUaClientService : IOpcUaClientService
ct);
}
private async Task HandleKeepAliveFailureAsync()
/// <summary>A bad keep-alive from the session of connection <paramref name="epoch"/>.</summary>
private void OnKeepAliveFailed(long epoch)
{
// Serialize failover: the OPC UA stack raises KeepAlive repeatedly
// while a session is down, so multiple bad keep-alives can fire before
// the first failover loop finishes. CompareExchange atomically claims
// the failover slot; a re-entrant call sees 1 and returns immediately,
// guaranteeing exactly one failover loop runs at a time.
// A late keep-alive from a previous connection's session is not about the current one.
if (!IsCurrentEpoch(epoch)) return;
// Serialize failover: the OPC UA stack raises KeepAlive repeatedly while a session is down, so several bad
// keep-alives can fire before the first failover loop finishes. CompareExchange claims the failover slot; a
// re-entrant call sees 1 and returns, so exactly one loop runs — and only the winner sets the test seam
// (review m8: a re-entrant call used to overwrite it with an already-completed no-op).
if (Interlocked.CompareExchange(ref _failoverInProgress, 1, 0) != 0)
return;
LastFailover = HandleKeepAliveFailureAsync(epoch);
}
private async Task HandleKeepAliveFailureAsync(long epoch)
{
try
{
await RunFailoverAsync();
await RunFailoverAsync(epoch);
}
finally
{
@@ -708,74 +796,136 @@ public sealed class OpcUaClientService : IOpcUaClientService
}
}
private async Task RunFailoverAsync()
private bool IsCurrentEpoch(long epoch)
{
lock (_subscriptionLock) return _connectionEpoch == epoch && !_disposed;
}
/// <summary>
/// A state transition made only while <paramref name="epoch"/> is still current (checked and applied under
/// <see cref="_subscriptionLock"/>, the event raised outside it).
/// </summary>
/// <returns><see langword="false"/> when the epoch has moved on (the caller must abandon).</returns>
private bool TransitionStateIfCurrent(long epoch, ConnectionState newState, string endpointUrl)
{
ConnectionState oldState;
lock (_subscriptionLock)
{
if (_connectionEpoch != epoch || _disposed) return false;
oldState = _state;
if (oldState == newState) return true;
_state = newState;
}
ConnectionStateChanged?.Invoke(this, new ConnectionStateChangedEventArgs(oldState, newState, endpointUrl));
return true;
}
private async Task RunFailoverAsync(long epoch)
{
if (_state == ConnectionState.Reconnecting || _state == ConnectionState.Disconnected)
return;
var oldEndpoint = _session?.EndpointUrl ?? string.Empty;
TransitionState(ConnectionState.Reconnecting, oldEndpoint);
var oldSession = _session;
var oldEndpoint = oldSession?.EndpointUrl ?? string.Empty;
if (!TransitionStateIfCurrent(epoch, ConnectionState.Reconnecting, oldEndpoint)) return;
Logger.Warning("Session lost on {EndpointUrl}. Attempting failover...", oldEndpoint);
// Close old session
if (_session != null)
// Dispose the dead session BEFORE taking the gate: an in-flight subscribe holding the gate is blocked on
// that session, and disposing it is what makes the call fail fast and release the gate.
if (oldSession != null)
{
try
{
_session.Dispose();
oldSession.Dispose();
}
catch
{
}
_session = null;
}
_dataSubscription = null;
_alarmSubscription = null;
if (_settings == null || _allEndpointUrls == null)
// The rest runs under the subscription gate (and its null-outs under _subscriptionLock): no subscribe can
// interleave with the teardown, the reconnect or the replay.
await _subscriptionGate.WaitAsync();
try
{
TransitionState(ConnectionState.Disconnected, oldEndpoint);
return;
}
// Try each endpoint
for (var attempt = 0; attempt < _allEndpointUrls.Length; attempt++)
{
_currentEndpointIndex = (_currentEndpointIndex + 1) % _allEndpointUrls.Length;
var url = _allEndpointUrls[_currentEndpointIndex];
try
// The user's settings / endpoint list as of THIS epoch: a Connect during the failover replaces the fields.
ConnectionSettings? settings;
string[]? endpoints;
lock (_subscriptionLock)
{
Logger.Information("Failover attempt to {EndpointUrl}", url);
var session = await ConnectToEndpointAsync(_settings, url, CancellationToken.None);
_session = session;
if (_connectionEpoch != epoch || _disposed) return; // the user disconnected / reconnected meanwhile
if (ReferenceEquals(_session, oldSession)) _session = null;
_dataSubscription = null;
_alarmSubscription = null;
settings = _settings;
endpoints = _allEndpointUrls;
}
session.RegisterKeepAliveHandler(isGood =>
{
if (!isGood) _ = HandleKeepAliveFailureAsync();
});
CurrentConnectionInfo = BuildConnectionInfo(session);
TransitionState(ConnectionState.Connected, url);
Logger.Information("Failover succeeded to {EndpointUrl}", url);
// Replay subscriptions
await ReplaySubscriptionsAsync();
if (settings == null || endpoints == null)
{
TransitionStateIfCurrent(epoch, ConnectionState.Disconnected, oldEndpoint);
return;
}
catch (Exception ex)
{
Logger.Warning(ex, "Failover to {EndpointUrl} failed", url);
}
}
Logger.Error("All failover endpoints unreachable");
TransitionState(ConnectionState.Disconnected, oldEndpoint);
// Try each endpoint
for (var attempt = 0; attempt < endpoints.Length; attempt++)
{
if (!IsCurrentEpoch(epoch)) return; // stop trying once the user has moved on (m5)
_currentEndpointIndex = (_currentEndpointIndex + 1) % endpoints.Length;
var url = endpoints[_currentEndpointIndex];
try
{
Logger.Information("Failover attempt to {EndpointUrl}", url);
var session = await ConnectToEndpointAsync(settings, url, CancellationToken.None);
// A Disconnect / Connect / Dispose that ran while this attempt was connecting wins: never
// resurrect the old connection or overwrite the user's new session. Checked and installed
// atomically against the epoch.
bool installed;
lock (_subscriptionLock)
{
installed = _connectionEpoch == epoch && !_disposed;
if (installed) _session = session;
}
if (!installed)
{
session.Dispose();
return;
}
session.RegisterKeepAliveHandler(isGood =>
{
if (!isGood) OnKeepAliveFailed(epoch);
});
CurrentConnectionInfo = BuildConnectionInfo(session);
if (!TransitionStateIfCurrent(epoch, ConnectionState.Connected, url)) return;
Logger.Information("Failover succeeded to {EndpointUrl}", url);
// Replay subscriptions
await ReplaySubscriptionsAsync(session);
return;
}
catch (Exception ex)
{
Logger.Warning(ex, "Failover to {EndpointUrl} failed", url);
}
}
if (TransitionStateIfCurrent(epoch, ConnectionState.Disconnected, oldEndpoint))
Logger.Error("All failover endpoints unreachable");
}
finally
{
_subscriptionGate.Release();
}
}
private async Task ReplaySubscriptionsAsync()
/// <summary>Re-creates the recorded subscriptions on <paramref name="session"/>. Runs under the subscription gate.</summary>
private async Task ReplaySubscriptionsAsync(ISessionAdapter session)
{
// Snapshot the bookkeeping state under the lock, then clear it so the
// replayed handles can be recorded fresh as each monitored item is
@@ -791,13 +941,17 @@ public sealed class OpcUaClientService : IOpcUaClientService
}
// Replay data subscriptions
ISubscriptionAdapter? dataSubscription = null;
foreach (var (nodeIdStr, (nodeId, intervalMs, _)) in subscriptions)
try
{
if (_dataSubscription == null)
_dataSubscription = await _session!.CreateSubscriptionAsync(intervalMs, CancellationToken.None);
if (dataSubscription == null)
{
dataSubscription = await session.CreateSubscriptionAsync(intervalMs, CancellationToken.None);
lock (_subscriptionLock) _dataSubscription = dataSubscription;
}
var handle = await _dataSubscription.AddDataChangeMonitoredItemAsync(
var handle = await dataSubscription.AddDataChangeMonitoredItemAsync(
nodeId, intervalMs, OnDataChangeNotification, CancellationToken.None);
lock (_subscriptionLock)
@@ -817,9 +971,10 @@ public sealed class OpcUaClientService : IOpcUaClientService
try
{
var monitorNode = sourceNodeId ?? ObjectIds.Server;
_alarmSubscription = await _session!.CreateSubscriptionAsync(intervalMs, CancellationToken.None);
var replayedAlarms = await session.CreateSubscriptionAsync(intervalMs, CancellationToken.None);
lock (_subscriptionLock) _alarmSubscription = replayedAlarms;
var filter = CreateAlarmEventFilter();
await _alarmSubscription.AddEventMonitoredItemAsync(
await replayedAlarms.AddEventMonitoredItemAsync(
monitorNode, intervalMs, filter, OnAlarmEventNotification, CancellationToken.None);
lock (_subscriptionLock)
@@ -884,6 +1039,8 @@ public sealed class OpcUaClientService : IOpcUaClientService
catch { return false; }
}
/// <summary>Builds the event filter (select clauses for the alarm and condition fields) used by alarm subscriptions.</summary>
/// <returns>The populated <see cref="EventFilter"/>.</returns>
internal static EventFilter CreateAlarmEventFilter()
{
var filter = new EventFilter();
@@ -935,9 +1092,13 @@ public sealed class OpcUaClientService : IOpcUaClientService
private void TransitionState(ConnectionState newState, string endpointUrl)
{
var oldState = _state;
if (oldState == newState) return;
_state = newState;
ConnectionState oldState;
lock (_subscriptionLock)
{
oldState = _state;
if (oldState == newState) return;
_state = newState;
}
ConnectionStateChanged?.Invoke(this, new ConnectionStateChangedEventArgs(oldState, newState, endpointUrl));
}
@@ -43,23 +43,9 @@ public partial class HistoryViewModel : ObservableObject
_dispatcher = dispatcher;
}
/// <summary>Available aggregate types (null means "Raw").</summary>
/// <summary>Available aggregate types (null means "Raw"): every client aggregate, which is the served set.</summary>
public IReadOnlyList<AggregateType?> AggregateTypes { get; } =
[
null,
AggregateType.Average,
AggregateType.Minimum,
AggregateType.Maximum,
AggregateType.Count,
AggregateType.Start,
AggregateType.End,
AggregateType.StandardDeviation,
AggregateType.TimeAverage,
AggregateType.TimeAverage2,
AggregateType.Total,
AggregateType.Total2,
AggregateType.Interpolative
];
[null, .. Enum.GetValues<AggregateType>().Select(a => (AggregateType?)a)];
/// <summary>Gets a value indicating whether an aggregate read is selected.</summary>
public bool IsAggregateRead => SelectedAggregateType != null;
@@ -1,4 +1,5 @@
using System.Collections.ObjectModel;
using System.Collections.Specialized;
using CommunityToolkit.Mvvm.ComponentModel;
using CommunityToolkit.Mvvm.Input;
using Opc.Ua;
@@ -19,6 +20,11 @@ public partial class SubscriptionsViewModel : ObservableObject
private readonly IUiDispatcher _dispatcher;
private readonly IOpcUaClientService _service;
// NodeId -> its rows, kept in step with ActiveSubscriptions by OnActiveSubscriptionsChanged, so a notification is
// one lookup instead of a scan of every row on the UI thread (2 000 items at 1 Hz was 4M string compares/s).
// Touched only on the UI thread, like the collection itself.
private readonly Dictionary<string, List<SubscriptionItemViewModel>> _rowsByNodeId = new(StringComparer.Ordinal);
/// <summary>
/// Last user-visible status message — set when a subscribe/unsubscribe operation fails so the
/// shell can surface the diagnostic instead of silently dropping the error. Cleared on success.
@@ -49,9 +55,13 @@ public partial class SubscriptionsViewModel : ObservableObject
{
_service = service;
_dispatcher = dispatcher;
ActiveSubscriptions.CollectionChanged += OnActiveSubscriptionsChanged;
_service.DataChanged += OnDataChanged;
}
/// <summary>The number of distinct NodeIds in the notification index (test seam).</summary>
internal int IndexedNodeCount => _rowsByNodeId.Count;
/// <summary>Currently active subscriptions.</summary>
public ObservableCollection<SubscriptionItemViewModel> ActiveSubscriptions { get; } = [];
@@ -62,16 +72,42 @@ public partial class SubscriptionsViewModel : ObservableObject
{
_dispatcher.Post(() =>
{
foreach (var item in ActiveSubscriptions)
if (item.NodeId == e.NodeId)
{
item.Value = Helpers.ValueFormatter.Format(e.Value.Value);
item.Status = Helpers.StatusCodeFormatter.Format(e.Value.StatusCode);
item.Timestamp = e.Value.SourceTimestamp.ToString("O");
}
if (!_rowsByNodeId.TryGetValue(e.NodeId, out var rows)) return;
foreach (var item in rows)
{
item.Value = Helpers.ValueFormatter.Format(e.Value.Value);
item.Status = Helpers.StatusCodeFormatter.Format(e.Value.StatusCode);
item.Timestamp = e.Value.SourceTimestamp.ToString("O");
}
});
}
private void OnActiveSubscriptionsChanged(object? sender, NotifyCollectionChangedEventArgs e)
{
if (e.Action == NotifyCollectionChangedAction.Reset)
{
_rowsByNodeId.Clear();
foreach (var item in ActiveSubscriptions) Index(item);
return;
}
if (e.OldItems != null)
foreach (SubscriptionItemViewModel item in e.OldItems)
if (_rowsByNodeId.TryGetValue(item.NodeId, out var rows) && rows.Remove(item) && rows.Count == 0)
_rowsByNodeId.Remove(item.NodeId);
if (e.NewItems != null)
foreach (SubscriptionItemViewModel item in e.NewItems)
Index(item);
}
private void Index(SubscriptionItemViewModel item)
{
if (!_rowsByNodeId.TryGetValue(item.NodeId, out var rows))
_rowsByNodeId[item.NodeId] = rows = [];
rows.Add(item);
}
private bool CanAddSubscription()
{
return IsConnected && !string.IsNullOrWhiteSpace(NewNodeIdText);
@@ -149,7 +185,7 @@ public partial class SubscriptionsViewModel : ObservableObject
if (!IsConnected || string.IsNullOrWhiteSpace(nodeIdStr)) return;
// Skip if already subscribed
if (ActiveSubscriptions.Any(s => s.NodeId == nodeIdStr)) return;
if (_rowsByNodeId.ContainsKey(nodeIdStr)) return;
try
{
@@ -58,7 +58,7 @@ public sealed class AkkaClusterOptions
/// <see cref="RoleParser.ResolveEffectiveRoles(IReadOnlyList{string}?)"/>. That fallback was
/// described here — and relied upon in the field — for a year before it was implemented; a
/// node configured only through the env var joined the cluster carrying no roles at all.
/// See Gitea #532 and <c>docs/deployments/wonder-app-vd03-makino-z-34184.md</c>.
/// See <c>docs/deployments/wonder-app-vd03-makino-z-34184.md</c>.
/// </para>
/// <para>
/// The two sources normalize differently: values bound from <c>Cluster:Roles</c> are taken
@@ -44,7 +44,7 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so constructor
/// dependencies are safe here. Both <paramref name="effectiveRoles"/> and
/// <paramref name="configuration"/> are optional (default <see langword="null"/>) so a caller
/// with neither to hand — including every test in this file that pre-dates R3-04 — keeps
/// with neither to hand — including every test in this file that pre-dates the post-overlay guard — keeps
/// constructing this validator parameterlessly; the post-overlay guard below (choice 4) is
/// simply skipped when either is null.
/// </summary>
@@ -116,7 +116,7 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
}
/// <summary>
/// R3-04 (choice 4, the residual divergence surface): <c>Program.cs</c> resolves
/// Post-overlay guard (choice 4, the residual divergence surface): <c>Program.cs</c> resolves
/// <see cref="EffectiveRoles"/> BEFORE the per-role appsettings overlay loads (the overlay's
/// filename depends on the answer), so if a role overlay — or a <c>${secret:}</c> expansion —
/// ever set <c>Cluster:Roles</c> itself, that value would silently disagree with the
@@ -46,7 +46,9 @@ public sealed class ClusterBootstrapCoordinator : IHostedService
_log = log ?? throw new ArgumentNullException(nameof(log));
}
/// <inheritdoc/>
/// <summary>Starts the bootstrap join in the background when the bootstrap guard is enabled.</summary>
/// <param name="cancellationToken">Not used; the join runs under the coordinator's own cancellation source.</param>
/// <returns>A completed task; the join itself continues asynchronously.</returns>
public Task StartAsync(CancellationToken cancellationToken)
{
if (!_options.BootstrapGuard.Enabled)
@@ -59,7 +61,9 @@ public sealed class ClusterBootstrapCoordinator : IHostedService
return Task.CompletedTask;
}
/// <inheritdoc/>
/// <summary>Cancels the background join and waits for it to finish.</summary>
/// <param name="cancellationToken">Not used; shutdown is driven by the coordinator's own cancellation source.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task StopAsync(CancellationToken cancellationToken)
{
await _cts.CancelAsync().ConfigureAwait(false);
@@ -63,7 +63,7 @@ public static class ClusterBootstrapGuard
/// <summary>
/// Analyzes the configured seed list to decide this node's bootstrap role. The guard engages only
/// for the Phase-6 pair shape (exactly two seeds, one of them this node); any other shape
/// for the pair shape (exactly two seeds, one of them this node); any other shape
/// (single-seed legacy site node, three+ seeds, self absent) returns <see cref="BootstrapRole.Applies"/>
/// = false and the caller joins the configured seeds unchanged.
/// </summary>
@@ -23,7 +23,7 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
private readonly string? _clusterRole;
private readonly string? _clusterId;
private readonly object _lock = new();
// R3-04 (01/C-9, census row 17) — the two dictionaries below have DIFFERENT correct comparers,
// The two dictionaries below have DIFFERENT correct comparers,
// and that split is deliberate, not an oversight (review finding, 2026-08-20).
//
// _membersByRole answers "every member carrying (a spelling of) this role" — OrdinalIgnoreCase,
@@ -53,7 +53,7 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
// sharing a bind IP) stay distinct. Production hosts have unique DNS names so the port
// suffix is harmless redundancy.
_localNode = CommonsNodeId.Parse($"{options.Value.PublicHostname}:{options.Value.Port}");
// OrdinalIgnoreCase (Gitea #532): Cluster:Roles binds verbatim, so a node authored with 'Driver'
// OrdinalIgnoreCase: Cluster:Roles binds verbatim, so a node authored with 'Driver'
// used to answer HasRole("driver") false while gossiping a role Akka would still match against
// its own ordinal comparison — the node was a driver to the cluster and not to itself.
_localRoles = new HashSet<string>(options.Value.Roles, StringComparer.OrdinalIgnoreCase);
@@ -128,7 +128,7 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
{
lock (_lock)
{
// Cluster-002 (review R1 m6): seed only USABLE members. A member already Leaving / Exiting / Down
// Seed only USABLE members. A member already Leaving / Exiting / Down
// (or still Joining) at seed time is not one, and a departing member never fires a later
// MemberLeft / MemberExited / MemberDowned that HandleMemberEvent would drop it on — it would
// linger until MemberRemoved, the very delay the member-event handling exists to avoid. A
@@ -165,7 +165,7 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
set.Add(up.Member);
}
break;
// Cluster-002: a leaving / exiting / downed member is no longer a usable member, so
// A leaving / exiting / downed member is no longer a usable member, so
// drop it as soon as that transition lands rather than waiting for MemberRemoved
// (several seconds later). MembersWithRole now has a production caller (the
// historization scope-peer check), which must not defer to a departing partner.
@@ -210,8 +210,8 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
// Record the event's own address. The leader used to be resolved by looking the address up
// in Cluster.State.Members, but that snapshot is published asynchronously and can lag the
// event: a miss stored null and nothing re-ran, so RoleLeader(role) stayed null for good
// (Gitea #590). Only the address is ever read, so no Member lookup is needed.
// event: a miss stored null and nothing re-ran, so RoleLeader(role) stayed null for good.
// Only the address is ever read, so no Member lookup is needed.
_roleLeaders[evt.Role] = evt.Leader;
if (evt.Leader is not null)
next = ToNodeId(evt.Leader);
@@ -21,14 +21,18 @@ public sealed class ConfigSourceOptionsValidator : IValidateOptions<ConfigSource
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so a
/// constructor dependency is safe here. This validator's only use of configuration is this
/// node's roles, so it takes <see cref="EffectiveRoles"/> — the single role authority
/// (archreview R3-04) — directly rather than <see cref="IConfiguration"/>.
/// — directly rather than <see cref="IConfiguration"/>.
/// </summary>
/// <param name="effectiveRoles">The node's effective roles, used to decide which fetch settings are required.</param>
public ConfigSourceOptionsValidator(EffectiveRoles effectiveRoles)
{
_effectiveRoles = effectiveRoles;
}
/// <inheritdoc />
/// <summary>Validates the <see cref="ConfigSourceOptions"/> for this node's roles.</summary>
/// <param name="name">The options instance name (unused).</param>
/// <param name="options">The options to validate.</param>
/// <returns>A success result, or a failure listing every violated rule.</returns>
public ValidateOptionsResult Validate(string? name, ConfigSourceOptions options)
{
ArgumentNullException.ThrowIfNull(options);
@@ -92,8 +96,8 @@ public sealed class ConfigSourceOptionsValidator : IValidateOptions<ConfigSource
// Per-cluster mesh Phase 4: a driver-only node has no central ConfigDb connection to fall back
// on, so Direct is only ever valid on a fused admin+driver node (admin holds the SQL
// connection). EffectiveRoles is the single role authority (archreview R3-04): its precedence
// already applies the OTOPCUA_ROLES fallback (Gitea #532) and its normalization already folds
// connection). EffectiveRoles is the single role authority: its precedence
// already applies the OTOPCUA_ROLES fallback and its normalization already folds
// case, so neither an env-only driver-only node nor one spelled 'Driver' can sail through here
// on Direct to read its configuration from nothing.
var isDriver = _effectiveRoles.HasDriver;
@@ -69,6 +69,7 @@ public sealed class EffectiveRoles
/// <summary>True when <paramref name="role"/> is among <see cref="Gossiped"/>, compared case-insensitively.</summary>
/// <param name="role">The role to test for.</param>
/// <returns><see langword="true"/> when the role is present; otherwise <see langword="false"/>.</returns>
public bool Has(string role) =>
Gossiped.Any(r => string.Equals(r, role, StringComparison.OrdinalIgnoreCase));
@@ -23,7 +23,7 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so constructor
/// dependencies are safe here. <see cref="IConfiguration"/> is still needed for the
/// <c>ConfigServe:GrpcListenPort</c> cross-read (the admin-side obligation below) —
/// <see cref="EffectiveRoles"/> is the single role authority (archreview R3-04) this rule reads
/// <see cref="EffectiveRoles"/> is the single role authority this rule reads
/// roles through instead of re-deriving them.
/// </summary>
/// <param name="configuration">The application configuration, read for <c>ConfigServe:GrpcListenPort</c>.</param>
@@ -34,7 +34,10 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
_effectiveRoles = effectiveRoles;
}
/// <inheritdoc />
/// <summary>Validates the <see cref="MeshTransportOptions"/> for this node's roles.</summary>
/// <param name="name">The options instance name (unused).</param>
/// <param name="options">The options to validate.</param>
/// <returns>A success result, or a failure listing every violated rule.</returns>
public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
{
ArgumentNullException.ThrowIfNull(options);
@@ -71,7 +74,7 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
// dials nodes rather than being dialled, and resolves per-cluster keys at runtime through
// MeshTransportOptions.ClusterPsks / the Secrets store, so requiring a single GrpcPsk and an
// endpoint list of it would demand configuration it never reads. EffectiveRoles is the single
// role authority (archreview R3-04) — its precedence already applies the OTOPCUA_ROLES
// role authority — its precedence already applies the OTOPCUA_ROLES
// fallback and its normalization already folds case, so a node configured only through the env
// var, or spelled 'Driver', is still the driver that has to dial central.
var isDriver = _effectiveRoles.HasDriver;
@@ -24,7 +24,7 @@ public static class RoleParser
/// <summary>Returns true when <paramref name="role"/> is a cluster-scoped role — starts with
/// <see cref="ClusterRolePrefix"/> and carries a non-empty ClusterId suffix.</summary>
/// <remarks>
/// The prefix test is <b>case-insensitive</b> (Gitea #532). It was ordinal, which meant a
/// The prefix test is <b>case-insensitive</b>. It was ordinal, which meant a
/// <c>Cluster:Roles</c> entry spelled <c>Cluster-SITE-A</c> — that path binds verbatim, unlike
/// the <see cref="Parse"/> path, which lower-cases — classified as an ordinary role. The node
/// then resolved a <see langword="null"/> ClusterId while still gossiping a cluster-scoped role,
@@ -33,6 +33,7 @@ public static class RoleParser
/// <c>ClusterRoleInfo</c> did not, and got the null.
/// </remarks>
/// <param name="role">The role name to test.</param>
/// <returns><see langword="true"/> if the role is a cluster-scoped role.</returns>
public static bool IsClusterRole(string role) =>
role is not null
&& role.StartsWith(ClusterRolePrefix, StringComparison.OrdinalIgnoreCase)
@@ -47,9 +48,10 @@ public static class RoleParser
/// <c>OTOPCUA_ROLES</c> path (⇒ <c>site-a</c>) — and normalizing one side would silently rename
/// the Akka role of a running deployment (docker-dev's rig configures <c>cluster-MAIN</c> /
/// <c>cluster-SITE-A</c> verbatim today). Every consumer of a ClusterId therefore compares it
/// case-insensitively; see the identity rule on <c>AclScopeRef</c> (Gitea #533).
/// case-insensitively; see the identity rule on <c>AclScopeRef</c>.
/// </remarks>
/// <param name="role">The cluster-scoped role name.</param>
/// <returns>The ClusterId portion of the role, with the role prefix removed.</returns>
public static string ClusterIdFromRole(string role) => role[ClusterRolePrefix.Length..];
/// <summary>
@@ -72,7 +74,7 @@ public static class RoleParser
/// cluster-scoped role (a legacy / single-mesh node).
/// </summary>
/// <remarks>
/// The single home for a derivation that was open-coded at four call sites before Gitea #532
/// The single home for a derivation that was open-coded at four call sites before
/// (<c>ClusterRoleInfo</c>, the ControlPlane singleton scope, <c>SplitTopologyTransportValidator</c>
/// and <c>ConfigDbApiKeyStore</c>) — each with its own view of whether the classification was
/// case-sensitive, which is how the mixed-case gap above survived in three of the four.
@@ -91,7 +93,7 @@ public static class RoleParser
/// </summary>
/// <remarks>
/// <para>
/// ⚠️ <b>This fallback was documented for a year before it existed</b> (Gitea #532).
/// ⚠️ <b>This fallback was documented for a year before it existed</b>.
/// <c>AkkaClusterOptions.Roles</c>' own XML doc and <see cref="SplitTopologyTransportValidator"/>
/// both described it; only the validator implemented it, privately. A node configured solely
/// through <c>OTOPCUA_ROLES</c> — which drives every bit of <c>Program.cs</c>'s conditional
@@ -114,7 +116,7 @@ public static class RoleParser
/// <exception cref="ArgumentException">Either source names an unknown role.</exception>
/// <remarks>
/// Thin delegation to <see cref="EffectiveRoles.Resolve(IReadOnlyList{string}?, string?)"/> —
/// the single role authority (archreview R3-04). <see cref="EffectiveRoles"/> is the type to
/// the single role authority. <see cref="EffectiveRoles"/> is the type to
/// reach for directly; this overload is retained so existing callers of the array-returning
/// shape keep compiling.
/// </remarks>
@@ -35,7 +35,7 @@ public static class ServiceCollectionExtensions
services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
configuration, AkkaClusterOptions.SectionName);
// R3-04 (structural identity, choice 3): EffectiveRoles is resolved ONCE and registered as a
// Structural identity: EffectiveRoles is resolved ONCE and registered as a
// singleton, off the SAME IConfiguration this call received — not off a DI-resolved
// IConfiguration, which a plain ServiceCollection (as opposed to a WebApplicationBuilder) never
// registers. Every consumer (Program.cs's hasAdmin/hasDriver, the four role-scoped validators,
@@ -54,7 +54,7 @@ public static class ServiceCollectionExtensions
// otherwise; TryAddSingleton is a no-op on a real host, where it is already registered.
services.TryAddSingleton(configuration);
// Gitea #532 (superseded by R3-04): apply the OTOPCUA_ROLES fallback the Roles property has
// Apply the OTOPCUA_ROLES fallback the Roles property has
// always DOCUMENTED, in the one place every consumer reads through. Before this, a node
// configured only through the env var — the knob that drives all of Program.cs's role wiring —
// bound an EMPTY Roles array and so joined the cluster role-less: invisible to
@@ -14,7 +14,7 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// Phase 2/3/5 dark switches keep their DPS branches alive, but DPS is mesh-wide — it delivers
/// nothing between a central node and a site node that no longer share a gossip ring. The
/// marker of the split topology is a <c>cluster-{ClusterId}</c> role
/// (<see cref="RoleParser.IsClusterRole"/>); those roles exist only post-Phase-6. A node that
/// (<see cref="RoleParser.IsClusterRole"/>); those roles exist only in the per-cluster mesh topology. A node that
/// carries one <b>must</b> use the mesh-crossing transports, and this validator refuses to
/// boot it otherwise — the alternative is a node that runs, listens, and silently delivers no
/// commands or telemetry, which has no stack trace and no failing node to point at.
@@ -44,16 +44,21 @@ public sealed class SplitTopologyTransportValidator : IValidateOptions<MeshTrans
/// transport-mode cross-reads (<c>MeshTransport:Mode</c>, <c>Telemetry:Mode</c>,
/// <c>TelemetryDial:Mode</c>) — not <c>IClusterRoleInfo</c>, whose implementation needs the
/// <c>ActorSystem</c>, which does not exist yet at <c>ValidateOnStart</c> time.
/// <see cref="EffectiveRoles"/> is the single role authority (archreview R3-04) — this rule no
/// <see cref="EffectiveRoles"/> is the single role authority — this rule no
/// longer re-derives the effective, normalized role list itself.
/// </summary>
/// <param name="configuration">The host configuration used to cross-read the transport-mode settings.</param>
/// <param name="effectiveRoles">The node's effective, normalized role list.</param>
public SplitTopologyTransportValidator(IConfiguration configuration, EffectiveRoles effectiveRoles)
{
_configuration = configuration;
_effectiveRoles = effectiveRoles;
}
/// <inheritdoc />
/// <summary>Fails startup when a split-topology node is left on a transport mode that cannot work there.</summary>
/// <param name="name">The options instance name; unused.</param>
/// <param name="options">The mesh transport options; unused because every input is cross-read from configuration.</param>
/// <returns>A success result, or a failure naming the offending transport setting.</returns>
public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
{
// The options instance is intentionally unused: every value this rule depends on is cross-read
@@ -72,7 +72,13 @@ public sealed class TelemetryDialOptions
/// <summary>How often central refreshes its set of dialable node contacts, in seconds.</summary>
public int ContactRefreshSeconds { get; set; } = 60;
/// <summary>Per-call deadline for a gRPC telemetry dial, in seconds.</summary>
/// <summary>
/// Deadline, in seconds, for a gRPC telemetry dial to deliver its first event. A live node answers a
/// fresh subscribe at once (it replays its health/resilience snapshots), so a dial still silent at the
/// deadline is one that never connected — it is cancelled and re-dialled. Bounds establishment only;
/// an established stream is kept alive by HTTP/2 keepalive pings, not by this value. Values below 1
/// are treated as 1.
/// </summary>
public int CallTimeoutSeconds { get; set; } = 30;
}
@@ -93,14 +99,18 @@ public sealed class TelemetryOptionsValidator : IValidateOptions<TelemetryOption
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so a
/// constructor dependency is safe here. This validator's only use of configuration is this
/// node's roles, so it takes <see cref="EffectiveRoles"/> — the single role authority
/// (archreview R3-04) — directly rather than <see cref="IConfiguration"/>.
/// — directly rather than <see cref="IConfiguration"/>.
/// </summary>
/// <param name="effectiveRoles">The node's resolved roles, the single role authority.</param>
public TelemetryOptionsValidator(EffectiveRoles effectiveRoles)
{
_effectiveRoles = effectiveRoles;
}
/// <inheritdoc />
/// <summary>Validates the telemetry options against this node's roles.</summary>
/// <param name="name">The named options instance, or <see langword="null"/> for the default.</param>
/// <param name="options">The options to validate.</param>
/// <returns>Success, or a failure listing every invalid setting.</returns>
public ValidateOptionsResult Validate(string? name, TelemetryOptions options)
{
ArgumentNullException.ThrowIfNull(options);
@@ -117,8 +127,8 @@ public sealed class TelemetryOptionsValidator : IValidateOptions<TelemetryOption
+ $"Expected '{TelemetryOptions.ModeDps}' or '{TelemetryOptions.ModeGrpc}'.");
}
// EffectiveRoles is the single role authority (archreview R3-04) — its precedence already
// applies the OTOPCUA_ROLES fallback (Gitea #532) and its normalization already folds case, so
// EffectiveRoles is the single role authority — its precedence already
// applies the OTOPCUA_ROLES fallback and its normalization already folds case, so
// a node configured only through the env var, or spelled 'Driver', still satisfies the
// Grpc-mode requirements below.
var isDriver = _effectiveRoles.HasDriver;
@@ -157,7 +167,10 @@ public sealed class TelemetryOptionsValidator : IValidateOptions<TelemetryOption
/// </summary>
public sealed class TelemetryDialOptionsValidator : IValidateOptions<TelemetryDialOptions>
{
/// <inheritdoc />
/// <summary>Validates the telemetry dial options.</summary>
/// <param name="name">The named options instance, or <see langword="null"/> for the default.</param>
/// <param name="options">The options to validate.</param>
/// <returns>Success, or a failure listing every invalid setting.</returns>
public ValidateOptionsResult Validate(string? name, TelemetryDialOptions options)
{
ArgumentNullException.ThrowIfNull(options);
@@ -56,7 +56,8 @@ public sealed class CapturedTreeBrowseSession(CapturedTree tree) : IBrowseSessio
return Task.FromResult<IReadOnlyList<AttributeInfo>>(attrs);
}
/// <inheritdoc />
/// <summary>Drops this session's reference to the captured tree; the tree itself may be shared.</summary>
/// <returns>A completed <see cref="ValueTask"/>.</returns>
public ValueTask DisposeAsync()
{
_tree = null; // drop this session's reference only — tree may be shared (coalescing)
@@ -68,7 +69,7 @@ public sealed class CapturedTreeBrowseSession(CapturedTree tree) : IBrowseSessio
/// <summary>
/// Surfaces a leaf's driver-native type name as the <c>dataType</c> address field, so a
/// browse-commit can stamp the driver-typed <c>TagConfig</c> with the type the controller
/// actually reported (Gitea #536).
/// actually reported.
/// </summary>
/// <param name="a">The captured attribute.</param>
/// <returns>The address fields, or null when the driver reported no native type.</returns>
@@ -80,7 +81,7 @@ public sealed class CapturedTreeBrowseSession(CapturedTree tree) : IBrowseSessio
/// reverse map would bind a 16-bit INT as a 32-bit DINT and read four bytes for two.
/// <para>
/// Null when the driver stamped nothing — the mapper then leaves the model's own default in
/// place, which is exactly the pre-#536 behaviour rather than a guess.
/// place, which is exactly the previous behaviour rather than a guess.
/// </para>
/// </remarks>
private static IReadOnlyDictionary<string, string>? NativeTypeFields(DriverAttributeInfo a)
@@ -17,6 +17,8 @@ public sealed class CapturingAddressSpaceBuilder : IAddressSpaceBuilder
public bool Truncated;
public readonly int NodeCap = nodeCap;
public readonly Dictionary<string, CapturedNode> ById = new(StringComparer.Ordinal);
/// <summary>Reserves a slot for one more node, marking the capture truncated when the cap is reached.</summary>
/// <returns><see langword="true"/> when the node may be recorded; <see langword="false"/> when the cap is reached.</returns>
public bool TryReserve()
{
if (Count >= NodeCap) { Truncated = true; return false; }
@@ -96,7 +98,10 @@ public sealed class CapturingAddressSpaceBuilder : IAddressSpaceBuilder
private sealed class CapturedVariableHandle(CapturedNode node) : IVariableHandle
{
/// <inheritdoc />
public string FullReference => node.Id;
/// <inheritdoc />
public IAlarmConditionSink MarkAsAlarmCondition(AlarmConditionInfo info)
{
node.IsAlarmMarked = true;
@@ -107,6 +112,7 @@ public sealed class CapturingAddressSpaceBuilder : IAddressSpaceBuilder
private sealed class NoopAlarmSink : IAlarmConditionSink
{
public static readonly NoopAlarmSink Instance = new();
/// <inheritdoc />
public void OnTransition(AlarmEventArgs args) { } // browse never delivers live transitions
}
}
@@ -12,7 +12,7 @@ public sealed class DiscoveryDriverBrowser : IUniversalDriverBrowser
/// 20 s per-call browse timeout, which only covers serving from memory.</summary>
public static readonly TimeSpan OpenTimeout = TimeSpan.FromSeconds(60);
/// <summary>Bound on the cleanup ShutdownAsync (R2-01: no unbounded waits — a driver wedged
/// <summary>Bound on the cleanup ShutdownAsync (no unbounded waits — a driver wedged
/// in discovery may be equally wedged in shutdown).</summary>
public static readonly TimeSpan ShutdownTimeout = TimeSpan.FromSeconds(10);
@@ -46,7 +46,11 @@ public sealed class DiscoveryDriverBrowser : IUniversalDriverBrowser
public DiscoveryDriverBrowser(IDriverFactory driverFactory, ILogger<DiscoveryDriverBrowser> logger)
: this(driverFactory, logger, OpenTimeout, ShutdownTimeout) { }
// Test seam: short timeouts.
/// <summary>Test seam: constructor with explicit open and shutdown bounds.</summary>
/// <param name="driverFactory">Factory used to construct throwaway/capture driver instances.</param>
/// <param name="logger">Logger for shutdown-timeout warnings.</param>
/// <param name="openTimeout">Upper bound on opening a capture driver and running discovery.</param>
/// <param name="shutdownTimeout">Upper bound on the cleanup shutdown of a capture driver.</param>
internal DiscoveryDriverBrowser(IDriverFactory driverFactory, ILogger<DiscoveryDriverBrowser> logger,
TimeSpan openTimeout, TimeSpan shutdownTimeout)
{
@@ -111,6 +115,10 @@ public sealed class DiscoveryDriverBrowser : IUniversalDriverBrowser
}
}
/// <summary>Builds the in-flight de-duplication key for a capture request.</summary>
/// <param name="driverType">The driver type name.</param>
/// <param name="configJson">The driver configuration JSON.</param>
/// <returns>A key combining the lower-cased driver type and a hash of the configuration.</returns>
internal static string CaptureKey(string driverType, string configJson) =>
driverType.ToLowerInvariant() + ":" +
Convert.ToHexString(System.Security.Cryptography.SHA256.HashData(
@@ -180,6 +188,13 @@ public sealed class DiscoveryDriverBrowser : IUniversalDriverBrowser
}
}
/// <summary>
/// Final teardown of a throwaway capture/probe driver: a bounded <see cref="IDriver.ShutdownAsync"/>
/// followed by a bounded dispose. The dispose is NOT optional and runs even after a failed or timed-out
/// shutdown: a Galaxy driver's <c>ShutdownAsync</c> releases none of its gRPC runtime (channels, MX
/// session, stream loops) — only <c>DisposeAsync</c> does — so a browse that stopped at shutdown leaked a
/// full runtime per <c>/raw</c> browse and per <see cref="CanBrowse"/> probe (02/L1, 06/G-1).
/// </summary>
private async Task BoundedShutdownAsync(IDriver driver, string driverType)
{
try
@@ -191,10 +206,32 @@ public sealed class DiscoveryDriverBrowser : IUniversalDriverBrowser
catch (Exception ex)
{
_logger.LogWarning(ex,
"Browse-capture shutdown for {DriverType} failed/timed out; instance abandoned", driverType);
"Browse-capture shutdown for {DriverType} failed/timed out; disposing the instance anyway", driverType);
}
try
{
switch (driver)
{
case IAsyncDisposable asyncDisposable:
await asyncDisposable.DisposeAsync().AsTask().WaitAsync(_shutdownTimeout).ConfigureAwait(false);
break;
case IDisposable disposable:
disposable.Dispose();
break;
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Browse-capture dispose for {DriverType} failed/timed out; instance abandoned", driverType);
}
}
/// <summary>Applies the driver-specific browse-time patch (if any) to the configuration JSON.</summary>
/// <param name="driverType">The driver type name.</param>
/// <param name="configJson">The driver configuration JSON.</param>
/// <returns>The patched configuration JSON, or the input unchanged when the driver type has no patch.</returns>
internal static string PatchForBrowse(string driverType, string configJson)
{
if (!BrowsePatches.TryGetValue(driverType, out var patch)) return configJson;
@@ -13,7 +13,7 @@ public enum ApplyAckOutcome { Applied, Failed }
/// <param name="FailureReason">Why the apply failed; null on <see cref="ApplyAckOutcome.Applied"/>.</param>
/// <param name="CorrelationId">The dispatch's correlation id.</param>
/// <param name="Degraded">
/// Gitea #601: <see langword="true"/> when the node applied the deployment (the revision advanced) but
/// <see langword="true"/> when the node applied the deployment (the revision advanced) but
/// its OPC UA address-space apply was DEGRADED — swallowed per-node sink faults, a failed rebuild, or no
/// outcome reported in time — so the served address space may be partial. Only meaningful with
/// <see cref="ApplyAckOutcome.Applied"/>. Trailing and defaulted so an ack from a node built before
@@ -30,7 +30,7 @@ public sealed record ApplyAck(
string? DegradedReason = null)
{
/// <summary>
/// Gitea #601 — prefix of <c>NodeDeploymentState.FailureReason</c> on an <b>Applied</b> row whose
/// The prefix of <c>NodeDeploymentState.FailureReason</c> on an <b>Applied</b> row whose
/// address-space apply was degraded. Shared by central (which writes it from the ack) and a node with
/// a ConfigDb (which writes its own row), so both spell the row identically.
/// </summary>
@@ -11,11 +11,11 @@ public static class DeployTimings
/// (<c>ConfigPublishCoordinator.DefaultApplyDeadline</c>).</summary>
public static readonly TimeSpan ApplyDeadline = TimeSpan.FromMinutes(2);
/// <summary>Gitea #601 — how far inside <see cref="ApplyDeadline"/> a node's held Applied ack is released
/// <summary>How far inside <see cref="ApplyDeadline"/> a node's held Applied ack is released
/// when the address-space outcome has not arrived, leaving room for transport latency.</summary>
public static readonly TimeSpan ApplyOutcomeSafetyMargin = TimeSpan.FromSeconds(15);
/// <summary>Gitea #601 — the node's default apply-outcome timeout: <see cref="ApplyDeadline"/> −
/// <summary>The node's default apply-outcome timeout: <see cref="ApplyDeadline"/> −
/// <see cref="ApplyOutcomeSafetyMargin"/> (105 s). Long enough that a slow-but-clean rebuild of a large
/// address space is not labelled degraded, short enough to ack before central gives up.</summary>
/// <remarks>The timer is armed after the artifact read and driver reconcile, so if those alone took
@@ -38,7 +38,7 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
/// so a driver-only node has no ConfigDb connection to write rows to. This channel already reaches
/// <c>/hosts</c>, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already
/// replays a last-value snapshot on every (re)subscribe — so per-host state re-primes after a
/// reconnect without a durable store. The table was dropped; see Gitea #521.</para>
/// reconnect without a durable store. The table was dropped.</para>
/// <para><b>Value-equality caveat.</b> A record's generated <c>Equals</c> compares this list by
/// REFERENCE, so two <c>DriverHealthChanged</c> carrying equal-but-distinct lists are not equal.
/// Nothing dedups on record equality — <c>DriverInstanceActor</c> keeps its own flattened fingerprint
@@ -48,13 +48,13 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
/// What this driver skipped when it last built its tag table, from
/// <c>ISkippedTagReporter.GetSkippedTags()</c>; null when the driver does not implement that
/// capability. All twelve production drivers do.
/// <para><b>Why this is on the health snapshot (Gitea #541).</b> A skip happens at init, is not an
/// <para><b>Why this is on the health snapshot.</b> A skip happens at init, is not an
/// error, and does not stop the driver's OTHER tags from reading — so <c>State</c> stays Healthy,
/// <c>ErrorCount5Min</c> stays 0, <c>LastError</c> stays null and <c>LastSuccessfulReadUtc</c> keeps
/// advancing while the driver ignores part or all of its configuration. On the Makino in Gitea #540
/// advancing while the driver ignores part or all of its configuration. On the Makino
/// that combination held while 20 of 22 tags were ignored. This field is what contradicts it.</para>
/// <para><b>Additive, never a state change.</b> See <c>ISkippedTagReporter</c> for why a skipped tag
/// must not latch the driver non-Healthy — it would satisfy both halves of the #527 comms-loss
/// must not latch the driver non-Healthy — it would satisfy both halves of the comms-loss
/// conjunction and reconnect-loop a healthy device.</para>
/// <para><b>Value-equality caveat</b> — the same one <paramref name="HostStatuses"/> carries: the
/// record's generated <c>Equals</c> compares the sample list by REFERENCE.</para>
@@ -62,7 +62,7 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
/// <param name="Diagnostics">
/// Operator-facing driver diagnostics from <c>IDriverDiagnostics.GetDiagnostics()</c> (e.g. S7's
/// negotiated PDU size), bounded and in ordinal key order; null when the driver does not implement the
/// capability (Gitea #640). Additive, never a state change. Same by-reference value-equality caveat as
/// capability. Additive, never a state change. Same by-reference value-equality caveat as
/// <paramref name="HostStatuses"/>.
/// </param>
public sealed record DriverHealthChanged(
@@ -216,7 +216,7 @@ public static class MeshTransportDtoMapper
if (message.FailureReason is not null)
proto.FailureReason = message.FailureReason;
// #601 — additive fields; a clean ack leaves both unset, byte-identical to a pre-#601 encode.
// Additive fields; a clean ack leaves both unset, byte-identical to an encode that predates these fields.
proto.Degraded = message.Degraded;
if (message.DegradedReason is not null)
proto.DegradedReason = message.DegradedReason;
@@ -255,7 +255,7 @@ public static class MeshTransportDtoMapper
outcome,
proto.HasFailureReason ? proto.FailureReason : null,
CorrelationId.Parse(proto.CorrelationId),
// #601 — absent on an old node's ack, which decodes as false/null: today's clean ack.
// Absent on an old node's ack, which decodes as false/null: today's clean ack.
Degraded: proto.Degraded,
DegradedReason: proto.HasDegradedReason ? proto.DegradedReason : null);
}
@@ -77,7 +77,7 @@ public static class OtOpcUaTelemetry
public const string PollOverrunDriverInstanceTag = "driver_instance";
/// <summary>
/// Session/publish-level events of an OPC UA Client driver instance (#276): the operator's view of a
/// Session/publish-level events of an OPC UA Client driver instance: the operator's view of a
/// flapping upstream. Tagged <see cref="PollOverrunDriverInstanceTag"/> (<c>driver_instance</c>) +
/// <c>event</c> = <c>publish_response|notification|data_change|bad_status_value|publish_error|
/// keepalive_miss|keepalive_recovered|stale_notification_dropped</c>.
@@ -19,15 +19,15 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
/// <param name="StatusCode">
/// Status code of the condition's source data (Part 9 <c>ConditionType.Quality</c>). Carried so a
/// comms-lost native source can report a non-<c>Good</c> condition instead of the accidentally-Good
/// default (issue #477). It is a pure annotation — it never alters Active/Acked/Retain. Defaults to
/// default. It is a pure annotation — it never alters Active/Acked/Retain. Defaults to
/// <see cref="OpcUaStatus.Good"/> so scripted-alarm callers (which stay Good in v1) need not supply it.
/// Carries the full sub-code since #519, so a comms-lost condition reports
/// Carries the full sub-code, so a comms-lost condition reports
/// <see cref="OpcUaStatus.BadNoCommunication"/> rather than a bare <see cref="OpcUaStatus.Bad"/>.
/// </param>
/// <param name="UnshelveAtUtc">
/// For <see cref="AlarmShelvingKind.Timed"/>, the UTC instant the engine expires the shelve; otherwise
/// <c>null</c>. Carried so the node's Part 9 <c>ShelvingState.UnshelveTime</c> reports the real remaining
/// time instead of the SDK's ~24.8-day ceiling (Gitea #597). A timed snapshot without it (a caller that
/// time instead of the SDK's ~24.8-day ceiling. A timed snapshot without it (a caller that
/// predates the field) falls back to that ceiling — the engine still owns the expiry either way.
/// </param>
public sealed record AlarmConditionSnapshot(
@@ -44,6 +44,7 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
=> _inner.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc, realm);
/// <inheritdoc />
public void WriteAlarmQuality(string alarmNodeId, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
=> _inner.WriteAlarmQuality(alarmNodeId, statusCode, sourceTimestampUtc, realm);
@@ -52,7 +53,7 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
=> _inner.MaterialiseAlarmCondition(alarmNodeId, equipmentNodeId, displayName, alarmType, severity, realm, isNative);
/// <inheritdoc />
// Forward the WP4 multi-notifier wiring to the inner sink. Without this the native-alarm fan-out to the
// Forward the multi-notifier wiring to the inner sink. Without this the native-alarm fan-out to the
// referencing equipment folders ships INERT on every driver-role host (actors inject THIS wrapper, not the
// inner SdkAddressSpaceSink) — the F10b / PR#423 forwarding trap the reflection guard exists to catch.
public void WireAlarmNotifiers(string alarmNodeId, AddressSpaceRealm alarmRealm, IReadOnlyList<string> notifierFolderNodeIds, AddressSpaceRealm notifierFolderRealm)
@@ -100,7 +101,7 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
=> _inner is ISurgicalAddressSpaceSink surgical
&& surgical.UpdateFolderDisplayName(folderNodeId, displayName, realm);
// R2-07 T7 — surgical remove forwards. Same capability-sniffing pattern as UpdateTagAttributes /
// Surgical remove forwards. Same capability-sniffing pattern as UpdateTagAttributes /
// UpdateFolderDisplayName: forward when the inner sink supports the surgical capability; return false
// otherwise (before the real SdkAddressSpaceSink is swapped in, or any non-surgical inner) so the caller
// (AddressSpaceApplier) falls back to a full rebuild. Without these forwards the surgical remove path is
@@ -1,21 +1,38 @@
namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
/// <summary>
/// Late-binding adapter that holds an inner <see cref="IServiceLevelPublisher"/> reference
/// swappable at runtime. Mirrors <see cref="DeferredAddressSpaceSink"/>: Akka actors resolve
/// the publisher at DI time, but the production <c>SdkServiceLevelPublisher</c> only exists
/// after <c>StandardServer.Start</c>. The Host's hosted service swaps the inner once the SDK
/// is up; until then writes route through <see cref="NullServiceLevelPublisher"/>.
/// Late-binding adapter that accepts and retains the latest requested ServiceLevel until
/// the SDK publisher is available. Binding synchronously replays that request, including zero.
/// </summary>
/// <remarks>
/// Publishing and binding are serialized through the synchronous inner call. Inner publishers
/// must not call back into this adapter. Acceptance while unbound is not an SDK write confirmation.
/// A failed call retains both the requested level and the installed publisher for later recovery.
/// </remarks>
public sealed class DeferredServiceLevelPublisher : IServiceLevelPublisher
{
private volatile IServiceLevelPublisher _inner = NullServiceLevelPublisher.Instance;
private readonly object _gate = new();
private IServiceLevelPublisher? _inner;
private byte? _latest;
/// <summary>Swap the underlying publisher. Pass null to revert to the Null no-op.</summary>
/// <param name="inner">The publisher implementation to use, or null to use the null publisher.</param>
public void SetInner(IServiceLevelPublisher? inner) =>
_inner = inner ?? NullServiceLevelPublisher.Instance;
/// <summary>Bind and replay the latest request, or unbind while retaining that request.</summary>
/// <param name="inner">The publisher implementation to use, or null to unbind.</param>
public void SetInner(IServiceLevelPublisher? inner)
{
lock (_gate)
{
_inner = inner;
if (_latest is { } latest) _inner?.Publish(latest);
}
}
/// <inheritdoc />
public void Publish(byte serviceLevel) => _inner.Publish(serviceLevel);
public void Publish(byte serviceLevel)
{
lock (_gate)
{
_latest = serviceLevel;
_inner?.Publish(serviceLevel);
}
}
}
@@ -11,14 +11,14 @@ public interface IOpcUaAddressSpaceSink
/// <summary>Write a Variable node's current value + status code + source timestamp.</summary>
/// <param name="nodeId">The OPC UA node ID of the variable.</param>
/// <param name="value">The value to write.</param>
/// <param name="statusCode">The full OPC UA status code, sub-code included. Passed to the SDK verbatim
/// (#519): a driver's <see cref="OpcUaStatus.BadNoCommunication"/> reaches the client as
/// <param name="statusCode">The full OPC UA status code, sub-code included. Passed to the SDK verbatim, so
/// a driver's <see cref="OpcUaStatus.BadNoCommunication"/> reaches the client as
/// <c>BadNoCommunication</c>, not as a generic <see cref="OpcUaStatus.Bad"/>. Use
/// <see cref="OpcUaStatus"/>' named constants rather than a literal.</param>
/// <param name="sourceTimestampUtc">The source timestamp in UTC.</param>
/// <param name="realm">Which of the two v3 namespaces (<see cref="AddressSpaceRealm.Raw"/> device tree or
/// <see cref="AddressSpaceRealm.Uns"/> equipment tree) the <paramref name="nodeId"/> lives in — the sink
/// resolves the namespace index from the realm rather than parsing it out of the id string. WP3's fan-out
/// resolves the namespace index from the realm rather than parsing it out of the id string. The fan-out
/// calls this once per NodeId (raw + every referencing UNS node) with the matching realm.</param>
void WriteValue(string nodeId, object? value, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
@@ -81,11 +81,11 @@ public interface IOpcUaAddressSpaceSink
/// the condition was materialised under).</param>
void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
/// <summary>#477 — annotate a materialised condition's source-data quality OUT OF BAND from any alarm
/// <summary>Annotate a materialised condition's source-data quality OUT OF BAND from any alarm
/// transition (used by the driver-connectivity path: comms lost → <see cref="OpcUaStatus.BadNoCommunication"/>,
/// restored → <see cref="OpcUaStatus.Good"/>). Sets ONLY the condition's Quality — never
/// Active/Acked/Severity/Retain (a comms-lost active alarm stays active) — and fires one Part 9 event
/// only on a change to the status code itself — since #519 that includes a move between two Bad
/// only on a change to the status code itself — that now includes a move between two Bad
/// sub-codes, which genuinely changes what a subscriber reads. A no-op for an unmaterialised /
/// non-condition node.</summary>
/// <param name="alarmNodeId">The condition node id (RawPath for a native alarm).</param>
@@ -172,9 +172,9 @@ public interface IOpcUaAddressSpaceSink
/// <param name="realm">The namespace realm the variable (and its <paramref name="parentFolderNodeId"/>) live
/// in. A historized UNS reference node registers the SAME <paramref name="historianTagname"/> as its backing
/// raw node — both NodeIds map to one tagname — so this call carries the shared tagname per realm.</param>
/// <param name="refuseArrayOnScalar">Gitea #610 review (R6): when true and the node is SCALAR, a client write
/// <param name="refuseArrayOnScalar">When true and the node is SCALAR, a client write
/// of an array value is refused inline (<c>BadTypeMismatch</c>, nothing dispatched) because the owning
/// driver's scalar encoder cannot take one (Modbus / S7). False (default) keeps the pre-#610 behaviour — the
/// driver's scalar encoder cannot take one (Modbus / S7). False (default) keeps the earlier behaviour — the
/// write is dispatched and the driver decides (OpcUaClient forwards it to the upstream server, Galaxy encodes
/// it). Fixed at creation; an existing node keeps the value it was created with.</param>
void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, AddressSpaceRealm realm, string? historianTagname = null, bool isArray = false, uint? arrayLength = null, bool refuseArrayOnScalar = false);
@@ -186,7 +186,7 @@ public interface IOpcUaAddressSpaceSink
void RebuildAddressSpace();
/// <summary>
/// Gitea #641 — mark the END of one address-space apply pass (after <see cref="RebuildAddressSpace"/>, if
/// Mark the END of one address-space apply pass (after <see cref="RebuildAddressSpace"/>, if
/// one ran, and every materialise / notifier-wiring call that followed it). An SDK-backed sink carries each
/// native condition's live Part 9 state across a rebuild and restores it when the same condition is
/// re-materialised; this is where it reports the ONE condition event per restored condition (now that
@@ -229,7 +229,7 @@ public interface IOpcUaAddressSpaceSink
}
/// <summary>One value write within a <see cref="IOpcUaAddressSpaceSink.WriteValues"/> batch.
/// Same semantics per field as <see cref="IOpcUaAddressSpaceSink.WriteValue"/> (#519: status verbatim).</summary>
/// Same semantics per field as <see cref="IOpcUaAddressSpaceSink.WriteValue"/> (status verbatim).</summary>
public readonly record struct ValueWrite(
string NodeId, object? Value, uint StatusCode, DateTime SourceTimestampUtc, AddressSpaceRealm Realm);
@@ -237,7 +237,7 @@ public readonly record struct ValueWrite(
/// node's current VALUE is retained and only <paramref name="StatusCode"/> +
/// <paramref name="SourceTimestampUtc"/> change.</summary>
/// <param name="NodeId">The OPC UA node id of the variable, within <paramref name="Realm"/>.</param>
/// <param name="StatusCode">The full status code, sub-code included, passed to the SDK verbatim (#519).</param>
/// <param name="StatusCode">The full status code, sub-code included, passed to the SDK verbatim.</param>
/// <param name="SourceTimestampUtc">The source timestamp in UTC.</param>
/// <param name="Realm">Which of the two v3 namespaces <paramref name="NodeId"/> lives in.</param>
public readonly record struct StatusWrite(
@@ -279,6 +279,7 @@ public sealed class NullOpcUaAddressSpaceSink : IOpcUaAddressSpaceSink
/// <inheritdoc />
public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
/// <inheritdoc />
public void WriteAlarmQuality(string alarmNodeId, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
/// <inheritdoc />
@@ -2,7 +2,7 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
/// <summary>
/// The handful of OPC UA status codes the engine layer needs to spell, plus the Part 4 severity
/// classification helpers that replaced the retired <c>OpcUaQuality</c> 3-state enum (Gitea #519).
/// classification helpers that replaced the retired <c>OpcUaQuality</c> 3-state enum.
///
/// <para><b>Why these are bare <c>uint</c> literals.</b> Commons and Runtime deliberately do not
/// reference the OPC UA SDK — <see cref="IOpcUaAddressSpaceSink"/> exists precisely so the Runtime
@@ -10,11 +10,11 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
/// decision the driver assemblies make, and it carries the same single failure mode: nothing checks
/// the literal against the name. <c>StatusCodeParityTests</c> closes it by reflection, and it can
/// only see a <b>named</b> constant — so add codes here rather than writing a literal at a call
/// site, exactly as Gitea #497 concluded for the drivers.</para>
/// site, exactly as for the drivers.</para>
///
/// <para><b>Severity, not equality, is what callers should branch on.</b> A status code's top two
/// bits carry its Part 4 severity (00 = Good, 01 = Uncertain, 10/11 = Bad); everything below is the
/// sub-code. Before #519 the engine collapsed every value onto a 3-state enum here and the node
/// sub-code. Previously the engine collapsed every value onto a 3-state enum here and the node
/// manager re-expanded <c>Bad</c> to a bare <c>0x80000000</c>, so no driver's sub-code ever reached
/// a client. The sub-code now rides through to the SDK verbatim, and any code that only needs the
/// bucket uses <see cref="IsGood"/> / <see cref="IsUncertain"/> / <see cref="IsBad"/> rather than
@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration.Entities;
/// <summary>
/// One OPC UA data-plane API key. The OPC UA data plane is machine-to-machine (#520), so a
/// One OPC UA data-plane API key. The OPC UA data plane is machine-to-machine, so a
/// consuming system authenticates with a key rather than an LDAP user, and the key resolves to a
/// <see cref="Principal"/> that <see cref="NodeAcl"/> rows grant permissions to.
/// </summary>
@@ -22,7 +22,7 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Entities;
/// ⚠️ <b>Revocation requires a deploy.</b> Phase 4 leaves driver-only nodes with no ConfigDb, so
/// keys reach them inside the deployment artifact exactly as <see cref="NodeAcl"/> rows do.
/// Setting <see cref="Enabled"/> to false takes effect on a node only once a new artifact reaches
/// it. This is an operational sharp edge, not an oversight — see the #520 design doc.
/// it. This is an operational sharp edge, not an oversight.
/// </para>
/// </remarks>
public sealed class ApiKey
@@ -26,7 +26,11 @@ public sealed class Deployment
public DateTime CreatedAtUtc { get; init; } = DateTime.UtcNow;
/// <summary>Gets or sets the serialized artifact blob containing the configuration.</summary>
public byte[] ArtifactBlob { get; init; } = Array.Empty<byte>();
/// <remarks>
/// Settable (not init-only) so the retention sweep can EMPTY it on an old terminal row. An empty blob
/// is "no artifact" to every reader (#485), so the audit row survives with its bytes retired.
/// </remarks>
public byte[] ArtifactBlob { get; set; } = Array.Empty<byte>();
/// <summary>Gets or sets the row version for optimistic concurrency control.</summary>
public byte[] RowVersion { get; set; } = Array.Empty<byte>();
@@ -0,0 +1,36 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
{
/// <summary>
/// Enables <c>ALLOW_SNAPSHOT_ISOLATION</c> on the config database so the deploy path can read every
/// table inside one <c>SNAPSHOT</c> transaction (<c>DraftSnapshotFactory.FromConfigDbAsync</c>):
/// the validator and the artifact composer then see one consistent state instead of two reads
/// an AdminUI save can land between. No schema change — the model snapshot is untouched.
/// </summary>
/// <remarks>
/// <c>ALTER DATABASE</c> cannot run inside the user transaction the migrator wraps each migration in,
/// hence <c>suppressTransaction: true</c>. Turning the option ON does not need exclusive access (unlike
/// <c>READ_COMMITTED_SNAPSHOT</c>); it waits for in-flight transactions and then takes effect. Down
/// leaves it ON: a database that has run with it is not harmed by keeping it, and switching it OFF
/// would fail any deploy running at that instant.
/// </remarks>
public partial class EnableSnapshotIsolation : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.Sql(
"ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON;",
suppressTransaction: true);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
// Deliberately a no-op; see the class remarks.
}
}
}
@@ -41,7 +41,7 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
/// <summary>Gets the DbSet of node ACLs.</summary>
public DbSet<NodeAcl> NodeAcls => Set<NodeAcl>();
/// <summary>OPC UA data-plane API keys (#520). One principal per key; secret stored as a peppered hash.</summary>
/// <summary>OPC UA data-plane API keys. One principal per key; secret stored as a peppered hash.</summary>
public DbSet<ApiKey> ApiKeys => Set<ApiKey>();
/// <summary>Gets the DbSet of configuration audit logs.</summary>
public DbSet<ConfigAuditLog> ConfigAuditLogs => Set<ConfigAuditLog>();
@@ -563,14 +563,14 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
});
}
// ConfigureDriverHostStatus is GONE (Gitea #521). The DriverHostStatus table held per-host
// ConfigureDriverHostStatus is GONE. The DriverHostStatus table held per-host
// connectivity that a publisher hosted service was supposed to upsert from each driver node. That
// publisher was never written, and per-cluster mesh Phase 4 made it unwritable as designed: ConfigDb
// is registered only on the admin role, so a driver-only node has no connection to write rows with.
// Per-host connectivity now rides DriverHealthChanged.HostStatuses to /hosts over the Phase 5
// telemetry stream, which needs no DB and survives the mesh split.
//
// ConfigureDriverInstanceResilienceStatus is GONE for the same reason (Gitea #524). That table held
// ConfigureDriverInstanceResilienceStatus is GONE for the same reason. That table held
// per-(instance, host) breaker/in-flight counters and was equally never written: its doc described a
// HostedService sampler and an AdminUI left-join on DriverHostStatus, neither of which was built. The
// live data reaches the AdminUI's DriverStatusPanel as DriverResilienceStatusChanged over the
@@ -57,7 +57,7 @@ public sealed class DraftSnapshot
public IReadOnlyList<ScriptedAlarm> ScriptedAlarms { get; init; } = [];
/// <summary>User-authored scripts (shared by VirtualTags, ScriptedAlarms, and Calculation tags). Drives
/// the WP7 Calculation gates: <c>scriptId</c> existence + the calc→calc dependency-cycle check.</summary>
/// the Calculation gates: <c>scriptId</c> existence + the calc→calc dependency-cycle check.</summary>
public IReadOnlyList<Script> Scripts { get; init; } = [];
/// <summary>Gets the list of poll groups.</summary>
public IReadOnlyList<PollGroup> PollGroups { get; init; } = [];
@@ -66,6 +66,23 @@ public sealed class DraftSnapshot
/// silently dropped when the permission trie is built, so the grant has to be caught here instead.</summary>
public IReadOnlyList<NodeAcl> NodeAcls { get; init; } = [];
/// <summary>
/// Cluster rows. Not read by the row-level rules; carried so the deploy path validates the cluster
/// topology (<see cref="DraftValidator.ValidateClusterTopology"/>) and composes the artifact from
/// the SAME read as every other table — a second read between validate and compose is exactly how
/// an edit ships unvalidated.
/// </summary>
public IReadOnlyList<ServerCluster> ServerClusters { get; init; } = [];
/// <summary>Cluster-node rows; see <see cref="ServerClusters"/> for why they ride the snapshot.</summary>
public IReadOnlyList<ClusterNode> ClusterNodes { get; init; } = [];
/// <summary>
/// Data-plane API keys (peppered HMAC only, never a secret). Not validated; carried so the artifact
/// composer ships the keys a driver-only node authenticates against from the one consistent read.
/// </summary>
public IReadOnlyList<ApiKey> ApiKeys { get; init; } = [];
/// <summary>Prior Equipment rows (any generation, same cluster) for stability checks.</summary>
public IReadOnlyList<Equipment> PriorEquipment { get; init; } = [];
@@ -1,3 +1,4 @@
using System.Data;
using Microsoft.EntityFrameworkCore;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Validation;
@@ -21,6 +22,16 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Validation;
/// / <see cref="DraftSnapshot.Site"/> are left null so the path-length rule uses its
/// conservative upper bound.
/// </para>
/// <para>
/// <b>One consistent read.</b> On a relational provider the tables are read inside a
/// <see cref="IsolationLevel.Snapshot"/> transaction, so a <c>/uns</c> or <c>/raw</c> save that
/// lands mid-read cannot leave half the tables on the new state and half on the old. The deploy path
/// validates AND composes its artifact from this one snapshot, which is what makes "the artifact the
/// validator never saw" unrepresentable. SQL Server requires <c>ALLOW_SNAPSHOT_ISOLATION ON</c>
/// (migration <c>EnableSnapshotIsolation</c>); without it the first query fails with error 3952 and
/// the deploy is rejected rather than silently read unisolated. A non-relational provider (the
/// InMemory test provider) and a context already inside a transaction read without one.
/// </para>
/// </remarks>
public static class DraftSnapshotFactory
{
@@ -29,6 +40,29 @@ public static class DraftSnapshotFactory
/// <param name="ct">Cancellation token.</param>
/// <returns>A snapshot populated from the live DB, ready for <see cref="DraftValidator.Validate"/>.</returns>
public static async Task<DraftSnapshot> FromConfigDbAsync(OtOpcUaConfigDbContext db, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(db);
var ownsTransaction = db.Database.IsRelational() && db.Database.CurrentTransaction is null;
var tx = ownsTransaction
? await db.Database.BeginTransactionAsync(IsolationLevel.Snapshot, ct).ConfigureAwait(false)
: null;
await using (tx)
{
var snapshot = await ReadAsync(db, ct).ConfigureAwait(false);
// Read-only: committing releases the snapshot's version-store reservation; nothing was written.
if (tx is not null) await tx.CommitAsync(ct).ConfigureAwait(false);
return snapshot;
}
}
// Every composed table is read ORDERED BY ITS NATURAL KEY, server-side. That order IS the deployment
// artifact's order (ConfigComposer.Compose serialises the snapshot as given): it is exactly the order the
// pre-R03:3.2 composer's own queries produced — the database collation's, case-insensitive on SQL Server —
// so the revision hash of an unchanged configuration is byte-identical across the refactor (fix round 1;
// pinned by ConfigComposerSqlServerHashStabilityTests). Re-sorting in memory with any other comparer would
// re-hash every mixed-case configuration and make drift detection report the whole fleet drifted.
private static async Task<DraftSnapshot> ReadAsync(OtOpcUaConfigDbContext db, CancellationToken ct)
=> new DraftSnapshot
{
// GenerationId=0 and null Enterprise/Site are intentional conservative fallbacks at the
@@ -36,23 +70,26 @@ public static class DraftSnapshotFactory
// EquipmentUuid checks run in separate validator passes — no rule here reads these fields.
GenerationId = 0, // generation model dropped; placeholder (no rule reads it)
ClusterId = string.Empty, // global snapshot; rules compare entity ClusterId fields, not this
ServerClusters = await db.ServerClusters.AsNoTracking().OrderBy(x => x.ClusterId).ToListAsync(ct),
ClusterNodes = await db.ClusterNodes.AsNoTracking().OrderBy(x => x.NodeId).ToListAsync(ct),
// v3: Namespace entity retired — no namespace list to materialize. The Raw tree
// (RawFolders/TagGroups) + UnsTagReferences feed the v3 rules (charset, tagname length,
// effective-leaf uniqueness).
RawFolders = await db.RawFolders.AsNoTracking().ToListAsync(ct),
DriverInstances = await db.DriverInstances.AsNoTracking().ToListAsync(ct),
Devices = await db.Devices.AsNoTracking().ToListAsync(ct),
TagGroups = await db.TagGroups.AsNoTracking().ToListAsync(ct),
UnsAreas = await db.UnsAreas.AsNoTracking().ToListAsync(ct),
UnsLines = await db.UnsLines.AsNoTracking().ToListAsync(ct),
Equipment = await db.Equipment.AsNoTracking().ToListAsync(ct),
Tags = await db.Tags.AsNoTracking().ToListAsync(ct),
UnsTagReferences = await db.UnsTagReferences.AsNoTracking().ToListAsync(ct),
VirtualTags = await db.VirtualTags.AsNoTracking().ToListAsync(ct),
ScriptedAlarms = await db.ScriptedAlarms.AsNoTracking().ToListAsync(ct),
Scripts = await db.Scripts.AsNoTracking().ToListAsync(ct),
PollGroups = await db.PollGroups.AsNoTracking().ToListAsync(ct),
NodeAcls = await db.NodeAcls.AsNoTracking().ToListAsync(ct),
RawFolders = await db.RawFolders.AsNoTracking().OrderBy(x => x.RawFolderId).ToListAsync(ct),
DriverInstances = await db.DriverInstances.AsNoTracking().OrderBy(x => x.DriverInstanceId).ToListAsync(ct),
Devices = await db.Devices.AsNoTracking().OrderBy(x => x.DeviceId).ToListAsync(ct),
TagGroups = await db.TagGroups.AsNoTracking().OrderBy(x => x.TagGroupId).ToListAsync(ct),
UnsAreas = await db.UnsAreas.AsNoTracking().OrderBy(x => x.UnsAreaId).ToListAsync(ct),
UnsLines = await db.UnsLines.AsNoTracking().OrderBy(x => x.UnsLineId).ToListAsync(ct),
Equipment = await db.Equipment.AsNoTracking().OrderBy(x => x.EquipmentId).ToListAsync(ct),
Tags = await db.Tags.AsNoTracking().OrderBy(x => x.TagId).ToListAsync(ct),
UnsTagReferences = await db.UnsTagReferences.AsNoTracking().OrderBy(x => x.UnsTagReferenceId).ToListAsync(ct),
VirtualTags = await db.VirtualTags.AsNoTracking().OrderBy(x => x.VirtualTagId).ToListAsync(ct),
ScriptedAlarms = await db.ScriptedAlarms.AsNoTracking().OrderBy(x => x.ScriptedAlarmId).ToListAsync(ct),
Scripts = await db.Scripts.AsNoTracking().OrderBy(x => x.ScriptId).ToListAsync(ct),
PollGroups = await db.PollGroups.AsNoTracking().OrderBy(x => x.PollGroupId).ToListAsync(ct),
NodeAcls = await db.NodeAcls.AsNoTracking().OrderBy(x => x.NodeAclId).ToListAsync(ct),
ApiKeys = await db.ApiKeys.AsNoTracking().OrderBy(x => x.ApiKeyId).ToListAsync(ct),
PriorEquipment = [], // intentional: no prior-generation table to diff against at this level
ActiveReservations = await db.ExternalIdReservations
.AsNoTracking()
@@ -33,7 +33,7 @@ public static class DraftValidator
ValidateReservationPreflight(draft, errors);
ValidateEquipmentIdDerivation(draft, errors);
// v3 rules (WP4). The retired rules ValidateSameClusterNamespaceBinding (Namespace entity gone)
// v3 rules. The retired rules ValidateSameClusterNamespaceBinding (Namespace entity gone)
// and ValidateGalaxyTagFullName (blob-as-identity gone) are replaced by:
ValidateRawNameCharset(draft, errors);
ValidateHistorizedTagnameLength(draft, errors);
@@ -47,7 +47,7 @@ public static class DraftValidator
}
/// <summary>
/// #520 stage 3: every <see cref="NodeAcl"/> row's scope must name something the deployed address
/// every <see cref="NodeAcl"/> row's scope must name something the deployed address
/// space actually contains.
/// </summary>
/// <remarks>
@@ -74,7 +74,7 @@ public static class DraftValidator
/// <c>AclScopePathResolver</c>'s dictionary lookups use, so a row this rule passes is a row that
/// resolver resolves. The realm check mirrors <c>NodeAclRealms.TryCanonicalize</c> instead
/// (trimmed, case-insensitive) for the same reason. Both this rule and the resolver were
/// ordinal until Gitea #533; the cross-cluster comparison below was ordinal too, which was the
/// ordinal until that was fixed; the cross-cluster comparison below was ordinal too, which was the
/// worse half — it could reject a deploy as cross-cluster over a <c>ClusterId</c> the runtime
/// would have matched, and the two spellings genuinely occur (see
/// <c>RoleParser.ClusterIdFromRole</c>). The rule now is: identifiers compare
@@ -279,7 +279,7 @@ public static class DraftValidator
/// <summary>
/// The <c>Sql</c> driver's "a pasted literal connection string is never persisted" guarantee, enforced
/// at the deploy gate (Gitea #498).
/// at the deploy gate.
/// </summary>
/// <remarks>
/// <para>A Sql driver names its credentials indirectly — <c>connectionStringRef</c> resolves from the
@@ -294,7 +294,7 @@ public static class DraftValidator
/// Discarding a secret on read is not the same as refusing to store it.</para>
/// <para><b>Two of the three surfaces are checked here.</b> The third — a node's
/// <see cref="ClusterNode.DriverConfigOverridesJson"/> — is not reachable from
/// <see cref="DraftSnapshot"/> and is gated at its save instead (#499); see
/// <see cref="DraftSnapshot"/> and is gated at its save instead; see
/// <see cref="SqlCredentialGuard"/> for why a deploy gate is the wrong instrument for a value that
/// never enters the artifact.</para>
/// <para>Checked on <b>both</b> config surfaces a Sql driver reads: the instance's
@@ -344,7 +344,7 @@ public static class DraftValidator
}
}
/// <summary>Error code for an MTConnect driver instance that has more than one <c>/raw</c> Device (#628).</summary>
/// <summary>Error code for an MTConnect driver instance that has more than one <c>/raw</c> Device.</summary>
public const string MTConnectMultipleDevicesCode = "MTConnectMultipleDevices";
/// <summary>
@@ -352,6 +352,7 @@ public static class DraftValidator
/// use, so the two can never describe different rules.
/// </summary>
/// <param name="driverLabel">The driver instance id or name to cite.</param>
/// <returns>The validation message text.</returns>
public static string MTConnectMultipleDevicesMessage(string driverLabel) =>
$"MTConnect driver '{driverLabel}' has more than one Device. MTConnect binds all tags through one agent " +
"client per driver instance and does not route by /raw Device, and with several Devices the Device's " +
@@ -359,7 +360,7 @@ public static class DraftValidator
"supply the endpoint is ambiguous. Author one MTConnect driver per agent device, each with a single Device.";
/// <summary>
/// #628 (decision (b)): an MTConnect driver instance owns exactly one agent client, so a second <c>/raw</c>
/// An MTConnect driver instance owns exactly one agent client, so a second <c>/raw</c>
/// Device under it is a genuine ambiguity — the driver ignores <c>RawTagEntry.DeviceName</c>, and
/// <c>DriverDeviceConfigMerger</c> only merges a Device's <c>DeviceConfig</c> up to the driver config when
/// there is exactly ONE Device. The normal single-Device layout (every driver gets a default
@@ -380,7 +381,7 @@ public static class DraftValidator
}
}
/// <summary>WP7 Calculation-driver deploy gates. For every tag bound to a <c>Calculation</c> driver:
/// <summary>Calculation-driver deploy gates. For every tag bound to a <c>Calculation</c> driver:
/// <list type="number">
/// <item><b>scriptId existence</b> — the tag's <c>TagConfig.scriptId</c> must be present and resolve to
/// a <see cref="Script"/> row in the draft.</item>
@@ -451,6 +452,58 @@ public static class DraftValidator
"Calculation tags form a dependency cycle (members: " + string.Join(", ", cycle) +
"); a calc→calc cycle is a live oscillation loop and must be broken before deploy",
cycle.Count > 0 ? cycle[0] : null));
// Rule 3 — effective-trigger gate (R07:U5). A calc tag evaluates on a declared dependency
// changing (change-trigger: the literal ctx.GetTag paths) and/or on a timer. A change-triggered
// tag whose script has NO literal reads (`return ctx.Now.Hour;`, a dynamic-path read) and no timer
// never evaluates at all: it sits BadWaitingForInitialData forever while the driver reports
// Healthy, and nothing warned. Same for a tag that is neither change- nor timer-triggered.
foreach (var (t, _, scriptId) in calcTags)
{
var (changeTriggered, timerIntervalMs) = ParseTriggers(t.TagConfig);
if (timerIntervalMs is > 0) continue;
var source = scriptId is not null && scriptSourceById.TryGetValue(scriptId, out var s) ? s : null;
// An unresolved scriptId is already CalculationScriptMissing / CalculationScriptNotFound; with no
// source there are no reads to count, so a change-triggered tag would get a second, misleading
// "no literal ctx.GetTag read" error. (changeTriggered=false + no timer is source-independent.)
if (changeTriggered && source is null) continue;
var depCount = EquipmentScriptPaths.ExtractDependencyRefs(source).Count;
if (changeTriggered && depCount > 0) continue;
errors.Add(new("CalculationNoEffectiveTrigger",
changeTriggered
? $"Calculation tag '{t.TagId}' is change-triggered but its script has no literal ctx.GetTag read, " +
"and it has no timerIntervalMs — nothing would ever evaluate it. Add a dependency read or a timer."
: $"Calculation tag '{t.TagId}' has changeTriggered=false and no timerIntervalMs — nothing would ever evaluate it. " +
"Enable the change trigger or add a timer.",
t.TagId));
}
}
/// <summary>Read the trigger fields from a calc tag's <c>TagConfig</c>: <c>changeTriggered</c>
/// (default <c>true</c>, the driver's own default) and <c>timerIntervalMs</c> (null when absent or not a
/// number). Never throws — malformed JSON yields the defaults.</summary>
private static (bool ChangeTriggered, int? TimerIntervalMs) ParseTriggers(string? tagConfig)
{
if (string.IsNullOrWhiteSpace(tagConfig)) return (true, null);
try
{
using var doc = System.Text.Json.JsonDocument.Parse(tagConfig);
var root = doc.RootElement;
if (root.ValueKind != System.Text.Json.JsonValueKind.Object) return (true, null);
var changeTriggered = true;
if (root.TryGetProperty("changeTriggered", out var ct)
&& ct.ValueKind is System.Text.Json.JsonValueKind.False or System.Text.Json.JsonValueKind.True)
changeTriggered = ct.GetBoolean();
int? timer = null;
if (root.TryGetProperty("timerIntervalMs", out var tm)
&& tm.ValueKind == System.Text.Json.JsonValueKind.Number && tm.TryGetInt32(out var ms))
timer = ms;
return (changeTriggered, timer);
}
catch (System.Text.Json.JsonException)
{
return (true, null);
}
}
/// <summary>Extract the <c>scriptId</c> string from a calc tag's schemaless <c>TagConfig</c> JSON. Never
@@ -634,7 +687,7 @@ public static class DraftValidator
}
// The rule itself lives on UnsSegment, which the AdminUI's authoring guards also call — a second
// copy here is exactly how the two drifted apart (Gitea #535). Only the per-level wording is local.
// copy here is exactly how the two drifted apart. Only the per-level wording is local.
private static void ValidateUnsSegments(DraftSnapshot draft, List<ValidationError> errors)
{
foreach (var a in draft.UnsAreas)
@@ -4,7 +4,7 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Validation;
/// <summary>
/// The one place that decides whether a persisted config blob carries a literal Sql
/// <c>connectionString</c> (Gitea #498 / #499).
/// <c>connectionString</c>.
/// </summary>
/// <remarks>
/// <para>
@@ -12,7 +12,7 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Validation;
/// rule is not cosmetic: it is what keeps a NodeId parseable and stable.
/// </para>
/// <para>
/// ⚠️ <b>This exists because the rule was duplicated and drifted (Gitea #535).</b> The deploy gate
/// ⚠️ <b>This exists because the rule was duplicated and drifted.</b> The deploy gate
/// held it privately in <c>DraftValidator</c> and the AdminUI restated it on the equipment form
/// alone, so an area or line could be named <c>Packaging</c> or <c>Line 1</c>, saved, and then
/// fail the deploy with <c>UnsSegmentInvalid</c>. Because a deployment snapshots the whole
@@ -28,6 +28,10 @@
Directory.Packages.props. This is the one project where the DataProtection chain
enters the repo; every other consumer reaches it through a ProjectReference here. -->
<PackageReference Include="System.Security.Cryptography.Xml"/>
<!-- Surgical transitive pin — see the System.Security.Cryptography.Pkcs block in
Directory.Packages.props. Microsoft.Data.SqlClient (via EFCore.SqlServer) enters here
too and pulls a vulnerable Pkcs 9.0.4 wherever the shared framework prunes the Xml chain. -->
<PackageReference Include="System.Security.Cryptography.Pkcs"/>
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions"/>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions"/>
<PackageReference Include="Polly.Core"/>
@@ -2,7 +2,7 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// Shared capped-exponential backoff for poll-based / lazy-reconnect drivers, extracted from
/// the S7 driver's bespoke poll fork (05/STAB-8, CONV-1). Provides two surfaces:
/// the S7 driver's bespoke poll fork. Provides two surfaces:
/// <list type="bullet">
/// <item><see cref="ComputeDelay"/> — the pure schedule (base, 2×, 4×, … saturating at a
/// cap) used by <c>PollGroupEngine</c> to stretch its retry cadence under sustained
@@ -3,10 +3,10 @@ using System.Text.Json;
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// Gitea #610 — the single authority on whether a driver can WRITE a 1-D array tag. The address-space
/// the single authority on whether a driver can WRITE a 1-D array tag. The address-space
/// composer consults it at compose time (both compose seams run the same composer, so the live-edit and
/// artifact-decode sides cannot disagree) to decide whether an authored-<c>ReadWrite</c> array tag
/// materialises as a writable OPC UA node. Before #610 every array node was forced read-only regardless of
/// materialises as a writable OPC UA node. Previously every array node was forced read-only regardless of
/// driver; that remains the answer for every (driver, element type) pair NOT listed here.
/// <para>
/// <b>The rule is "never advertise a write the driver will refuse".</b> A pair is listed only when the
@@ -20,11 +20,24 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// until this table follows.
/// </para>
/// <list type="bullet">
/// <item><b>Modbus</b> — <c>region: Coils</c> (FC15, any element, at most 1968 coils) and
/// <c>region: HoldingRegisters</c> with a fixed-width numeric <c>dataType</c> (FC16) whose
/// <c>arrayLength × registers-per-element</c> fits the driver's <c>maxRegistersPerWrite</c> (default
/// 123 — the driver refuses rather than chunks a longer FC16). Bit-in-register, String and
/// Bool-in-register arrays have no array encoder; input regions are read-only.</item>
/// <item><b>Modbus</b> — tags addressed by the <c>region</c> + <c>address</c> KEYS (the
/// <c>ModbusTagAddressResolution</c> rule: keys win whenever both are present), read as strictly as the
/// driver reads them (<c>address</c> an integer 0–65535; a present <c>region</c> / <c>dataType</c> /
/// <c>byteOrder</c> / <c>stringByteOrder</c> string must name a member). <c>region: Coils</c>: FC15, at most
/// 1968 coils, with ANY <c>dataType</c> — the resolver coerces a bit region to Bool and never refuses it.
/// <c>region: HoldingRegisters</c> (FC16), never Bool: a fixed-width numeric <c>dataType</c>,
/// <c>String</c> (<c>(stringLength+1)/2</c> registers per element, <c>stringLength &gt;= 1</c>) or
/// <c>BitInRegister</c> (one register per element; <c>bitIndex</c> 0–15), whose
/// <c>arrayLength × registers-per-element</c> fits the driver's <c>maxRegistersPerWrite</c> (default 123 —
/// the driver refuses rather than chunks a longer FC16) and never more than FC16's protocol ceiling of 123,
/// however high the cap is configured (a conformant device refuses a longer FC16, so that write always
/// fails). A BitInRegister array is read-modify-written — one unchunked FC03 of the span, then one FC16 —
/// so it is also bounded by <c>maxRegistersPerRead</c> (default and protocol ceiling 125). Every span
/// (registers or coils) ends at or before address 65535. FC23 is opt-in, capped lower and falls back to
/// FC16, so it never changes the answer. Input
/// regions are read-only. A string-only tag (an <c>addressString</c> with no keys) is resolved by the
/// driver with ITS family through a parser this assembly cannot reference, so it cannot be proven
/// writable here and stays read-only.</item>
/// <item><b>S7</b> — <c>dataType</c> Byte / Int16 / UInt16 / Int32 / UInt32 / Float32 (one contiguous
/// block, <c>EncodeArrayBlock</c>). Bool arrays would clobber neighbouring bits; the wide / String /
/// DateTime element types have no array encoder.</item>
@@ -35,25 +48,68 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// DateTime — and only with a declared <c>arrayLength</c>: an undeclared length materialises an
/// unbounded node, and a short whole-array write to it would RESIZE the Galaxy attribute (the
/// mxaccessgw <c>gateway.md</c> contract), which is never what a client meant.</item>
/// <item>Everything else (AbCip, AbLegacy, TwinCAT, FOCAS, MQTT, MTConnect, Sql, Calculation) — scalar
/// write encoders only; array nodes stay read-only.</item>
/// <item><b>AbCip</b> — <c>dataType</c> SInt / USInt / Int / UInt / DInt / UDInt / Real / Dt / LInt / ULInt /
/// LReal (<c>AbCipArrayCodec</c>, both transports), and not a <c>safetyTag</c>. BOOL arrays (bit-packed
/// DWORDs), STRING and structure arrays have no array encoder.</item>
/// <item><b>AbLegacy</b> — <c>dataType</c> Int / AnalogInt on an N / B / A file, Long on an L file, Float
/// on an F file (the encode width is the DATA FILE's; a type that does not match its file is refused),
/// on a plain <c>&lt;file&gt;&lt;number&gt;:&lt;word&gt;</c> address of at most 256 elements (the PCCC
/// file ceiling the factory clamps to). Refused and so left out: Bit arrays (bit-packed words), String /
/// Timer / Counter / Control arrays (structured; the factory rejects them), <c>/bit</c> and <c>.sub</c>
/// addresses, a PLC-5 device on the native transport (every PLC-5 write is refused before the wire), and
/// the <c>S:</c> / <c>I:</c> / <c>O:</c> files (libplctag refuses their translated names). Indirect
/// addresses (<c>N7:[N7:0]</c>) are kept read-only deliberately: the element moves with the index word
/// and the index read is not atomic with the write.</item>
/// <item><b>TwinCAT</b> — every element type <c>TwinCATArrayCodec</c> lays out (all but Structure, which
/// the factory refuses), STRING / WSTRING included, except a BOOL bit-in-word path
/// (<c>MAIN.Word.3</c>), which is a scalar bit write. A multi-dimensional PLC array is written row-major
/// from the flat 1-D value. The PLC-side shape (element width, declared length ≤ the PLC's) is checked
/// against the resolved symbol at write time, which no compose-time table can know.</item>
/// <item>Everything else (FOCAS, MQTT, MTConnect, Sql, Calculation) — scalar write encoders only; array
/// nodes stay read-only.</item>
/// </list>
/// <para>Modbus, S7 and Galaxy require a declared length — <c>isArray: true</c> AND an integer
/// <c>arrayLength &gt;= 1</c>. The Modbus and S7 tag factories treat anything else as a SCALAR tag (whose
/// <para>Modbus, S7, Galaxy, AbCip, AbLegacy and TwinCAT require a declared length — <c>isArray: true</c> AND
/// an integer <c>arrayLength &gt;= 1</c>. Their tag factories treat anything else as a SCALAR tag (whose
/// scalar encoder cannot take an array value), though the composer still materialises an array node.</para>
/// <para><b>Elements convert; they are not exact-matched.</b> ScadaBridge writes CLR arrays such as
/// <c>int[]</c> / <c>List&lt;int&gt;</c> to Int16 nodes; the node manager converts a numeric array to the
/// node's element type, and the AbCip, AbLegacy and TwinCAT encoders then <c>Convert</c> each element to the
/// controller's type (an element that does not fit is <c>BadOutOfRange</c>); Modbus <c>Convert</c>s each element to
/// the register / coil type the same way. So those rows do not key on an exact node <c>DataType</c>; they leave
/// out only the node types whose elements the server cannot promise to convert: a <c>DateTime</c> node over a
/// numeric element (it never converts) or a <c>String</c> node over one (it converts only when the text happens
/// to parse, <c>"1"</c> / <c>"true"</c>) — on Modbus over a numeric or bit element, and a Modbus String element
/// takes any node type; likewise on AbCip, AbLegacy and TwinCAT — TwinCAT keeps
/// DATE / DT and WSTRING under a DateTime node, and STRING / WSTRING under a String node; a DateTime's text
/// form is culture-dependent, so a STRING under a DateTime node is left out), and a <c>Boolean</c> node on
/// AbLegacy, whose encoder refuses bool elements. An unknown DataType name (served as BaseDataType) hands the
/// driver whatever the client sent and is not refused.</para>
/// <para>Multi-dimensional arrays (<c>ValueRank &gt; 1</c>) are out of scope altogether — every array node
/// is materialised 1-D (<c>ValueRank = OneDimension</c>), so nothing here considers more than one rank.</para>
/// </summary>
public static class DriverArrayWriteSupport
{
/// <summary>The Modbus driver's FC16 write cap when <c>maxRegistersPerWrite</c> is absent or <c>0</c>
/// (<c>ModbusDriver</c> resolves it the same way).</summary>
/// (<c>ModbusDriver</c> resolves it the same way). A configured value only ever LOWERS it — see
/// <see cref="ModbusProtocolMaxRegistersPerWrite"/>.</summary>
public const int ModbusDefaultMaxRegistersPerWrite = 123;
/// <summary>Modbus FC15's protocol ceiling on coils per request (0x07B0). The driver sends one FC15 for the
/// whole array without chunking, so a longer coil array is refused by every conformant device.</summary>
public const int ModbusMaxCoilsPerWrite = 1968;
/// <summary>Modbus FC16's protocol ceiling on registers per request (0x7B). A configured
/// <c>maxRegistersPerWrite</c> above it never lifts it: a conformant device refuses a longer FC16.</summary>
public const int ModbusProtocolMaxRegistersPerWrite = 123;
/// <summary>Modbus FC03's protocol ceiling on registers per request (0x7D) — the bound on a BitInRegister[]
/// write's read-modify-write read, together with a configured <c>maxRegistersPerRead</c>.</summary>
public const int ModbusProtocolMaxRegistersPerRead = 125;
/// <summary>The last Modbus address (registers and coils alike): a span ending past it is refused by every
/// conformant device (exception 02).</summary>
private const long ModbusLastAddress = ushort.MaxValue;
// Element type → registers per element (ModbusDriver.RegisterCount).
private static readonly Dictionary<string, int> ModbusRegisterArrayTypes = new(StringComparer.OrdinalIgnoreCase)
{
@@ -62,6 +118,30 @@ public static class DriverArrayWriteSupport
["Int64"] = 4, ["UInt64"] = 4, ["Float64"] = 4,
};
// The Modbus enums' member names (ModbusRegion / ModbusDataType / ModbusByteOrder / ModbusStringByteOrder in
// Driver.Modbus.Addressing, which this assembly cannot reference). The parity tests enumerate the real enums, so a
// member added there without being added here fails a test.
private static readonly HashSet<string> ModbusRegions = new(StringComparer.OrdinalIgnoreCase)
{
"Coils", "DiscreteInputs", "InputRegisters", "HoldingRegisters",
};
private static readonly HashSet<string> ModbusDataTypes = new(StringComparer.OrdinalIgnoreCase)
{
"Bool", "Int16", "UInt16", "Int32", "UInt32", "Int64", "UInt64", "Float32", "Float64",
"BitInRegister", "String", "Bcd16", "Bcd32",
};
private static readonly HashSet<string> ModbusByteOrders = new(StringComparer.OrdinalIgnoreCase)
{
"BigEndian", "WordSwap", "ByteSwap", "FullReverse",
};
private static readonly HashSet<string> ModbusStringByteOrders = new(StringComparer.OrdinalIgnoreCase)
{
"HighByteFirst", "LowByteFirst",
};
private static readonly HashSet<string> S7ArrayTypes = new(StringComparer.OrdinalIgnoreCase)
{
"Byte", "Int16", "UInt16", "Int32", "UInt32", "Float32",
@@ -72,6 +152,34 @@ public static class DriverArrayWriteSupport
"Boolean", "Int32", "Int64", "Float", "Double", "String", "DateTime",
};
// AbCipArrayCodec.ElementWidth: the atomic element types with a fixed wire width.
private static readonly HashSet<string> AbCipArrayTypes = new(StringComparer.OrdinalIgnoreCase)
{
"SInt", "USInt", "Int", "UInt", "DInt", "UDInt", "Real", "Dt", "LInt", "ULInt", "LReal",
};
/// <summary>The PCCC data-file element count ceiling (<c>AbLegacyArray.MaxElements</c>): the AbLegacy tag
/// factory CLAMPS a longer <c>arrayLength</c> to it, so a node of the authored (longer) length never matches.</summary>
public const int AbLegacyMaxArrayElements = 256;
// AbLegacyTagDefinitionFactory.ArrayFileElementType for the files the table admits, keyed by file letter, to the
// authored element types that match that file's width (ArrayFileMismatch). I / O / S are deliberately absent:
// libplctag refuses a write to their translated names, and the table does not widen output-image / status-file
// writes on the native transport alone.
private static readonly Dictionary<string, string[]> AbLegacyFileElementTypes = new(StringComparer.OrdinalIgnoreCase)
{
["N"] = ["Int", "AnalogInt"], ["B"] = ["Int", "AnalogInt"], ["A"] = ["Int", "AnalogInt"],
["L"] = ["Long"],
["F"] = ["Float"],
};
// TwinCATArrayCodec.Encode: every TwinCATDataType but Structure (which the factory refuses outright).
private static readonly HashSet<string> TwinCATArrayTypes = new(StringComparer.OrdinalIgnoreCase)
{
"Bool", "SInt", "USInt", "Int", "UInt", "DInt", "UDInt", "LInt", "ULInt", "Real", "LReal",
"String", "WString", "Time", "Date", "DateTime", "TimeOfDay",
};
/// <summary>
/// Whether a driver of <paramref name="driverType"/> can write a 1-D array tag of the given shape.
/// </summary>
@@ -93,22 +201,7 @@ public static class DriverArrayWriteSupport
return dataType is not null && GalaxyArrayDataTypes.Contains(dataType)
&& TryParseObject(tagConfig, out var galaxyRoot) && ReadArrayLength(galaxyRoot) is not null;
case DriverTypeNames.Modbus:
{
// Same reads as ModbusTagDefinitionFactory: a blob with no `region` or no numeric `address` is not
// a Modbus tag at all (the factory refuses it). A present non-string region is read as absent ⇒
// HoldingRegisters, and an absent dataType ⇒ Int16 (TagConfigJson.TryReadEnumStrict).
if (!TryParseObject(tagConfig, out var root)
|| !root.TryGetProperty("region", out _)
|| !root.TryGetProperty("address", out var address) || address.ValueKind != JsonValueKind.Number
|| ReadArrayLength(root) is not { } count)
return false;
var region = ReadEnumName(root, "region") ?? "HoldingRegisters";
var native = ReadEnumName(root, "dataType") ?? "Int16";
if (string.Equals(region, "Coils", StringComparison.OrdinalIgnoreCase)) return count <= ModbusMaxCoilsPerWrite;
return string.Equals(region, "HoldingRegisters", StringComparison.OrdinalIgnoreCase)
&& ModbusRegisterArrayTypes.TryGetValue(native, out var registersPerElement)
&& (long)count * registersPerElement <= ModbusMaxRegistersPerWrite(driverConfigs);
}
return ModbusSupports(dataType, tagConfig, driverConfigs);
case DriverTypeNames.S7:
{
// S7TagDefinitionFactory requires a dataType key (absent ⇒ not an S7 tag at all); a present
@@ -119,22 +212,236 @@ public static class DriverArrayWriteSupport
var native = ReadEnumName(root, "dataType") ?? "Int16";
return S7ArrayTypes.Contains(native);
}
case DriverTypeNames.AbCip:
return AbCipSupports(dataType, tagConfig);
case DriverTypeNames.AbLegacy:
return AbLegacySupports(dataType, tagConfig, driverConfigs);
case DriverTypeNames.TwinCAT:
return TwinCATSupports(dataType, tagConfig);
default:
return false;
}
}
// ModbusTagAddressResolution + ModbusTagDefinitionFactory + ModbusDriver.WriteAsync, KEYS shape only:
// - `region` AND `address` both present (Classify == Keys). Anything else is a string-only (`addressString`) blob the
// driver resolves with ITS family through ModbusAddressParser, which this assembly cannot reference — so it is
// never proven writable here and stays read-only (conservative: a hidden write, never an advertised refusal).
// - the strict key reads: `address` an integer 0–65535; a present string `region` / `dataType` / `byteOrder` /
// `stringByteOrder` must be a member name (a non-string token is read as absent ⇒ the default).
// - Coils: the resolver COERCES any type to Bool (never refuses); one FC15 of at most 1968 coils.
// - HoldingRegisters: never Bool; fixed-width numeric, String ((stringLength+1)/2 registers, stringLength >= 1) or
// BitInRegister (one register each, bitIndex 0–15); arrayLength × registers per element within
// maxRegistersPerWrite clamped to FC16's 123 (the driver refuses rather than chunks; FC23 is opt-in and capped
// lower, falling back to FC16, so it never changes the answer). BitInRegister[] also READS its span first in one
// unchunked FC03, so it is bounded by maxRegistersPerRead (clamped to FC03's 125) too.
// - Every span ends at or before address 65535 (a conformant device refuses one past it with exception 02).
// - Elements CONVERT, so the node type is not exact-matched. A DateTime node over a numeric or bit element never
// converts; a String node over one converts only when the text happens to parse ("1", "true"), which the server
// cannot promise — both are left out. A String element Convert.ToString()s any node type.
private static bool ModbusSupports(string? dataType, string? tagConfig, string?[]? driverConfigs)
{
if (!TryParseObject(tagConfig, out var root)
|| !root.TryGetProperty("region", out _)
|| !root.TryGetProperty("address", out var addressEl) || addressEl.ValueKind != JsonValueKind.Number
|| !addressEl.TryGetInt32(out var address) || address is < 0 or > ushort.MaxValue
|| ReadArrayLength(root) is not { } count
|| !TryReadModbusEnum(root, "region", ModbusRegions, out var region)
|| !TryReadModbusEnum(root, "dataType", ModbusDataTypes, out var native)
|| !TryReadModbusEnum(root, "byteOrder", ModbusByteOrders, out _)
|| !TryReadModbusEnum(root, "stringByteOrder", ModbusStringByteOrders, out _))
return false;
region ??= "HoldingRegisters";
if (region.Equals("Coils", StringComparison.OrdinalIgnoreCase))
return count <= ModbusMaxCoilsPerWrite && address + (long)count - 1 <= ModbusLastAddress
&& dataType is not ("String" or "DateTime");
if (!region.Equals("HoldingRegisters", StringComparison.OrdinalIgnoreCase)) return false; // input regions are read-only
native ??= "Int16";
int registersPerElement;
if (native.Equals("String", StringComparison.OrdinalIgnoreCase))
{
// The factory narrows to ushort ((ushort)ReadInt) and refuses a zero length.
var stringLength = (ushort)ReadInt(root, "stringLength");
if (stringLength < 1) return false;
registersPerElement = (stringLength + 1) / 2;
}
else
{
if (dataType is "String" or "DateTime") return false;
if (native.Equals("BitInRegister", StringComparison.OrdinalIgnoreCase))
{
// The factory narrows to byte ((byte)ReadInt) and refuses a bit above 15. The RMW reads the whole span
// in ONE unchunked FC03 before writing it, so the read cap bounds it too.
if ((byte)ReadInt(root, "bitIndex") > 15 || count > ModbusMaxRegistersPerRead(driverConfigs)) return false;
registersPerElement = 1;
}
else if (!ModbusRegisterArrayTypes.TryGetValue(native, out registersPerElement))
{
return false; // Bool in a register region: the factory refuses the pair
}
}
var registers = (long)count * registersPerElement;
return registers <= ModbusMaxRegistersPerWrite(driverConfigs) && address + registers - 1 <= ModbusLastAddress;
}
// TagConfigJson.TryReadEnumStrict over the Modbus enum's member names: absent or a non-string token ⇒ null (the
// caller applies the default); a string must name a member (trimmed, case-insensitive), else the tag is refused.
// Numeric spellings are not member names, so they are refused exactly as ClassifyEnumText refuses them.
private static bool TryReadModbusEnum(JsonElement root, string name, HashSet<string> members, out string? value)
{
value = ReadEnumName(root, name)?.Trim();
return value is null || members.Contains(value);
}
private static int ReadInt(JsonElement root, string name) =>
root.TryGetProperty(name, out var el) && el.ValueKind == JsonValueKind.Number && el.TryGetInt32(out var v) ? v : 0;
// AbCipTagDefinitionFactory + AbCipDriver.WriteAsync: a non-blank string tagPath, an atomic element type the codec
// has a width for (BOOL arrays are bit-packed DWORDs, STRING / structure elements are structures), a declared
// length, and not a safety tag (refused BadNotWritable). Each element is CONVERTED, so any numeric / Boolean node
// type works; DateTime and (non-numeric) String elements never convert to a Logix integer or REAL.
private static bool AbCipSupports(string? dataType, string? tagConfig)
{
if (!TryParseObject(tagConfig, out var root) || !HasNonBlankString(root, "tagPath")
|| ReadArrayLength(root) is null
|| (root.TryGetProperty("safetyTag", out var safety) && safety.ValueKind == JsonValueKind.True))
return false;
var native = ReadEnumName(root, "dataType")?.Trim() ?? "DInt";
return AbCipArrayTypes.Contains(native) && dataType is not ("String" or "DateTime");
}
// AbLegacyTagDefinitionFactory + AbLegacyDriver.WriteAsync: a flat Int / AnalogInt / Long / Float array on a plain
// `<file><number>:<word>` address whose data file's element width matches (ArrayFileMismatch); no /bit, no .sub,
// no indirect [..] address; at most 256 elements (the factory clamps a longer length, so the node never matches);
// and not a PLC-5 on the native transport (refused before the wire). Elements CONVERT from any numeric node type;
// bool, string and DateTime elements are refused.
private static bool AbLegacySupports(string? dataType, string? tagConfig, string?[]? driverConfigs)
{
if (!TryParseObject(tagConfig, out var root)
|| !root.TryGetProperty("address", out var addressEl) || addressEl.ValueKind != JsonValueKind.String
|| ReadArrayLength(root) is not { } count || count > AbLegacyMaxArrayElements
|| dataType is "Boolean" or "String" or "DateTime")
return false;
var native = ReadEnumName(root, "dataType")?.Trim() ?? "Int";
if (!TryParsePlainPcccAddress(addressEl.GetString(), out var fileLetter)
|| !AbLegacyFileElementTypes.TryGetValue(fileLetter, out var matching)
|| !matching.Contains(native, StringComparer.OrdinalIgnoreCase))
return false;
return !AbLegacyIsNativePlc5(driverConfigs);
}
// `<letters><digits>:<digits>` and nothing else (trimmed) — what AbLegacyAddress.TryParse yields with no bit
// index, no sub-element and no indirection, for the file letters that carry a file number.
private static bool TryParsePlainPcccAddress(string? address, out string fileLetter)
{
fileLetter = "";
if (string.IsNullOrWhiteSpace(address)) return false;
var src = address.Trim();
var colon = src.IndexOf(':');
if (colon <= 0) return false;
var letterEnd = 0;
while (letterEnd < colon && char.IsAsciiLetter(src[letterEnd])) letterEnd++;
if (letterEnd == 0 || letterEnd == colon) return false; // no letter, or no file number
var fileNumber = src[letterEnd..colon];
var word = src[(colon + 1)..];
if (!fileNumber.All(char.IsAsciiDigit) || !int.TryParse(fileNumber, out _)
|| word.Length == 0 || !word.All(char.IsAsciiDigit) || !int.TryParse(word, out _))
return false;
fileLetter = src[..letterEnd];
return true;
}
// Conservative: ANY config naming the native transport AND ANY config naming a PLC-5 (a root `plcFamily`, as a
// DeviceConfig carries it, or one in a `devices` array, as a DriverConfig carries it). Keys and values are matched
// case-insensitively, like the driver's DTO binding and enum parse. An absent transport is libplctag for every
// family (AbLegacyTransportResolver), which writes PLC-5.
private static bool AbLegacyIsNativePlc5(string?[]? driverConfigs)
{
var native = false;
var plc5 = false;
foreach (var config in driverConfigs ?? [])
{
if (!TryParseObject(config, out var root)) continue;
foreach (var prop in root.EnumerateObject())
{
if (IsNamed(prop, "transport") && IsStringValue(prop.Value, "Native")) native = true;
else if (IsNamed(prop, "plcFamily") && IsStringValue(prop.Value, "Plc5")) plc5 = true;
else if (IsNamed(prop, "devices") && prop.Value.ValueKind == JsonValueKind.Array)
{
foreach (var device in prop.Value.EnumerateArray())
{
if (device.ValueKind != JsonValueKind.Object) continue;
foreach (var dp in device.EnumerateObject())
if (IsNamed(dp, "plcFamily") && IsStringValue(dp.Value, "Plc5")) plc5 = true;
}
}
}
}
return native && plc5;
}
// TwinCATTagDefinitionFactory + TwinCATDriver.WriteAsync: a non-blank string symbolPath, a declared length, any
// element type the codec lays out (not Structure, which the factory refuses), and not a BOOL bit-in-word path
// (`MAIN.Word.3`: a scalar bit write, which an array value cannot feed). Elements CONVERT. A String node only feeds
// a STRING / WSTRING element (anything else would depend on the text happening to parse). A DateTime node only
// feeds a DATE / DT element or a WSTRING: its text form is CULTURE-dependent and may carry characters a STRING's
// strict Windows-1252 encoding refuses (ICU's en-US "12:00:00 PM").
private static bool TwinCATSupports(string? dataType, string? tagConfig)
{
if (!TryParseObject(tagConfig, out var root) || !HasNonBlankString(root, "symbolPath")
|| ReadArrayLength(root) is null)
return false;
var native = ReadEnumName(root, "dataType")?.Trim() ?? "DInt";
if (!TwinCATArrayTypes.Contains(native)) return false;
var isWString = native.Equals("WString", StringComparison.OrdinalIgnoreCase);
var isString = isWString || native.Equals("String", StringComparison.OrdinalIgnoreCase);
var isDate = native.Equals("Date", StringComparison.OrdinalIgnoreCase) || native.Equals("DateTime", StringComparison.OrdinalIgnoreCase);
if (dataType == "String" && !isString) return false;
if (dataType == "DateTime" && !isWString && !isDate) return false;
return !(native.Equals("Bool", StringComparison.OrdinalIgnoreCase)
&& HasTwinCATBitSuffix(root.GetProperty("symbolPath").GetString()!));
}
// TwinCATSymbolPath.TryParse's bit-index rule: a last top-level `.` segment of 0..31 with no `[`, behind at least
// one other segment (a TwinCAT 2 global's leading dot belongs to its root segment).
private static bool HasTwinCATBitSuffix(string symbolPath)
{
var src = symbolPath.Trim();
if (src.StartsWith('.')) src = src[1..];
var depth = 0;
var lastDot = -1;
for (var i = 0; i < src.Length; i++)
{
if (src[i] == '[') depth++;
else if (src[i] == ']') depth--;
else if (src[i] == '.' && depth == 0) lastDot = i;
}
if (lastDot < 0) return false;
var last = src[(lastDot + 1)..];
return int.TryParse(last, out var bit) && bit is >= 0 and <= 31 && !last.Contains('[');
}
private static bool HasNonBlankString(JsonElement root, string name) =>
root.TryGetProperty(name, out var el) && el.ValueKind == JsonValueKind.String && !string.IsNullOrWhiteSpace(el.GetString());
private static bool IsNamed(JsonProperty prop, string name) => string.Equals(prop.Name, name, StringComparison.OrdinalIgnoreCase);
private static bool IsStringValue(JsonElement el, string value) =>
el.ValueKind == JsonValueKind.String && string.Equals(el.GetString()?.Trim(), value, StringComparison.OrdinalIgnoreCase);
/// <summary>
/// Gitea #610 review (R6 MINOR-1) — whether a driver of <paramref name="driverType"/> is known to REFUSE an
/// Whether a driver of <paramref name="driverType"/> is known to REFUSE an
/// array value written to one of its SCALAR tags, so the node manager may refuse such a write inline
/// (<c>BadTypeMismatch</c>, nothing dispatched) instead of storing it optimistically, dispatching it, and
/// reverting it when the scalar encoder throws.
/// <para>
/// Only Modbus and S7 qualify: their scalar encoders <c>Convert.ToXxx</c> the value, which throws on an
/// array. Every other driver keeps the pre-#610 behaviour — the write is dispatched and the driver
/// array. Every other driver keeps the previous behaviour — the write is dispatched and the driver
/// decides. That matters for the pass-through drivers: a browse-committed array attribute is committed
/// as a SCALAR tag (the <c>/raw</c> commit never sets <c>isArray</c>), the driver publishes an array into
/// it, and writing that array back worked before #610 — <c>OpcUaClient</c> forwards the
/// it, and writing that array back worked previously — <c>OpcUaClient</c> forwards the
/// <c>Variant</c> unchanged (the upstream server arbitrates shape) and Galaxy's <c>MxValueEncoder</c>
/// encodes the array. An unknown / null driver type is not listed ⇒ <see langword="false"/>.
/// </para>
@@ -144,9 +451,24 @@ public static class DriverArrayWriteSupport
public static bool RefusesArrayValueOnScalarTag(string? driverType) =>
driverType is DriverTypeNames.Modbus or DriverTypeNames.S7;
// The smallest Modbus write cap configured in any config; none configured (absent / non-positive) ⇒ the driver
// default (123). The key is matched case-insensitively, like the driver's DTO binding.
private static int ModbusMaxRegistersPerWrite(string?[]? driverConfigs)
// The effective Modbus FC16 write cap: the smallest maxRegistersPerWrite configured in any config (absent /
// non-positive ⇒ the driver default, 123), CLAMPED to FC16's protocol ceiling of 123. The driver trusts a larger
// configured cap and sends the over-ceiling FC16 (past 127 registers its one-byte byte-count field even wraps), which
// every conformant device refuses — so such a write always fails and is never advertised.
private static int ModbusMaxRegistersPerWrite(string?[]? driverConfigs) =>
Math.Min(SmallestPositive(driverConfigs, "maxRegistersPerWrite") ?? ModbusDefaultMaxRegistersPerWrite,
ModbusProtocolMaxRegistersPerWrite);
// The effective Modbus FC03 read cap: the smallest maxRegistersPerRead configured (absent / non-positive ⇒ the driver
// default, 125), clamped to FC03's protocol ceiling of 125. Only the BitInRegister[] write reads (its RMW issues one
// unchunked FC03 of the whole span), so only that shape is bounded by it.
private static int ModbusMaxRegistersPerRead(string?[]? driverConfigs) =>
Math.Min(SmallestPositive(driverConfigs, "maxRegistersPerRead") ?? ModbusProtocolMaxRegistersPerRead,
ModbusProtocolMaxRegistersPerRead);
// The smallest positive Int32 value of `key` (matched case-insensitively, like the driver's DTO binding) at the root
// of any config; null when none carries one.
private static int? SmallestPositive(string?[]? driverConfigs, string key)
{
int? cap = null;
foreach (var config in driverConfigs ?? [])
@@ -154,7 +476,7 @@ public static class DriverArrayWriteSupport
if (!TryParseObject(config, out var root)) continue;
foreach (var prop in root.EnumerateObject())
{
if (string.Equals(prop.Name, "maxRegistersPerWrite", StringComparison.OrdinalIgnoreCase)
if (string.Equals(prop.Name, key, StringComparison.OrdinalIgnoreCase)
&& prop.Value.ValueKind == JsonValueKind.Number
&& prop.Value.TryGetInt32(out var configured) && configured > 0)
{
@@ -162,7 +484,7 @@ public static class DriverArrayWriteSupport
}
}
}
return cap ?? ModbusDefaultMaxRegistersPerWrite;
return cap;
}
// The declared element count: isArray is the JSON literal true AND arrayLength is an Int32 >= 1 — the rule the
@@ -53,7 +53,7 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// Sparkplug births, MTConnect probe inference).
/// <para>
/// <b>It exists because <paramref name="DriverDataType"/> is lossy in exactly the direction a
/// browse-commit needs</b> (Gitea #536): <c>AbCipDataType.SInt</c>, <c>.Int</c> and <c>.DInt</c>
/// browse-commit needs</b>: <c>AbCipDataType.SInt</c>, <c>.Int</c> and <c>.DInt</c>
/// all map to <see cref="DriverDataType.Int32"/>, so a commit that reverse-mapped the
/// driver-agnostic type would bind a 16-bit INT as a 32-bit DINT. Nothing downstream of the
/// address-space build reads this — it is authoring metadata carried alongside the type the
@@ -38,7 +38,7 @@ public enum DriverCapability
AlarmAcknowledge,
/// <summary>
/// <c>IDriver.InitializeAsync</c> — connection establishment (R2-10 T10 closure: without this,
/// <c>IDriver.InitializeAsync</c> — connection establishment (without this,
/// a driver that cannot connect contributes zero breaker input and the resilience panel stays
/// quiet for the failure operators most care about). Does NOT retry — the
/// <c>DriverInstanceActor</c> reconnect timer is the retry loop, and pipeline-level retries
@@ -0,0 +1,21 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// Thrown by a driver that deliberately refuses an operation before contacting its backend — a
/// configuration / authorization verdict (e.g. no Galaxy ack credential is available for the
/// caller, lmxopcua#749), not a transport or device fault.
/// </summary>
/// <remarks>
/// The resilience pipeline neither retries it nor counts it toward the circuit breaker: a refusal
/// says nothing about the backend's health, so letting five misconfigured acks open the breaker
/// would block every correctly-configured caller on the same host.
/// </remarks>
public sealed class DriverOperationRefusedException : InvalidOperationException
{
/// <summary>Initializes a new instance of the <see cref="DriverOperationRefusedException"/> class.</summary>
/// <param name="message">Why the operation was refused. Must never contain a secret.</param>
public DriverOperationRefusedException(string message)
: base(message)
{
}
}
@@ -1,6 +1,6 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
// OPC UA Part 11 modified-value history (Gitea #668): every HistoryUpdate a backend applies leaves an audit row
// OPC UA Part 11 modified-value history: every HistoryUpdate a backend applies leaves an audit row
// carrying the PRIOR value, who made the change and when; HistoryRead with IsReadModified=true serves them.
/// <summary>Who is performing a HistoryUpdate. Threaded into every <see cref="IHistorianDataUpdater"/> call.</summary>
@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
// Shared history result contracts used by IHistorianDataSource and the historian gateway.
// Drivers carry no history capability (Gitea #655): HistoryRead is served server-side.
// Drivers carry no history capability: HistoryRead is served server-side.
/// <summary>Result of a HistoryRead call.</summary>
@@ -13,7 +13,7 @@ public sealed record HistoryReadResult(
/// <summary>Aggregate function for processed history reads. Mirrors OPC UA Part 13 standard aggregates.</summary>
/// <remarks>
/// The first five members are the original set every historian backend serves. The remainder (#285) are
/// The first five members are the original set every historian backend serves. The remainder are
/// the further standard Part 13 aggregates a server-side HistoryRead can forward to a backend that
/// supports them (a gateway backend maps them to its own aggregate modes). A backend that cannot serve a
/// member must throw <see cref="HistoryAggregateNotSupportedException"/> — the server maps that to
@@ -48,14 +48,14 @@ public enum HistoryAggregateType
StandardDeviationPopulation,
VariancePopulation,
/// <summary>Part 13 TimeAverage2: time-weighted average over the interval's non-Bad data only (#671).</summary>
/// <summary>Part 13 TimeAverage2: time-weighted average over the interval's non-Bad data only.</summary>
TimeAverage2,
/// <summary>Part 13 Total2: time-weighted integral (value·seconds) over the interval's non-Bad data only (#671).</summary>
/// <summary>Part 13 Total2: time-weighted integral (value·seconds) over the interval's non-Bad data only.</summary>
Total2,
}
/// <summary>Shared <see cref="HistoryAggregateType"/> sets for <see cref="IHistorianDataSource.SupportedAggregates"/> (#682).</summary>
/// <summary>Shared <see cref="HistoryAggregateType"/> sets for <see cref="IHistorianDataSource.SupportedAggregates"/>.</summary>
public static class HistoryAggregateTypes
{
/// <summary>Every defined <see cref="HistoryAggregateType"/> — the interface default.</summary>
@@ -85,7 +85,7 @@ public sealed class HistoryAggregateNotSupportedException : NotSupportedExceptio
/// <summary>
/// Thrown by a history backend when a read would exceed a configured size limit (e.g. the raw-row cap of the SQL
/// backends' time-weighted aggregates, #671). The Server HistoryRead path translates it to
/// backends' time-weighted aggregates). The Server HistoryRead path translates it to
/// <c>BadTooManyOperations</c> for that node only; it is a client-request limit, not a backend fault.
/// </summary>
public sealed class HistoryReadLimitExceededException : InvalidOperationException
@@ -94,7 +94,7 @@ public sealed class HistoryReadLimitExceededException : InvalidOperationExceptio
public long Limit { get; }
/// <summary>
/// Gitea #694: a raw read refused because rows BEFORE its window (inside the backend's look-back margin — a tie
/// a raw read refused because rows BEFORE its window (inside the backend's look-back margin — a tie
/// cluster just before a page start, the HistorianGateway's boundary row) crowded a ceiling-bounded row budget: how
/// many such rows the read saw (a lower bound when the read was itself truncated). <c>0</c> = not a crowding refusal.
/// The server re-reads such a page once, smaller by this many values (<c>RawPageLimit</c>).
@@ -127,7 +127,7 @@ public sealed record HistoricalEvent(
ushort Severity);
/// <summary>Result of a <see cref="IHistorianDataSource.ReadEventsAsync(string?, DateTime, DateTime, int, CancellationToken)"/> call.</summary>
/// <param name="Events">Events ordered by <c>EventTimeUtc</c> in the read's direction — ascending, or descending for a reverse read (#675).</param>
/// <param name="Events">Events ordered by <c>EventTimeUtc</c> in the read's direction — ascending, or descending for a reverse read.</param>
/// <param name="ContinuationPoint">Opaque token for the next call when more events are available; null when complete.</param>
public sealed record HistoricalEventsResult(
IReadOnlyList<HistoricalEvent> Events,
@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// </summary>
/// <remarks>
/// This is a *server registration*, not a driver capability: drivers carry no history
/// surface (Gitea #655). The server calls it directly, so a single historian can serve
/// surface. The server calls it directly, so a single historian can serve
/// many drivers' nodes, and drivers can restart without dropping history availability.
/// All values returned use the shared <see cref="DataValueSnapshot"/> /
/// <see cref="HistoricalEvent"/> shapes; backend-specific quality / type encodings
@@ -20,7 +20,7 @@ public interface IHistorianDataSource : IDisposable
/// <remarks>
/// Forward (<c>startUtc &lt; endUtc</c>) is <c>[start, end)</c> ascending; <c>startUtc == endUtc</c> is a point read.
/// <c>startUtc &gt; endUtc</c> is the Part 11 reverse read <c>(end, start]</c> newest-first — but only a backend that
/// declares <see cref="SupportsReverseRaw"/> is ever sent one (Gitea #686); the server serves a reverse read on any
/// declares <see cref="SupportsReverseRaw"/> is ever sent one; the server serves a reverse read on any
/// other backend from bounded forward reads.
/// </remarks>
/// <param name="fullReference">The full reference of the tag to read.</param>
@@ -38,7 +38,7 @@ public interface IHistorianDataSource : IDisposable
/// <summary>
/// Whether this backend can serve a reverse (newest-first) raw read itself — <see cref="ReadRawAsync"/> with
/// <c>startUtc &gt; endUtc</c> (Gitea #686). The server checks it before every raw read. When <c>false</c> it never
/// <c>startUtc &gt; endUtc</c>. The server checks it before every raw read. When <c>false</c> it never
/// sends <c>start &gt; end</c>: it serves the reverse read — an explicit one, a resumed reverse page, and the
/// end-only capped read Part 11 reads backward from EndTime — from a series of bounded FORWARD reads walking back
/// from the later bound (<c>ReverseRawReader</c> in the OPC UA server). Default <c>false</c>, the safe direction: a
@@ -48,11 +48,11 @@ public interface IHistorianDataSource : IDisposable
/// <summary>
/// The most values one capped <see cref="ReadRawAsync"/> can currently return without failing
/// <see cref="HistoryReadLimitExceededException"/> on a BACKEND row ceiling (Gitea #694); <c>0</c> = no known limit.
/// <see cref="HistoryReadLimitExceededException"/> on a BACKEND row ceiling; <c>0</c> = no known limit.
/// The server sizes its raw pages to it — a page is at most this many values and carries a continuation point when
/// full (Part 11 allows fewer than <c>NumValuesPerNode</c> with a continuation point), instead of failing a page size
/// the backend cannot read in one go. It may change at runtime (a strict HistorianGateway's ceiling is learned from
/// its first refusal, #691.3), so it is read per read. A backend whose reads are never ceiling-bound returns <c>0</c>.
/// its first refusal), so it is read per read. A backend whose reads are never ceiling-bound returns <c>0</c>.
/// The source still never truncates: a read asking for more than this is answered in full or refused. Default
/// <c>0</c>; a backend declares it itself (a guard test pins that).
/// </summary>
@@ -94,7 +94,7 @@ public interface IHistorianDataSource : IDisposable
CancellationToken cancellationToken);
/// <summary>
/// Gitea #693: <see cref="ReadAtTimeAsync(string, IReadOnlyList{DateTime}, CancellationToken)"/> with the value type of
/// <see cref="ReadAtTimeAsync(string, IReadOnlyList{DateTime}, CancellationToken)"/> with the value type of
/// the node being read, for a backend whose stored values do not carry their type (the Gateway: AVEVA stores every
/// historized type as a numeric analog, so a Boolean comes back as a 0/1 Double). Such a backend uses the hint to apply
/// the AtTime rules exactly as the typed backends do — a Boolean (or String / DateTime) node is stepped, an integer node
@@ -156,17 +156,17 @@ public interface IHistorianDataSource : IDisposable
/// <summary>
/// Whether this backend can serve a reverse (newest-first) events read — the reverse overload of
/// <see cref="ReadEventsAsync(string?, DateTime, DateTime, int, bool, CancellationToken)"/> (Gitea #675). The server
/// <see cref="ReadEventsAsync(string?, DateTime, DateTime, int, bool, CancellationToken)"/>. The server
/// checks it BEFORE calling the source: a reverse read (explicit <c>StartTime &gt; EndTime</c>, or an end-only capped
/// read, which Part 11 reads backward) on a backend that cannot is never sent the reverse overload — since Gitea
/// #691.4 the server walks it back from bounded FORWARD reads instead (#675 refused it). Default <c>false</c>; an
/// the server walks it back from bounded FORWARD reads instead (it was once refused). Default <c>false</c>; an
/// implementation (or a wrapper) declares it itself — a guard test pins that.
/// </summary>
bool SupportsReverseEvents => false;
/// <summary>
/// Read historical alarm / event records in a chosen direction — the overload the server's HistoryReadEvents
/// paging calls (Gitea #675).
/// paging calls.
/// </summary>
/// <remarks>
/// <para>
@@ -182,7 +182,7 @@ public interface IHistorianDataSource : IDisposable
/// The default forwards a forward read to the five-argument overload and throws
/// <see cref="NotSupportedException"/> for a reverse one — a backend that cannot read newest-first (the
/// Gateway). The server never relies on that throw: it consults <see cref="SupportsReverseEvents"/> first and walks
/// such a read back from forward reads (#691.4), never returning the earliest events in ascending order as if
/// such a read back from forward reads, never returning the earliest events in ascending order as if
/// they were the answer. A backend that serves reverse natively must override BOTH members.
/// </para>
/// </remarks>
@@ -228,7 +228,7 @@ public interface IHistorianDataSource : IDisposable
long TimestampResolutionTicks => 1;
/// <summary>
/// The Part 13 aggregates this backend serves for a processed HistoryRead (Gitea #682) — the ONE authority for
/// The Part 13 aggregates this backend serves for a processed HistoryRead — the ONE authority for
/// both what the server advertises under <c>HistoryServerCapabilities/AggregateFunctions</c> (i=11201) and what
/// its HistoryRead-Processed answers <c>BadAggregateNotSupported</c> up front without calling the backend.
/// (<c>ServerCapabilities/AggregateFunctions</c>, i=2997, is not affected: it lists every aggregate the server
@@ -236,7 +236,7 @@ public interface IHistorianDataSource : IDisposable
/// </summary>
/// <remarks>
/// The default is <see cref="HistoryAggregateTypes.All"/> — "every aggregate the server can name; a member I cannot
/// serve throws <see cref="HistoryAggregateNotSupportedException"/> at read time", which is exactly the pre-#682
/// serve throws <see cref="HistoryAggregateNotSupportedException"/> at read time", which is exactly the earlier
/// behaviour, so a test fake or an out-of-tree source keeps working unchanged. It is a concrete set rather than a
/// <c>null</c> "all" sentinel so every consumer handles one shape. Every PRODUCTION backend declares its set itself
/// (a reflection guard pins that): a wrapper that forgot to forward this would otherwise advertise every aggregate.
@@ -247,7 +247,7 @@ public interface IHistorianDataSource : IDisposable
IReadOnlySet<HistoryAggregateType> SupportedAggregates => HistoryAggregateTypes.All;
/// <summary>
/// Read modified-value history — OPC UA HistoryReadRaw with <c>IsReadModified = true</c> (Part 11, Gitea #668):
/// Read modified-value history — OPC UA HistoryReadRaw with <c>IsReadModified = true</c> (Part 11):
/// every modification a HistoryUpdate applied to the tag inside the window, each carrying the PRIOR value.
/// </summary>
/// <remarks>

Some files were not shown because too many files have changed in this diff Show More