Commit Graph

273 Commits

Author SHA1 Message Date
Joseph Doherty a5a0f96d49 fix(drivers): harden operator TimeoutMs handling + AbLegacy evict parity
v2-ci / build (pull_request) Successful in 4m5s
v2-ci / unit-tests (pull_request) Failing after 10m45s
Follow-ups from the fleet-wide read-timeout audit that the S7 R2-01 read-leg
fix (PR #453) prompted. The audit confirmed S7 was the ONLY driver with the
async-read-ignores-socket-timeout hang; these are the two adjacent (non-hang)
findings it surfaced.

1. FOCAS TimeoutMs:0 footgun (the risky one): a non-positive Timeout made
   SynchronizedFocasClient DISABLE its per-call wall-clock ceiling, reverting to
   the caller's long-lived poll token — reintroducing exactly the frozen-peer
   wedge S7 just eliminated, under misconfig. Clamp non-positive TimeoutMs to the
   2s default at the config boundary so the deadline can never be authored away.

2. TimeoutMs validation symmetry: apply the same clamp in the AbCip + AbLegacy
   factories. libplctag's Tag.Timeout setter throws on <=0, faulting tag creation
   on every read/write; clamping keeps a misconfigured TimeoutMs:0 running on the
   default bound instead. (Shared PositiveTimeoutOrDefault helper per factory.)

3. AbLegacy reconnect parity with AbCip: AbLegacy evicted the cached libplctag
   runtime on neither the non-zero-status nor transport-exception read/write path
   (AbCip evicts on both), so a data-path fault recovered only via the probe loop
   / libplctag internals. Added EvictRuntime + wired it into both read and write
   failure paths so a fresh handle is created on the next call.

Tests: FOCAS 265->269 (clamp theory + positive), AbCip 336->339 (clamp theory +
positive), AbLegacy 209->212 (read-nonzero / read-exception / write-nonzero evict).
No production regressions; all three driver suites green.
2026-07-15 07:35:17 -04:00
Joseph Doherty 88e0977ae6 fix(s7): bound async wire ops with a wall-clock deadline (R2-01 read leg)
v2-ci / build (pull_request) Successful in 5m5s
v2-ci / unit-tests (pull_request) Failing after 13m50s
The R2-01 live gate (S7_1500ConnectTimeoutOutageTests, docker-pause blackhole)
surfaced a real gap the offline fakes couldn't: S7.Net's ReadTimeout/WriteTimeout
map to the TcpClient's ReceiveTimeout/SendTimeout, which govern only SYNCHRONOUS
socket calls — the async read/write paths S7.Net uses ignore them. On an
established-but-frozen peer (frozen PLC / firewall DROP / cable pulled mid-flow,
TCP session still open) a read blocked until the OS TCP stack gave up (minutes),
silently wedging the poll loop: no Bad tick, no reconnect. STAB-14 fixed only the
CONNECT leg (EnsureConnectedAsync CancelAfter); this is its READ-leg sibling.

- New S7OperationDeadline: bounds every data-plane wire op with a wall-clock
  ceiling (= _options.Timeout), surfacing an overrun as TimeoutException. Applied
  in S7PlcAdapter to Read/ReadBytes/Write/WriteBytes/ReadStatus. OpenAsync is left
  to EnsureConnectedAsync's own CancelAfter (not double-bounded).
- IsS7ConnectionFatal now classifies TimeoutException fatal → handle marked dead →
  next EnsureConnectedAsync reopens (connect-timeout fix takes over from there).
- Tests: 5 S7OperationDeadline unit tests (deadline / token-honouring / caller-
  cancel-passthrough / resultless), 1 driver-reaction test (read TimeoutException
  → reopen). Driver.S7.Tests 260/260.
- Live gate S7_1500ConnectTimeoutOutageTests now GREEN against the real snap7 sim
  (8s: baseline Good -> pause blackhole -> Bad tick -> unpause -> recovered Good).
2026-07-15 07:11:42 -04:00
Joseph Doherty 7982043673 fix(focas-tests): honour fixture endpoint + fix mock timer endianness (9F/1P -> 10/10)
v2-ci / build (pull_request) Successful in 4m31s
v2-ci / unit-tests (pull_request) Failing after 10m26s
Two test/fixture bugs; NO OtOpcUa driver change.

1. Topology: FocasSimFixture reads an env-overridable endpoint
   (OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port,
   but WireBackendTests + WireBackendCoverageTests hardcoded
   focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed
   (remote reachable) but the driver dialed localhost -> KeyNotFound. Added a
   DeviceUri property to the fixture; each test now derives DeviceHost from it.
   Recovered 8 of 9.

2. Mock timer endianness (surfaced once #1 let the tests reach the mock): the
   last failure read 60397977600 for a 3600 s power-on timer (= 0x3C000000 x 60).
   The focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32),
   but cnc_rdtimer's timer fields are LITTLE-endian on real hardware — a
   documented, live-31i-B-validated quirk that FocasWireClient.ParseTimer decodes
   LE (docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was
   correct; the mock was wrong. _wire_timer now emits little-endian (_u32_le).

Suite 9F/1P -> 10/10 against the remote fixture at ~/otopcua-focas.
Integration-sweep follow-up #3.
2026-07-15 05:51:47 -04:00
Joseph Doherty 00f39c849e docs(modbus-tests): document profile-cycling gotcha; sweep #2 was a false alarm
v2-ci / build (pull_request) Successful in 4m13s
v2-ci / unit-tests (pull_request) Failing after 9m41s
The 2026-07 integration sweep flagged the Modbus non-standard profiles as
8F/5F/3F/8F and guessed 'pymodbus 4.x drift'. Re-investigation shows that was
wrong on both counts:

- The image runs the PINNED pymodbus 3.13.0 (verified via import pymodbus;
  __version__). The '...removed in v4' log is 3.13.0's forward-deprecation
  warning, misread as running 4.x.
- The failures were a fixture-cycling artifact: each profile is a separate
  profile-gated compose service sharing :5020, and a plain 'docker compose down'
  (no --profile) does NOT stop the running profile container. It kept holding
  :5020; the next '--profile up' container silently failed to bind (compose
  still says 'Started'), so every later profile run hit the STALE sim. Proven by
  a raw Modbus probe: the 'mitsubishi' sim served DL205's 0xCAFE at HR0.

After force-removing all modbus containers and bringing up each profile cleanly
(verifying it actually published :5020), all four pass: dl205 16/16,
mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17. No OtOpcUa or test
code change needed.

Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md and
corrected the sweep record. Integration-sweep follow-up #2.
2026-07-15 05:12:52 -04:00
Joseph Doherty 326b22a7ba fix(abcip-tests): expression-body KnownProfiles.All to fix static-init NRE
KnownProfiles.All used a { get; } = [...] initialiser declared textually
before the static readonly profile fields it references, so C# static-init
order captured an all-null list. ForFamily() then NRE'd on p.Family, faulting
AbServerFixture..ctor and blocking the whole AbCip.IntegrationTests suite.

Expression-body it so it evaluates on access, after the fields are set.
Live-verified against the controllogix fixture: suite goes 6F/1P/3skip ->
10 pass / 0 fail / 3 skip (skips = emulator-mode tests).

Integration-sweep follow-up #1 (archreview/plans/INTEGRATION-SWEEP-STATUS.md).
2026-07-15 03:28:04 -04:00
Joseph Doherty 9f62310b49 fix(historian): map Boolean historization to Int2, not degenerate Int1 (#441)
v2-ci / build (pull_request) Successful in 3m24s
v2-ci / unit-tests (pull_request) Failing after 8m19s
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.

Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.

- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
  (Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
  retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
  archreview STATUS.md (R2-06 live gate -> 6/6)

Resolves #441.
2026-07-14 11:57:27 -04:00
Joseph Doherty 4da2d62b9a fix(historian): stamp a positive StorageRateMs on provisioned tags
v2-ci / build (pull_request) Successful in 4m7s
v2-ci / unit-tests (pull_request) Failing after 10m50s
GatewayTagProvisioner built HistorianTagDefinition without a StorageRateMs, so
the proto uint32 defaulted to 0. The gateway's EnsureTags maps StorageRateMs
straight to the AVEVA SDK's storageRateMs, which requires a positive value —
0 makes the SDK throw ArgumentOutOfRangeException ('Storage rate must be > 0 ms'),
so EVERY historized-tag auto-provision (the deploy-time IHistorianProvisioning
hook) silently failed at the gateway with failed=N.

Stamp DefaultStorageRateMs=1000 (matches the gateway's own live provisioner
convention; quantized 1000/5000/10000). Add a regression unit test asserting the
definition carries a positive rate, and set StorageRateMs on the live-suite's
inline definitions so the retype probe reaches a real EnsureTags.

Found via the R2-06 live gate against a real 0.2.0 gateway + AVEVA historian:
with this fixed, Float/Int2/UInt2/Int4/UInt4/Double all provision and the live
suite goes 3/6 -> 5/6. (Int1/Boolean remains blocked by the gateway histsdk's
ProtocolEvidenceMissingException for Int1 — a gateway-side type-support gap, not
this bug.)

Gateway driver unit 103/103.
2026-07-13 19:29:49 -04:00
Joseph Doherty 56ddf45e1c chore(core): flatten Historian sub-namespace to Core.Abstractions root
v2-ci / build (pull_request) Successful in 4m32s
v2-ci / unit-tests (pull_request) Failing after 10m2s
Four continuous-historization types (IHistorizationOutbox, HistorizationOutboxEntry,
IHistorianValueWriter/HistorizationValue, HistorizationCommitMode) drifted into a
ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian sub-namespace while the other six files
in the same folder correctly use the root namespace. This violated decision #59's
flat-public-surface rule and left InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace
red on master (pre-existing, predates round-2 remediation).

Move all four to the root namespace and drop/retarget the now-redundant
'using ...Core.Abstractions.Historian;' in the ~11 consumers. No behavioral change.

Verified: Core.Abstractions.Tests 129/129 (was 128/129), Runtime.Tests Historian+DI 72/72,
Gateway driver unit 102/102, full solution build 0 errors.
2026-07-13 18:29:01 -04:00
Joseph Doherty 6ee74d26fc docs(r2-09): record execution deviations; clear CS8604 nullable warning in the 5 PollErrorHealthTests
v2-ci / build (pull_request) Successful in 3m32s
v2-ci / unit-tests (pull_request) Failing after 9m27s
2026-07-13 12:45:50 -04:00
Joseph Doherty 502a327f25 fix(twincat): probe recycle honors cancellation — no uncancellable gate wait (05/STAB-12 compounded part); B6 complete (TwinCAT suite 189 green) 2026-07-13 12:43:06 -04:00
Joseph Doherty 082ee99b6b fix(twincat): retract the fresh ADS notification when unsubscribe raced the replay (05/STAB-17) 2026-07-13 12:39:39 -04:00
Joseph Doherty de29759643 fix(twincat): retry failed replay intents on the next successful probe tick (05/STAB-16) 2026-07-13 12:37:20 -04:00
Joseph Doherty 4f16620948 test(twincat): failing repro for STAB-16 silent replay failure 2026-07-13 12:33:35 -04:00
Joseph Doherty e69d3b3b0f fix(modbus): equipment tags carry unitId; ResolveHost keys per-unit via the resolver (CONV-4) 2026-07-13 12:29:13 -04:00
Joseph Doherty 9262dd25b7 fix(drivers): ResolveHost resolves equipment-tag refs via EquipmentTagRefResolver — per-host breaker isolation (CONV-4) 2026-07-13 12:26:42 -04:00
Joseph Doherty 8a7701db3f test(drivers): failing repro for CONV-4 equipment-tag ResolveHost mis-keying 2026-07-13 12:23:43 -04:00
Joseph Doherty 86742ad8cb fix(abcip): throttle evict-recreate Forward-Open attempts per device (05/STAB-8) 2026-07-13 12:20:10 -04:00
Joseph Doherty e54af9b948 fix(focas): per-device connect-attempt backoff across the three loops (05/STAB-8) 2026-07-13 12:16:31 -04:00
Joseph Doherty 962afed241 fix(twincat): per-device connect-attempt backoff, probe-driven recovery (05/STAB-8) 2026-07-13 12:13:23 -04:00
Joseph Doherty 85ef74ea86 refactor(s7): retire the bespoke poll fork onto PollGroupEngine v2 (CONV-1; fixes PERF-3 + STAB-6-S7; deletes the fork's bare-OCE poll-death — cross-ref plan R2-01) 2026-07-13 12:07:39 -04:00
Joseph Doherty 0d879363f8 test(s7): failing guards for the poll-fork retirement (PERF-3, STAB-6) 2026-07-13 12:05:08 -04:00
Joseph Doherty b494d09d9b fix(drivers): wire PollGroupEngine onError->health + 30s poll backoff (05/STAB-9) [twincat,focas] 2026-07-13 11:59:49 -04:00
Joseph Doherty 9213c037b0 fix(drivers): wire PollGroupEngine onError->health + 30s poll backoff (05/STAB-9) [modbus,abcip,ablegacy] 2026-07-13 11:57:25 -04:00
Joseph Doherty bb8386d126 test(focas): failing repro for STAB-5 cross-thread Dictionary mutation 2026-07-13 11:46:12 -04:00
Joseph Doherty 32f248c93d test(abcip): failing repro for STAB-4 unserialized shared-handle ops 2026-07-13 11:40:17 -04:00
Joseph Doherty 141c143586 test(modbus): failing repro for STAB-3 timeout/framing stream desync 2026-07-13 11:35:20 -04:00
Joseph Doherty 1a698cbb97 Merge R2-11 TagConfig consolidation (arch-review round 2) [PR #434]
v2-ci / build (push) Successful in 3m55s
v2-ci / unit-tests (push) Failing after 8m14s
Findings 01/C-1, 01/P-1 (single TagConfigIntent.Parse in Commons, 4 copies
delegate, parse-once) + 05 CONV-2/UNDER-1/UNDER-6 (shared strict TagConfigJson
readers, per-driver Inspect() + writable key, FOCAS capability pre-flight,
Modbus probe timeoutMs, deploy-gate Warn|Error). Phase C runtime-strict flip
deferred. T22/T24 deferred. Auto-merged AddressSpaceApplier.cs + DriverHostActor.cs
with R2-04/R2-10; verified OpcUaServer.Tests 286/286 + Runtime.Tests 396/396.
Build clean.
2026-07-13 11:29:32 -04:00
Joseph Doherty b2a56ebf3b fix(driver-modbus): probe parses the factory DTO shape — timeoutMs no longer silently dropped (R2-11, 05/CONV-2) 2026-07-13 11:07:44 -04:00
Joseph Doherty 24435efa8b fix(driver-focas): equipment tags run the capability-matrix pre-flight authored tags get (R2-11, 05/UNDER-6) 2026-07-13 11:05:02 -04:00
Joseph Doherty e294416087 fix(driver-focas): equipment tags forced read-only until PMC writes exist; shared readers + Inspect (R2-11, 05/UNDER-1) 2026-07-13 11:01:55 -04:00
Joseph Doherty 4f5bbd253c feat(driver-twincat): shared TagConfigJson readers, writable key, Inspect warnings (R2-11) 2026-07-13 11:00:31 -04:00
Joseph Doherty 6e2415bb4c feat(driver-ablegacy): shared TagConfigJson readers, writable key, Inspect warnings (R2-11) 2026-07-13 10:59:27 -04:00
Joseph Doherty d374ba332b feat(driver-abcip): shared TagConfigJson readers + Inspect warnings (R2-11) 2026-07-13 10:58:17 -04:00
Joseph Doherty 1cceef406e feat(driver-s7): shared TagConfigJson readers, writable key, Inspect warnings (R2-11) 2026-07-13 10:57:01 -04:00
Joseph Doherty 64b9bba9e9 feat(driver-modbus): shared TagConfigJson readers, writable key, Inspect warnings (R2-11) 2026-07-13 10:55:41 -04:00
Joseph Doherty edbbc0438c test(r2-04): live guard - advised write remains Good under fail-closed advise (skip-gated) 2026-07-13 10:54:13 -04:00
Joseph Doherty 084c6e3cee feat(r2-04): meter unconfirmed + advise-failed Galaxy writes 2026-07-13 10:53:17 -04:00
Joseph Doherty 610306734c fix(r2-04): Galaxy fail-closed write when AdviseSupervisory fails - Bad status feeds #5 node revert (06/S-1) 2026-07-13 10:51:16 -04:00
Joseph Doherty a537d29f44 refactor(r2-04): internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) 2026-07-13 10:48:30 -04:00
Joseph Doherty fea6477f45 Merge R2-01 S7 fault-path hardening (arch-review round 2) [PR #429]
v2-ci / build (push) Successful in 3m37s
v2-ci / unit-tests (push) Failing after 6m49s
Findings 05/STAB-14 (High regression — connect-timeout OCE no longer silently
kills S7 subscriptions), STAB-15 (framing/desync + mid-PDU cancel fatal
classification), STAB-8 seam note. All prod changes in S7Driver.cs; T11 live
outage gate deferred. STATUS.md conflict resolved additively. Build clean.
2026-07-13 10:14:59 -04:00
Joseph Doherty a1b22f979d test(r2-01): env-gated live connect-timeout outage test — the STAB-14 outage class the bounce test cannot produce (task 11) 2026-07-13 10:06:08 -04:00
Joseph Doherty 8999c722a0 fix(r2-01): probe failures mark the handle dead under the gate so the next tick reopens (STAB-15b, task 9) 2026-07-13 10:03:36 -04:00
Joseph Doherty 33044061c3 fix(r2-01): cancellation observed mid-PDU marks the connection dead before propagating (STAB-15a, task 8) 2026-07-13 10:01:20 -04:00
Joseph Doherty 1aa89d754f test(r2-06): HISTGW_BOOL_SANDBOX_TAG fixture + Boolean EnsureTags leg + retype probe (skip-clean offline) (06/U-7) 2026-07-13 10:01:12 -04:00
Joseph Doherty dc2f3fa970 test(r2-06): pin 0.2.0 Boolean->Int1 explicit-presence (HasDataType) provisioning (06/U-7) 2026-07-13 09:58:07 -04:00
Joseph Doherty 8aed9ac365 fix(r2-01): RED framing-fault classification Theory T5 (STAB-15a, task 6) 2026-07-13 09:57:08 -04:00
Joseph Doherty 7813304199 fix(r2-01): RED poll-loop-survival T2 + caller-cancel pinning T4 (task 2) 2026-07-13 09:51:57 -04:00
Joseph Doherty 3a049a13a4 fix(r2-06): RED repro — misconfigured ServerHistorian:Endpoint should fail with a named error, not UriFormatException (06/S-11) 2026-07-13 09:51:31 -04:00
Joseph Doherty 9dedb7936b fix(r2-01): RED connect-timeout regression tests T1+T3 (task 1) 2026-07-13 09:50:07 -04:00
Joseph Doherty fcdcf0e2fb fix(r2-01): extend reconnect fakes with token-honouring hang + probe-fault modes (task 0) 2026-07-13 09:48:42 -04:00