7 Commits

Author	SHA1	Message	Date
Joseph Doherty	5a7a08deca	fix(drivers): detect mid-run comms loss and recover from it (#527) ... DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely: publishing Bad for every tag, never re-initialising, and NOT recovering when the device returned. PublishHealthSnapshot read GetHealth() for reporting only and never acted on a driver reporting itself unhealthy. Both comms-loss quality paths -- #477's native-condition annotation and #481's scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were correctly wired to a signal that never fired. The Connected health tick now evaluates a watchdog and self-sends DisconnectObserved when the driver reports a non-Healthy state AND its LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s). The conjunction is the design, and the issue's own proposal -- "non-Healthy for N consecutive polls" -- is wrong. Degraded means *connected but some data is bad*; several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device, and a state-only watchdog would reconnect that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is the honest evidence: preserved across Degraded/Faulted writes, advanced by any success. The state check remains the second conjunct so an idle Healthy driver with nothing subscribed is not declared dead for not reading. Verified across all 12 drivers that a sustained failure does reach a non-Healthy state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip, which is correct: it has no transport. Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync. ModbusDriver (_transport) and MqttDriver (_connection) overwrite an IAsyncDisposable field inside InitializeAsync without disposing the previous one, so re-initialising without a teardown leaks a socket per attempt -- latent while only a manual Force Reconnect reached this path, but the watchdog makes it automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry). Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence assertion carrying a positive control (a proven count of processed health-poll ticks). Falsified five ways -- dropping either conjunct reds exactly the test written for it, and removing the teardown, the null baseline, or the watchdog call each reds its own guard. Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice: - fixture stopped -> watchdog fires at ~108s, "declaring comms lost" - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)" (#481 Layer 4) and the #477 native-condition annotation on a driver owning conditions - fixture restarted -> reconnected in ~8s with NO node restart; both the raw (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo	2026-07-31 10:58:38 -04:00
Joseph Doherty	b815446940	refactor(drivers): delete GetMemoryFootprint / FlushOptionalCachesAsync (#525) ... Both `IDriver` members had no production call site. Their only consumer was `MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and `GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which had not been true since v3, even before #522 removed the poller. Owner decision on #525 was (a) delete rather than (b) build the process-wide watchdog that would consume them. The driver-side implementations are in git history and were mostly reusable; what was always missing is the trigger, and a member nobody calls is how the misleading doc-comment survived this long. Removed from `IDriver`, all 12 driver implementations, and every test stub — plus the support code that existed only to feed them, which is the part a member-only deletion would have left behind as dead weight: - MTConnect's three per-entry byte-weight constants, its cached `ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run on every cache fill to populate a number with no remaining reader. - OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it. - MQTT's `ApproxBytesPerAuthoredTag`. Two behavioural notes, neither a change in what the software does: - `AbCipTemplateCache` now has no production reset path. Its only caller was AB CIP's flush, which itself had no caller — so nothing changes except how visible the fact is. Recorded on the type, along with the correction that its comment's claim about a device `ReinitializeAsync` also flushing it was never true. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt. - MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an Agent restart. That is now `internal` rather than `private` so the tests which used the flush purely to empty the cache drive the real path instead — they now cover a case that actually occurs in production. Several unrelated doc comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are repointed to it. Tests: the ones that existed only to assert on the removed members are deleted (3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a footprint of 0 before the flush and 0 after with no discovery in between, so it would have passed against a flush that did nothing at all. Full suite green except 7 failures verified identical on master before and after (4 AbLegacy integration with the compose profile down, 3 Host.IntegrationTests), checked by name rather than assumed. Docs corrected where they described this as live: `DriverLifecycle.md` (which carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and `v2/driver-stability.md` — whose "polls this every 30 s" + per-driver `DriverConfig` memory budget were never built as written and are now banner-ed as design intent rather than description. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo	2026-07-31 09:07:11 -04:00
Joseph Doherty	2d20aeb779	fix(opcua,drivers): carry driver status sub-codes to clients (#519) + health-write ordering (#517) ... #519 — a driver's OPC UA status code now reaches a client with its sub-code intact. DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication, BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in logs only. Resolved by option (b): the raw uint rides AttributeValuePublished -> AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad). The surface was wider than the issue costed (41 files, not "the sink and 3 implementations"): OpcUaQuality had non-driver producers with no uint to carry. Each got its natural code — BadInternalError for a failed virtual-tag script, BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the engine already held. AddressSpaceApplier's removal writes deliberately keep the generic Bad: not losing a code the driver had is this issue; inventing one for the deploy path is a separate call. Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change rather than a severity-bucket change. While the annotation was 3-state those were the same predicate; now a move between two Bad reasons changes what a subscriber reads. New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so it can only see a named constant, never an inline literal. #517 — the symptom is real but the issue named the wrong path and its fix was unsafe. ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth, has no production call site.) InitializeAsync can: it runs on a fire-and-forget Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a catch which disposes the client before writing Faulted. Moving the write earlier, as suggested, inverts the terminal state — teardown drains in-flight loops that write health as they unwind, so a late loop could leave Healthy at rest with every connection closed. The fix is a bracket (conservative before, terminal after), applied to MTConnect's init-failure path only; shutdown keeps its single trailing write, and the rule plus the reachability table are documented on IDriver.ShutdownAsync. CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing — Healthy before initialize, after shutdown and after dispose. Now derived from _running. Its test, named Health_is_always_Connected, asserted the defect and is replaced. Every new guard was falsified by breaking the production code and watching it go red. The two sub-code tests each assert two different Bad codes, since a single-code assertion could pass against a hard-coded constant under the old projection. Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in Host.IntegrationTests. Closes #517 Closes #519 Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo	2026-07-31 05:25:27 -04:00
Joseph Doherty	cbb882293b	refactor(drivers): delete the inert driver-tier recycle machinery (#522) ... The tier system was documented, operator-authorable, and inert. Deleted rather than activated, because its premise is gone rather than merely unused. "Tier C" meant a driver running out-of-process behind an IDriverSupervisor that could restart its Host without tearing down the OPC UA session. No such process exists anywhere: Galaxy reaches MXAccess over gRPC to the external mxaccessgw sidecar (PR 7.2 retired the in-process Galaxy.Host/Proxy/Shared projects) and FOCAS has run in-process since its managed wire client landed 2026-04-24. Consistently, IDriverSupervisor had ZERO implementations and there was nothing for one to implement against. The issue understated the inertness. It says the Tier-C-only protections never engaged, implying the Tier A/B parts did. They did not: nothing constructs MemoryTracking, MemoryRecycle or ScheduledRecycleScheduler outside their own unit tests, so the whole Core/Stability recycle layer was dead — meaning option (a), "pass real tiers", was never a flag flip. It would have meant writing the wiring that never existed AND arming it. Deleted: MemoryTracking, MemoryRecycle, ScheduledRecycleScheduler, IDriverSupervisor, the vestigial DriverTypeRegistry (referenced only by its own tests), and the RecycleIntervalSeconds knob — which the AdminUI let an operator author and the parser validated while it configured nothing. DriverTier itself SURVIVES and is load-bearing: DriverResilienceOptions.GetTierDefaults supplies the real per-capability timeout/retry/breaker policies via DriverFactoryRegistry.GetTier and DriverCapabilityInvokerFactory. Only the isolation-and-recycle layer above it is gone. Deliberately NOT deleted: - WedgeDetector came along in the same directory and is equally dead in production, but it is tier-agnostic and is not recycle machinery — it only shares the folder. Restored rather than swept up in a decision that was not about it. - IDriver.GetMemoryFootprint() and FlushOptionalCachesAsync() lose their only consumer here. Removing them touches all 12 drivers and every test stub, so they are documented as consumerless and filed as #525 instead of buried in this diff. Compatibility: a deployed ResilienceConfig blob still carrying "recycleIntervalSeconds" parses cleanly (unknown keys are ignored — guarded by a new test, because a blob that suddenly failed to parse would fall back to tier defaults and silently discard the operator's real overrides), and the AdminUI's preserve-unknown-keys bag keeps the key rather than rewriting stored config on an unrelated edit. The 01/U-6 knob-inertness guard carried an explicit carve-out admitting RecycleIntervalSeconds was dormant and out of scope; that carve-out is now gone, so the expected set is literally what the test's name claims. Note: Host.IntegrationTests has 2 failures (DriverProbeRegistrationTests.is_idempotent, PrimaryGateFailoverTests) — verified pre-existing by reproducing both on clean master dc9d947b. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo	2026-07-30 04:44:04 -04:00
Joseph Doherty	2124f21ab6	docs(historian-gateway): document gateway backend, config keys, EnsureTags hook, known gates; retire Wonderware from docs ... HistorianGateway is now the sole historian backend (read + alarm SendEvent + continuous WriteLiveValues). Document the final state and retire the Wonderware sidecar from the docs/config/labels: - CLAUDE.md: rewrite the Historian section — ServerHistorian / ContinuousHistorization / AlarmHistorian config keys, the IHistorianProvisioning EnsureTags hook, the GatewayAlarmHistorianWriter SendEvent path + ReadEvents dependency on gateway RuntimeDb:EventReadsEnabled=true, gateway-side prerequisites (RuntimeDb flags + historian:read/write/tags:write scopes), migration note, and two KNOWN-LIMITATION callouts (live-validation gate + empty historized-ref-set recorder follow-on). - appsettings.json: fix the stale ServerHistorian block (Host/Port/SharedSecret/ ServerCertThumbprint -> Endpoint/ApiKey/UseTls/AllowUntrustedServerCertificate/ CaCertificatePath/CallTimeout, keep MaxTieClusterOverfetch); add a disabled ContinuousHistorization block; prune the orphaned Wonderware keys from AlarmHistorian (keep the SQLite knobs). ApiKey env-supplied via ServerHistorian__ApiKey (commented; valid strict JSON via _comment keys). - README.md + docs (Historian.md, AlarmHistorian.md, Configuration.md, ServiceHosting.md, DriverLifecycle.md, drivers/README.md, Uns.md, VirtualTags.md, AlarmTracking.md, Client.UI.md, README.md, TestConnectProbes.md): retire the Wonderware historian backend from current-backend descriptions; fix the stale ServerHistorian/AlarmHistorian config tables (now gateway shape); convert drivers/Historian.Wonderware.md to a retired stub pointing at the gateway. - Source/UI labels (descriptive text only, no behavior change): OtOpcUaServerHostedService.cs, HistoryPaging.cs, OtOpcUaSdkServer.cs, HistorianAdapterActor.cs, VirtualTagModal.razor, ScriptedAlarmModal.razor, AlarmsHistorian.razor now name the HistorianGateway backend. Build clean (0 errors); AdminUI.Tests green (514 passed). Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii	2026-06-26 19:46:27 -04:00
Joseph Doherty	1be06502c7	fix(historian): correct AlarmHistorian config-key refs in docs + install (review)	2026-06-12 12:25:13 -04:00
Joseph Doherty	33912694fd	docs(audit): G1 completeness — driver-lifecycle + alarm-historian reference pages	2026-06-03 15:55:28 -04:00