Commit Graph

2977 Commits

Author SHA1 Message Date
Joseph Doherty 678498a4a6 fix(opcuaclient): split the per-endpoint connect budget into reachability + session phases (#912)
PerEndpointConnectTimeout (3 s) now bounds only TCP + discovery, so failover past a
dead endpoint stays fast. The new SessionCreateTimeout (10 s) bounds OpenSecureChannel
+ CreateSession on an endpoint that answered discovery, where the client-side cert and
crypto work lives (#910). 2 x (3 + 10) s stays inside the host's 30 s initialise bound.
Expiries name the budget that ran out; both budgets must be positive; AdminUI form
carries the new field so a save does not drop it; driver docs updated.

Closes #912
2026-11-15 16:55:22 -05:00
Joseph Doherty c8edd65f3e Keep driver address builders inline within their owning dialog 2026-11-14 02:14:13 -05:00
Joseph Doherty cb26ed9179 Keep cloned collection edits inside their owning driver dialog 2026-11-14 02:10:43 -05:00
Joseph Doherty 03ace0f9ee fix(admin-ui): close measured cluster field and narrow layout gaps 2026-11-14 02:00:20 -05:00
Joseph Doherty 455c874875 fix(ui): name role grant fields and wrap security details 2026-11-14 01:55:00 -05:00
Joseph Doherty ef1609f523 Observe serialized modal activation through its actual JS invocation 2026-11-14 01:46:16 -05:00
Joseph Doherty 50ab406de7 fix(admin-ui): bind equipment callers and recovery to presentations 2026-11-14 01:44:09 -05:00
Joseph Doherty 6756ffab85 fix(admin-ui): associate alert actions and guard requests 2026-11-14 01:40:29 -05:00
Joseph Doherty e411f89f6b Align destructive-action source guards with captured page confirmations 2026-11-14 01:33:27 -05:00
Joseph Doherty fc4053f0c9 Harden Hosts driver-controls dialog lifecycle and reflow 2026-11-14 01:18:22 -05:00
Joseph Doherty aa919ceef2 fix(admin-ui): isolate UNS callers and recover confirmed-write refreshes 2026-11-14 01:14:15 -05:00
Joseph Doherty 7bd013a57c Bind script editing and deletion to captured route intent 2026-11-14 01:11:35 -05:00
Joseph Doherty 8929f5f32e fix(admin-ui): bind Raw callers and lazy reads to their presentations 2026-11-14 01:00:12 -05:00
Joseph Doherty 3ae92f4d6e Guard driver operations and inline restart confirmation 2026-11-14 00:47:11 -05:00
Joseph Doherty f953d230f8 fix(admin-ui): harden read-only script source dialog lifecycle 2026-11-14 00:39:03 -05:00
Joseph Doherty ea5bb6a0bc Guard and name Raw manual entry and browsing workflows 2026-11-14 00:19:09 -05:00
Joseph Doherty a0aa4f023c fix(admin-ui): scope UNS modal lifecycles and script identity 2026-11-14 00:14:30 -05:00
Joseph Doherty c9f191d5fa Isolate staged CSV imports in named managed dialogs 2026-11-13 23:54:11 -05:00
Joseph Doherty 6bb5a31337 feat(admin-ui): pause live-tail snapshots while retaining bounded updates 2026-11-13 23:40:55 -05:00
Joseph Doherty 50942b4a75 Retire pending Monaco creation before editor registration 2026-11-13 23:36:55 -05:00
Joseph Doherty c6fe9c46e0 fix(admin-ui): recover root loads and clarify loaded-tree filtering 2026-11-13 23:32:11 -05:00
Joseph Doherty 7c888d1dca Preserve Fleet and Hosts readings through refresh failures 2026-11-13 23:30:42 -05:00
Joseph Doherty b4f7672d2a Synchronize shared Monaco read-only state without replacing drafts 2026-11-13 23:29:35 -05:00
Joseph Doherty 79acb80da9 fix(admin-ui): scope configuration modal lifecycles and validation 2026-11-13 23:24:56 -05:00
Joseph Doherty f7bc69e9c3 fix(admin-ui): distinguish deployment acceptance and application progress 2026-11-13 23:22:22 -05:00
Joseph Doherty 43a3512baf Guard shared OT browse and probe lifecycle entry points 2026-11-13 23:20:20 -05:00
Joseph Doherty c5a288b080 Adapt Raw hierarchy density and menu placement to touch layouts 2026-11-13 23:09:26 -05:00
Joseph Doherty 78bad3a6c9 Expose complete diagnostic text through native disclosures 2026-11-13 22:14:57 -05:00
Joseph Doherty c086f97757 fix(admin-ui): harden simple modal field and pending lifecycles 2026-11-13 22:07:54 -05:00
Joseph Doherty b990ebe3ea Complete common picker and tree field accessibility 2026-11-13 21:52:00 -05:00
Joseph Doherty 25e78ebe92 fix(admin-ui): gate cluster and ACL actions by destination policies 2026-11-13 21:47:16 -05:00
Joseph Doherty 1fe4dbf51c Standardize cluster anatomy and gate authoring destinations 2026-11-13 21:43:30 -05:00
Joseph Doherty cb3400c13c fix(admin-ui): align task orientation and navigation with page policies 2026-11-13 21:42:30 -05:00
Joseph Doherty 28add408c4 Name common identity resilience and static builder fields 2026-11-13 21:38:19 -05:00
Joseph Doherty 27bca683c1 Complete protocol field associations and verify rendered inventory 2026-11-13 21:29:03 -05:00
Joseph Doherty 169149f1eb Improve admin field names and generation-safe dialog lifecycle 2026-11-13 21:15:58 -05:00
Joseph Doherty 8cc053ceb7 fix(admin-ui): pass contextual names to native Monaco editor 2026-11-13 21:04:30 -05:00
Joseph Doherty a47ffb9151 test(admin): replace bUnit preview with stable 2.7.2 2026-11-13 15:23:32 -05:00
Joseph Doherty 79371402e4 Merge branches 'worktree-agent-a41f08fe289ebe7a8', 'worktree-agent-a21d19bd14da138e7' and 'worktree-agent-a8d85b6d5ef424006'
v2-ci / unit-tests (push) Failing after 1h32m12s
2026-11-13 12:43:14 -05:00
Joseph Doherty 73132b7d78 fix(opcuaclient): #910 name the per-endpoint connect budget; widen it in the #816 host-name tests
Cause: the client-side PerEndpointConnectTimeout (3 s), not opc-plc. The
validate:null case is the first test the assembly runs, so it pays the
cold-process cost of the first Basic256Sha256 connect: loading the fresh
throwaway certificate, validating the server's, and the asymmetric handshake.
In the G2 full-solution run that local work used up the budget. The opc-plc
log at 13:13:50-53Z shows the discovery channel closing at 50.709, then nothing
from the client for 3 s, then OpenSecureChannel answered at 53.775 and the
client closing the channel 48 ms later. The SDK reports a wait cancelled by
that budget as BadRequestInterrupted. opc-plc auto-accepted the certificate
(--aa, "Trusting certificate ... command line option") with no reject and no
store round-trip.

- Driver: a per-endpoint budget expiry now surfaces as TimeoutException naming
  PerEndpointConnectTimeout instead of BadRequestInterrupted or a bare
  TaskCanceledException. There is no behaviour change. Unit test added
  against a silent TCP listener.
- Test: the ServerHostNameValidationTests secure options use a 15 s budget,
  the same as the browser integration tests' 10 s. Connect latency is not
  what these tests check.
- docs/drivers/OpcUaClient.md: documents what the budget covers.
2026-11-13 12:42:45 -05:00
Joseph Doherty fc08892eb3 fix(modbus-fixture): run pymodbus sim without asyncio debug mode; FC03 healthcheck + watchdog (#909)
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes
asyncio.run(run_main(), debug=True). Debug mode captures a traceback per
scheduled callback and logs callbacks slower than 100 ms (the "took 0.146
seconds" lines in #909). Side by side on the shared docker host (closed-loop
FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s
with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it.
The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of
central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the
debug-mode capacity, so a test run on top was answered late ("Cancel send,
because not connected!") and timed out.

- run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog
  thread that exits after 3 failed FC03 probes so restart: unless-stopped
  replaces the container (docker has no restart-on-unhealthy).
- fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the
  compose healthcheck.
- docker-compose.yml: shared anchor for every pymodbus service: new entrypoint,
  `project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded
  json-file logs.
- README: why, the numbers, and how to diagnose load.
2026-11-13 11:03:08 -05:00
Joseph Doherty f1da809f3c test(scripted-alarms): wait for the sibling run to finish before asserting InFlightCount (#911)
ScriptRunGuard counts a run on StartedRunCount and InFlightCount together the
moment it is admitted, but releases InFlightCount only in the run's finally.
The runaway test waited on StartedRunCount >= 6 and then asserted
InFlightCount == 1, so the healthy sibling's 4th-push run - admitted but still
executing - was counted alongside the runaway orphan (2). Test race, not an
engine defect: widening only the healthy run reproduced it 5/5.

- wait until StartedRunCount == 6 AND InFlightCount == 1
- healthy predicate spins ~100 ms so the window is open on every run
- positive controls: the final push must be refused once more (the in-flight
  run is the orphan) and the quality-once absence check runs only after the
  healthy Activated event from the later pass has been delivered

Under 28 busy-loops on 14 cores: before 2/60 failed, after 0/60.
2026-11-13 10:46:19 -05:00
Joseph Doherty c9bc358dc9 feat(cli): print array values; docs: #615 G9 PASS with otopcua-cli, G8 waived
v2-ci / unit-tests (push) Has been cancelled
G9 ran on an isolated 2-node pair (otopcua-g9, 003c7f89, Mode unset as
deployed) with otopcua-cli as the owner-approved client: A1-A3, A5, A7, A8
(graceful, ~11 s gap), B2 (docker kill, failover <1 s, ~2 s gap, 0 Bad,
survivor 250 at +44 s) and B3 (no failback) pass; A4/A5b unmet under Unset,
met under Warm (#717). G8 (CTT) waived by the owner for GA; G5 is the only
open exit criterion.

The run found the CLI printing every array as System.String[] (ServerArray
unreadable); ValueFormatter now renders [a, b] / hex for read, subscribe and
historyread, with tests. Runbook A4/A5b text updated for the #717 modes.
Evidence: docs/plans/evidence/2026-11-13-615-g9/RESULTS.md.
2026-11-13 07:46:56 -05:00
Joseph Doherty 76e73e7dbf chore(historian): HistorianGateway Client/Contracts 1.1.0, typed error reason (#900)
v2-ci / unit-tests (push) Has been cancelled
Bump ZB.MOM.WW.HistorianGateway.Client + Contracts 1.0.0 -> 1.1.0 (historiangw#135,
release 6365f34). GatewayErrorReasons now reads Client 1.1.0's typed
HistorianGatewayException.StatusCode / ErrorReason / ErrorDomain (constants from
HistorianGatewayErrorReasons) instead of parsing grpc-status-details-bin itself; the
direct Grpc.StatusProto pin and reference are dropped (the client brings it). The
published client maps every RpcException to a HistorianGatewayException, so the
trailer fallback is not needed. Behaviour unchanged: FailedPrecondition +
AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) ->
BadAggregateNotSupported, everything else keeps its handling.

Adds GatewayErrorReasonsTests (typed-property-only exceptions, so they fail against
the old trailer implementation); wire tests and docs/Historian.md updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:17:10 -05:00
Joseph Doherty 47f0b98c6e chore(deps): pin ZB.MOM.WW.LocalDb 0.3.8 (scadaproj#57)
v2-ci / unit-tests (push) Has been cancelled
LocalDb 0.3.8 ends open sync streams on ApplicationStopping and refuses
new ones Unavailable while stopping (lmxopcua#906, library side). The
host-side ShutdownStreamCancellationInterceptor stays: it also covers
the mesh telemetry/command streams.

GracefulStopWithOpenSyncStreamTests' positive control started passing
the stop in ~950 ms under 0.3.8, because the library alone now ends
the stream. The control registers LocalDbSyncService without the host
lifetime (the 0.3.7 behaviour), so it still reproduces the 8 s drain.

Release build 0 warnings. Host.Tests 288/288, Host.IntegrationTests
350 passed / 2 skipped, Runtime.Tests 1063 passed / 3 skipped,
Core.AlarmHistorian.Tests 53/53.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 06:55:27 -05:00
Joseph Doherty c569f80694 fix(717): Warm crash survivor stays Active through auto-down; verdict reads its own events
Two defects on the crash-failover path, found by the T13 live gate on an isolated docker pair:

1. When auto-down removed the unreachable Primary (~15 s after the survivor promoted), the
   snapshot's Primary left the unreachable set, so (3') failed and the survivor demoted to
   Standby (BadOutOfService, ServiceLevel 50) for ~15 s, until its own singleton published
   after the hand-over retries. A Primary that is Down, or that this node saw removed, now
   counts as gone (WarmRoleAuthority.SnapshotPrimaryGone). A member that is merely
   Leaving/Exiting, or that this node never saw, is not.

2. The verdict read Cluster.State / SelfMember (Akka's ClusterReadView, a separate subscriber
   to the same events), which can lag the event being handled. Live, the survivor stayed
   Standby 4.3 s after its own UnreachableMember event (promotion waited for the next health
   tick). The Warm verdict now uses a membership view built from this actor's own
   member/reachability subscription, falling back to the read view only before the first event.

Tests: WarmRoleAuthorityTests.A_snapshot_Primary_that_auto_down_removed_is_still_gone_and_the_survivor_stays_Active,
OpcUaPublishActorWarmTests.The_crash_survivor_promotes_on_its_own_unreachable_event_and_stays_Active_through_the_removal
(synthetic events the read view never sees). Mutations: read-view-only verdict and the removed
arm switched off each fail the actor test. Runtime.Tests 1058 passed, 3 skipped.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 04:36:58 -05:00
Joseph Doherty 687256d68e test(717): deferred Warm coverage - 2-node handover, DataEncoding, deleted/reused items, clone matrix, ACL stand-down, per-counter metrics
v2-ci / unit-tests (push) Has been cancelled
- T3 WarmSingletonHandoverTests: real 2-node cluster, singleton under ClusterSingletonManager on both nodes; the old
  incarnation's held-back publish (older revision, higher sequence) reaches a real Warm OpcUaPublishActor after the
  handover and never demotes the new Primary. Pin: a removed node cannot rejoin from its old ActorSystem; a new one
  rejoins younger (higher up-number).
- T6 DataEncoding: Binary/XML encoded reads of a structured tag projected on the Standby; scalar encoding error kept;
  a DefaultXml item is projected, then promoted through its own encoding (TypeId i=888).
- T7 item deleted between queue and publish; ClientHandle reused by i=2267 on a payload recycled through the SDK 2.0.1
  pool (precondition asserted); WarmStandbyCloneMatrixTests over DataChange+DiagnosticInfos/EventNotificationList/
  StatusChange, clone/cached immutability, encode round-trip, real-pool stale marker.
- T8 ACL-stood-down item receives nothing from demotion or promotion; restored grant reaches it again.
- T10 per-counter MeterListener tests (by instrument instance): warm_transition, warm_role gauge, snapshot_rejected
  (publish actor both reasons + DriverHostActor's ordering gate, first actor test of it), projected_notifications,
  suppressed_alarm_events. Process-wide state tests run in non-parallel collections.
- Every test mutation-checked (26 mutations killed); no product bug found. Plan status updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 04:31:11 -05:00
Joseph Doherty f817c77e02 test(717): deferred-event boundary, ACL-filtered promotion refresh, native-ack refusal, effective-URI XML load; docs + T13 overlay
v2-ci / unit-tests (push) Has been cancelled
- Wire: an event reported while Active but deferred (#838) is delivered after demotion; the promotion refresh
  withholds an ACL-denied condition (positive control alongside); ConditionRefresh2 and model-change events on the
  Standby; a native Acknowledge is refused and never routed; Warm validates and advertises the EFFECTIVE
  ApplicationUri from an ApplicationConfigPath XML load.
- docs/Redundancy.md: Warm section, the Unset/None i=11314 behaviour (orphan readable by NodeId under Unset,
  deleted under None), instruments; OpcUaServer.md, Configuration.md, security.md, cluster-topology rule.
- docker-dev/docker-compose.warm.yml for the T13 gate (central pair, Mode=Warm).
- Plan: implementation status and the test gaps left for T13/follow-up.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:39:50 -05:00
Joseph Doherty 7829b937d1 feat(717): Warm non-transparent redundancy, default-off (T1-T10)
OpcUa:Redundancy:Mode (Unset default = today's server exactly; None; Warm; Hot and unknown values fail start).
- T1 options + validator + effective-URI check + boot banner; Warm needs the driver role.
- T2 metadata in OnNodeManagerStartedAsync: Warm -> NonTransparentRedundancyType, read-only
  ServerUriArray [self, peers], RedundancySupport=Warm, boot ServiceLevel 1; None -> RedundancySupport=None
  and the SDK's orphan i=11314 deleted; Unset untouched.
- T3 RedundancyStateChanged.Revision (up-number, incarnation tick, sequence); subscribers drop older
  revisions under Warm (OpcUaPublishActor, DriverHostActor); unrevisioned never makes a node Active.
- T4 WarmRoleAuthority + ComputeWarm bands (Standby 90/50, Active 250/200/100), PromoteWhilePrimaryUnreachable.
- T5 IStandbyProjection + DeferredStandbyProjection; OpcUaPublishActor runs promotion/demotion in the spec
  4.7 order (Active level last and retried; Standby level first, data degraded regardless).
- T6 EquipmentVariableState read projection (BadOutOfService/null); Warm Standby refuses writes BadInvalidState.
- T7 D4 mechanism adopted after T0: ordinary-tag marker on MonitoredItemNotification.Message validated by
  (ClientHandle, DataValue); first-delivery decision keyed by (SubscriptionId, SequenceNumber); the SDK's
  cached copy is never mutated.
- T8 queue current/BadOutOfService to live ordinary-tag items ignoring filters.
- T9 alarm stamp/deliver split (suppressed on Standby), Part 9 methods refused on Standby, promotion
  refresh via ISubscription.ConditionRefresh2Async on every live event item.
- T10 deploy-time RedundancyMode cross-check + six instruments.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:39:50 -05:00
Joseph Doherty dcff5ae994 fix(host): leave the cluster before Kestrel drains and end open streams on stop (#906)
v2-ci / unit-tests (push) Waiting to run
A planned stop of the Primary took ~30 s on QA: WebApplication stops the web
server before Akka.Hosting's service, and Kestrel's graceful drain waited the
whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's
LocalDbSyncService only observes the call token, which fires at the END of the
drain). Only then did CoordinatedShutdown run, so the stopping node held the
Primary role and the peer sat at ServiceLevel 240 for the whole window.

- GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs
  before any hosted service's StopAsync; it raises ApplicationStopping, then
  runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave,
  whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster
  is left while Kestrel is still up.
- ShutdownStreamCancellationInterceptor (innermost on the shared gRPC
  pipeline): every streaming call gets a context whose token is linked with
  ApplicationStopping; streams arriving once stopping are refused Unavailable.
  Host-side mitigation until LocalDb 0.3.8 is published and pinned
  (scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams.

Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c +
MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain
budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive
control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain
began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order
assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the
original context -> stop 8006 ms, fails.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:32:13 -05:00