PerEndpointConnectTimeout (3 s) now bounds only TCP + discovery, so failover past a
dead endpoint stays fast. The new SessionCreateTimeout (10 s) bounds OpenSecureChannel
+ CreateSession on an endpoint that answered discovery, where the client-side cert and
crypto work lives (#910). 2 x (3 + 10) s stays inside the host's 30 s initialise bound.
Expiries name the budget that ran out; both budgets must be positive; AdminUI form
carries the new field so a save does not drop it; driver docs updated.
Closes#912
Cause: the client-side PerEndpointConnectTimeout (3 s), not opc-plc. The
validate:null case is the first test the assembly runs, so it pays the
cold-process cost of the first Basic256Sha256 connect: loading the fresh
throwaway certificate, validating the server's, and the asymmetric handshake.
In the G2 full-solution run that local work used up the budget. The opc-plc
log at 13:13:50-53Z shows the discovery channel closing at 50.709, then nothing
from the client for 3 s, then OpenSecureChannel answered at 53.775 and the
client closing the channel 48 ms later. The SDK reports a wait cancelled by
that budget as BadRequestInterrupted. opc-plc auto-accepted the certificate
(--aa, "Trusting certificate ... command line option") with no reject and no
store round-trip.
- Driver: a per-endpoint budget expiry now surfaces as TimeoutException naming
PerEndpointConnectTimeout instead of BadRequestInterrupted or a bare
TaskCanceledException. There is no behaviour change. Unit test added
against a silent TCP listener.
- Test: the ServerHostNameValidationTests secure options use a 15 s budget,
the same as the browser integration tests' 10 s. Connect latency is not
what these tests check.
- docs/drivers/OpcUaClient.md: documents what the budget covers.
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes
asyncio.run(run_main(), debug=True). Debug mode captures a traceback per
scheduled callback and logs callbacks slower than 100 ms (the "took 0.146
seconds" lines in #909). Side by side on the shared docker host (closed-loop
FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s
with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it.
The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of
central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the
debug-mode capacity, so a test run on top was answered late ("Cancel send,
because not connected!") and timed out.
- run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog
thread that exits after 3 failed FC03 probes so restart: unless-stopped
replaces the container (docker has no restart-on-unhealthy).
- fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the
compose healthcheck.
- docker-compose.yml: shared anchor for every pymodbus service: new entrypoint,
`project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded
json-file logs.
- README: why, the numbers, and how to diagnose load.
ScriptRunGuard counts a run on StartedRunCount and InFlightCount together the
moment it is admitted, but releases InFlightCount only in the run's finally.
The runaway test waited on StartedRunCount >= 6 and then asserted
InFlightCount == 1, so the healthy sibling's 4th-push run - admitted but still
executing - was counted alongside the runaway orphan (2). Test race, not an
engine defect: widening only the healthy run reproduced it 5/5.
- wait until StartedRunCount == 6 AND InFlightCount == 1
- healthy predicate spins ~100 ms so the window is open on every run
- positive controls: the final push must be refused once more (the in-flight
run is the orphan) and the quality-once absence check runs only after the
healthy Activated event from the later pass has been delivered
Under 28 busy-loops on 14 cores: before 2/60 failed, after 0/60.
G9 ran on an isolated 2-node pair (otopcua-g9, 003c7f89, Mode unset as
deployed) with otopcua-cli as the owner-approved client: A1-A3, A5, A7, A8
(graceful, ~11 s gap), B2 (docker kill, failover <1 s, ~2 s gap, 0 Bad,
survivor 250 at +44 s) and B3 (no failback) pass; A4/A5b unmet under Unset,
met under Warm (#717). G8 (CTT) waived by the owner for GA; G5 is the only
open exit criterion.
The run found the CLI printing every array as System.String[] (ServerArray
unreadable); ValueFormatter now renders [a, b] / hex for read, subscribe and
historyread, with tests. Runbook A4/A5b text updated for the #717 modes.
Evidence: docs/plans/evidence/2026-11-13-615-g9/RESULTS.md.
Bump ZB.MOM.WW.HistorianGateway.Client + Contracts 1.0.0 -> 1.1.0 (historiangw#135,
release 6365f34). GatewayErrorReasons now reads Client 1.1.0's typed
HistorianGatewayException.StatusCode / ErrorReason / ErrorDomain (constants from
HistorianGatewayErrorReasons) instead of parsing grpc-status-details-bin itself; the
direct Grpc.StatusProto pin and reference are dropped (the client brings it). The
published client maps every RpcException to a HistorianGatewayException, so the
trailer fallback is not needed. Behaviour unchanged: FailedPrecondition +
AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) ->
BadAggregateNotSupported, everything else keeps its handling.
Adds GatewayErrorReasonsTests (typed-property-only exceptions, so they fail against
the old trailer implementation); wire tests and docs/Historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
LocalDb 0.3.8 ends open sync streams on ApplicationStopping and refuses
new ones Unavailable while stopping (lmxopcua#906, library side). The
host-side ShutdownStreamCancellationInterceptor stays: it also covers
the mesh telemetry/command streams.
GracefulStopWithOpenSyncStreamTests' positive control started passing
the stop in ~950 ms under 0.3.8, because the library alone now ends
the stream. The control registers LocalDbSyncService without the host
lifetime (the 0.3.7 behaviour), so it still reproduces the 8 s drain.
Release build 0 warnings. Host.Tests 288/288, Host.IntegrationTests
350 passed / 2 skipped, Runtime.Tests 1063 passed / 3 skipped,
Core.AlarmHistorian.Tests 53/53.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Two defects on the crash-failover path, found by the T13 live gate on an isolated docker pair:
1. When auto-down removed the unreachable Primary (~15 s after the survivor promoted), the
snapshot's Primary left the unreachable set, so (3') failed and the survivor demoted to
Standby (BadOutOfService, ServiceLevel 50) for ~15 s, until its own singleton published
after the hand-over retries. A Primary that is Down, or that this node saw removed, now
counts as gone (WarmRoleAuthority.SnapshotPrimaryGone). A member that is merely
Leaving/Exiting, or that this node never saw, is not.
2. The verdict read Cluster.State / SelfMember (Akka's ClusterReadView, a separate subscriber
to the same events), which can lag the event being handled. Live, the survivor stayed
Standby 4.3 s after its own UnreachableMember event (promotion waited for the next health
tick). The Warm verdict now uses a membership view built from this actor's own
member/reachability subscription, falling back to the read view only before the first event.
Tests: WarmRoleAuthorityTests.A_snapshot_Primary_that_auto_down_removed_is_still_gone_and_the_survivor_stays_Active,
OpcUaPublishActorWarmTests.The_crash_survivor_promotes_on_its_own_unreachable_event_and_stays_Active_through_the_removal
(synthetic events the read view never sees). Mutations: read-view-only verdict and the removed
arm switched off each fail the actor test. Runtime.Tests 1058 passed, 3 skipped.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- T3 WarmSingletonHandoverTests: real 2-node cluster, singleton under ClusterSingletonManager on both nodes; the old
incarnation's held-back publish (older revision, higher sequence) reaches a real Warm OpcUaPublishActor after the
handover and never demotes the new Primary. Pin: a removed node cannot rejoin from its old ActorSystem; a new one
rejoins younger (higher up-number).
- T6 DataEncoding: Binary/XML encoded reads of a structured tag projected on the Standby; scalar encoding error kept;
a DefaultXml item is projected, then promoted through its own encoding (TypeId i=888).
- T7 item deleted between queue and publish; ClientHandle reused by i=2267 on a payload recycled through the SDK 2.0.1
pool (precondition asserted); WarmStandbyCloneMatrixTests over DataChange+DiagnosticInfos/EventNotificationList/
StatusChange, clone/cached immutability, encode round-trip, real-pool stale marker.
- T8 ACL-stood-down item receives nothing from demotion or promotion; restored grant reaches it again.
- T10 per-counter MeterListener tests (by instrument instance): warm_transition, warm_role gauge, snapshot_rejected
(publish actor both reasons + DriverHostActor's ordering gate, first actor test of it), projected_notifications,
suppressed_alarm_events. Process-wide state tests run in non-parallel collections.
- Every test mutation-checked (26 mutations killed); no product bug found. Plan status updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- Wire: an event reported while Active but deferred (#838) is delivered after demotion; the promotion refresh
withholds an ACL-denied condition (positive control alongside); ConditionRefresh2 and model-change events on the
Standby; a native Acknowledge is refused and never routed; Warm validates and advertises the EFFECTIVE
ApplicationUri from an ApplicationConfigPath XML load.
- docs/Redundancy.md: Warm section, the Unset/None i=11314 behaviour (orphan readable by NodeId under Unset,
deleted under None), instruments; OpcUaServer.md, Configuration.md, security.md, cluster-topology rule.
- docker-dev/docker-compose.warm.yml for the T13 gate (central pair, Mode=Warm).
- Plan: implementation status and the test gaps left for T13/follow-up.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A planned stop of the Primary took ~30 s on QA: WebApplication stops the web
server before Akka.Hosting's service, and Kestrel's graceful drain waited the
whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's
LocalDbSyncService only observes the call token, which fires at the END of the
drain). Only then did CoordinatedShutdown run, so the stopping node held the
Primary role and the peer sat at ServiceLevel 240 for the whole window.
- GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs
before any hosted service's StopAsync; it raises ApplicationStopping, then
runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave,
whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster
is left while Kestrel is still up.
- ShutdownStreamCancellationInterceptor (innermost on the shared gRPC
pipeline): every streaming call gets a context whose token is linked with
ApplicationStopping; streams arriving once stopping are refused Unavailable.
Host-side mitigation until LocalDb 0.3.8 is published and pinned
(scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams.
Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c +
MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain
budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive
control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain
began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order
assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the
original context -> stop 8006 ms, fails.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi