Cause: the client-side PerEndpointConnectTimeout (3 s), not opc-plc. The
validate:null case is the first test the assembly runs, so it pays the
cold-process cost of the first Basic256Sha256 connect: loading the fresh
throwaway certificate, validating the server's, and the asymmetric handshake.
In the G2 full-solution run that local work used up the budget. The opc-plc
log at 13:13:50-53Z shows the discovery channel closing at 50.709, then nothing
from the client for 3 s, then OpenSecureChannel answered at 53.775 and the
client closing the channel 48 ms later. The SDK reports a wait cancelled by
that budget as BadRequestInterrupted. opc-plc auto-accepted the certificate
(--aa, "Trusting certificate ... command line option") with no reject and no
store round-trip.
- Driver: a per-endpoint budget expiry now surfaces as TimeoutException naming
PerEndpointConnectTimeout instead of BadRequestInterrupted or a bare
TaskCanceledException. There is no behaviour change. Unit test added
against a silent TCP listener.
- Test: the ServerHostNameValidationTests secure options use a 15 s budget,
the same as the browser integration tests' 10 s. Connect latency is not
what these tests check.
- docs/drivers/OpcUaClient.md: documents what the budget covers.
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes
asyncio.run(run_main(), debug=True). Debug mode captures a traceback per
scheduled callback and logs callbacks slower than 100 ms (the "took 0.146
seconds" lines in #909). Side by side on the shared docker host (closed-loop
FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s
with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it.
The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of
central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the
debug-mode capacity, so a test run on top was answered late ("Cancel send,
because not connected!") and timed out.
- run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog
thread that exits after 3 failed FC03 probes so restart: unless-stopped
replaces the container (docker has no restart-on-unhealthy).
- fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the
compose healthcheck.
- docker-compose.yml: shared anchor for every pymodbus service: new entrypoint,
`project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded
json-file logs.
- README: why, the numbers, and how to diagnose load.
ScriptRunGuard counts a run on StartedRunCount and InFlightCount together the
moment it is admitted, but releases InFlightCount only in the run's finally.
The runaway test waited on StartedRunCount >= 6 and then asserted
InFlightCount == 1, so the healthy sibling's 4th-push run - admitted but still
executing - was counted alongside the runaway orphan (2). Test race, not an
engine defect: widening only the healthy run reproduced it 5/5.
- wait until StartedRunCount == 6 AND InFlightCount == 1
- healthy predicate spins ~100 ms so the window is open on every run
- positive controls: the final push must be refused once more (the in-flight
run is the orphan) and the quality-once absence check runs only after the
healthy Activated event from the later pass has been delivered
Under 28 busy-loops on 14 cores: before 2/60 failed, after 0/60.
The phase-6-*.ps1 scripts checked v2 Phase 6 code that later commits deliberately
deleted (76310b88 v1 Server/Admin, #522, #524, LiteDB cache, v3 migration squash):
56/56 failures were retired files, so they could not pass. Owner retired G1; G2
plus the per-area suites cover the invariants. phase-7-compliance.ps1 is untouched.
Two service-env entries per node (Mode=Warm, PeerApplicationUris__0), pair stopped and
started together. Gates: 250/90, RedundancySupport Warm, standby BadOutOfService,
LocalDb sync, canary Good on the Primary; standby Write could only show BadNotWritable
(QA tags are read-only). ScadaBridge WarmCanary moved off the Standby by ServiceLevel in
0.1 s (connect-time preference); graceful Primary stop still took the 66 s connectivity
path (ScadaBridge#141). End state: Warm, vq04 Primary.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).
- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
until the service's host process (PID + start time captured before the
stop) and every process imaged under InstallRoot have exited; otherwise
throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
PowerShell 7's Move-Item falls back to an item-by-item move when the
rename fails because a file under the folder is open, and leaves a partial
PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
while it fails because something under InstallRoot is open (a backstop
for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.
Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
c569f806 passed the bare "driver" literal to WarmRoleAuthority.OldestUpMember
at OpcUaPublishActor.cs:1875-1876, which RoleAuthorityGuardTests forbids outside
RoleParser. Same value; found by the #615 G2 full-suite run. Cluster.Tests
255/255, Runtime Warm tests 23/23.
Executed 2026-11-13 12:27-12:48Z; target moved from the pack's 47f0b98c to the tip 76e73e7d
(HG Client 1.1.0, behaviour unchanged; note added to the pack).
- vd03: ~14 s downtime (was ~101 s; #905 window 5.7 s), SL 250, FOCAS 21/22, Mode Unset,
HG Probe 494 -> 536 with no write RPCs (added gate for #900).
- QA: pair outage 36.5 s (was ~91 s), 250/240, both failover legs, no failback, sync
connected, canary Good. #906: vq04 (sync server) shutdown -> Leaving 30.0 s on c9640f5c,
21 ms on 76e73e7d; peer at 250 in 0.47 s (leg 1: 0.36 s).
- Deviation: the first Move-Item of app\ failed "file in use" right after the stop on both
QA nodes; a retry with the service stopped succeeded (lmxopcua#908).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
G9 ran on an isolated 2-node pair (otopcua-g9, 003c7f89, Mode unset as
deployed) with otopcua-cli as the owner-approved client: A1-A3, A5, A7, A8
(graceful, ~11 s gap), B2 (docker kill, failover <1 s, ~2 s gap, 0 Bad,
survivor 250 at +44 s) and B3 (no failback) pass; A4/A5b unmet under Unset,
met under Warm (#717). G8 (CTT) waived by the owner for GA; G5 is the only
open exit criterion.
The run found the CLI printing every array as System.String[] (ServerArray
unreadable); ValueFormatter now renders [a, b] / hex for read, subscribe and
historyread, with tests. Runbook A4/A5b text updated for the #717 modes.
Evidence: docs/plans/evidence/2026-11-13-615-g9/RESULTS.md.
The ignitionoee CR capture (6d2aec40: 7 h read-only sample of 34 production
controls on the Java LSV2 driver) and the 2026-11-11 iTNC 530 run did not
change the driver's code: ProgramState 0 = cutting (11,865 of 11,869 motion
pairs), 7 = idle; TNC 426/430 BadNotSupported; overrides BadUserAccessDenied
under INSPECT; capabilities Unknown until first poll; alarm classes 1/7/8 seen,
class 5 not. Owner accepted the driver on that basis; remaining limits stay
documented in Heidenhain.md §Hardware-gated.
Bump ZB.MOM.WW.HistorianGateway.Client + Contracts 1.0.0 -> 1.1.0 (historiangw#135,
release 6365f34). GatewayErrorReasons now reads Client 1.1.0's typed
HistorianGatewayException.StatusCode / ErrorReason / ErrorDomain (constants from
HistorianGatewayErrorReasons) instead of parsing grpc-status-details-bin itself; the
direct Grpc.StatusProto pin and reference are dropped (the client brings it). The
published client maps every RpcException to a HistorianGatewayException, so the
trailer fallback is not needed. Behaviour unchanged: FailedPrecondition +
AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) ->
BadAggregateNotSupported, everything else keeps its handling.
Adds GatewayErrorReasonsTests (typed-property-only exceptions, so they fail against
the old trailer implementation); wire tests and docs/Historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Range, settings (no migration; OpcUa:Redundancy:Mode unset everywhere), read-only
ConfigDb/SMB/OPC UA preflight on both tiers, #905 scoped swap with -NoStart on both
tiers, #906 failover timing gate (stop the sync-server node as Primary), Unset
metadata gate, rollback with the vd03 #903 drift note. Not executed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi