Cause: the client-side PerEndpointConnectTimeout (3 s), not opc-plc. The
validate:null case is the first test the assembly runs, so it pays the
cold-process cost of the first Basic256Sha256 connect: loading the fresh
throwaway certificate, validating the server's, and the asymmetric handshake.
In the G2 full-solution run that local work used up the budget. The opc-plc
log at 13:13:50-53Z shows the discovery channel closing at 50.709, then nothing
from the client for 3 s, then OpenSecureChannel answered at 53.775 and the
client closing the channel 48 ms later. The SDK reports a wait cancelled by
that budget as BadRequestInterrupted. opc-plc auto-accepted the certificate
(--aa, "Trusting certificate ... command line option") with no reject and no
store round-trip.
- Driver: a per-endpoint budget expiry now surfaces as TimeoutException naming
PerEndpointConnectTimeout instead of BadRequestInterrupted or a bare
TaskCanceledException. There is no behaviour change. Unit test added
against a silent TCP listener.
- Test: the ServerHostNameValidationTests secure options use a 15 s budget,
the same as the browser integration tests' 10 s. Connect latency is not
what these tests check.
- docs/drivers/OpcUaClient.md: documents what the budget covers.
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes
asyncio.run(run_main(), debug=True). Debug mode captures a traceback per
scheduled callback and logs callbacks slower than 100 ms (the "took 0.146
seconds" lines in #909). Side by side on the shared docker host (closed-loop
FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s
with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it.
The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of
central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the
debug-mode capacity, so a test run on top was answered late ("Cancel send,
because not connected!") and timed out.
- run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog
thread that exits after 3 failed FC03 probes so restart: unless-stopped
replaces the container (docker has no restart-on-unhealthy).
- fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the
compose healthcheck.
- docker-compose.yml: shared anchor for every pymodbus service: new entrypoint,
`project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded
json-file logs.
- README: why, the numbers, and how to diagnose load.
ScriptRunGuard counts a run on StartedRunCount and InFlightCount together the
moment it is admitted, but releases InFlightCount only in the run's finally.
The runaway test waited on StartedRunCount >= 6 and then asserted
InFlightCount == 1, so the healthy sibling's 4th-push run - admitted but still
executing - was counted alongside the runaway orphan (2). Test race, not an
engine defect: widening only the healthy run reproduced it 5/5.
- wait until StartedRunCount == 6 AND InFlightCount == 1
- healthy predicate spins ~100 ms so the window is open on every run
- positive controls: the final push must be refused once more (the in-flight
run is the orphan) and the quality-once absence check runs only after the
healthy Activated event from the later pass has been delivered
Under 28 busy-loops on 14 cores: before 2/60 failed, after 0/60.
The phase-6-*.ps1 scripts checked v2 Phase 6 code that later commits deliberately
deleted (76310b88 v1 Server/Admin, #522, #524, LiteDB cache, v3 migration squash):
56/56 failures were retired files, so they could not pass. Owner retired G1; G2
plus the per-area suites cover the invariants. phase-7-compliance.ps1 is untouched.
Two service-env entries per node (Mode=Warm, PeerApplicationUris__0), pair stopped and
started together. Gates: 250/90, RedundancySupport Warm, standby BadOutOfService,
LocalDb sync, canary Good on the Primary; standby Write could only show BadNotWritable
(QA tags are read-only). ScadaBridge WarmCanary moved off the Standby by ServiceLevel in
0.1 s (connect-time preference); graceful Primary stop still took the 66 s connectivity
path (ScadaBridge#141). End state: Warm, vq04 Primary.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).
- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
until the service's host process (PID + start time captured before the
stop) and every process imaged under InstallRoot have exited; otherwise
throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
PowerShell 7's Move-Item falls back to an item-by-item move when the
rename fails because a file under the folder is open, and leaves a partial
PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
while it fails because something under InstallRoot is open (a backstop
for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.
Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
c569f806 passed the bare "driver" literal to WarmRoleAuthority.OldestUpMember
at OpcUaPublishActor.cs:1875-1876, which RoleAuthorityGuardTests forbids outside
RoleParser. Same value; found by the #615 G2 full-suite run. Cluster.Tests
255/255, Runtime Warm tests 23/23.
Executed 2026-11-13 12:27-12:48Z; target moved from the pack's 47f0b98c to the tip 76e73e7d
(HG Client 1.1.0, behaviour unchanged; note added to the pack).
- vd03: ~14 s downtime (was ~101 s; #905 window 5.7 s), SL 250, FOCAS 21/22, Mode Unset,
HG Probe 494 -> 536 with no write RPCs (added gate for #900).
- QA: pair outage 36.5 s (was ~91 s), 250/240, both failover legs, no failback, sync
connected, canary Good. #906: vq04 (sync server) shutdown -> Leaving 30.0 s on c9640f5c,
21 ms on 76e73e7d; peer at 250 in 0.47 s (leg 1: 0.36 s).
- Deviation: the first Move-Item of app\ failed "file in use" right after the stop on both
QA nodes; a retry with the service stopped succeeded (lmxopcua#908).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
G9 ran on an isolated 2-node pair (otopcua-g9, 003c7f89, Mode unset as
deployed) with otopcua-cli as the owner-approved client: A1-A3, A5, A7, A8
(graceful, ~11 s gap), B2 (docker kill, failover <1 s, ~2 s gap, 0 Bad,
survivor 250 at +44 s) and B3 (no failback) pass; A4/A5b unmet under Unset,
met under Warm (#717). G8 (CTT) waived by the owner for GA; G5 is the only
open exit criterion.
The run found the CLI printing every array as System.String[] (ServerArray
unreadable); ValueFormatter now renders [a, b] / hex for read, subscribe and
historyread, with tests. Runbook A4/A5b text updated for the #717 modes.
Evidence: docs/plans/evidence/2026-11-13-615-g9/RESULTS.md.
The ignitionoee CR capture (6d2aec40: 7 h read-only sample of 34 production
controls on the Java LSV2 driver) and the 2026-11-11 iTNC 530 run did not
change the driver's code: ProgramState 0 = cutting (11,865 of 11,869 motion
pairs), 7 = idle; TNC 426/430 BadNotSupported; overrides BadUserAccessDenied
under INSPECT; capabilities Unknown until first poll; alarm classes 1/7/8 seen,
class 5 not. Owner accepted the driver on that basis; remaining limits stay
documented in Heidenhain.md §Hardware-gated.
Bump ZB.MOM.WW.HistorianGateway.Client + Contracts 1.0.0 -> 1.1.0 (historiangw#135,
release 6365f34). GatewayErrorReasons now reads Client 1.1.0's typed
HistorianGatewayException.StatusCode / ErrorReason / ErrorDomain (constants from
HistorianGatewayErrorReasons) instead of parsing grpc-status-details-bin itself; the
direct Grpc.StatusProto pin and reference are dropped (the client brings it). The
published client maps every RpcException to a HistorianGatewayException, so the
trailer fallback is not needed. Behaviour unchanged: FailedPrecondition +
AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) ->
BadAggregateNotSupported, everything else keeps its handling.
Adds GatewayErrorReasonsTests (typed-property-only exceptions, so they fail against
the old trailer implementation); wire tests and docs/Historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Range, settings (no migration; OpcUa:Redundancy:Mode unset everywhere), read-only
ConfigDb/SMB/OPC UA preflight on both tiers, #905 scoped swap with -NoStart on both
tiers, #906 failover timing gate (stop the sync-server node as Primary), Unset
metadata gate, rollback with the vd03 #903 drift note. Not executed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
LocalDb 0.3.8 ends open sync streams on ApplicationStopping and refuses
new ones Unavailable while stopping (lmxopcua#906, library side). The
host-side ShutdownStreamCancellationInterceptor stays: it also covers
the mesh telemetry/command streams.
GracefulStopWithOpenSyncStreamTests' positive control started passing
the stop in ~950 ms under 0.3.8, because the library alone now ends
the stream. The control registers LocalDbSyncService without the host
lifetime (the 0.3.7 behaviour), so it still reproduces the 8 s drain.
Release build 0 warnings. Host.Tests 288/288, Host.IntegrationTests
350 passed / 2 skipped, Runtime.Tests 1063 passed / 3 skipped,
Core.AlarmHistorian.Tests 53/53.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
T13b: the isolated otopcua-t13b pair with OpcUa:Redundancy:Mode absent, on origin/master f826af8e
(it contains c569f806) and A/B on c569f806^ (687256d6). Every check passes on both builds:
- levels 250/240;
- Good data on both nodes;
- writes, Acknowledge, AddComment and alarm delivery on the Secondary;
- graceful stop: the survivor reaches 250 259 ms after Leaving (pre-fix 270 ms);
- crash: 250 at +43.6 s (pre-fix +42.8 s), with no demotion at auto-down;
- rejoin at 240 with no failback.
The role-decision changes are gated on IsWarm and do not run under Unset.
Plan L5, the design's section 4.3 and the Warm table in Redundancy.md now say a Leaving/Exiting node
reports ServiceLevel 0, not 90 (ComputeWarm returns 0 for any member that is not Up/Joining; T13 note A).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Every applicable T13 leg passes on 0c19f26c + c569f806. The leg list covers:
- metadata and ServiceLevel bands;
- standby projection;
- alarm suppression with Standby ConditionRefresh and the promotion refresh;
- the three D4 demotion timings, run live;
- standby write and Part 9 refusal (BadInvalidState);
- graceful and crash failover, with no failback;
- subscription transfer;
- the Mode=Unset regression.
The two crash-path defects the gate found are fixed in c569f806. Evidence, rig files and the
raw-client harness are in docs/plans/evidence/2026-11-13-717-t13/.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Two defects on the crash-failover path, found by the T13 live gate on an isolated docker pair:
1. When auto-down removed the unreachable Primary (~15 s after the survivor promoted), the
snapshot's Primary left the unreachable set, so (3') failed and the survivor demoted to
Standby (BadOutOfService, ServiceLevel 50) for ~15 s, until its own singleton published
after the hand-over retries. A Primary that is Down, or that this node saw removed, now
counts as gone (WarmRoleAuthority.SnapshotPrimaryGone). A member that is merely
Leaving/Exiting, or that this node never saw, is not.
2. The verdict read Cluster.State / SelfMember (Akka's ClusterReadView, a separate subscriber
to the same events), which can lag the event being handled. Live, the survivor stayed
Standby 4.3 s after its own UnreachableMember event (promotion waited for the next health
tick). The Warm verdict now uses a membership view built from this actor's own
member/reachability subscription, falling back to the read view only before the first event.
Tests: WarmRoleAuthorityTests.A_snapshot_Primary_that_auto_down_removed_is_still_gone_and_the_survivor_stays_Active,
OpcUaPublishActorWarmTests.The_crash_survivor_promotes_on_its_own_unreachable_event_and_stays_Active_through_the_removal
(synthetic events the read view never sees). Mutations: read-view-only verdict and the removed
arm switched off each fail the actor test. Runtime.Tests 1058 passed, 3 skipped.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- T3 WarmSingletonHandoverTests: real 2-node cluster, singleton under ClusterSingletonManager on both nodes; the old
incarnation's held-back publish (older revision, higher sequence) reaches a real Warm OpcUaPublishActor after the
handover and never demotes the new Primary. Pin: a removed node cannot rejoin from its old ActorSystem; a new one
rejoins younger (higher up-number).
- T6 DataEncoding: Binary/XML encoded reads of a structured tag projected on the Standby; scalar encoding error kept;
a DefaultXml item is projected, then promoted through its own encoding (TypeId i=888).
- T7 item deleted between queue and publish; ClientHandle reused by i=2267 on a payload recycled through the SDK 2.0.1
pool (precondition asserted); WarmStandbyCloneMatrixTests over DataChange+DiagnosticInfos/EventNotificationList/
StatusChange, clone/cached immutability, encode round-trip, real-pool stale marker.
- T8 ACL-stood-down item receives nothing from demotion or promotion; restored grant reaches it again.
- T10 per-counter MeterListener tests (by instrument instance): warm_transition, warm_role gauge, snapshot_rejected
(publish actor both reasons + DriverHostActor's ordering gate, first actor test of it), projected_notifications,
suppressed_alarm_events. Process-wide state tests run in non-parallel collections.
- Every test mutation-checked (26 mutations killed); no product bug found. Plan status updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and
-Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again)
with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about
3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read,
not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s.
- Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only
GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed.
- Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only).
- Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply,
-Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest,
including the firewall rule (its query costs 3-5 s; its exe path does not change).
The two partition the plan. Prints the scan time.
- Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a
failure stops before anything moves), -Scope InstallRoot inside the window; preflight
checks the candidate carries the worker exe; prints the stopped window and its parts.
-NoStart, -NoCarry and the pre-#901 refusal unchanged.
windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1),
101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving.
Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- Wire: an event reported while Active but deferred (#838) is delivered after demotion; the promotion refresh
withholds an ACL-denied condition (positive control alongside); ConditionRefresh2 and model-change events on the
Standby; a native Acknowledge is refused and never routed; Warm validates and advertises the EFFECTIVE
ApplicationUri from an ApplicationConfigPath XML load.
- docs/Redundancy.md: Warm section, the Unset/None i=11314 behaviour (orphan readable by NodeId under Unset,
deleted under None), instruments; OpcUaServer.md, Configuration.md, security.md, cluster-topology rule.
- docker-dev/docker-compose.warm.yml for the T13 gate (central pair, Mode=Warm).
- Plan: implementation status and the test gaps left for T13/follow-up.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi