Cause: the client-side PerEndpointConnectTimeout (3 s), not opc-plc. The
validate:null case is the first test the assembly runs, so it pays the
cold-process cost of the first Basic256Sha256 connect: loading the fresh
throwaway certificate, validating the server's, and the asymmetric handshake.
In the G2 full-solution run that local work used up the budget. The opc-plc
log at 13:13:50-53Z shows the discovery channel closing at 50.709, then nothing
from the client for 3 s, then OpenSecureChannel answered at 53.775 and the
client closing the channel 48 ms later. The SDK reports a wait cancelled by
that budget as BadRequestInterrupted. opc-plc auto-accepted the certificate
(--aa, "Trusting certificate ... command line option") with no reject and no
store round-trip.
- Driver: a per-endpoint budget expiry now surfaces as TimeoutException naming
PerEndpointConnectTimeout instead of BadRequestInterrupted or a bare
TaskCanceledException. There is no behaviour change. Unit test added
against a silent TCP listener.
- Test: the ServerHostNameValidationTests secure options use a 15 s budget,
the same as the browser integration tests' 10 s. Connect latency is not
what these tests check.
- docs/drivers/OpcUaClient.md: documents what the budget covers.
c569f806 passed the bare "driver" literal to WarmRoleAuthority.OldestUpMember
at OpcUaPublishActor.cs:1875-1876, which RoleAuthorityGuardTests forbids outside
RoleParser. Same value; found by the #615 G2 full-suite run. Cluster.Tests
255/255, Runtime Warm tests 23/23.
G9 ran on an isolated 2-node pair (otopcua-g9, 003c7f89, Mode unset as
deployed) with otopcua-cli as the owner-approved client: A1-A3, A5, A7, A8
(graceful, ~11 s gap), B2 (docker kill, failover <1 s, ~2 s gap, 0 Bad,
survivor 250 at +44 s) and B3 (no failback) pass; A4/A5b unmet under Unset,
met under Warm (#717). G8 (CTT) waived by the owner for GA; G5 is the only
open exit criterion.
The run found the CLI printing every array as System.String[] (ServerArray
unreadable); ValueFormatter now renders [a, b] / hex for read, subscribe and
historyread, with tests. Runbook A4/A5b text updated for the #717 modes.
Evidence: docs/plans/evidence/2026-11-13-615-g9/RESULTS.md.
Bump ZB.MOM.WW.HistorianGateway.Client + Contracts 1.0.0 -> 1.1.0 (historiangw#135,
release 6365f34). GatewayErrorReasons now reads Client 1.1.0's typed
HistorianGatewayException.StatusCode / ErrorReason / ErrorDomain (constants from
HistorianGatewayErrorReasons) instead of parsing grpc-status-details-bin itself; the
direct Grpc.StatusProto pin and reference are dropped (the client brings it). The
published client maps every RpcException to a HistorianGatewayException, so the
trailer fallback is not needed. Behaviour unchanged: FailedPrecondition +
AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) ->
BadAggregateNotSupported, everything else keeps its handling.
Adds GatewayErrorReasonsTests (typed-property-only exceptions, so they fail against
the old trailer implementation); wire tests and docs/Historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Two defects on the crash-failover path, found by the T13 live gate on an isolated docker pair:
1. When auto-down removed the unreachable Primary (~15 s after the survivor promoted), the
snapshot's Primary left the unreachable set, so (3') failed and the survivor demoted to
Standby (BadOutOfService, ServiceLevel 50) for ~15 s, until its own singleton published
after the hand-over retries. A Primary that is Down, or that this node saw removed, now
counts as gone (WarmRoleAuthority.SnapshotPrimaryGone). A member that is merely
Leaving/Exiting, or that this node never saw, is not.
2. The verdict read Cluster.State / SelfMember (Akka's ClusterReadView, a separate subscriber
to the same events), which can lag the event being handled. Live, the survivor stayed
Standby 4.3 s after its own UnreachableMember event (promotion waited for the next health
tick). The Warm verdict now uses a membership view built from this actor's own
member/reachability subscription, falling back to the read view only before the first event.
Tests: WarmRoleAuthorityTests.A_snapshot_Primary_that_auto_down_removed_is_still_gone_and_the_survivor_stays_Active,
OpcUaPublishActorWarmTests.The_crash_survivor_promotes_on_its_own_unreachable_event_and_stays_Active_through_the_removal
(synthetic events the read view never sees). Mutations: read-view-only verdict and the removed
arm switched off each fail the actor test. Runtime.Tests 1058 passed, 3 skipped.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A planned stop of the Primary took ~30 s on QA: WebApplication stops the web
server before Akka.Hosting's service, and Kestrel's graceful drain waited the
whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's
LocalDbSyncService only observes the call token, which fires at the END of the
drain). Only then did CoordinatedShutdown run, so the stopping node held the
Primary role and the peer sat at ServiceLevel 240 for the whole window.
- GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs
before any hosted service's StopAsync; it raises ApplicationStopping, then
runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave,
whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster
is left while Kestrel is still up.
- ShutdownStreamCancellationInterceptor (innermost on the shared gRPC
pipeline): every streaming call gets a context whose token is linked with
ApplicationStopping; streams arriving once stopping are refused Unavailable.
Host-side mitigation until LocalDb 0.3.8 is published and pinned
(scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams.
Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c +
MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain
budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive
control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain
began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order
assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the
original context -> stop 8006 ms, fails.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A picker-form native alarm (attributeRef tag.Attr) only learns the provider-qualified reference
Galaxy!<Area>.tag.Attr from the alarm feed. After a host restart the condition can be restored from
LocalDb before the feed's first snapshot reached the new driver, and an ack in that window dialled the
bare tag.Attr, which the gateway's AcknowledgeAlarm refuses.
The feed now exposes InitialSnapshotCompleted (set on snapshot_complete, after every snapshot entry has
been raised). AcknowledgeAsync resolves the reference through ResolveAlarmAckReferenceAsync: when a
picker-form tag has learned nothing yet and the feed is running, it waits for that snapshot (2 s, and the
caller's token) and dials the reference the snapshot names, so the area always comes from the feed and a
re-areaed object acks under its current area. No reference is guessed: if the snapshot does not name the
alarm, or does not arrive in time, the authored reference is dialled as before and refused visibly.
Chosen over the alternatives in the issue: a GR-browse area lookup can be stale against the runtime
area and is not loaded for authored tags, and a gateway-side tag.Attr resolution needs a cross-repo
deploy and would resolve from the same monitor cache the snapshot already carries.
Tests: 5 driver cases + 1 feed case (fail with the call site / sentinel reverted), and a skip-gated live
gate GalaxyPickerAckBeforeSnapshotLiveTests, run against windev's TEST Galaxy (TestMachine_015.TestAlarm003):
an ack issued before the snapshot dials Galaxy!TestArea.TestMachine_015.TestAlarm003 and is confirmed; with
the call site reverted it fails with the gateway's InvalidRequest refusal.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The #815 warn-only compatibility window is over. A running Galaxy driver
(InitializeAsync, every gateway ReopenAsync, the ack fallback password) and a
running OpcUaClient driver (session open) now refuse an out-of-namespace
credential reference with the same CredentialReferenceException that save,
deploy (CredentialReferenceOutOfScope), Test Connect and browse already raise,
before any store / environment / file read and before any endpoint is dialled.
The one existing scope check (CredentialReferences.Parse/ResolveAsync) is now
the only path; nothing new was added.
Removed:
- LegacyCredentialReferenceTracker (Core.Abstractions), its rate-limited WARN
and the credential-compat:<field> diagnostics entry.
- The per-driver Legacy* resolve methods and every `legacy` parameter
(GalaxySecretRef, GalaxyAckCredentialResolver, OpcUaUserIdentityFactory,
OpcUaUserCertificatePaths, OpcUaClientSecretResolution).
- IDriverDiagnostics on GalaxyDriver / OpcUaClientDriver (it only carried the
tracker's entries).
- The tracker also kept a pre-#836 absolute UserCertificatePath loading on a
running OpcUaClient driver; that is refused now too.
Preflight / banner decision: KEPT, as a hard error. The Running source
(driver-health credential-compat entries) and CredentialReferenceBannerModel
are gone with the tracker, but the ConfigDb scan still tells an operator
something no other surface does: a Deployed finding is now a driver that is
not connecting, a Draft finding is a deploy the gate will refuse - named
before anyone deploys, e.g. after restoring an old ConfigDb backup or
importing a pre-#815 config. It also carries the #835 cross-instance and
#836/#850 path checks. CredentialReferencePreflightService now logs one Error
per finding (was Warning) with no warn-only wording, and the /raw banner is
alert-danger.
DEV and QA ConfigDbs hold no Galaxy/OpcUaClient driver (checked 2026-11-13);
no production install exists.
Tests: GalaxyDriverLegacyCredentialTests replaced by
GalaxyDriverCredentialScopeTests; new OpcUaClientDriverCredentialScopeTests;
tracker tests in OpcUaClientSecretResolutionTests /
OpcUaClientUserCertificatePathTests deleted; AdminUI preflight tests drop the
Running/banner-model cases and assert Error level. Against the old code the
new refusal tests fail (Galaxy 5/6, OpcUaClient 4/5; the two positive controls
pass on both).
docs/security.md: the migration table now says running drivers refuse; the
warn-only section is replaced by "The compatibility window is over".
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Owner decision 2026-11-13 on #872 item 1. StatusCodeMap.FromMxStatus now maps a
Pending or Warning row to Uncertain (0x40000000) for any detail, ahead of the
detail vocabulary, keeping the once-per-(detail, category) warning. MXAccess sets
success = -1 on Pending (accepted, in progress); neither category is a failure.
Detail 8001 (automation-object ConfigurationError, write rejected for type
conversion, live-captured on windev 2026-11-13) maps to BadTypeMismatch. 8018
stays BadUserAccessDenied via its SecurityError category, now pinned by a test.
Data change: the value is published Uncertain. Write reply: the result is
Uncertain, never Good, so the server (which commits only Good) reports the write
not committed and reverts the node; a later MXAccess commit arrives as a data
change. Host status is unchanged (probe decode treats Uncertain and Bad alike,
quality < 192 -> Unknown); no alarm path calls the map.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Root cause: DeploymentDriftProbe (a singleton shared by every circuit) cached its verdict until
the ConfigEdit audit watermark (MAX(EditedAtUtc), COUNT(*)) moved, assuming every configuration
write appends a ConfigEdit row. Only deployments and driver restart/reconnect do; the AdminUI
editors (RawTreeService, UnsTreeService, ScriptEdit, cluster/node/ACL pages) write the config
tables directly. So after a draft change the page kept showing the cached "In sync" until
something unrelated moved the watermark.
- DeploymentDriftProbe: the watermark cache is gone; every verdict comes from a composition that
started after the caller asked. Compositions stay serialised behind the gate, and callers queued
behind one share a verdict only when that composition started after they arrived (start
sequence number) for the same sealed hash, so sharing is exact, never stale.
- Deployments page: the drift check runs in OnAfterRenderAsync (interactive render only, so a
visit costs one composition, not two with prerender); the first render shows a neutral
"Checking for changes..." chip, never a verdict; a failed check shows "Drift check failed" and
logs a warning instead of crashing the page. A deploy re-runs the check.
- Tests: probe tests for an un-audited draft change (real ConfigComposer over InMemory), fresh
composition per call, and queued callers not reusing an older composition; bUnit page tests that
a second page load after a direct draft change shows "Configuration drift" and that the first
render shows "Checking" until the comparison answers. All fail against the previous code.
DeploymentDriftProbeSqlServerTests is removed: it pinned the SQL translation of the watermark
query, which no longer exists.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Closing a tab with the script editor open ended the circuit; MonacoEditor.DisposeAsync then
called MonacoBlazor.dispose through SafeInvokeAsync, which caught only JSDisconnectedException,
so the TaskCanceledException from the torn-down circuit escaped and was logged at [ERR].
DisposeAsync now treats JSDisconnectedException, OperationCanceledException (incl.
TaskCanceledException) and ObjectDisposedException as the expected gone-circuit outcome and logs
them at Debug; a real JSException from a live page is still logged (Warning); the
DotNetObjectReference is released in a finally. MonacoEditor is the only component in the repo
that uses JS interop, so there is nothing else to align.
bUnit tests: dispose throwing each of the three gone-circuit exceptions neither escapes nor logs
above Debug (with a positive control that the failing call ran), and a JSException still warns.
They fail against the previous DisposeAsync.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi