Bump ZB.MOM.WW.HistorianGateway.Client + Contracts 1.0.0 -> 1.1.0 (historiangw#135,
release 6365f34). GatewayErrorReasons now reads Client 1.1.0's typed
HistorianGatewayException.StatusCode / ErrorReason / ErrorDomain (constants from
HistorianGatewayErrorReasons) instead of parsing grpc-status-details-bin itself; the
direct Grpc.StatusProto pin and reference are dropped (the client brings it). The
published client maps every RpcException to a HistorianGatewayException, so the
trailer fallback is not needed. Behaviour unchanged: FailedPrecondition +
AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) ->
BadAggregateNotSupported, everything else keeps its handling.
Adds GatewayErrorReasonsTests (typed-property-only exceptions, so they fail against
the old trailer implementation); wire tests and docs/Historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
LocalDb 0.3.8 ends open sync streams on ApplicationStopping and refuses
new ones Unavailable while stopping (lmxopcua#906, library side). The
host-side ShutdownStreamCancellationInterceptor stays: it also covers
the mesh telemetry/command streams.
GracefulStopWithOpenSyncStreamTests' positive control started passing
the stop in ~950 ms under 0.3.8, because the library alone now ends
the stream. The control registers LocalDbSyncService without the host
lifetime (the 0.3.7 behaviour), so it still reproduces the 8 s drain.
Release build 0 warnings. Host.Tests 288/288, Host.IntegrationTests
350 passed / 2 skipped, Runtime.Tests 1063 passed / 3 skipped,
Core.AlarmHistorian.Tests 53/53.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Two defects on the crash-failover path, found by the T13 live gate on an isolated docker pair:
1. When auto-down removed the unreachable Primary (~15 s after the survivor promoted), the
snapshot's Primary left the unreachable set, so (3') failed and the survivor demoted to
Standby (BadOutOfService, ServiceLevel 50) for ~15 s, until its own singleton published
after the hand-over retries. A Primary that is Down, or that this node saw removed, now
counts as gone (WarmRoleAuthority.SnapshotPrimaryGone). A member that is merely
Leaving/Exiting, or that this node never saw, is not.
2. The verdict read Cluster.State / SelfMember (Akka's ClusterReadView, a separate subscriber
to the same events), which can lag the event being handled. Live, the survivor stayed
Standby 4.3 s after its own UnreachableMember event (promotion waited for the next health
tick). The Warm verdict now uses a membership view built from this actor's own
member/reachability subscription, falling back to the read view only before the first event.
Tests: WarmRoleAuthorityTests.A_snapshot_Primary_that_auto_down_removed_is_still_gone_and_the_survivor_stays_Active,
OpcUaPublishActorWarmTests.The_crash_survivor_promotes_on_its_own_unreachable_event_and_stays_Active_through_the_removal
(synthetic events the read view never sees). Mutations: read-view-only verdict and the removed
arm switched off each fail the actor test. Runtime.Tests 1058 passed, 3 skipped.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- T3 WarmSingletonHandoverTests: real 2-node cluster, singleton under ClusterSingletonManager on both nodes; the old
incarnation's held-back publish (older revision, higher sequence) reaches a real Warm OpcUaPublishActor after the
handover and never demotes the new Primary. Pin: a removed node cannot rejoin from its old ActorSystem; a new one
rejoins younger (higher up-number).
- T6 DataEncoding: Binary/XML encoded reads of a structured tag projected on the Standby; scalar encoding error kept;
a DefaultXml item is projected, then promoted through its own encoding (TypeId i=888).
- T7 item deleted between queue and publish; ClientHandle reused by i=2267 on a payload recycled through the SDK 2.0.1
pool (precondition asserted); WarmStandbyCloneMatrixTests over DataChange+DiagnosticInfos/EventNotificationList/
StatusChange, clone/cached immutability, encode round-trip, real-pool stale marker.
- T8 ACL-stood-down item receives nothing from demotion or promotion; restored grant reaches it again.
- T10 per-counter MeterListener tests (by instrument instance): warm_transition, warm_role gauge, snapshot_rejected
(publish actor both reasons + DriverHostActor's ordering gate, first actor test of it), projected_notifications,
suppressed_alarm_events. Process-wide state tests run in non-parallel collections.
- Every test mutation-checked (26 mutations killed); no product bug found. Plan status updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- Wire: an event reported while Active but deferred (#838) is delivered after demotion; the promotion refresh
withholds an ACL-denied condition (positive control alongside); ConditionRefresh2 and model-change events on the
Standby; a native Acknowledge is refused and never routed; Warm validates and advertises the EFFECTIVE
ApplicationUri from an ApplicationConfigPath XML load.
- docs/Redundancy.md: Warm section, the Unset/None i=11314 behaviour (orphan readable by NodeId under Unset,
deleted under None), instruments; OpcUaServer.md, Configuration.md, security.md, cluster-topology rule.
- docker-dev/docker-compose.warm.yml for the T13 gate (central pair, Mode=Warm).
- Plan: implementation status and the test gaps left for T13/follow-up.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A planned stop of the Primary took ~30 s on QA: WebApplication stops the web
server before Akka.Hosting's service, and Kestrel's graceful drain waited the
whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's
LocalDbSyncService only observes the call token, which fires at the END of the
drain). Only then did CoordinatedShutdown run, so the stopping node held the
Primary role and the peer sat at ServiceLevel 240 for the whole window.
- GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs
before any hosted service's StopAsync; it raises ApplicationStopping, then
runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave,
whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster
is left while Kestrel is still up.
- ShutdownStreamCancellationInterceptor (innermost on the shared gRPC
pipeline): every streaming call gets a context whose token is linked with
ApplicationStopping; streams arriving once stopping are refused Unavailable.
Host-side mitigation until LocalDb 0.3.8 is published and pinned
(scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams.
Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c +
MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain
budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive
control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain
began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order
assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the
original context -> stop 8006 ms, fails.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The #815 warn-only compatibility window is over. A running Galaxy driver
(InitializeAsync, every gateway ReopenAsync, the ack fallback password) and a
running OpcUaClient driver (session open) now refuse an out-of-namespace
credential reference with the same CredentialReferenceException that save,
deploy (CredentialReferenceOutOfScope), Test Connect and browse already raise,
before any store / environment / file read and before any endpoint is dialled.
The one existing scope check (CredentialReferences.Parse/ResolveAsync) is now
the only path; nothing new was added.
Removed:
- LegacyCredentialReferenceTracker (Core.Abstractions), its rate-limited WARN
and the credential-compat:<field> diagnostics entry.
- The per-driver Legacy* resolve methods and every `legacy` parameter
(GalaxySecretRef, GalaxyAckCredentialResolver, OpcUaUserIdentityFactory,
OpcUaUserCertificatePaths, OpcUaClientSecretResolution).
- IDriverDiagnostics on GalaxyDriver / OpcUaClientDriver (it only carried the
tracker's entries).
- The tracker also kept a pre-#836 absolute UserCertificatePath loading on a
running OpcUaClient driver; that is refused now too.
Preflight / banner decision: KEPT, as a hard error. The Running source
(driver-health credential-compat entries) and CredentialReferenceBannerModel
are gone with the tracker, but the ConfigDb scan still tells an operator
something no other surface does: a Deployed finding is now a driver that is
not connecting, a Draft finding is a deploy the gate will refuse - named
before anyone deploys, e.g. after restoring an old ConfigDb backup or
importing a pre-#815 config. It also carries the #835 cross-instance and
#836/#850 path checks. CredentialReferencePreflightService now logs one Error
per finding (was Warning) with no warn-only wording, and the /raw banner is
alert-danger.
DEV and QA ConfigDbs hold no Galaxy/OpcUaClient driver (checked 2026-11-13);
no production install exists.
Tests: GalaxyDriverLegacyCredentialTests replaced by
GalaxyDriverCredentialScopeTests; new OpcUaClientDriverCredentialScopeTests;
tracker tests in OpcUaClientSecretResolutionTests /
OpcUaClientUserCertificatePathTests deleted; AdminUI preflight tests drop the
Running/banner-model cases and assert Error level. Against the old code the
new refusal tests fail (Galaxy 5/6, OpcUaClient 4/5; the two positive controls
pass on both).
docs/security.md: the migration table now says running drivers refuse; the
warn-only section is replaced by "The compatibility window is over".
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Root cause: DeploymentDriftProbe (a singleton shared by every circuit) cached its verdict until
the ConfigEdit audit watermark (MAX(EditedAtUtc), COUNT(*)) moved, assuming every configuration
write appends a ConfigEdit row. Only deployments and driver restart/reconnect do; the AdminUI
editors (RawTreeService, UnsTreeService, ScriptEdit, cluster/node/ACL pages) write the config
tables directly. So after a draft change the page kept showing the cached "In sync" until
something unrelated moved the watermark.
- DeploymentDriftProbe: the watermark cache is gone; every verdict comes from a composition that
started after the caller asked. Compositions stay serialised behind the gate, and callers queued
behind one share a verdict only when that composition started after they arrived (start
sequence number) for the same sealed hash, so sharing is exact, never stale.
- Deployments page: the drift check runs in OnAfterRenderAsync (interactive render only, so a
visit costs one composition, not two with prerender); the first render shows a neutral
"Checking for changes..." chip, never a verdict; a failed check shows "Drift check failed" and
logs a warning instead of crashing the page. A deploy re-runs the check.
- Tests: probe tests for an un-audited draft change (real ConfigComposer over InMemory), fresh
composition per call, and queued callers not reusing an older composition; bUnit page tests that
a second page load after a direct draft change shows "Configuration drift" and that the first
render shows "Checking" until the comparison answers. All fail against the previous code.
DeploymentDriftProbeSqlServerTests is removed: it pinned the SQL translation of the watermark
query, which no longer exists.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Closing a tab with the script editor open ended the circuit; MonacoEditor.DisposeAsync then
called MonacoBlazor.dispose through SafeInvokeAsync, which caught only JSDisconnectedException,
so the TaskCanceledException from the torn-down circuit escaped and was logged at [ERR].
DisposeAsync now treats JSDisconnectedException, OperationCanceledException (incl.
TaskCanceledException) and ObjectDisposedException as the expected gone-circuit outcome and logs
them at Debug; a real JSException from a live page is still logged (Warning); the
DotNetObjectReference is released in a finally. MonacoEditor is the only component in the repo
that uses JS interop, so there is nothing else to align.
bUnit tests: dispose throwing each of the three gone-circuit exceptions neither escapes nor logs
above Debug (with a positive control that the failing call ran), and a JSException still warns.
They fail against the previous DisposeAsync.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A node declared numeric whose AVEVA tag is discrete (or string) is not refused up front by #897, so its native
TimeAverage / Total still reach AVEVA, which refuses them. A HistorianGateway at or after historiangw#135
(a220ec0) answers that FailedPrecondition with a google.rpc.ErrorInfo (reason AGGREGATE_NOT_SUPPORTED_FOR_TAG,
domain historian_gateway.v1) in grpc-status-details-bin. The Gateway backend's native processed read now maps
exactly that to HistoryAggregateNotSupportedException (BadAggregateNotSupported for the node) without recording
a health failure or treating it as transport. Any other FailedPrecondition and every Internal (an older gateway's
answer for the same case) keep today's handling.
Detection lives in one place, GatewayErrorReasons. The HistorianGateway.Client pin stays 1.0.0 (1.1.0 is not on
the feed): 1.0.0 keeps the gateway's RpcException, trailers included, as the InnerException of the
HistorianGatewayException it throws, so the reason is read with Grpc.StatusProto 2.76.0 (added via central
package management; nuget.org; same version as Grpc.Core.Api; depends on Google.Api.CommonProtos 2.16.0, which
accepts Google.Protobuf 3.34.1). With Client 1.1.0, ReasonOf becomes
`(ex as HistorianGatewayException)?.ErrorReason` and the StatusProto reference can go.
Tests:
- GatewayAggregateNotSupportedWireTests: in-process gRPC gateway sending the real status + ErrorInfo shape
(copied from historiangw HistorianRpcFaults.ToRpcException), read through the PUBLISHED Client 1.0.0 via the
production adapter: TimeAverage / Total -> HistoryAggregateNotSupportedException, 0 health failures; controls
(Internal; FailedPrecondition without ErrorInfo; other reason; other domain; the reason on Internal) are
rethrown, not IOException, and counted; positive control served.
- DiscreteTagTimeWeightedWireTests: over a real SDK session, a Double / Int32 node on the discrete tag ->
BadAggregateNotSupported with 0 failures; the older gateway's Internal answer keeps its status and counts.
- AvevaGatewayAggregateRefusalLiveGateTests: opt-in read-only live gate (OTOPCUA_AVEVA_GATE_900=true).
Mutations (catch removed; status ignored; domain ignored; reason ignored; inner-exception walk removed) each
fail at least one test. Release build 0 warnings; Historian.Gateway.Tests 601/607 (6 skipped), OpcUaServer
IntegrationTests 323/361 (38 skipped).
Live, read-only: private HistorianGateway container from a220ec0 (API-key auth disabled, live writes off,
removed after) against wonder-sql-vd03: Double and Int32 nodes on SysPulse, TimeAverage and Total ->
BadAggregateNotSupported, health failures unchanged (0); SysTimeSec TimeAverage Good (2 rows); SysPulse Average
served. Gateway traffic was ReadAggregate / ReadRaw / GetTagMetadata only.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
#899 — EnsureTags is an upsert, so binding a node to an existing AVEVA tag via
historianTagname re-typed/re-described it on every add and host start.
- TagConfig `historianReadOnly: true` (TagConfigIntent -> RawTagPlan.HistorianReadOnly):
HistoryRead from an existing historian tag that OtOpcUa never provisions and never
records into (AddressSpaceApplier.IsOwnedHistorizedValue gates both the EnsureTags
hook and the recorder feed). Node-inert: surgical, not mirrored on UNS refs, in the
RawTagPlan equality/hash. A non-bool value is the deploy error HistorianReadOnlyInvalid.
AdminUI: CSV column HistorianReadOnly (strict), TagHistorizeConfig reads via the
runtime parse.
- ServerHistorian:ProvisionTags = CreateOrUpdate (default, unchanged) | CreateOnly
(Gateway only: TagExistsAsync first, existing tags left alone, failed lookup not
sent) | Off. Unknown value / CreateOnly on a SQL backend fail host start.
- Default for an explicit historianTagname stays OWNED: overrides pin owned history
across renames, a read-only default would silently stop provisioning + recording.
#898 — the hook parsed Tag.DataType (an OPC UA name) as DriverDataType, so historized
Float/Double/Byte/SByte tags were never provisioned. New shared
HistorianDataTypeNames.TryResolve: Float->Float32, Double->Float64, Byte/SByte->Int16
(lossless, provisioned as Int2); unmapped names logged at Information. Gateway now skips
Int64/UInt64 (HistorianGateway cannot create Int8/UInt8: lossless-cast-or-reject).
UPGRADE: the first host start after this provisions (upserts) every existing historized
Float/Double/Byte/SByte tag. Mark foreign-tag bindings historianReadOnly, or set
ServerHistorian:ProvisionTags=CreateOnly/Off, before upgrading (docs/Historian.md).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
TimeAverage / Total on a Boolean node over the Gateway backend (AVEVA discrete tag, live
SysPulse) went to the gateway since native TimeAverage/Total became default-on (#694). AVEVA
refuses both modes at StartQuery; the gateway reports that only as Internal /
HistorianServerOperationException (the same answer as a historian storage fault), so the
node answered BadHistoryOperationUnsupported and the read counted as a backend failure.
HistoryRead-Processed now checks the node's scalar DataType before calling any backend and
answers BadAggregateNotSupported (Part 11 6.5.4: "If the Aggregate is not valid for the Node
then the StatusCode shall be Bad_AggregateNotSupported"). Nothing is sent, nothing is counted.
- Shared, every backend: HistoryAggregateDataTypes (Core.Abstractions) applies Part 13
5.4.2.3 Table 50: the numeric-only aggregates (Average, TimeAverage(2), Total(2), Min/Max,
ActualTime, Range, Delta, DeltaBounds, Interpolative, stddev/variance) on a String or
DateTime node are refused. Client-visible: the SQL backends used to answer every interval
BadAggregateInvalidInputs.
- Per backend: IHistorianDataSource.SupportsAggregateFor(aggregate, TypeCode?) (default true;
every production backend declares it, guard-tested). Gateway: false for native
TimeAverage/Total on a Boolean node. SqlServer/QuestDb: true, since they serve Boolean as
0/1 (stepped), so Boolean stays served there, as do the Gateway's raw-evaluated aggregates.
The residual case (a node declared numeric whose AVEVA tag is discrete) cannot be told apart
from a historian fault on the wire and still counts; that needs the gateway to surface
AVEVA's native error code as a deterministic status.
Tests: HistoryAggregateDataTypesTests, guard test, node-manager String/DateTime and per-type
refusals, Gateway capability matrix, DiscreteTagTimeWeightedWireTests (production Gateway
backend over an AVEVA-shaped fake: unsent, 0 health failures, positive controls). Each
mutation (no check, no shared rule, no backend hint, Gateway hint true, shared rule always
true, SQL inheriting the default) fails at least one. Live gate re-run read-only against
AVEVA 2023 R2 via a private HistorianGateway fa9cf31 container: SysPulse TimeAverage on the
wire = BadAggregateNotSupported, health failures unchanged; #801 gate 244 cases pass.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
#694 item 3 — historiangw#27 is fixed and now live-verified through OtOpcUa:
- ServerHistorian:Gateway:EnableNativeTimeWeighted defaults to true (false is
the kill switch for a gateway older than 9f711c1). TimeAverage / Total are
served and advertised again (12 aggregates).
- The gateway sends the interval as max(1, round(ms)) and AVEVA clips its last
interval at the request end (live). NativeAggregatePlan therefore sends a
multi-interval grid only for whole-ms intervals; a fractional interval is read
one interval per request (at most 1,000, else BadTooManyOperations), and every
one-interval request goes out rounded UP to whole ms so AVEVA answers one
clipped row. Live, a 3.0004 s clipped interval sent at its own width lost its
last 0.4 ms (Total 123 vs 123.0172).
#746 — HistorianGateway Contracts/Client 1.0.0 surface:
- Alarm history: GatewayAlarmHistorianWriter sends SendEvents batches of <=200.
A failed per-event result is re-sent alone via SendEvent (classified as
before); a refused batch (InvalidArgument etc.) goes event by event so a
poison event dead-letters alone; a systemic fault retries the rest without
sending more; Unimplemented (older gateway) falls back to SendEvent for good.
IHistorianGatewayClient.SendEventsAsync defaults to Unimplemented.
- Events reads: a capped page sends cap + 1 as ReadEventsRequest.max_events, so
the gateway completes the stream instead of the client cancelling it. An
ambiguous capped answer (foreign rows filtered out of the server's cut) or a
refused cap is re-read uncapped with the existing EventDrainBudget drain.
Tests: GatewayServerCappedEventsTests, GatewayAlarmWriterBatchTests,
GatewayNativeWholeMillisecondGridTests (gateway-faithful rounding fake), gate
tests flipped to the new default; every new test mutation-checked (each
mutation of the production change fails at least one). Live harness
AvevaGatewayHistorianFollowupsLiveGateTests (read-only) for #694/#700/#746.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A UnsTagReference to a raw tag that carries an "alarm" object produced a UNS
value Variable that read BadWaitingForInitialData forever (G6 live gate,
TestAlarm002Condition on testmachine-015). BuildUnsReferenceVariables now skips
those references; the equipment gets the alarm only as the raw condition wired
as an event notifier of its folder, which is unchanged (the reference still
feeds RawTagPlan.ReferencingEquipmentPaths).
Why condition-only and not "feed the mirror":
- The raw side has no value node for such a tag: AddressSpaceApplier
materialises an alarm tag as a condition OR a value variable, never both.
There is no raw value for the UNS mirror to mirror.
- DriverHostActor skips alarm tags when it builds the value fan-out and write
routing maps (`continue` before the UNS registration), and the driver leaves
them out of its value subscription ("subscribed to 5 refs" with 6 tags in
G6). Feeding the mirror would mean subscribing a value the design says the
tag does not have.
- Docs already state condition-only: ScriptedAlarms.md ("instead of a value
variable"), AlarmTracking.md ("only the condition, with no sibling value
variable"), the v3 design (native alarm = one condition at the raw tag,
surfaced at equipment via notifiers).
- The composer tests that asserted the mirror were incidental: f4f3e17e
(v3 B4-WP1) put the alarm on the fixture's Speed tag to test raw alarm
attachment, and later tests reused Speed for writable / historian / array
checks of the mirror. No commit or doc chose a value mirror for alarm tags.
- ScadaBridge does not read it: its OtOpcUa alarm path is event subscriptions
on the equipment/raw folders (v3 alarm B/C live gate), and a node that never
had a value cannot be a working binding. G6 itself authored the alarm's
Boolean as a separate value tag (TestAlarm002), which is the supported way to
read it.
Tests: the composer fixture moves the alarm onto its own HighTemp tag
(Speed is now a plain historized value tag), with a new test asserting no UNS
Variable for the alarm reference, with positive controls (the raw tag still
carries the equipment path; the value references still emit). New real-SDK
test Alarm_bearing_reference_surfaces_as_the_condition_only_against_real_SDK:
no Variable at the UNS path, the Speed mirror present, one condition, the
equipment folder wired as its notifier. Both new tests fail with the skip
removed. Docs: Uns.md, ScriptedAlarms.md, G6 evidence note.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi