A planned stop of the Primary took ~30 s on QA: WebApplication stops the web
server before Akka.Hosting's service, and Kestrel's graceful drain waited the
whole ShutdownTimeout for the peer's open LocalDb sync stream (the library's
LocalDbSyncService only observes the call token, which fires at the END of the
drain). Only then did CoordinatedShutdown run, so the stopping node held the
Primary role and the peer sat at ServiceLevel 240 for the whole window.
- GracefulClusterLeaveService (IHostedLifecycleService): StoppingAsync runs
before any hosted service's StopAsync; it raises ApplicationStopping, then
runs and awaits CoordinatedShutdown with ClrExitReason (not Cluster.Leave,
whose ClusterLeavingReason makes Akka.Hosting set ExitCode -1). The cluster
is left while Kestrel is still up.
- ShutdownStreamCancellationInterceptor (innermost on the shared gRPC
pipeline): every streaming call gets a context whose token is linked with
ApplicationStopping; streams arriving once stopping are refused Unavailable.
Host-side mitigation until LocalDb 0.3.8 is published and pinned
(scadaproj 3ddb083, scadaproj#57); it also covers the mesh streams.
Test GracefulStopWithOpenSyncStreamTests (real WebApplication + Kestrel h2c +
MapZbLocalDbSync + single-node Akka cluster + real peer initiator, 8 s drain
budget): stop 895 ms, cluster left at 892 ms, drain began at 894 ms. Positive
control without the fix: stop 8863 ms, cluster left at 8841 ms after the drain
began at 0 ms. Mutations: no CoordinatedShutdown in StoppingAsync -> order
assertion fails (drain at 1 ms, removed at 893 ms); interceptor passes the
original context -> stop 8006 ms, fails.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The #815 warn-only compatibility window is over. A running Galaxy driver
(InitializeAsync, every gateway ReopenAsync, the ack fallback password) and a
running OpcUaClient driver (session open) now refuse an out-of-namespace
credential reference with the same CredentialReferenceException that save,
deploy (CredentialReferenceOutOfScope), Test Connect and browse already raise,
before any store / environment / file read and before any endpoint is dialled.
The one existing scope check (CredentialReferences.Parse/ResolveAsync) is now
the only path; nothing new was added.
Removed:
- LegacyCredentialReferenceTracker (Core.Abstractions), its rate-limited WARN
and the credential-compat:<field> diagnostics entry.
- The per-driver Legacy* resolve methods and every `legacy` parameter
(GalaxySecretRef, GalaxyAckCredentialResolver, OpcUaUserIdentityFactory,
OpcUaUserCertificatePaths, OpcUaClientSecretResolution).
- IDriverDiagnostics on GalaxyDriver / OpcUaClientDriver (it only carried the
tracker's entries).
- The tracker also kept a pre-#836 absolute UserCertificatePath loading on a
running OpcUaClient driver; that is refused now too.
Preflight / banner decision: KEPT, as a hard error. The Running source
(driver-health credential-compat entries) and CredentialReferenceBannerModel
are gone with the tracker, but the ConfigDb scan still tells an operator
something no other surface does: a Deployed finding is now a driver that is
not connecting, a Draft finding is a deploy the gate will refuse - named
before anyone deploys, e.g. after restoring an old ConfigDb backup or
importing a pre-#815 config. It also carries the #835 cross-instance and
#836/#850 path checks. CredentialReferencePreflightService now logs one Error
per finding (was Warning) with no warn-only wording, and the /raw banner is
alert-danger.
DEV and QA ConfigDbs hold no Galaxy/OpcUaClient driver (checked 2026-11-13);
no production install exists.
Tests: GalaxyDriverLegacyCredentialTests replaced by
GalaxyDriverCredentialScopeTests; new OpcUaClientDriverCredentialScopeTests;
tracker tests in OpcUaClientSecretResolutionTests /
OpcUaClientUserCertificatePathTests deleted; AdminUI preflight tests drop the
Running/banner-model cases and assert Error level. Against the old code the
new refusal tests fail (Galaxy 5/6, OpcUaClient 4/5; the two positive controls
pass on both).
docs/security.md: the migration table now says running drivers refuse; the
warn-only section is replaced by "The compatibility window is over".
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Root cause: DeploymentDriftProbe (a singleton shared by every circuit) cached its verdict until
the ConfigEdit audit watermark (MAX(EditedAtUtc), COUNT(*)) moved, assuming every configuration
write appends a ConfigEdit row. Only deployments and driver restart/reconnect do; the AdminUI
editors (RawTreeService, UnsTreeService, ScriptEdit, cluster/node/ACL pages) write the config
tables directly. So after a draft change the page kept showing the cached "In sync" until
something unrelated moved the watermark.
- DeploymentDriftProbe: the watermark cache is gone; every verdict comes from a composition that
started after the caller asked. Compositions stay serialised behind the gate, and callers queued
behind one share a verdict only when that composition started after they arrived (start
sequence number) for the same sealed hash, so sharing is exact, never stale.
- Deployments page: the drift check runs in OnAfterRenderAsync (interactive render only, so a
visit costs one composition, not two with prerender); the first render shows a neutral
"Checking for changes..." chip, never a verdict; a failed check shows "Drift check failed" and
logs a warning instead of crashing the page. A deploy re-runs the check.
- Tests: probe tests for an un-audited draft change (real ConfigComposer over InMemory), fresh
composition per call, and queued callers not reusing an older composition; bUnit page tests that
a second page load after a direct draft change shows "Configuration drift" and that the first
render shows "Checking" until the comparison answers. All fail against the previous code.
DeploymentDriftProbeSqlServerTests is removed: it pinned the SQL translation of the watermark
query, which no longer exists.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Closing a tab with the script editor open ended the circuit; MonacoEditor.DisposeAsync then
called MonacoBlazor.dispose through SafeInvokeAsync, which caught only JSDisconnectedException,
so the TaskCanceledException from the torn-down circuit escaped and was logged at [ERR].
DisposeAsync now treats JSDisconnectedException, OperationCanceledException (incl.
TaskCanceledException) and ObjectDisposedException as the expected gone-circuit outcome and logs
them at Debug; a real JSException from a live page is still logged (Warning); the
DotNetObjectReference is released in a finally. MonacoEditor is the only component in the repo
that uses JS interop, so there is nothing else to align.
bUnit tests: dispose throwing each of the three gone-circuit exceptions neither escapes nor logs
above Debug (with a positive control that the failing call ran), and a JSException still warns.
They fail against the previous DisposeAsync.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
#899 — EnsureTags is an upsert, so binding a node to an existing AVEVA tag via
historianTagname re-typed/re-described it on every add and host start.
- TagConfig `historianReadOnly: true` (TagConfigIntent -> RawTagPlan.HistorianReadOnly):
HistoryRead from an existing historian tag that OtOpcUa never provisions and never
records into (AddressSpaceApplier.IsOwnedHistorizedValue gates both the EnsureTags
hook and the recorder feed). Node-inert: surgical, not mirrored on UNS refs, in the
RawTagPlan equality/hash. A non-bool value is the deploy error HistorianReadOnlyInvalid.
AdminUI: CSV column HistorianReadOnly (strict), TagHistorizeConfig reads via the
runtime parse.
- ServerHistorian:ProvisionTags = CreateOrUpdate (default, unchanged) | CreateOnly
(Gateway only: TagExistsAsync first, existing tags left alone, failed lookup not
sent) | Off. Unknown value / CreateOnly on a SQL backend fail host start.
- Default for an explicit historianTagname stays OWNED: overrides pin owned history
across renames, a read-only default would silently stop provisioning + recording.
#898 — the hook parsed Tag.DataType (an OPC UA name) as DriverDataType, so historized
Float/Double/Byte/SByte tags were never provisioned. New shared
HistorianDataTypeNames.TryResolve: Float->Float32, Double->Float64, Byte/SByte->Int16
(lossless, provisioned as Int2); unmapped names logged at Information. Gateway now skips
Int64/UInt64 (HistorianGateway cannot create Int8/UInt8: lossless-cast-or-reject).
UPGRADE: the first host start after this provisions (upserts) every existing historized
Float/Double/Byte/SByte tag. Mark foreign-tag bindings historianReadOnly, or set
ServerHistorian:ProvisionTags=CreateOnly/Off, before upgrading (docs/Historian.md).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
TimeAverage / Total on a Boolean node over the Gateway backend (AVEVA discrete tag, live
SysPulse) went to the gateway since native TimeAverage/Total became default-on (#694). AVEVA
refuses both modes at StartQuery; the gateway reports that only as Internal /
HistorianServerOperationException (the same answer as a historian storage fault), so the
node answered BadHistoryOperationUnsupported and the read counted as a backend failure.
HistoryRead-Processed now checks the node's scalar DataType before calling any backend and
answers BadAggregateNotSupported (Part 11 6.5.4: "If the Aggregate is not valid for the Node
then the StatusCode shall be Bad_AggregateNotSupported"). Nothing is sent, nothing is counted.
- Shared, every backend: HistoryAggregateDataTypes (Core.Abstractions) applies Part 13
5.4.2.3 Table 50: the numeric-only aggregates (Average, TimeAverage(2), Total(2), Min/Max,
ActualTime, Range, Delta, DeltaBounds, Interpolative, stddev/variance) on a String or
DateTime node are refused. Client-visible: the SQL backends used to answer every interval
BadAggregateInvalidInputs.
- Per backend: IHistorianDataSource.SupportsAggregateFor(aggregate, TypeCode?) (default true;
every production backend declares it, guard-tested). Gateway: false for native
TimeAverage/Total on a Boolean node. SqlServer/QuestDb: true, since they serve Boolean as
0/1 (stepped), so Boolean stays served there, as do the Gateway's raw-evaluated aggregates.
The residual case (a node declared numeric whose AVEVA tag is discrete) cannot be told apart
from a historian fault on the wire and still counts; that needs the gateway to surface
AVEVA's native error code as a deterministic status.
Tests: HistoryAggregateDataTypesTests, guard test, node-manager String/DateTime and per-type
refusals, Gateway capability matrix, DiscreteTagTimeWeightedWireTests (production Gateway
backend over an AVEVA-shaped fake: unsent, 0 health failures, positive controls). Each
mutation (no check, no shared rule, no backend hint, Gateway hint true, shared rule always
true, SQL inheriting the default) fails at least one. Live gate re-run read-only against
AVEVA 2023 R2 via a private HistorianGateway fa9cf31 container: SysPulse TimeAverage on the
wire = BadAggregateNotSupported, health failures unchanged; #801 gate 244 cases pass.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
#694 item 3 — historiangw#27 is fixed and now live-verified through OtOpcUa:
- ServerHistorian:Gateway:EnableNativeTimeWeighted defaults to true (false is
the kill switch for a gateway older than 9f711c1). TimeAverage / Total are
served and advertised again (12 aggregates).
- The gateway sends the interval as max(1, round(ms)) and AVEVA clips its last
interval at the request end (live). NativeAggregatePlan therefore sends a
multi-interval grid only for whole-ms intervals; a fractional interval is read
one interval per request (at most 1,000, else BadTooManyOperations), and every
one-interval request goes out rounded UP to whole ms so AVEVA answers one
clipped row. Live, a 3.0004 s clipped interval sent at its own width lost its
last 0.4 ms (Total 123 vs 123.0172).
#746 — HistorianGateway Contracts/Client 1.0.0 surface:
- Alarm history: GatewayAlarmHistorianWriter sends SendEvents batches of <=200.
A failed per-event result is re-sent alone via SendEvent (classified as
before); a refused batch (InvalidArgument etc.) goes event by event so a
poison event dead-letters alone; a systemic fault retries the rest without
sending more; Unimplemented (older gateway) falls back to SendEvent for good.
IHistorianGatewayClient.SendEventsAsync defaults to Unimplemented.
- Events reads: a capped page sends cap + 1 as ReadEventsRequest.max_events, so
the gateway completes the stream instead of the client cancelling it. An
ambiguous capped answer (foreign rows filtered out of the server's cut) or a
refused cap is re-read uncapped with the existing EventDrainBudget drain.
Tests: GatewayServerCappedEventsTests, GatewayAlarmWriterBatchTests,
GatewayNativeWholeMillisecondGridTests (gateway-faithful rounding fake), gate
tests flipped to the new default; every new test mutation-checked (each
mutation of the production change fails at least one). Live harness
AvevaGatewayHistorianFollowupsLiveGateTests (read-only) for #694/#700/#746.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A UnsTagReference to a raw tag that carries an "alarm" object produced a UNS
value Variable that read BadWaitingForInitialData forever (G6 live gate,
TestAlarm002Condition on testmachine-015). BuildUnsReferenceVariables now skips
those references; the equipment gets the alarm only as the raw condition wired
as an event notifier of its folder, which is unchanged (the reference still
feeds RawTagPlan.ReferencingEquipmentPaths).
Why condition-only and not "feed the mirror":
- The raw side has no value node for such a tag: AddressSpaceApplier
materialises an alarm tag as a condition OR a value variable, never both.
There is no raw value for the UNS mirror to mirror.
- DriverHostActor skips alarm tags when it builds the value fan-out and write
routing maps (`continue` before the UNS registration), and the driver leaves
them out of its value subscription ("subscribed to 5 refs" with 6 tags in
G6). Feeding the mirror would mean subscribing a value the design says the
tag does not have.
- Docs already state condition-only: ScriptedAlarms.md ("instead of a value
variable"), AlarmTracking.md ("only the condition, with no sibling value
variable"), the v3 design (native alarm = one condition at the raw tag,
surfaced at equipment via notifiers).
- The composer tests that asserted the mirror were incidental: f4f3e17e
(v3 B4-WP1) put the alarm on the fixture's Speed tag to test raw alarm
attachment, and later tests reused Speed for writable / historian / array
checks of the mirror. No commit or doc chose a value mirror for alarm tags.
- ScadaBridge does not read it: its OtOpcUa alarm path is event subscriptions
on the equipment/raw folders (v3 alarm B/C live gate), and a node that never
had a value cannot be a working binding. G6 itself authored the alarm's
Boolean as a separate value tag (TestAlarm002), which is the supported way to
read it.
Tests: the composer fixture moves the alarm onto its own HighTemp tag
(Speed is now a plain historized value tag), with a new test asserting no UNS
Variable for the alarm reference, with positive controls (the raw tag still
carries the equipment path; the value references still emit). New real-SDK
test Alarm_bearing_reference_surfaces_as_the_condition_only_against_real_SDK:
no Variable at the UNS path, the Speed mirror present, one condition, the
equipment folder wired as its notifier. Both new tests fail with the skip
removed. Docs: Uns.md, ScriptedAlarms.md, G6 evidence note.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Found on the docker-dev rig during the #761 Wave-4 gate run (2026-11-12): the documented
bodyless headless deploy (docker-dev/deploy.sh, README "Headless deploy") returned 404 on
both centrals, while the same POST with a JSON body was accepted.
Cause: the optional body was an inferred `DeployRequest?` parameter, which stamps
Accepts(application/json) metadata on the endpoint. For a request with no Content-Type the
routing AcceptsMatcherPolicy prefers endpoints that declare no consumes metadata, and on a
fused admin+driver host MapOtOpcUaMeshGrpcServices has mapped grpc-dotnet's
`{unimplementedService}/{unimplementedMethod}` catch-all, so the request was routed away
from the deploy handler and 404'd. The deploy API on its own does not reproduce it.
The handler now reads the optional body itself: no body (absent or zero-length) is the
headless form, a non-JSON body answers 415 and malformed JSON 400, as the framework's
binding did. New DeployApiBodylessPostTests compose the deploy API with the mesh gRPC
services and drive real HTTP; with the fix reverted the two bodyless cases fail with 404.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.Telemetry(.Serilog) 0.1.0 -> 0.3.0. The OTel Resource now names the
node: site.id = its application Cluster id (the cluster-<ClusterId> role suffix,
omitted on a legacy node), node.role = its plain roles in canonical order
(admin / driver / admin,driver), service.version = the entry assembly's informational
version (the git SHA). Program passes its resolved EffectiveRoles. The whole
OtOpcUa:Telemetry section goes through the shared ApplyConfiguration, so SiteId /
NodeRole / ServiceVersion / ResourceAttributes:* are operator overrides, and
Exporter=Otlp now also sends the Serilog logs to the same collector.
ObservabilityResourceIdentityTests cover the mapping and the built container's
Resource. Host.Tests 288/288, Host.IntegrationTests 325/327 (2 skipped).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ZbTelemetryOptions.Meters/ActivitySources are strict allowlists. The host named only
ZB.MOM.WW.OtOpcUa and the LocalDb meter, so the in-process Galaxy driver's galaxy.* spans
and its event pump's galaxy.events.* counters (both on "ZB.MOM.WW.OtOpcUa.Driver.Galaxy")
were silently dropped from /metrics and OTLP. Found exercising OTLP end to end against a
collector (scadaproj#13).
- Galaxy driver: public GalaxyDriverTelemetryNames (MeterName / ActivitySourceName);
GalaxyTelemetry and EventPump reference it.
- Host: ObservabilityExtensions.ObservedMeters / ObservedActivitySources include it.
- Host.IntegrationTests: ObservabilityAllowlistTests assert on the listener side (span
sampled, instrument enabled) with an unregistered-name negative control; dropping the
Galaxy entries fails 3 of 8.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.LocalDb and .Replication at 0.3.7 (published 2026-11-12).
Wire-compatible; no code change needed.
- 0.3.6 (scadaproj#36): the library sync service enforces
LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
its refuse-every-stream-when-blank rule; doc comment updated. docker-dev
site-a pair: both nodes now read one variable,
OTOPCUA_SITE_A_LOCALDB_SYNC_KEY (same dev default as before), so an
override can never split the pair.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
node offline past TombstoneRetention no longer brings a pruned
deployment back. The ledger grows while a peer is down; optional
TombstoneHardCap (documented in Configuration.md) bounds it.
Runbook, Configuration.md and the cluster-topology rule updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK memoizes INodeManager3.ValidateEventRolePermissions per (item, event type,
SourceNode). #826 kept it fresh by reflecting into the private
MonitoredNode2.InvalidatePermissionCaches (plus a detach/re-attach fallback). Since #855
condition events never take that channel, and the search for #878 found nothing
production-raised that does except a condition's SDK method audits, which reach it only
with auditing on, SignAndEncrypt and a SecurityAdmin session (nothing grants that role).
Under enforcement the channel verdict is now a flat deny for items on our nodes, so no
cached verdict depends on a grant, and InvalidateFolderEventVerdicts, the reflection,
the fallback, its counter and the guard test are removed. Method audits still reach
Server-object items, decided per event.
Tests: FolderEventVerdictInvalidationTests -> FolderEventVerdictFreshnessTests; new
revoked-grant test (positive control: a still-granted session) and method-audit test
(SecurityAdmin mapped for the test; positive controls: Server-object item, flag off);
the plain-event test runs each mode on a fresh server. Mutation-checked.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A notifier folder read one event source, its NodeId identifier (a RawPath or an EquipmentId). AVEVA stores a
Galaxy object's A&E history under the object's tag_name (Source_Object), so AVEVA's own alarm and event
history reached no production folder. Its lower-case property keys were also missed, so every row read
Severity 1 with no ConditionName.
Sources. A folder's event history now reads its own source plus the Galaxy object of every native condition
that fires on it live: those parented under it and those WireAlarmNotifiers wired to it. Live and history use
the same membership. The object comes from the condition's Galaxy alarm reference, its raw tag's attributeRef,
up to the first '.':
- The composer reads it with the driver's own parser (GalaxyAttributeRef, shared with GalaxyDriver), only for
a GalaxyMxGateway tag with an alarm (RawTagPlan.NativeAlarmReference, part of plan equality).
- The applier binds it after materialising the condition (IOpcUaAddressSpaceSink.BindNativeAlarmHistory,
forwarded by the deferred sink). A rebuild or the condition's removal forgets it.
Each object is read with the #882 rule: its own events and <Object>.<Attribute> rows, never a prefix sibling.
Fan-out. historiangw's ReadEvents filter is a single predicate, so there is one request per source; the wire
contract is unchanged. EventSourceFanOut merges the reads as one source:
- order is (time, source position, arrival), reversed for a reverse read; ties are never re-sorted by EventId,
so the boundary skip stays stable;
- duplicates by EventId are dropped and the merge is cut to the cap;
- a truncated source bounds the page at the time it stopped.
Paging, continuation points (bound to the whole source set), the walked-back reverse read and the tie-cluster
over-fetch are unchanged. A folder with one source is read exactly as before.
ACL. Every source is read inside the folder's own HistoryRead, after AuthorizeHistoryRead (#826). A denied
folder reads nothing and fails closed.
Mapping (EventMapper, property names now case-insensitive):
- Severity: an AVEVA row's priority goes through GalaxyAlarmPriority, the table live Galaxy alarms use
(MxAccessSeverityMapper now delegates to it), so history matches live. Otherwise severity (OtOpcUa's own
rows, which AVEVA stores lower-cased), clamped. AVEVA's own 1-4 severity is ignored. The table ranks a
higher priority as more severe, which inverts AVEVA's convention: filed as #887.
- An AVEVA row whose source_conditionvariable names a native condition that fires on the folder takes that
condition's SourceNode and live ConditionName. Otherwise ConditionName is alarmname or the condition
variable's attribute.
- alarm_id/alarmid map to AlarmId.
Tests:
- GalaxyEquipmentEventHistoryWireTests (4, real SDK session, production Gateway backend, AVEVA-shaped fake
gateway): equipment EqA backed by objects Oven_007 + OvenDoor_07. Covers object and attribute events from
both objects, the prefix sibling excluded, cross-object and in-object ties, forward paging at 1-4 a page,
reverse whole and paged, SourceNode/ConditionName/Severity, the raw device folder, ACL deny with no backend
call plus a granted positive control, and a rebuild forgetting the bindings.
- EventSourceFanOutTests (8), EventMapper AVEVA cases (11), GalaxyAttributeRefTests (16), a composer test and
an applier ordering test.
- Mutation-checked, 17 mutants, all killed. Covered: parent-only membership, no scope, a case-sensitive
condition match, no condition rewrite, no truncation bound, EventId tie order, no sort, no cap, no stop at
the bound, unreversed source order, no dedupe, no '.' boundary, priority ignored, case-sensitive keys, no
applier bind, no deferred forward, and a composer that ignores the driver type. The forget-on-rematerialise
guard survives as belt-and-braces: the rebuild clear covers it.
- Live, read-only, no WinRM, vd03 HistorianGateway ea9f21c: equipment folder EQ-886-ZipsenLeft (objects
DevWinPlatform + Left_001). It read 0 events before and 92 now, equal to the unfiltered read's rows for both
objects. 1/7-event pages and reverse whole/5-event pages agree, and 73 rows resolve to their live condition.
diff_runs vs the #882 run: 221/226 identical. The 5 that differ are expected: DevWinPlatform Severity
1 -> 750/1000, and #881 capabilities.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ConfigurationBinder appends configured collection items to an existing list, so
the pre-filled [None, Sign, SignAndEncrypt] default made "drop None" a no-op:
every deployment that listed only the secure profiles still advertised an
unsigned/unencrypted None endpoint.
EnabledSecurityProfiles is now null by default; ResolveEnabledSecurityProfiles()
applies the unchanged three-profile default only when nothing is configured.
A set-but-empty list fails startup validation and BuildSecurityPolicies no
longer falls back to a None endpoint on empty input (it throws). Sweep of all
options classes found no other non-empty pre-filled collection default.
Tests: options binding (indexed, JSON, env forms, default, empty, bad name), a
booted-server wire test (no None endpoint, None connection refused with
BadSecurityPolicyRejected, secure connection works; default case is the control).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK starts Server.ServiceLevel at 255 (full service), and OpcUaPublishActor publishes nothing
until the node's first redundancy snapshot arrives, so a restarting pair node advertised 255 for the
~10-13 s it took to join, out-ranking the real Primary (250) for any client that picks the highest
ServiceLevel.
OtOpcUaSdkServer now writes StartupServiceLevel = 0 in OnNodeManagerStartedAsync: after the SDK's
CreateServerObjectAsync wrote 255 and before the server goes Running, so no session can ever read
255. 0 is the design's existing Down / Detached tier (ServiceLevelCalculator returns 0 for a member
that is not Up/Joining, and a node with no snapshot has no role yet). It is below the SDK 2
session-headroom floor (200), so it stands through session churn before the #884 claim, and the
claim's bind replays any level already requested. Single-node deployments are unaffected beyond
the window: the first snapshot publishes 250 as before.
New Node_with_no_requested_level_advertises_the_startup_level_until_one_is_published: reads 0
before the bind, 0 after a bind with nothing to replay, then 250 once published. Without the
startup write it reads 255 on the first session (mutation-checked). docs/Redundancy.md updated for
#884 and #885.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
SDK 2.0.1 recomputes Server.ServiceLevel from session headroom on every session create/close and
overwrites any value >= 200 unless a provider has claimed it through
IServerServiceLevelControl.ClaimServiceLevelControl. SdkServiceLevelPublisher never claimed it, so on
the docker-dev pair both nodes read 255 as soon as a client connected while their own metrics showed
250 (Primary) and 240 (Secondary); the Stale tier's 200 was lifted to 255 too. Levels below 200 (100,
0) were untouched, which is why the bootstrap 100 showed briefly.
The publisher now claims the value at construction and writes through the SDK's writer; a server
without the claim (test doubles) keeps the diagnostics-guarded write. New
Published_level_survives_later_sessions (250/240/200) fails without the claim (reads 255) and passes
with it.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Every Boolean under Server/ServerCapabilities/HistoryServerCapabilities read
false (the SDK default) even though data and event history were served, so a
spec-following client skipped HistoryRead entirely.
Each backend now declares what it serves through its own interface:
- IHistorianDataSource.ReadCapabilities (Data / Events / ServerTimestamps):
Gateway Data|Events; SqlServer + QuestDb add ServerTimestamps (they store
ServerTimeUtc, the Gateway stamps read time); Null declares None. A
reflection guard requires every production source to declare it itself.
- IHistorianDataUpdater.Capabilities (required member): SqlServer and QuestDb
declare Insert/Replace/Update/DeleteRaw/DeleteAtTime; the Gateway provider
has no updater.
The node manager writes the flags on every source/updater assignment
(AdvertisedHistoryCapabilities), and the same declarations drive dispatch:
an update the updater does not declare is BadHistoryOperationUnsupported
without reaching it, the HistoryWrite AccessLevel bit follows a non-empty
declaration, and event-history notifier promotion follows the Events flag.
Event and annotation update flags stay false (nothing dispatches them).
Tests: booted-server wire test per backend + Null + a SqlServer->Gateway
rewire pairing on one session; dispatch gating unit test; the #801 live
gate now asserts the Gateway flags (not run here).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
OpcUaApplicationHost left ServerConfiguration.MaxRegistrationInterval at the
SDK default (30000) with a null RegistrationEndpoint, so SDK 2.0.1's
StandardServer armed its registration timer against opc.tcp://localhost:4840,
which on an OtOpcUa node is the server itself (no LDS). It answered
BadServiceUnsupported and the SDK retried forever, visible since #874.
Registration is now opt-in: ApplyDiscoveryRegistration sets
MaxRegistrationInterval = 0 (the value that keeps StandardServer from arming
the timer) unless OpcUa:DiscoveryServerUrl is set, in which case the server
registers against that URL every OpcUa:DiscoveryRegistrationIntervalMs
(default 30000) over SignAndEncrypt/Basic256Sha256, like the SDK's own
default endpoint. The validator requires an absolute opc.tcp:// URL and a
positive interval when the URL is set. Documented in Configuration.md and
OpcUaServer.md.
Co-Authored-By: Claude <noreply@anthropic.com>
- Host reader: the frame read and its handling are caught separately. Only the
read's I/O exceptions mean the pipe ended; anything HandleFrame's inline
completion work throws now kills that worker as ProtocolFault (outstanding
runs -> worker restarted / BadResourceUnavailable) instead of escaping a
dedicated thread (process crash) or being swallowed as a clean end (an
InvalidOperationException used to be, leaving the worker unread until a Crash
2 s later).
- Worker reader: the same catch-all around HandleFrame (exit ProtocolFault).
- Worker: Cancel() runs on the pool, so a script's blocking
ctx.CancellationToken callback cannot stop the worker reading its pipe.
- Host write-stall check: ScriptWorkerFrameWriter.WriteInProgressFor; the
heartbeat monitor kills a worker whose pipe write has been blocked longer than
max(10 s, silence limit) as KilledUnresponsive, which fails the write and
releases the producer (seam submit, deploy load, Cancel). Fixes the wrong
'heartbeat monitor ends it' comment on the pipe buffer quota.
- StopAsync sends Shutdown off the caller so its 1 s budget holds on a full pipe.
- WindowsBlockingPipeStream refuses the async surface (Stream's base versions
serialise a write behind a waiting read).
- Tests: PipeFaultTests (throwing completion callback -> ProtocolFault and a
BadResourceUnavailable sibling, shard serves again; a worker that stops reading
but heartbeats is killed within the bound and the blocked producer returns,
via the Hello.test_stall_reader_on_diag seam); BlockingPipeStreamTests dispose
with a write blocked on a full pipe, and with a read and a write both pending;
Category=ScriptWorker on the protocol, handshake and pipe test classes.
Mutation checks: no stall kill -> stall test red; catch-all without the fault
-> callback test red (Crash).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A .NET pipe opened Asynchronous completes even its synchronous Read/Write
through the thread pool's completion port, so every frame on Windows cost a
completion packet plus a pool work item (and the pool's spin-wait). New
WindowsBlockingPipeStream issues its own overlapped ReadFile/WriteFile with the
OVERLAPPED.hEvent low bit set (no completion packet) and waits on that event on
the calling thread; reads and writes stay concurrent, unlike a synchronous
handle. ScriptWorkerFrameCodec.BlockingStream picks it on Windows and the pipe
itself on Unix, so both ends now use the dedicated blocking reader + synchronous
combining writer on every OS (UseBlockingIo and the async read loops are gone).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.
DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The host handed the SDK a provider-less telemetry context, so SDK-side
refusals — identity-token decryption (the SDK 2 RsaEncryptedSecret clock
window, #873), certificate validation, secure-channel faults — left no server
log line (found on windev during the #800 Windows validation). The hosted
service now creates one ServerSdkTelemetry context from its ILoggerFactory
(Warning and above, SDK categories, rate-limited 20/min per category) and
shares it between OtOpcUaSdkServer and the ApplicationInstance; released on
dispose. Test: an untrusted client's rejection reaches the captured log.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Since #855 condition events are queued to folder event items under the node
manager Lock. At a full DiscardOldest queue the SDK's EventQueueHandler
dequeues the head of EventMonitoredItemQueue, a List.RemoveAt(0) shift of the
whole queue (~1 us/item at QueueSize 1000, ~10 us at 10000).
OtOpcUaSdkServer.CreateMonitoredItemQueueFactory now wraps the SDK factory in
RingEventQueueFactory: event queues are RingEventMonitoredItemQueue (O(1)
discard), data-change queues, durable requests and disposal go to the SDK
factory unchanged. A factory set through StandardServer.MonitoredItemQueueFactory
is used as-is.
The SDK queue is storage only (decompiled 2.0.1): DiscardOldest, overflow-event
positions and discarded/overflow counters live in EventQueueHandler, which is
unchanged. The ring reproduces QueueSize 0 initial + BadInternalError enqueue,
drop-oldest at Count == QueueSize, FIFO dequeue, SetQueueSize keep-newest /
keep-oldest, the newest-1000 duplicate window via IsEventRaisedFor, no-op
Dispose. RingEventMonitoredItemQueueParityTests drive both queues, and the SDK
EventQueueHandler over both factories, with the same random sequences.
FolderEventDeliveryPerfTests gains Ring_vs_sdk_event_queue (A/B in one process
via the internal OtOpcUaSdkServer.RingEventQueues switch).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The sweep threshold never rises above the 4096 cap (a threshold the count
could not reach stopped sweeping, and sharing, for good); at the cap a
sweep for dead shapes runs after every 256 refusals.
- The per-report "queued" mark is taken after the filter group resolves,
so a failure there leaves the item unmarked.
- The bucket key also carries TypeDefinitionId and IndexRange.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The filter-group registry holds groups weakly (the filter->group table is
the only strong reference), sweeps dead entries amortised and counts only
live shapes against the 4096 cap; filters past the cap are counted on the
new meter otopcua.opcua.condition_events.filter_groups_unshared.
- A group validates its where clause too (restored, never-validated filters
evaluate like validated ones).
- Test: value-equal filters cost one projection per locale set per report.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Filter-group buckets include the where clause's (value-based) hash, so
per-source filters sharing a select list are not all scanned under Lock.
- Locale keys are cached per session array.
- Tests: a notifier reached twice in one walk (Line above two wired
equipment) gets each event once on both paths (fails without the
per-report set); queued events re-projected by a modify are identical on
both paths; locale keys unambiguous and where clauses hash by value.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.
Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
handler returned is reported; one pending at actor stop is reported; a
confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
reconnect, UNACK_RTN-as-Clear and RPC-budget limits.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- RefreshDirectItemsWithCut keys its "decided off Lock" set per (item,
condition), so a condition one item's walk saw is still decided for another.
- The deferral warning no longer blames slow folder subscribers (no folder of
ours defers since #855).
- FolderEventFloodWireTests' removal test now asserts the folder item got
nothing and the Server object exactly the pre-stall event (it passed with
the post-stall liveness re-check deleted).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.
- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
(default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
after AcknowledgeAsync, off its mailbox and outside the resilience
pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Both plain ConditionRefresh paths (Server object and folder) skip a
condition a deploy removed or replaced between the off-Lock walk and the
queueing (the SDK base did not either; only the cut path checked).
- A failure delivering at one notifier hop is logged and the walk continues
to the remaining hops; a failure deciding the Server-object items withholds
only them.
- Test: a condition removed during a plain refresh (folder + Server object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay's verdict refuses a condition a deploy removed or replaced
since the cut (Server-object and folder), so the cut path cannot replay a
ghost or judge a replaced scripted alarm on its old folder.
- DeliverToNotifierItems resolves the #826 decision's item-independent part
(live condition, notifier check, scopes) once per hop; only the grant and
SDK role checks stay per item.
- A deferred delivery's drain starts on the pool, never inline under Lock.
- Test: a condition removed between the cut and its replay (folder + Server
object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review findings on 02503a4a (ConfigPublishCoordinator):
1 (HIGH) A recovered deployment whose every expected ack was already in
NodeDeploymentState (a seal whose SaveChanges failed, a fused node's own row)
was never completed: inside the deadline it waited for acks that had landed,
past it HandleDeadline's "race" guard returned without sealing — stuck in
AwaitingApplyAcks, refusing every later deploy. ApplyRecovery now completes
it (Seal / PartiallyFailed), and HandleDeadline completes a full ack set
instead of returning.
2 (MED-HIGH) The past-deadline recovery of a populated expected set ran
HandleDeadline inline before the stash replay, discarding on-time acks queued
during the outage. It is now self-sent, behind the replay. The zero-rows
past-deadline branch stays inline on purpose: no ack can count there, and a
self-sent deadline would time out a queued dispatch of that same row right
after publishing it.
3 (MED) A failed PersistNodeAck threw, restarted the actor and dropped the ack
(driver-only nodes never write their own row and do not resend). Acks are now
persisted before being counted, and a failed persist is re-queued on a timer.
The class remarks document that the Loading stash is node-local (lost on a
hand-over; queued dispatches are recovered from their rows).
Minor: MarkReady/Become(Ready) only after the recovery applied; a throw while
applying is a failed load (backoff) rather than a tight restart loop; a queued
dispatch matching the recovered Dispatching row logs Info, not the "lost
dispatch" Warning.
Tests (ConfigPublishCoordinatorDbOutageTests): fully-acked recovery seals /
partially fails inside and past the deadline (4 cases); queued ack applied
before a past-deadline recovery; failed ack persist retried. All 6 fail on
02503a4a.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay re-asks each item's verdict right before each entry's gate
hold (Server-object and folder cut paths): a grant revoked between the
pre-Lock verdicts and the replay no longer lets the replay through. A denied
entry is also removed from the cut's re-sent set.
- ReportEventWithoutBlocking logs and returns on a delivery that faulted
inline instead of booking it as a deferral episode (which ran the drain and
ClearStaleRefreshCuts under the event-report gate while Lock was held).
- OnConditionReportEvent isolates failures per Server-object item.
- Tests: revocation between cut and replay (folder + Server object), throwing
hop not booked as a deferral; production-overload sampling inside Lock.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
When the data starts inside an interval (no start bound) and its first value is
Uncertain, the frozen 1.5.378.182 AverageAggregateCalculator integrated the
uncovered span as a ramp from 0 and divided TimeAverage by the whole interval:
GetRegionsInValueSet (ignoreBadData, sloped) overwrites the BadNoData start
region's status with UncertainDataSubNormal in its one branch without a not-Bad
guard. Live AggregateFilter therefore disagreed with HistoryRead and Part 13
§5.4.3.6/§5.4.3.8 ("No Start Bound: Calculate Partial Interval"); QP: Total
188.96 vs 163.96, TimeAverage 9.448 vs 10.931. SDK 2.0.1 stock does the same.
Owner-approved 2026-11-11.
- New app-owned CoveredTimeAverageAggregateCalculator subclasses the baseline
and recomputes only TimeAverage/Total, restoring the Bad status of a region
that starts at a Bad point (only the start bound can). Only such an interval's
value changes; status, timestamps, type and bits are the baseline's. Stepped
and TreatUncertainAsBad=true never reached the defect. CompatibilityAggregate-
Factory routes exactly those two IDs; copied SDK files untouched.
- Golden test: 0 fixture vectors change (every data set has a usable value at
-10 s), asserted. New sweep over late-start variants of every data set (cut
+5 s / +25 s, first non-Bad sample Uncertain, all directions/stepping/configs/
incremental): 150 / 142 of 384 change, each pinned to a baseline-derived
oracle (untouched baseline's value with that sample Good, its status/bits on
the variant itself) - never our calculator. Plus a hand-computed case.
- Live-vs-HistoryRead parity (TimeWeightedAggregator) forward + reverse, no
masks, with discrimination against the pre-#821 baseline; QualityMix now
compares QP's TimeAverage/Total with the live oracle on both SQL backends;
booted-runtime facts test expects 163.96 / 10.93.
- README recorded decision, ServerAggregateConfiguration remarks, oracle
remarks, docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Joseph Doherty