Found on the docker-dev rig during the #761 Wave-4 gate run (2026-11-12): the documented
bodyless headless deploy (docker-dev/deploy.sh, README "Headless deploy") returned 404 on
both centrals, while the same POST with a JSON body was accepted.
Cause: the optional body was an inferred `DeployRequest?` parameter, which stamps
Accepts(application/json) metadata on the endpoint. For a request with no Content-Type the
routing AcceptsMatcherPolicy prefers endpoints that declare no consumes metadata, and on a
fused admin+driver host MapOtOpcUaMeshGrpcServices has mapped grpc-dotnet's
`{unimplementedService}/{unimplementedMethod}` catch-all, so the request was routed away
from the deploy handler and 404'd. The deploy API on its own does not reproduce it.
The handler now reads the optional body itself: no body (absent or zero-length) is the
headless form, a non-JSON body answers 415 and malformed JSON 400, as the framework's
binding did. New DeployApiBodylessPostTests compose the deploy API with the mesh gRPC
services and drive real HTTP; with the fix reverted the two bodyless cases fail with 404.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.Telemetry(.Serilog) 0.1.0 -> 0.3.0. The OTel Resource now names the
node: site.id = its application Cluster id (the cluster-<ClusterId> role suffix,
omitted on a legacy node), node.role = its plain roles in canonical order
(admin / driver / admin,driver), service.version = the entry assembly's informational
version (the git SHA). Program passes its resolved EffectiveRoles. The whole
OtOpcUa:Telemetry section goes through the shared ApplyConfiguration, so SiteId /
NodeRole / ServiceVersion / ResourceAttributes:* are operator overrides, and
Exporter=Otlp now also sends the Serilog logs to the same collector.
ObservabilityResourceIdentityTests cover the mapping and the built container's
Resource. Host.Tests 288/288, Host.IntegrationTests 325/327 (2 skipped).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ZbTelemetryOptions.Meters/ActivitySources are strict allowlists. The host named only
ZB.MOM.WW.OtOpcUa and the LocalDb meter, so the in-process Galaxy driver's galaxy.* spans
and its event pump's galaxy.events.* counters (both on "ZB.MOM.WW.OtOpcUa.Driver.Galaxy")
were silently dropped from /metrics and OTLP. Found exercising OTLP end to end against a
collector (scadaproj#13).
- Galaxy driver: public GalaxyDriverTelemetryNames (MeterName / ActivitySourceName);
GalaxyTelemetry and EventPump reference it.
- Host: ObservabilityExtensions.ObservedMeters / ObservedActivitySources include it.
- Host.IntegrationTests: ObservabilityAllowlistTests assert on the listener side (span
sampled, instrument enabled) with an unregistered-name negative control; dropping the
Galaxy entries fails 3 of 8.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.LocalDb and .Replication at 0.3.7 (published 2026-11-12).
Wire-compatible; no code change needed.
- 0.3.6 (scadaproj#36): the library sync service enforces
LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
its refuse-every-stream-when-blank rule; doc comment updated. docker-dev
site-a pair: both nodes now read one variable,
OTOPCUA_SITE_A_LOCALDB_SYNC_KEY (same dev default as before), so an
override can never split the pair.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
node offline past TombstoneRetention no longer brings a pruned
deployment back. The ledger grows while a peer is down; optional
TombstoneHardCap (documented in Configuration.md) bounds it.
Runbook, Configuration.md and the cluster-topology rule updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK memoizes INodeManager3.ValidateEventRolePermissions per (item, event type,
SourceNode). #826 kept it fresh by reflecting into the private
MonitoredNode2.InvalidatePermissionCaches (plus a detach/re-attach fallback). Since #855
condition events never take that channel, and the search for #878 found nothing
production-raised that does except a condition's SDK method audits, which reach it only
with auditing on, SignAndEncrypt and a SecurityAdmin session (nothing grants that role).
Under enforcement the channel verdict is now a flat deny for items on our nodes, so no
cached verdict depends on a grant, and InvalidateFolderEventVerdicts, the reflection,
the fallback, its counter and the guard test are removed. Method audits still reach
Server-object items, decided per event.
Tests: FolderEventVerdictInvalidationTests -> FolderEventVerdictFreshnessTests; new
revoked-grant test (positive control: a still-granted session) and method-audit test
(SecurityAdmin mapped for the test; positive controls: Server-object item, flag off);
the plain-event test runs each mode on a fresh server. Mutation-checked.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A notifier folder read one event source, its NodeId identifier (a RawPath or an EquipmentId). AVEVA stores a
Galaxy object's A&E history under the object's tag_name (Source_Object), so AVEVA's own alarm and event
history reached no production folder. Its lower-case property keys were also missed, so every row read
Severity 1 with no ConditionName.
Sources. A folder's event history now reads its own source plus the Galaxy object of every native condition
that fires on it live: those parented under it and those WireAlarmNotifiers wired to it. Live and history use
the same membership. The object comes from the condition's Galaxy alarm reference, its raw tag's attributeRef,
up to the first '.':
- The composer reads it with the driver's own parser (GalaxyAttributeRef, shared with GalaxyDriver), only for
a GalaxyMxGateway tag with an alarm (RawTagPlan.NativeAlarmReference, part of plan equality).
- The applier binds it after materialising the condition (IOpcUaAddressSpaceSink.BindNativeAlarmHistory,
forwarded by the deferred sink). A rebuild or the condition's removal forgets it.
Each object is read with the #882 rule: its own events and <Object>.<Attribute> rows, never a prefix sibling.
Fan-out. historiangw's ReadEvents filter is a single predicate, so there is one request per source; the wire
contract is unchanged. EventSourceFanOut merges the reads as one source:
- order is (time, source position, arrival), reversed for a reverse read; ties are never re-sorted by EventId,
so the boundary skip stays stable;
- duplicates by EventId are dropped and the merge is cut to the cap;
- a truncated source bounds the page at the time it stopped.
Paging, continuation points (bound to the whole source set), the walked-back reverse read and the tie-cluster
over-fetch are unchanged. A folder with one source is read exactly as before.
ACL. Every source is read inside the folder's own HistoryRead, after AuthorizeHistoryRead (#826). A denied
folder reads nothing and fails closed.
Mapping (EventMapper, property names now case-insensitive):
- Severity: an AVEVA row's priority goes through GalaxyAlarmPriority, the table live Galaxy alarms use
(MxAccessSeverityMapper now delegates to it), so history matches live. Otherwise severity (OtOpcUa's own
rows, which AVEVA stores lower-cased), clamped. AVEVA's own 1-4 severity is ignored. The table ranks a
higher priority as more severe, which inverts AVEVA's convention: filed as #887.
- An AVEVA row whose source_conditionvariable names a native condition that fires on the folder takes that
condition's SourceNode and live ConditionName. Otherwise ConditionName is alarmname or the condition
variable's attribute.
- alarm_id/alarmid map to AlarmId.
Tests:
- GalaxyEquipmentEventHistoryWireTests (4, real SDK session, production Gateway backend, AVEVA-shaped fake
gateway): equipment EqA backed by objects Oven_007 + OvenDoor_07. Covers object and attribute events from
both objects, the prefix sibling excluded, cross-object and in-object ties, forward paging at 1-4 a page,
reverse whole and paged, SourceNode/ConditionName/Severity, the raw device folder, ACL deny with no backend
call plus a granted positive control, and a rebuild forgetting the bindings.
- EventSourceFanOutTests (8), EventMapper AVEVA cases (11), GalaxyAttributeRefTests (16), a composer test and
an applier ordering test.
- Mutation-checked, 17 mutants, all killed. Covered: parent-only membership, no scope, a case-sensitive
condition match, no condition rewrite, no truncation bound, EventId tie order, no sort, no cap, no stop at
the bound, unreversed source order, no dedupe, no '.' boundary, priority ignored, case-sensitive keys, no
applier bind, no deferred forward, and a composer that ignores the driver type. The forget-on-rematerialise
guard survives as belt-and-braces: the rebuild clear covers it.
- Live, read-only, no WinRM, vd03 HistorianGateway ea9f21c: equipment folder EQ-886-ZipsenLeft (objects
DevWinPlatform + Left_001). It read 0 events before and 92 now, equal to the unfiltered read's rows for both
objects. 1/7-event pages and reverse whole/5-event pages agree, and 73 rows resolve to their live condition.
diff_runs vs the #882 run: 221/226 identical. The 5 that differ are expected: DevWinPlatform Severity
1 -> 750/1000, and #881 capabilities.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ConfigurationBinder appends configured collection items to an existing list, so
the pre-filled [None, Sign, SignAndEncrypt] default made "drop None" a no-op:
every deployment that listed only the secure profiles still advertised an
unsigned/unencrypted None endpoint.
EnabledSecurityProfiles is now null by default; ResolveEnabledSecurityProfiles()
applies the unchanged three-profile default only when nothing is configured.
A set-but-empty list fails startup validation and BuildSecurityPolicies no
longer falls back to a None endpoint on empty input (it throws). Sweep of all
options classes found no other non-empty pre-filled collection default.
Tests: options binding (indexed, JSON, env forms, default, empty, bad name), a
booted-server wire test (no None endpoint, None connection refused with
BadSecurityPolicyRejected, secure connection works; default case is the control).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The SDK starts Server.ServiceLevel at 255 (full service), and OpcUaPublishActor publishes nothing
until the node's first redundancy snapshot arrives, so a restarting pair node advertised 255 for the
~10-13 s it took to join, out-ranking the real Primary (250) for any client that picks the highest
ServiceLevel.
OtOpcUaSdkServer now writes StartupServiceLevel = 0 in OnNodeManagerStartedAsync: after the SDK's
CreateServerObjectAsync wrote 255 and before the server goes Running, so no session can ever read
255. 0 is the design's existing Down / Detached tier (ServiceLevelCalculator returns 0 for a member
that is not Up/Joining, and a node with no snapshot has no role yet). It is below the SDK 2
session-headroom floor (200), so it stands through session churn before the #884 claim, and the
claim's bind replays any level already requested. Single-node deployments are unaffected beyond
the window: the first snapshot publishes 250 as before.
New Node_with_no_requested_level_advertises_the_startup_level_until_one_is_published: reads 0
before the bind, 0 after a bind with nothing to replay, then 250 once published. Without the
startup write it reads 255 on the first session (mutation-checked). docs/Redundancy.md updated for
#884 and #885.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
SDK 2.0.1 recomputes Server.ServiceLevel from session headroom on every session create/close and
overwrites any value >= 200 unless a provider has claimed it through
IServerServiceLevelControl.ClaimServiceLevelControl. SdkServiceLevelPublisher never claimed it, so on
the docker-dev pair both nodes read 255 as soon as a client connected while their own metrics showed
250 (Primary) and 240 (Secondary); the Stale tier's 200 was lifted to 255 too. Levels below 200 (100,
0) were untouched, which is why the bootstrap 100 showed briefly.
The publisher now claims the value at construction and writes through the SDK's writer; a server
without the claim (test doubles) keeps the diagnostics-guarded write. New
Published_level_survives_later_sessions (250/240/200) fails without the claim (reads 255) and passes
with it.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Every Boolean under Server/ServerCapabilities/HistoryServerCapabilities read
false (the SDK default) even though data and event history were served, so a
spec-following client skipped HistoryRead entirely.
Each backend now declares what it serves through its own interface:
- IHistorianDataSource.ReadCapabilities (Data / Events / ServerTimestamps):
Gateway Data|Events; SqlServer + QuestDb add ServerTimestamps (they store
ServerTimeUtc, the Gateway stamps read time); Null declares None. A
reflection guard requires every production source to declare it itself.
- IHistorianDataUpdater.Capabilities (required member): SqlServer and QuestDb
declare Insert/Replace/Update/DeleteRaw/DeleteAtTime; the Gateway provider
has no updater.
The node manager writes the flags on every source/updater assignment
(AdvertisedHistoryCapabilities), and the same declarations drive dispatch:
an update the updater does not declare is BadHistoryOperationUnsupported
without reaching it, the HistoryWrite AccessLevel bit follows a non-empty
declaration, and event-history notifier promotion follows the Events flag.
Event and annotation update flags stay false (nothing dispatches them).
Tests: booted-server wire test per backend + Null + a SqlServer->Gateway
rewire pairing on one session; dispatch gating unit test; the #801 live
gate now asserts the Gateway flags (not run here).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
OpcUaApplicationHost left ServerConfiguration.MaxRegistrationInterval at the
SDK default (30000) with a null RegistrationEndpoint, so SDK 2.0.1's
StandardServer armed its registration timer against opc.tcp://localhost:4840,
which on an OtOpcUa node is the server itself (no LDS). It answered
BadServiceUnsupported and the SDK retried forever, visible since #874.
Registration is now opt-in: ApplyDiscoveryRegistration sets
MaxRegistrationInterval = 0 (the value that keeps StandardServer from arming
the timer) unless OpcUa:DiscoveryServerUrl is set, in which case the server
registers against that URL every OpcUa:DiscoveryRegistrationIntervalMs
(default 30000) over SignAndEncrypt/Basic256Sha256, like the SDK's own
default endpoint. The validator requires an absolute opc.tcp:// URL and a
positive interval when the URL is set. Documented in Configuration.md and
OpcUaServer.md.
Co-Authored-By: Claude <noreply@anthropic.com>
- Host reader: the frame read and its handling are caught separately. Only the
read's I/O exceptions mean the pipe ended; anything HandleFrame's inline
completion work throws now kills that worker as ProtocolFault (outstanding
runs -> worker restarted / BadResourceUnavailable) instead of escaping a
dedicated thread (process crash) or being swallowed as a clean end (an
InvalidOperationException used to be, leaving the worker unread until a Crash
2 s later).
- Worker reader: the same catch-all around HandleFrame (exit ProtocolFault).
- Worker: Cancel() runs on the pool, so a script's blocking
ctx.CancellationToken callback cannot stop the worker reading its pipe.
- Host write-stall check: ScriptWorkerFrameWriter.WriteInProgressFor; the
heartbeat monitor kills a worker whose pipe write has been blocked longer than
max(10 s, silence limit) as KilledUnresponsive, which fails the write and
releases the producer (seam submit, deploy load, Cancel). Fixes the wrong
'heartbeat monitor ends it' comment on the pipe buffer quota.
- StopAsync sends Shutdown off the caller so its 1 s budget holds on a full pipe.
- WindowsBlockingPipeStream refuses the async surface (Stream's base versions
serialise a write behind a waiting read).
- Tests: PipeFaultTests (throwing completion callback -> ProtocolFault and a
BadResourceUnavailable sibling, shard serves again; a worker that stops reading
but heartbeats is killed within the bound and the blocked producer returns,
via the Hello.test_stall_reader_on_diag seam); BlockingPipeStreamTests dispose
with a write blocked on a full pipe, and with a read and a write both pending;
Category=ScriptWorker on the protocol, handshake and pipe test classes.
Mutation checks: no stall kill -> stall test red; catch-all without the fault
-> callback test red (Crash).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A .NET pipe opened Asynchronous completes even its synchronous Read/Write
through the thread pool's completion port, so every frame on Windows cost a
completion packet plus a pool work item (and the pool's spin-wait). New
WindowsBlockingPipeStream issues its own overlapped ReadFile/WriteFile with the
OVERLAPPED.hEvent low bit set (no completion packet) and waits on that event on
the calling thread; reads and writes stay concurrent, unlike a synchronous
handle. ScriptWorkerFrameCodec.BlockingStream picks it on Windows and the pipe
itself on Unix, so both ends now use the dedicated blocking reader + synchronous
combining writer on every OS (UseBlockingIo and the async read loops are gone).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Gate F (#761) against the VD03 gateway: Galaxy initialize (session open +
platform seed) took 4–5 s against Tier A's 5 s Initialize timeout, so 10 of 40
restarts failed, paid a 10 s retry and left a killed gateway worker.
DriverFactoryRegistry.Register takes optional capabilityDefaults per driver
type (GetCapabilityDefaults); DriverResilienceOptionsParser layers them over the
tier table and under the per-instance ResilienceConfig; the Host wires the
registry into DriverCapabilityInvokerFactory. Galaxy registers Initialize =
20 s (no retry, breaker 2), under the actor's 30 s pessimistic bound.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The host handed the SDK a provider-less telemetry context, so SDK-side
refusals — identity-token decryption (the SDK 2 RsaEncryptedSecret clock
window, #873), certificate validation, secure-channel faults — left no server
log line (found on windev during the #800 Windows validation). The hosted
service now creates one ServerSdkTelemetry context from its ILoggerFactory
(Warning and above, SDK categories, rate-limited 20/min per category) and
shares it between OtOpcUaSdkServer and the ApplicationInstance; released on
dispose. Test: an untrusted client's rejection reaches the captured log.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Since #855 condition events are queued to folder event items under the node
manager Lock. At a full DiscardOldest queue the SDK's EventQueueHandler
dequeues the head of EventMonitoredItemQueue, a List.RemoveAt(0) shift of the
whole queue (~1 us/item at QueueSize 1000, ~10 us at 10000).
OtOpcUaSdkServer.CreateMonitoredItemQueueFactory now wraps the SDK factory in
RingEventQueueFactory: event queues are RingEventMonitoredItemQueue (O(1)
discard), data-change queues, durable requests and disposal go to the SDK
factory unchanged. A factory set through StandardServer.MonitoredItemQueueFactory
is used as-is.
The SDK queue is storage only (decompiled 2.0.1): DiscardOldest, overflow-event
positions and discarded/overflow counters live in EventQueueHandler, which is
unchanged. The ring reproduces QueueSize 0 initial + BadInternalError enqueue,
drop-oldest at Count == QueueSize, FIFO dequeue, SetQueueSize keep-newest /
keep-oldest, the newest-1000 duplicate window via IsEventRaisedFor, no-op
Dispose. RingEventMonitoredItemQueueParityTests drive both queues, and the SDK
EventQueueHandler over both factories, with the same random sequences.
FolderEventDeliveryPerfTests gains Ring_vs_sdk_event_queue (A/B in one process
via the internal OtOpcUaSdkServer.RingEventQueues switch).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The sweep threshold never rises above the 4096 cap (a threshold the count
could not reach stopped sweeping, and sharing, for good); at the cap a
sweep for dead shapes runs after every 256 refusals.
- The per-report "queued" mark is taken after the filter group resolves,
so a failure there leaves the item unmarked.
- The bucket key also carries TypeDefinitionId and IndexRange.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- The filter-group registry holds groups weakly (the filter->group table is
the only strong reference), sweeps dead entries amortised and counts only
live shapes against the 4096 cap; filters past the cap are counted on the
new meter otopcua.opcua.condition_events.filter_groups_unshared.
- A group validates its where clause too (restored, never-validated filters
evaluate like validated ones).
- Test: value-equal filters cost one projection per locale set per report.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Filter-group buckets include the where clause's (value-based) hash, so
per-source filters sharing a select list are not all scanned under Lock.
- Locale keys are cached per session array.
- Tests: a notifier reached twice in one walk (Line above two wired
equipment) gets each event once on both paths (fails without the
per-report set); queued events re-projected by a modify are identical on
both paths; locale keys unambiguous and where clauses hash by value.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
A pending local ack can outlive its own delivery (sources that send no
Acknowledge back; a Clear now only carries it through), so a later failed
re-ack — or a stale Galaxy confirmation timing out after the alarm re-occurred
and was acked again — could revert an ack it does not own. MarkAcknowledged
now issues a sequence (0 when the ack changed nothing), carried on
RouteAlarmAck and echoed on AlarmAckFailed; TryRevertAcknowledge reverts only
on a matching sequence.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
NativeAlarmProjector dropped the pending local ack on EVERY driver transition,
so an operator ack followed by the alarm returning to normal (a Clear carrying
the optimistic ack) inside the confirmation window made a later "not
confirmed" report a no-op — the condition stayed inactive+acked while Galaxy
showed UNACK_RTN. Only a transition that states the ack (Acknowledge,
ActiveAcknowledged, or an un-acking Raise/Retrigger/InactiveUnacknowledged)
now settles it; a Clear / same-message Raise / Unspecified update leaves it
revertible, and the revert projects inactive+unacked.
Tests: projector + host end-to-end (ack, Clear, confirmation fails -> inactive
+ unacked); tightened the confirmed-ack positive controls; pinned that the
confirmation receives the same request instance as AcknowledgeAsync.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Start the ack confirmation with Task.Run + ConfigureAwait(false) so it never
depends on the ReceiveAsync handler's ActorTaskScheduler (the PostStop
abandon-and-revert report must run after the actor stopped).
- DriverInstanceActorAckConfirmationTests: a confirmation that faults after the
handler returned is reported; one pending at actor stop is reported; a
confirmed ack reports nothing (positive control).
- Deterministic tracker matching test (Raise/Clear/Retrigger/other alarm do not
confirm; case-insensitive match does).
- AdminUI min/max on the confirm-timeout field; Galaxy.md documents the
reconnect, UNACK_RTN-as-Clear and RPC-budget limits.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- RefreshDirectItemsWithCut keys its "decided off Lock" set per (item,
condition), so a condition one item's walk saw is still decided for another.
- The deferral warning no longer blames slow folder subscribers (no folder of
ours defers since #855).
- FolderEventFloodWireTests' removal test now asserts the folder item got
nothing and the Server object exactly the pre-stall event (it passed with
the post-stall liveness re-check deleted).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.
- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
(default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
after AcknowledgeAsync, off its mailbox and outside the resilience
pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- Both plain ConditionRefresh paths (Server object and folder) skip a
condition a deploy removed or replaced between the off-Lock walk and the
queueing (the SDK base did not either; only the cut path checked).
- A failure delivering at one notifier hop is logged and the walk continues
to the remaining hops; a failure deciding the Server-object items withholds
only them.
- Test: a condition removed during a plain refresh (folder + Server object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay's verdict refuses a condition a deploy removed or replaced
since the cut (Server-object and folder), so the cut path cannot replay a
ghost or judge a replaced scripted alarm on its old folder.
- DeliverToNotifierItems resolves the #826 decision's item-independent part
(live condition, notifier check, scopes) once per hop; only the grant and
SDK role checks stay per item.
- A deferred delivery's drain starts on the pool, never inline under Lock.
- Test: a condition removed between the cut and its replay (folder + Server
object).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review findings on 02503a4a (ConfigPublishCoordinator):
1 (HIGH) A recovered deployment whose every expected ack was already in
NodeDeploymentState (a seal whose SaveChanges failed, a fused node's own row)
was never completed: inside the deadline it waited for acks that had landed,
past it HandleDeadline's "race" guard returned without sealing — stuck in
AwaitingApplyAcks, refusing every later deploy. ApplyRecovery now completes
it (Seal / PartiallyFailed), and HandleDeadline completes a full ack set
instead of returning.
2 (MED-HIGH) The past-deadline recovery of a populated expected set ran
HandleDeadline inline before the stash replay, discarding on-time acks queued
during the outage. It is now self-sent, behind the replay. The zero-rows
past-deadline branch stays inline on purpose: no ack can count there, and a
self-sent deadline would time out a queued dispatch of that same row right
after publishing it.
3 (MED) A failed PersistNodeAck threw, restarted the actor and dropped the ack
(driver-only nodes never write their own row and do not resend). Acks are now
persisted before being counted, and a failed persist is re-queued on a timer.
The class remarks document that the Loading stash is node-local (lost on a
hand-over; queued dispatches are recovered from their rows).
Minor: MarkReady/Become(Ready) only after the recovery applied; a throw while
applying is a failed load (backoff) rather than a tight restart loop; a queued
dispatch matching the recovered Dispatching row logs Info, not the "lost
dispatch" Warning.
Tests (ConfigPublishCoordinatorDbOutageTests): fully-acked recovery seals /
partially fails inside and past the deadline (4 cases); queued ack applied
before a past-deadline recovery; failed ack persist retried. All 6 fail on
02503a4a.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- QueueFrozenReplay re-asks each item's verdict right before each entry's gate
hold (Server-object and folder cut paths): a grant revoked between the
pre-Lock verdicts and the replay no longer lets the replay through. A denied
entry is also removed from the cut's re-sent set.
- ReportEventWithoutBlocking logs and returns on a delivery that faulted
inline instead of booking it as a deferral episode (which ran the drain and
ClearStaleRefreshCuts under the event-report gate while Lock was held).
- OnConditionReportEvent isolates failures per Server-object item.
- Tests: revocation between cut and replay (folder + Server object), throwing
hop not booked as a deferral; production-overload sampling inside Lock.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
When the data starts inside an interval (no start bound) and its first value is
Uncertain, the frozen 1.5.378.182 AverageAggregateCalculator integrated the
uncovered span as a ramp from 0 and divided TimeAverage by the whole interval:
GetRegionsInValueSet (ignoreBadData, sloped) overwrites the BadNoData start
region's status with UncertainDataSubNormal in its one branch without a not-Bad
guard. Live AggregateFilter therefore disagreed with HistoryRead and Part 13
§5.4.3.6/§5.4.3.8 ("No Start Bound: Calculate Partial Interval"); QP: Total
188.96 vs 163.96, TimeAverage 9.448 vs 10.931. SDK 2.0.1 stock does the same.
Owner-approved 2026-11-11.
- New app-owned CoveredTimeAverageAggregateCalculator subclasses the baseline
and recomputes only TimeAverage/Total, restoring the Bad status of a region
that starts at a Bad point (only the start bound can). Only such an interval's
value changes; status, timestamps, type and bits are the baseline's. Stepped
and TreatUncertainAsBad=true never reached the defect. CompatibilityAggregate-
Factory routes exactly those two IDs; copied SDK files untouched.
- Golden test: 0 fixture vectors change (every data set has a usable value at
-10 s), asserted. New sweep over late-start variants of every data set (cut
+5 s / +25 s, first non-Bad sample Uncertain, all directions/stepping/configs/
incremental): 150 / 142 of 384 change, each pinned to a baseline-derived
oracle (untouched baseline's value with that sample Good, its status/bits on
the variant itself) - never our calculator. Plus a hand-computed case.
- Live-vs-HistoryRead parity (TimeWeightedAggregator) forward + reverse, no
masks, with discrimination against the pre-#821 baseline; QualityMix now
compares QP's TimeAverage/Total with the live oracle on both SQL backends;
booted-runtime facts test expects 163.96 / 10.93.
- README recorded decision, ServerAggregateConfiguration remarks, oracle
remarks, docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.
- PreStart now only subscribes and kicks off an async recovery load piped back
to the actor. A failure retries with exponential backoff (1 s doubling to
30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
stashes every message (DispatchDeployment logged as queued, ApplyAck too);
they are replayed in order after the recovered state is applied. Recovery
decisions (re-dispatch / time out) now run inline before the replay instead
of via self-tells, and a duplicate dispatch of the in-flight deployment with
a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
"config-publish" ready-tier health check: Degraded while loading (with the
attempt count and last error), Healthy when ready or not hosted here.
Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events reaching one of our notifiers no longer go through the SDK
MonitoredNode2 channel: the walk serves the attached items itself, per item and
per event (audit guard, uncached INodeManager3 verdict, item gate/cut,
QueueEvent), as the Server-object path does. No notifier of ours defers any
more; the #838 deferral queue and the #854 cut stay as the safety net and
folder items share the cut (RefreshDirectItems, flag on and off). An event
whose walk suspends is marked deferred before it suspends.
Tests produce deferral with a notifier-stall probe instead of holding a folder
item's SDK lock (which would now block the reporter). New
FolderConditionRefreshWireTests; design note in docs/plans.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The frozen 1.5.378.182 StatusAggregateCalculator counts Uncertain regions as
Good under TreatUncertainAsBad=false, so live AggregateFilter PercentGood /
DurationGood disagreed with HistoryRead (Part 13 §5.4.3.31/§5.4.3.33, Mantis
0006166) whenever Uncertain data was present. Owner-approved 2026-11-11.
- New app-owned GoodOnlyStatusAggregateCalculator subclasses the baseline and
replaces only the PercentGood/DurationGood computation (Good regions only);
CompatibilityAggregateFactory routes exactly those two IDs to it. The copied
SDK files are untouched; the other 35 aggregates are byte-identical.
- Golden test: the 64 changed vectors (38 PercentGood + 26 DurationGood) are
exempted exactly — those whose baseline output differs from the same case's
baseline TreatUncertainAsBad=true twin — and pinned to that twin (baseline
evidence, not regenerated from the implementation); plus a full Good-only
pin over all 352 PercentGood/DurationGood vectors and a hand-computed case.
- Live-vs-history parity over Good/Uncertain/Bad mixes (no masks), with a
discrimination check against the pre-#809 baseline; the booted-runtime facts
test now expects 8000 ms / 40 %.
- SdkAggregateOracle: Run/RunLive = the registered factory; RunSdkAsIs = the
untouched baseline; its Part13GoodCalculator duplicate is removed.
- README (recorded decision), ServerAggregateConfiguration remarks,
docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11: /fleet/diagnostics "Faulted devices" counts DISTINCT
devices across the nodes of the selected node's cluster, Faulted state only.
- DriverHostActor: FaultedDevices = HostState.Faulted only (Stopped no longer
counted); new StoppedDevices + FaultedHostNames. Counting extracted to the
testable DiagnosticsFromHealth.
- DriverInstanceDiagnostics: StoppedDevices (default 0) and FaultedHostNames
(default null) are additive with defaults, so a pre-#786 sender's snapshot
still deserializes over Akka's default JSON serializer (pinned by a test).
- FleetDiagnosticsAggregation (pure): identity key = (driver-instance string
id, HostName), case-insensitive; a single-host driver / nameless pre-#786
count is combined by MAX across nodes, never summed. Cluster peers come from
enabled ClusterNode rows of the selected node's ClusterId.
- FleetDiagnostics.razor: tile fetches the cluster peers and shows the
distinct count + "n of m nodes answered"; per-driver table gains a Stopped
devices column.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11: certificateValidation.validateServerHostName stays
opt-in (unset = off). The AdminUI toggle moves from the "Subscriptions, limits &
discovery" panel to the Security panel beside mode/policy, with help text on the
NAT/IP caveat, IPv6 zone ids and the loose first-label match. Binding unchanged
(tri-state, "Default (off)" leaves the key absent). OpcUaClient.md states the
decision; drivers rule updated; source-scan test pins placement + caveats.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The Controls panel primed its per-node view from the snapshot store with
ReceivedUtc = now, so a node that stopped publishing before the dialog
opened looked live for the first 30 s stale window.
InMemoryDriverStatusSnapshotStore now records each snapshot's LOCAL
receive time (TimeProvider-injectable) and exposes it via
GetAllWithReceiveTime(); the panel primes through the pure seam
DriverStatusByNode.Prime/PrimedReceiveTime, which keeps the store's
receive time (clamped to now; still skew-proof). A store that does not
track one (default interface impl -> null) falls back to the publish
time only when that is already older than the stale window, else now.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Joseph Doherty