#641 keeps the projector's prior state across a deploy instead of resetting it to the
acked default. The projector never saw a client's ack (the SDK applies it to the node and
the host only routes it), so a Clear after ack+deploy now projected Acked=false and
un-acknowledged the operator's ack - previously masked by the per-deploy reset.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
DriverHostActor cleared NativeAlarmProjector's per-condition prior state on
EVERY apply, so after any deploy a driver Acknowledge of a still-active alarm
projected Active=false (writing it inactive) and a Clear of an unacknowledged
alarm projected Acked=true (silently acknowledging it). The projector is now
pruned to the conditions the new composition still carries (RetainOnly).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
A full RebuildAddressSpace recreated every native Part 9 condition inactive +
acknowledged, and when the alarm-tag set was unchanged (e.g. one
UnsTagReference added to an alarm-bearing raw tag) nothing re-told the new node
its state: the host does not re-subscribe for an unchanged set, and Galaxy /
OpcUaClient have no snapshot to replay anyway. Active alarms read normal until
their next device transition.
OtOpcUaNodeManager now captures each NATIVE condition's live state (from the
node, so a client-applied ack is kept) before the rebuild clears, restores it
when the same id is re-materialised native in the same apply, and reports ONE
condition event per restored off-normal condition at the new
IOpcUaAddressSpaceSink.CompleteAddressSpaceApply (after notifier wiring).
Unclaimed captures are discarded there, so a later re-add is fresh. Quality is
carried silently for normal conditions too (it is only re-annotated on a
connectivity edge). Scripted conditions are untouched (#487 re-asserts them).
OpcUaPublishActor calls CompleteAddressSpaceApply at the end of every apply.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- Unavailable keeps an ACTIVE condition active: quality-only BadNoCommunication; on data
return same statement => quality-only Good, Normal => Clear, different => Retrigger.
- A changed alarm reference set (alarm-bearing delta = full rebuild) clears the whole
last-told record so every active condition is re-asserted; the same set keeps it.
- Quality-only events go through the host's AlarmQualityUpdate path, never a projected
snapshot (which could ack a retained unacked condition or rewrite message/severity).
- Deterministic tie-break between equal-level conditions (ordinal nativeCode, then type).
- Dedup key is level+nativeCode+nativeSeverity+qualifier; a text-only change updates nothing.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
AlarmEventArgs gains optional OpcUaSeverity (supersedes the four-bucket mapping, clamped
1..1000) and QualityStatusCode (projected onto the condition's Quality). An Unspecified-kind
event carrying a quality is a quality-only annotation: the host updates the condition but
publishes no alerts row. Both default null, so every existing native source is unchanged.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.
- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
real invoker's bracket across the removal and signals when its late writes are done (positive
control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
(incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
host test.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The drain gate's 'is my queue-sharing peer Primary?' used only the host of
LocalDb:Replication:PeerAddress, which the passive half (ApiKey +
SyncListenPort) does not set - so as Secondary it kept draining beside its
Primary partner and every alarm event reached the historian twice.
DriverHostActor takes a sharesAlarmQueueWithLocal predicate used when no
peer host is known; WithOtOpcUaRuntimeActors builds it
(BuildAlarmQueuePeerPredicate) for a SharesRowsWithPeer node with no
PeerAddress as 'a member of my own cluster-{ClusterId} mesh other than
me'. A node sharing no rows, or with no mesh role, gets none and keeps
draining. AlarmHistorian.md / Redundancy.md / pair-replication runbook
updated.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
MINOR-3: new DriverHostActorPrimaryRoleSourceTests case drives the
deny flip with a real ClusterEvent.MemberUp on the EventStream (how the
cluster's domain-event publisher reaches a Cluster.Subscribe subscriber)
instead of a direct RepublishPrimaryVerdict tell; it fails when the
Cluster.Subscribe(IMemberEvent) call is deleted.
MINOR-2: the published verdict MIRRORS the live gate; before this node has
joined its member count is 0 and both allow (seconds; up to the bootstrap
guard's 25 s probe), so the Sparkplug primary-host claim can be held on
both nodes in that window (native ack is unaffected - view and route
agree). Behaviour deliberately unchanged: denying until self is Up would
change the live gate too and delay writes/acks on a single-node cluster's
own join. Comments + AlarmTracking.md / Redundancy.md / Mqtt.md corrected.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- Native-condition Acknowledge on a non-Primary node returns BadInvalidState (nothing
routed/applied/reported) via NativeAlarmAckPrimaryGate, fed from
IDriverPrimaryRoleSource by OtOpcUaServerHostedService.
- A re-ack carrying a comment reports one comment event (Acked/Confirmed unchanged).
- TimedShelve applies its requested duration and the scripted projection carries
UnshelveAtUtc, so ShelvingState/UnshelveTime counts down instead of ~24.8 days.
- Native re-ack still routed to the driver (deliberate: documented retry path), pinned.
- AdminUI /alerts client ack/shelve path test; docs/AlarmTracking.md decisions incl.
accepted SDK EventsMonitored residual (item 6).
- Also #599/03/S20: RemoveEquipmentSubtree severs inverse Organizes edges.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- AnnounceAddedNodes raises NodeAdded in the Raw realm for the existing parents of
added raw containers/tags (ComputeRawAddAnnouncements); Uns side unchanged.
- ApplyPureRemove removes UNS reference variables BEFORE the equipment subtree, so a
reference the subtree's id-prefix scope reaches no longer ratchets to a full rebuild
(03/P5). A NotSubsumed filter was not used: in production a reference NodeId is
Area/Line/Equipment/Name, outside the subtree scope, so filtering would leak it.
The matching RemoveEquipmentSubtree severing (03/S20) is in the node-manager commit.
- UnsReferenceBackingParityTests: reflection guard discovering the classifier's surgical
raw-tag whitelist and requiring a same-typed mirror field (03/U11).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The drain gate counted SyncListenPort; a port with no ApiKey shares nothing (fail-closed
passive endpoint), and an ApiKey-only passive node shares its queue but was never gated.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.
Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.
Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.
Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
The alarm subscribe gated on 'a handle exists', which is correct for an idempotent
re-push and wrong for a deploy that widens/narrows the set. Combined with the
spawn-time desired set, a respawning deploy could latch the PREVIOUS revision's
alarm refs for the life of the connection; sourceNodeIds-filtering drivers
(OpcUaClient/AbCip/Focas) would never deliver the new alarm.
Seen RED: LastSubscribedAlarmRefs stayed [alarmX] after a deploy widening to
[alarmX, alarmY].
Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
The batched ingest's append-failure catch had zero coverage, and two doc
statements overstated the loss: the Warning said the burst's values were
"dropped", and RecorderStatus.OutboxAppendFailures said "the value was
dropped, not recorded". The outbox contract is prefix-never-orphan, so a
partial-prefix failure leaves entries durably queued that drain normally.
- New test pins the branch: one 3-item burst whose group commit commits a
1-entry prefix then throws => OutboxAppendFailures == 1 (per burst, not
per value), TotalRecorded == 0, actor alive, and the prefix still reaches
the writer via the post-failure DrainTick (drainInterval 30s, so the
steady cadence cannot account for it). Falsified twice: counting per item
and dropping the drain nudge each turn it red.
- Second test pins the OperationCanceledException filter from its
observable side - a cancel that is NOT shutdown is metered, not swallowed.
Removing the 'when (_lifetime.IsCancellationRequested)' turns it red. The
shutdown side has no cheap seam (the counter dies with the actor).
- Docs corrected: OutboxAppendFailures counts FAILED BATCH APPENDS, one per
burst whatever its size (a granularity change from the per-value era), and
is not a count of values lost; TotalRecorded under-counts when a commit
fails after a partial prefix. Log message says "failed to append" and
names the prefix. No functional change beyond the message string.
Also folds in the DrainBatchSize default alignment behind 0298a0b9: the
recorder's two ctor defaults and its non-positive fallback still said 64
while ContinuousHistorizationOptions.DrainBatchSize now defaults to 1024.
No test pinned the old 64.
C1 — a non-rebuild apply reporting failedNodes > 0 no longer advances _lastApplied.
Pre-delta-materialisation the full passes re-ensured every node on every apply, which
silently repaired any prior swallowed sink fault. The delta path removed that accidental
self-healing: a failed node sat inside _lastApplied, never appeared in a diff set again,
and EnsureVariable early-returns on the ones that landed — so it was lost until an
unrelated change forced a rebuild. The advance also moved to AFTER the materialise
passes, so a throwing pass leaves _lastApplied at the previous composition too. A
rebuild still advances even with failures (the ratchet is its resync path).
I2 — a UnsReferenceDelta whose identity is unchanged (same NodeId, EquipmentId and
BackingRawPath) is now applied in place via UpdateTagAttributes(Uns) instead of
remove+recreate, so the mirror's MonitoredItems survive an operator historizing or
reshaping the raw tag behind them. Remove+recreate stays for identity-changing deltas
(re-point, effective-name edit).
I3 — RemoveVariableNode severs the inverse half of every outgoing Organizes edge, the
variable-node counterpart of UnwireAlarmNotifiers. Rebuilds used to sweep these up;
surgical applies made rebuilds rare, so they accumulated on the raw node for the life of
the process.
I5 — added UNS reference variables are announced (their equipment folder is the parent,
already the Uns realm). Raw adds still are not; the gap comment now says so honestly
rather than calling it neutral and pre-existing.
I4 + M1 — corrected two comments that no longer described the code: the "re-point →
rebuild" claim in ApplyPureRemove, and the "N lock acquisitions" rationale for the delta
passes (Ensure* early-return on a dictionary probe before the lock).
Joseph Doherty