- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
batched p99 <= 53 ms at 1,000.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverHostActor built ScriptedAlarmEngine without a scriptTimeout, so every
production alarm predicate fell back to TimedScriptEvaluator.DefaultTimeout
(250 ms) while virtual tags and Calculation tags ran on 2 s. History shows no
decision behind it: 250 ms is the generic Phase-7 A.4 default, the VT
evaluator moved to 2 s in F8b (219d10a2) and the alarm engine was simply never
given a value.
- New node-wide Scripting:RunTimeout (default 2 s, clamped to (0, 60 s],
non-positive -> default, each clamp a startup Warning) on ScriptingOptions.
- The VT evaluator and the alarm engine default to it; DriverHostActor now
builds the engine through CreateScriptedAlarmEngine, which passes it
explicitly. Calculation keeps its per-driver runTimeoutMs.
- Docs: ScriptEditor (config table), ScriptedAlarms, VirtualTags.
Tests: ScriptedAlarmEngineBudgetTests (a 600 ms predicate now completes and
activates; default and configured budgets), ScriptedAlarmEngineBudgetWiringTests
(the host-built engine), ScriptingOptionsBindingTests (bind + clamps). The
engine and wiring tests fail with the old 250 ms default restored.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.
DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
token cancel = Timeout
abandon = Timeout + min(Timeout/2, 2 s) (RunBound)
queue budget = RunBound
child replies <= queue budget + RunBound (ReplyBound)
host Ask = ReplyBound + 1 s
gateway Ask = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.
- DriverHostActor takes writeBounds + writeClock, spawns children with them,
and runs its Ask deadline as a token on the write clock (Infinite Akka
timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
host): the queue budget counts mailbox time too, and a write past it is
failed BadTimeout and never started - including on arrival. Stale queued
writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
style as ScriptingOptionsBinding): unparsable / non-positive -> default,
above 5 min -> clamped, and each tier whose default Write timeout exceeds
the write timeout is named in a Warning (its policy would never take
effect). The derived bounds are logged at Information. Per-instance
ResilienceConfig Write overrides arrive with a deployment, so
DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.
Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
an 8.5 s step and succeeds; a token-ignoring write is answered by the
child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
the queue budget is failed and never reaches the driver; positive control
at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
caught at startup with Tier B and C named (not A); bad values fall back
with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.
Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.
ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverInstanceActor.RunWriteAsync cancelled every write with a hard-coded,
undilated 5 s CancellationTokenSource, so writeTimeout / DefaultWriteTimeout
(then 10 s) only ever served as the pessimistic abandon bound: a configured
bound above 5 s never took effect.
The write's token is now cancelled at the configured write timeout; a write
that ignores its token is abandoned WriteAbandonGrace later (half the
timeout, at most 2 s) so a token-honouring driver still gets to answer the
cancellation itself. Both bounds run on an injectable TimeProvider
(Props writeClock, System by default).
DefaultWriteTimeout is now 5 s (cancel 5 s, abandon 7 s), not 10 s: the
host's write Ask is 8 s, and a write allowed to run past it would already
have been reported as a timeout and its node reverted while the device could
still take the value - the exact failure the off-mailbox write work fixed.
Production behaviour is therefore unchanged for the cancel (5 s) and tighter
for a token-ignoring driver (abandoned at 7 s instead of 10 s, i.e. inside
the Ask). DriverHostActor's Ask comment updated to match.
Tests (DriverInstanceActorConcurrentWriteTests, manual clock copied from
Ab.Transport.Tests into Runtime.Tests/Harness/ManualTimeProvider):
- a write held past 6 s under a 10 s write timeout succeeds;
- positive control: at 10 s the token is cancelled ("write timeout");
- a token-ignoring write is still pending at 10 s and abandoned (BadTimeout)
after the grace.
Mutation checks: a fixed 5 s token on the same clock fails the first test;
dropping the grace fails the third (and the second, where the abandon then
races the cancellation).
342a825d re-check: its held-write tests were written around the undilated
5 s cap. Connect() now defaults the write timeout to the DILATED presence
budget, so a held, explicitly-released write no longer races any undilated
deadline; tests about the bound pass their own. AwaitWritesEntered keeps its
20 ms interval (doc updated).
Runtime.Tests: 946 passed / 3 skipped / 0 failed at AKKA_TEST_TIMEFACTOR 3
and 1. Solution Release build: 0 warnings.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Residuals of #760:
1. An ack routed to a disabled / not-yet-spawned driver, or to a stubbed child
(driver does not run on this node, or its factory rejected the config), was
dropped and left the condition Acknowledged. DriverHostActor now reverts it
through the same NativeAlarmProjector.TryRevertAcknowledge path as a refused
ack.
2. A native ack arriving while the host is Stale was dropped before the projector
recorded it, so it could never be reverted. Stale now marks the whole
NativeAlarmAckPreflightView unavailable (SetHostUnavailable), so the node
manager refuses every native ack synchronously; BecomeSteady clears it.
3. (keyed sessions without a Galaxy identity) is not addressed.
Tests: DriverHostActorNativeAlarmTests' client-ack test and the FOCAS respawn
theory now start the host's own driver (AckingAlarmDriver; a factory building a
real FocasDriver per spawn over per-instance fake CNCs, with a real config-change
respawn) - both fail against the fix in their old shape. New: ack to a disabled
driver reverts, ack to a stubbed driver reverts, a Stale host refuses through the
pre-check view - all three fail with the fix mutated out. Runtime.Tests 943/0/3.
Docs: AlarmTracking.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so
a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess,
left the client with Good and the condition showing Acknowledged while Galaxy
was never acked. Both options from the issue, layered:
1. Synchronous refusal for what the driver can decide without I/O. New optional
IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an
anonymous session when the driver has no AlarmAck fallback account.
DriverHostActor publishes a condition NodeId -> owning-driver snapshot into
NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new
NativeAlarmAckPreflight seam, consulted after the authz and Primary gates,
returns BadUserAccessDenied with the reason - nothing routed, applied or
reported. A keyed session is not pre-checked (its identity lives in the
Secrets store, which must not be read under the SDK lock).
2. Revert for everything else. RouteAlarmAck carries the condition NodeId;
DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure,
a driver without an alarm source, or an ack dropped while connecting /
reconnecting. DriverHostActor re-projects the condition unacknowledged via
NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is
still pending - any driver transition projected since then wins. A disabled
or unspawned driver is not reverted (unchanged).
GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a
non-zero native hresult (wrong password, no ack permission) instead of only
warning, so that case reverts too; as a refusal it neither retries nor trips
the breaker.
Tests (each fails with its fix mutated out): projector revert/supersede,
DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful
ack control), preflight view snapshot, node-manager preflight unit test,
over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no
event; positive control), Galaxy preflight + hresult refusal.
Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:
1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
held ONLY in the encrypted Secrets store
(otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
operator_user. Refusals are excluded from the resilience retry/breaker.
- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
acknowledger scrubs it from every gateway message it echoes and withholds
any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
path is fire-and-forget after the Part 9 method returns).
Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.
Closes#749
Review I-2. Under factory Props a supervision restart of
ScriptedAlarmHostActor builds an empty engine, and nothing re-sent
ApplyScriptedAlarms. Until the next deploy the host had no mux interest
(inputs stopped), and its ownership filter silently dropped every
operator ack and shelve.
The host now tells its parent ScriptedAlarmHostRestarted from
PostRestart. DriverHostActor keeps the last plans it applied and
re-sends ApplyScriptedAlarms when the signal comes from the current
host, in Steady, Applying and Stale. This mirrors how a driver child's
ReprovisionRequested is answered. The OnAlarmsLoaded comment, which
assumed a re-apply that never happened, now says where it comes from.
DriverHostActorScriptedAlarmRestartTests deploys one scripted alarm
through a real host, crashes the alarm host, does NOT re-apply by hand,
and asserts that interest is re-registered, a dependency push activates
the alarm, and an operator ack reaches the engine. It is red without
the signal (times out waiting for the re-registration).
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review I-1. Context.Stop is asynchronous, so a removed child kept
publishing through the rest of the apply handler and its own stop.
Those bursts were forwarded after SourceRemoved and re-retained the
removed driver's last Good in the mux, where nothing would ever clear
it while a replay subscriber held the ref (a removed driver never
publishes again, and its PostStop Bad fan resolves no refs once the
index is rebuilt without it).
StopChild and HandleRestartDriver now retire the child (_retiredChildren
+ Context.Watch) before stopping it. ForwardBatchToMux drops a burst
whose Sender is retired, and Terminated (handled in Steady, Applying and
Stale) removes the entry. Akka delivers Terminated after the watched
child's last message, so the set is bounded and a live incarnation is
never filtered. ConnectivityChanged from a retired child is
deliberately not filtered, because the respawn path relies on its Bad
fan.
Tests (review m-3 folded in): a stale burst spoofed as the removed child,
held alive in PostStop by a new RecordingSubscribableDriver.ShutdownGate,
no longer re-retains (red with the guard disabled). New cases cover a
config-change respawn and an operator Restart, each with the new
driver's value replaying as the positive control. The arrange step now
waits for the real subscription set and fishes for both drivers'
values at once (two sequential fishes could discard the other burst).
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
It had no production sender: every driver data change is already a
batch, and the comms-loss fan was converted earlier. DependencyMuxActor
drops its per-value receive, DriverInstanceActor drops the record, and
DependencyMuxActorTests' 13 sends move to the ValuePublishes.Single
one-item batch (same semantics through the batch path). XML docs and
the living docs (DriverLifecycle, VirtualTags, ScriptEditor) are
reworded; historical plan docs are left as written.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The parameter was retained only for signature stability after the
EfAlarmConditionStateStore fallback that consumed it was retired. It is
removed from Props, the constructor and the positional Props.Create
forwarding list together, and from its one caller in
WithOtOpcUaRuntimeActors. No test passed it.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverHostActor now tells the mux SourceRemoved(driverInstanceId)
wherever a driver object ends: StopChild (deploy removal and the stop
half of a config-change respawn) and an operator Restart. It is not
sent on a supervision restart, which re-uses the same driver. Before
this nothing sent it, so a removed driver's last Good stayed in the
mux and replayed to the next replay-opting subscriber as current.
DriverHostActorSourceRemovedTests drives a real mux with two real
driver children: after a deploy drops DI-1 only DI-2's value replays
(positive control: both replay before the removal), and a supervision
restart of DI-1 leaves its value replayable.
Residual (documented on EvictRetainedValues): a burst the old child
enqueued just before it stopped can still be forwarded after the
eviction; the next publish, a comms-loss fan or the grace sweep
bounds it.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverHostActor now hands ScriptedAlarmHostActor an engine factory
instead of a pre-built engine, so every incarnation (a supervision
restart included) builds, owns and disposes its own upstream + engine.
The legacy overload was already restart-safe in the detach sense, so
this is hygiene rather than a live hole; the factory path is the one
ScriptedAlarmHostActorRestartTests covers end to end. The closure
captures locals only and carries the node-wide ScriptingOptions.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Program.cs binds ScriptingOptions from Scripting:* once, clamps every
numeric knob through ScriptingOptions.Sanitized (logging a Warning per
adjustment) and registers the one instance. It now reaches all three
script seams: RoslynVirtualTagEvaluator, the scripted-alarm engine
built in DriverHostActor (new trailing Props parameter, wired in
WithOtOpcUaRuntimeActors) and every Calculation driver (node options
as the base, per-driver config keys override one knob on a copy).
ScriptRunGuard sanitizes again in its ctor and computes the stuck
threshold with a saturating multiply, so an absurd
StuckOrphanQuarantineFactor can no longer throw OverflowException on
a refused run.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.
IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.
Review ref: 02/U1.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
(and its deferred-reprovision timer): the map grew per removed id and a
re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
outcome back (single-flight), with a 2 s command timeout on relational
providers, so a hung SQL Server no longer pins the actor thread and the
cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
driver already uses UnsubscribeAsync (no change).
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
- HistoryUpdate Primary gate now reads IHistoryUpdateRoleView (PrimaryGatePolicy.ShouldServeHistoryUpdate):
the data-plane verdict but CLOSED until this node's first redundancy snapshot, so both pair nodes cannot
accept in the pre-join boot window. Single nodes open on their own singleton's first snapshot; admin-only
nodes register no view => no gate. DriverHostActor publishes it at every Primary-verdict publish point.
- SqlServer v1: the updater throws NotSupportedException on every arm; the node manager maps it to a
node-level BadHistoryOperationUnsupported (was per-item under a Good node for UpdateData/DeleteAtTime).
HistoryWrite bit still advertised on v1 (not knowable at materialise time) — documented.
- The update plan is evaluated before the Primary gate: an unservable request (Remove, IsDeleteModified,
not historized, no updater) is BadHistoryOperationUnsupported off-Primary, not BadInvalidState.
- Local-kind SourceTimestamp converted to UTC before the future-skew compare and sent as UTC.
- Hosted-service wiring extracted to BuildHistoryUpdatePosture and tested (gate source, skew, clock).
- Docs: Historian.md (boot window, ordering, node-level v1, HistoryWrite bit), CLAUDE.md.
Co-Authored-By: Claude <noreply@anthropic.com>
A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).
Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
Since #516 a DriverConfig change is stop + respawn, so DriverSpawnPlan.ToApplyDelta
was always empty and the in-place path had no producer. #663 and #664 were both
latent bugs in it. This deletes it:
- host: ApplyChildDelta, _pendingDelta, HandleApplyResult and HandleDeltaApplied
- plan: DriverSpawnPlan.ToApplyDelta
- actor: the ApplyDelta, ApplyResult and DeltaApplied messages, the Connected
reinit handler and the mid-connect AdoptConfigDuringInit
- tests: the actor tests that drove the seam
IDriver.ReinitializeAsync and each driver's implementation and unit tests stay.
Comments and docs that cited AdoptConfigDuringInit as the reason a driver must
release prior state at init now cite the IDriver contract and overlapping
retries, which still apply.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
After an OpcUaClient respawn an upstream inactive+unacked condition went out as
Retrigger+Clear (duplicate Activated+Cleared rows, Active flicker), and a
first-sight inactive+acked one as Unspecified (an Activated row for an inactive
alarm; a retained-Active host condition never cleared).
- New AlarmTransitionKind.InactiveUnacknowledged: one projection, rows only for
what changed (Activated+Cleared from acked, Cleared from active+unacked, none
when already inactive+unacked).
- OpcUaClient uses it for moves into inactive+unacked; first sight of an
inactive condition is InactiveUnacknowledged or Clear+Acknowledge, flagged.
- Projector: a flagged Acknowledge on an already-acked condition writes no row.
- E2E tests through DriverHostActor.ForwardNativeAlarm (red-proven).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- AlarmTransitionKind.ActiveAcknowledged (appended): active AND acked applied as ONE projection.
Galaxy ActiveAcked snapshot entries and OpcUaClient inactive->(active,acked) use it; the
projector writes 'Activated' only if not already active, 'Acknowledged' only if not acked.
- AlarmEventArgs.IsRestatement (optional, default false): set only by Galaxy snapshot entries
and OpcUaClient first sight. A flagged Retrigger on an Active+unacked condition with the same
message and severity, or a flagged Clear on an inactive condition, writes no /alerts row.
- NativeAlarmProjector reports the history rows per transition; ForwardNativeAlarm writes one
row per entry. Unflagged sources (FOCAS/MTConnect/AbCip, live Galaxy) are unchanged.
- Tests: projector rules each with a must-write control; OpcUaClient transition table;
Galaxy feed/driver mapping; Galaxy + OpcUaClient end-to-end through DriverHostActor.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
MAJOR-1: a deploy that changes a driver's config respawns it, and the fresh instance's empty told
record re-sent Raise for every still-active alarm, un-acking acknowledged conditions and writing a
duplicate "Activated" alerts row. NativeAlarmProjector now treats Raise on an already-active
condition as a re-statement (keeps Acked; ForwardNativeAlarm skips the alerts row); only Retrigger
(or inactive->active) un-acks. Sources whose upstream ack state is authoritative send Retrigger for
an unacked re-statement: OpcUaClient on first sight (no prior told), Galaxy for an unacked active
snapshot entry. Runtime E2E drives a FRESH FOCAS driver after a redeploy and asserts the ack survives.
MAJOR-2: a failed FOCAS alarm read annotates each subscribed condition BadNoCommunication once per
outage (#626 quality-only path) with one Warning (MINOR-6), lifted to Good on recovery; failed alarm
reads degrade driver health with LastSuccessfulRead frozen (successful ones advance it), so an
alarm-only driver trips the #527 comms-loss watchdog.
MINOR-3 CLAUDE.md #630 bullet; MINOR-4 split line; MINOR-5 a non-empty cnc_rdalmmsg2 payload that
fits neither stride throws; MINOR-7 value-tag + parsed-address tables are built and swapped, never
cleared in place under the retained poll engine. Docs corrected (FOCAS.md, AlarmTracking.md, CLAUDE.md).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
60 s labelled a slow-but-clean large rebuild degraded. The deadline moves to a
shared Commons DeployTimings constant that ConfigPublishCoordinator reads, and
the node's timeout is deadline - 15 s margin, so the held ack still lands in
time. A timed-out ack's reason now says the outcome was not reported in time,
distinct from a real sink failure.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The apply writes the node's row Applied/null before the address-space outcome
is known; central adds the reason only when it ACCEPTS the ack, so a discarded
(stale/timed-out) ack left a fused node's row reading clean. The prefix moves
to ApplyAck.DegradedReasonPrefix so both writers spell the row identically.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
DriverHostActor no longer acks Applied the moment the revision advances. It
asks OpcUaPublishActor for the outcome (RebuildAddressSpace.ReportOutcome) and
acks Applied+Degraded(reason) when the apply was degraded; a missing outcome
times out (60 s, inside the coordinator's 2 min deadline) to a degraded ack,
never a clean one. Re-dispatching a revision whose apply was degraded or is
still unconfirmed re-applies instead of taking the clean idempotent
short-circuit - that re-apply is also what repairs the degraded nodes.
Tests with a bare publish probe use a clean-outcome auto-pilot probe.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
#641 keeps the projector's prior state across a deploy instead of resetting it to the
acked default. The projector never saw a client's ack (the SDK applies it to the node and
the host only routes it), so a Clear after ack+deploy now projected Acked=false and
un-acknowledged the operator's ack - previously masked by the per-deploy reset.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
DriverHostActor cleared NativeAlarmProjector's per-condition prior state on
EVERY apply, so after any deploy a driver Acknowledge of a still-active alarm
projected Active=false (writing it inactive) and a Clear of an unacknowledged
alarm projected Acked=true (silently acknowledging it). The projector is now
pruned to the conditions the new composition still carries (RetainOnly).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- Unavailable keeps an ACTIVE condition active: quality-only BadNoCommunication; on data
return same statement => quality-only Good, Normal => Clear, different => Retrigger.
- A changed alarm reference set (alarm-bearing delta = full rebuild) clears the whole
last-told record so every active condition is re-asserted; the same set keeps it.
- Quality-only events go through the host's AlarmQualityUpdate path, never a projected
snapshot (which could ack a retained unacked condition or rewrite message/severity).
- Deterministic tie-break between equal-level conditions (ordinal nativeCode, then type).
- Dedup key is level+nativeCode+nativeSeverity+qualifier; a text-only change updates nothing.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
AlarmEventArgs gains optional OpcUaSeverity (supersedes the four-bucket mapping, clamped
1..1000) and QualityStatusCode (projected onto the condition's Quality). An Unspecified-kind
event carrying a quality is a quality-only annotation: the host updates the condition but
publishes no alerts row. Both default null, so every existing native source is unchanged.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.
- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
real invoker's bracket across the removal and signals when its late writes are done (positive
control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
(incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
host test.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The drain gate's 'is my queue-sharing peer Primary?' used only the host of
LocalDb:Replication:PeerAddress, which the passive half (ApiKey +
SyncListenPort) does not set - so as Secondary it kept draining beside its
Primary partner and every alarm event reached the historian twice.
DriverHostActor takes a sharesAlarmQueueWithLocal predicate used when no
peer host is known; WithOtOpcUaRuntimeActors builds it
(BuildAlarmQueuePeerPredicate) for a SharesRowsWithPeer node with no
PeerAddress as 'a member of my own cluster-{ClusterId} mesh other than
me'. A node sharing no rows, or with no mesh role, gets none and keeps
draining. AlarmHistorian.md / Redundancy.md / pair-replication runbook
updated.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
MINOR-3: new DriverHostActorPrimaryRoleSourceTests case drives the
deny flip with a real ClusterEvent.MemberUp on the EventStream (how the
cluster's domain-event publisher reaches a Cluster.Subscribe subscriber)
instead of a direct RepublishPrimaryVerdict tell; it fails when the
Cluster.Subscribe(IMemberEvent) call is deleted.
MINOR-2: the published verdict MIRRORS the live gate; before this node has
joined its member count is 0 and both allow (seconds; up to the bootstrap
guard's 25 s probe), so the Sparkplug primary-host claim can be held on
both nodes in that window (native ack is unaffected - view and route
agree). Behaviour deliberately unchanged: denying until self is Up would
change the live gate too and delay writes/acks on a single-node cluster's
own join. Comments + AlarmTracking.md / Redundancy.md / Mqtt.md corrected.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.
Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.
Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.
Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
Joseph Doherty