Commit Graph

138 Commits

Author SHA1 Message Date
Joseph Doherty 89204ac0fa fix(alarms): re-apply scripted alarms after a supervision restart of their host
Review I-2. Under factory Props a supervision restart of
ScriptedAlarmHostActor builds an empty engine, and nothing re-sent
ApplyScriptedAlarms. Until the next deploy the host had no mux interest
(inputs stopped), and its ownership filter silently dropped every
operator ack and shelve.

The host now tells its parent ScriptedAlarmHostRestarted from
PostRestart. DriverHostActor keeps the last plans it applied and
re-sends ApplyScriptedAlarms when the signal comes from the current
host, in Steady, Applying and Stale. This mirrors how a driver child's
ReprovisionRequested is answered. The OnAlarmsLoaded comment, which
assumed a re-apply that never happened, now says where it comes from.

DriverHostActorScriptedAlarmRestartTests deploys one scripted alarm
through a real host, crashes the alarm host, does NOT re-apply by hand,
and asserts that interest is re-registered, a dependency push activates
the alarm, and an operator ack reaches the engine. It is red without
the signal (times out waiting for the re-registration).

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:13:02 -04:00
Joseph Doherty acdf6d83a3 fix(runtime): drop value bursts from driver children the host already stopped
Review I-1. Context.Stop is asynchronous, so a removed child kept
publishing through the rest of the apply handler and its own stop.
Those bursts were forwarded after SourceRemoved and re-retained the
removed driver's last Good in the mux, where nothing would ever clear
it while a replay subscriber held the ref (a removed driver never
publishes again, and its PostStop Bad fan resolves no refs once the
index is rebuilt without it).

StopChild and HandleRestartDriver now retire the child (_retiredChildren
+ Context.Watch) before stopping it. ForwardBatchToMux drops a burst
whose Sender is retired, and Terminated (handled in Steady, Applying and
Stale) removes the entry. Akka delivers Terminated after the watched
child's last message, so the set is bounded and a live incarnation is
never filtered. ConnectivityChanged from a retired child is
deliberately not filtered, because the respawn path relies on its Bad
fan.

Tests (review m-3 folded in): a stale burst spoofed as the removed child,
held alive in PostStop by a new RecordingSubscribableDriver.ShutdownGate,
no longer re-retains (red with the guard disabled). New cases cover a
config-change respawn and an operator Restart, each with the new
driver's value replaying as the positive control. The arrange step now
waits for the real subscription set and fishes for both drivers'
values at once (two sequential fishes could discard the other burst).

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:06:23 -04:00
Joseph Doherty 6636de1199 refactor(runtime): delete the per-value AttributeValuePublished message
It had no production sender: every driver data change is already a
batch, and the comms-loss fan was converted earlier. DependencyMuxActor
drops its per-value receive, DriverInstanceActor drops the record, and
DependencyMuxActorTests' 13 sends move to the ValuePublishes.Single
one-item batch (same semantics through the batch path). XML docs and
the living docs (DriverLifecycle, VirtualTags, ScriptEditor) are
reworded; historical plan docs are left as written.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:42:08 -04:00
Joseph Doherty 0475603192 refactor(runtime): drop DriverHostActor's unused ILoggerFactory slot
The parameter was retained only for signature stability after the
EfAlarmConditionStateStore fallback that consumed it was retired. It is
removed from Props, the constructor and the positional Props.Create
forwarding list together, and from its one caller in
WithOtOpcUaRuntimeActors. No test passed it.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:38:33 -04:00
Joseph Doherty 5de1d661bb fix(runtime): evict a stopped driver's retained values from the dependency mux
DriverHostActor now tells the mux SourceRemoved(driverInstanceId)
wherever a driver object ends: StopChild (deploy removal and the stop
half of a config-change respawn) and an operator Restart. It is not
sent on a supervision restart, which re-uses the same driver. Before
this nothing sent it, so a removed driver's last Good stayed in the
mux and replayed to the next replay-opting subscriber as current.

DriverHostActorSourceRemovedTests drives a real mux with two real
driver children: after a deploy drops DI-1 only DI-2's value replays
(positive control: both replay before the removal), and a supervision
restart of DI-1 leaves its value replayable.

Residual (documented on EvictRetainedValues): a burst the old child
enqueued just before it stopped can still be forwarded after the
eviction; the next publish, a comms-loss fan or the grace sweep
bounds it.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:36:54 -04:00
Joseph Doherty a33145fe71 refactor(alarms): spawn the scripted-alarm host through its factory Props
DriverHostActor now hands ScriptedAlarmHostActor an engine factory
instead of a pre-built engine, so every incarnation (a supervision
restart included) builds, owns and disposes its own upstream + engine.
The legacy overload was already restart-safe in the detach sense, so
this is hygiene rather than a live hole; the factory path is the one
ScriptedAlarmHostActorRestartTests covers end to end. The closure
captures locals only and carries the node-wide ScriptingOptions.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:32:53 -04:00
Joseph Doherty 4615495f01 feat(scripting): bind the Scripting section node-wide and clamp its knobs
Program.cs binds ScriptingOptions from Scripting:* once, clamps every
numeric knob through ScriptingOptions.Sanitized (logging a Warning per
adjustment) and registers the one instance. It now reaches all three
script seams: RoslynVirtualTagEvaluator, the scripted-alarm engine
built in DriverHostActor (new trailing Props parameter, wired in
WithOtOpcUaRuntimeActors) and every Calculation driver (node options
as the base, per-driver config keys override one knob on a copy).

ScriptRunGuard sanitizes again in its ctor and computes the stuck
threshold with a saturating multiply, so an absurd
StuckOrphanQuarantineFactor can no longer throw OverflowException on
a refused run.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:24:53 -04:00
Joseph Doherty 59b72fe385 fix(runtime): diagnostics Name stays the string instance id; order-tolerant write-cap test (A6 follow-through)
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:15:01 -04:00
Joseph Doherty d1cc9e2cfc feat(runtime,adminui): real GetDiagnostics from child health + /fleet/diagnostics consumer (A6)
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.

IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.

Review ref: 02/U1.

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:44:38 -04:00
Joseph Doherty 79d6b0a4b2 fix(runtime): lifecycle housekeeping — reprovision stamp, Stale drops, off-thread DB probe, tombstone sweep (A4)
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
  (and its deferred-reprovision timer): the map grew per removed id and a
  re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
  AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
  letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
  outcome back (single-flight), with a 2 s command timeout on relational
  providers, so a hung SQL Server no longer pins the actor thread and the
  cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
  10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
  AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
  half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
  driver already uses UnsubscribeAsync (no change).

Refs #731

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
Joseph Doherty 232c03ed1b fix(historian): #674 review follow-ups — HistoryUpdate boot-window gate, node-level v1 refusal, plan-before-gate, UTC skew compare
- HistoryUpdate Primary gate now reads IHistoryUpdateRoleView (PrimaryGatePolicy.ShouldServeHistoryUpdate):
  the data-plane verdict but CLOSED until this node's first redundancy snapshot, so both pair nodes cannot
  accept in the pre-join boot window. Single nodes open on their own singleton's first snapshot; admin-only
  nodes register no view => no gate. DriverHostActor publishes it at every Primary-verdict publish point.
- SqlServer v1: the updater throws NotSupportedException on every arm; the node manager maps it to a
  node-level BadHistoryOperationUnsupported (was per-item under a Good node for UpdateData/DeleteAtTime).
  HistoryWrite bit still advertised on v1 (not knowable at materialise time) — documented.
- The update plan is evaluated before the Primary gate: an unservable request (Remove, IsDeleteModified,
  not historized, no updater) is BadHistoryOperationUnsupported off-Primary, not BadInvalidState.
- Local-kind SourceTimestamp converted to UTC before the future-skew compare and sent as UTC.
- Hosted-service wiring extracted to BuildHistoryUpdatePosture and tested (gate source, skew, clock).
- Docs: Historian.md (boot window, ordering, node-level v1, HistoryWrite bit), CLAUDE.md.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-11 11:41:11 -04:00
Joseph Doherty 1b2c36eb87 fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
2026-10-11 08:26:39 -04:00
Joseph Doherty 1ea1da8236 refactor(runtime): delete the dead in-place ApplyDelta seam; config change = respawn only (#665)
Since #516 a DriverConfig change is stop + respawn, so DriverSpawnPlan.ToApplyDelta
was always empty and the in-place path had no producer. #663 and #664 were both
latent bugs in it. This deletes it:
- host: ApplyChildDelta, _pendingDelta, HandleApplyResult and HandleDeltaApplied
- plan: DriverSpawnPlan.ToApplyDelta
- actor: the ApplyDelta, ApplyResult and DeltaApplied messages, the Connected
  reinit handler and the mid-connect AdoptConfigDuringInit
- tests: the actor tests that drove the seam

IDriver.ReinitializeAsync and each driver's implementation and unit tests stay.
Comments and docs that cited AdoptConfigDuringInit as the reason a driver must
release prior state at init now cite the IDriver contract and overlapping
retries, which still apply.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 11:23:16 -04:00
Joseph Doherty 48a80b5962 fix(alarms): atomic InactiveUnacknowledged + flagged Clear/Acknowledge first sight for OpcUaClient (#657)
After an OpcUaClient respawn an upstream inactive+unacked condition went out as
Retrigger+Clear (duplicate Activated+Cleared rows, Active flicker), and a
first-sight inactive+acked one as Unspecified (an Activated row for an inactive
alarm; a retained-Active host condition never cleared).

- New AlarmTransitionKind.InactiveUnacknowledged: one projection, rows only for
  what changed (Activated+Cleared from acked, Cleared from active+unacked, none
  when already inactive+unacked).
- OpcUaClient uses it for moves into inactive+unacked; first sight of an
  inactive condition is InactiveUnacknowledged or Clear+Acknowledge, flagged.
- Projector: a flagged Acknowledge on an already-acked condition writes no row.
- E2E tests through DriverHostActor.ForwardNativeAlarm (red-proven).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 06:08:39 -04:00
Joseph Doherty b39bb6794d fix(alarms): native-alarm re-statements no longer duplicate history or hide acked alarms (#656)
- AlarmTransitionKind.ActiveAcknowledged (appended): active AND acked applied as ONE projection.
  Galaxy ActiveAcked snapshot entries and OpcUaClient inactive->(active,acked) use it; the
  projector writes 'Activated' only if not already active, 'Acknowledged' only if not acked.
- AlarmEventArgs.IsRestatement (optional, default false): set only by Galaxy snapshot entries
  and OpcUaClient first sight. A flagged Retrigger on an Active+unacked condition with the same
  message and severity, or a flagged Clear on an inactive condition, writes no /alerts row.
- NativeAlarmProjector reports the history rows per transition; ForwardNativeAlarm writes one
  row per entry. Unflagged sources (FOCAS/MTConnect/AbCip, live Galaxy) are unchanged.
- Tests: projector rules each with a must-write control; OpcUaClient transition table;
  Galaxy feed/driver mapping; Galaxy + OpcUaClient end-to-end through DriverHostActor.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 23:35:50 -04:00
Joseph Doherty 89961af6e6 fix(alarms): #647 review — Raise on an active condition is a re-statement; FOCAS outage quality + health
MAJOR-1: a deploy that changes a driver's config respawns it, and the fresh instance's empty told
record re-sent Raise for every still-active alarm, un-acking acknowledged conditions and writing a
duplicate "Activated" alerts row. NativeAlarmProjector now treats Raise on an already-active
condition as a re-statement (keeps Acked; ForwardNativeAlarm skips the alerts row); only Retrigger
(or inactive->active) un-acks. Sources whose upstream ack state is authoritative send Retrigger for
an unacked re-statement: OpcUaClient on first sight (no prior told), Galaxy for an unacked active
snapshot entry. Runtime E2E drives a FRESH FOCAS driver after a redeploy and asserts the ack survives.

MAJOR-2: a failed FOCAS alarm read annotates each subscribed condition BadNoCommunication once per
outage (#626 quality-only path) with one Warning (MINOR-6), lifted to Good on recovery; failed alarm
reads degrade driver health with LastSuccessfulRead frozen (successful ones advance it), so an
alarm-only driver trips the #527 comms-loss watchdog.

MINOR-3 CLAUDE.md #630 bullet; MINOR-4 split line; MINOR-5 a non-empty cnc_rdalmmsg2 payload that
fits neither stride throws; MINOR-7 value-tag + parsed-address tables are built and swapped, never
cleared in place under the retained poll engine. Docs corrected (FOCAS.md, AlarmTracking.md, CLAUDE.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:18:03 -04:00
Joseph Doherty 8801631739 Merge b640-601: honest deploy acks — Applied only after the address-space outcome, degraded flag + reason (#601)
v2-ci / build (push) Successful in 4m42s
v2-ci / unit-tests (push) Failing after 20m19s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

# Conflicts:
#	src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs
2026-10-08 18:00:23 -04:00
Joseph Doherty 0a821417ea fix(deploy): derive the apply-outcome timeout from central's apply deadline (105 s) and name a timeout reason distinctly (#601 review)
60 s labelled a slow-but-clean large rebuild degraded. The deadline moves to a
shared Commons DeployTimings constant that ConfigPublishCoordinator reads, and
the node's timeout is deadline - 15 s margin, so the held ack still lands in
time. A timed-out ack's reason now says the outcome was not reported in time,
distinct from a real sink failure.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:54:09 -04:00
Joseph Doherty 345f40550b fix(deploy): a node with a ConfigDb records its degraded apply on its own NodeDeploymentState row (#601 review)
The apply writes the node's row Applied/null before the address-space outcome
is known; central adds the reason only when it ACCEPTS the ack, so a discarded
(stale/timed-out) ack left a fused node's row reading clean. The prefix moves
to ApplyAck.DegradedReasonPrefix so both writers spell the row identically.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:46:18 -04:00
Joseph Doherty 90d7f6dfca fix(deploy): hold the Applied ack until the address-space apply outcome is known (#601)
DriverHostActor no longer acks Applied the moment the revision advances. It
asks OpcUaPublishActor for the outcome (RebuildAddressSpace.ReportOutcome) and
acks Applied+Degraded(reason) when the apply was degraded; a missing outcome
times out (60 s, inside the coordinator's 2 min deadline) to a degraded ack,
never a clean one. Re-dispatching a revision whose apply was degraded or is
still unconfirmed re-applies instead of taking the clean idempotent
short-circuit - that re-apply is also what repairs the degraded nodes.
Tests with a bare publish probe use a clean-outcome auto-pilot probe.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:29:45 -04:00
Joseph Doherty 25f02e4102 fix(runtime): the native-alarm projector learns a routed client acknowledgement (#641 review)
#641 keeps the projector's prior state across a deploy instead of resetting it to the
acked default. The projector never saw a client's ack (the SDK applies it to the node and
the host only routes it), so a Clear after ack+deploy now projected Acked=false and
un-acknowledged the operator's ack - previously masked by the per-deploy reset.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:19:24 -04:00
Joseph Doherty 11f13f6326 fix(runtime): prune the native-alarm projector on deploy instead of clearing it (#641)
DriverHostActor cleared NativeAlarmProjector's per-condition prior state on
EVERY apply, so after any deploy a driver Acknowledge of a still-active alarm
projected Active=false (writing it inactive) and a Clear of an unacknowledged
alarm projected Acked=true (silently acknowledging it). The projector is now
pruned to the conditions the new composition still carries (RetainOnly).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:05:42 -04:00
Joseph Doherty 9c8e3da5da fix(mtconnect): review rulings for CONDITION alarms (#626)
- Unavailable keeps an ACTIVE condition active: quality-only BadNoCommunication; on data
  return same statement => quality-only Good, Normal => Clear, different => Retrigger.
- A changed alarm reference set (alarm-bearing delta = full rebuild) clears the whole
  last-told record so every active condition is re-asserted; the same set keeps it.
- Quality-only events go through the host's AlarmQualityUpdate path, never a projected
  snapshot (which could ack a retained unacked condition or rewrite message/severity).
- Deterministic tie-break between equal-level conditions (ordinal nativeCode, then type).
- Dedup key is level+nativeCode+nativeSeverity+qualifier; a text-only change updates nothing.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 15:38:52 -04:00
Joseph Doherty cb85a455c9 feat(alarms): exact OPC UA severity + per-condition quality on native alarm events (#626)
AlarmEventArgs gains optional OpcUaSeverity (supersedes the four-bucket mapping, clamped
1..1000) and QualityStatusCode (projected onto the condition's Quality). An Unspecified-kind
event carrying a quality is a quality-only annotation: the host updates the condition but
publishes no alerts row. Both default null, so every existing native source is unchanged.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 14:55:46 -04:00
Joseph Doherty 4a10ea1e1c fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.

- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
  Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
  survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
  re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
  fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
  real invoker's bracket across the removal and signals when its late writes are done (positive
  control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
  (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
  host test.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 09:07:11 -04:00
Joseph Doherty 3be8a1217e fix(runtime): the passive half of a LocalDb pair stands its alarm drain down for its mesh partner (review R6 MINOR-4)
The drain gate's 'is my queue-sharing peer Primary?' used only the host of
LocalDb:Replication:PeerAddress, which the passive half (ApiKey +
SyncListenPort) does not set - so as Secondary it kept draining beside its
Primary partner and every alarm event reached the historian twice.
DriverHostActor takes a sharesAlarmQueueWithLocal predicate used when no
peer host is known; WithOtOpcUaRuntimeActors builds it
(BuildAlarmQueuePeerPredicate) for a SharesRowsWithPeer node with no
PeerAddress as 'a member of my own cluster-{ClusterId} mesh other than
me'. A node sharing no rows, or with no mesh role, gets none and keeps
draining. AlarmHistorian.md / Redundancy.md / pair-replication runbook
updated.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:43:06 -04:00
Joseph Doherty e93d115cbe fix(runtime): pin the Primary-verdict member-event subscription with a test; document the pre-join allow window (review R6 MINOR-2/3)
MINOR-3: new DriverHostActorPrimaryRoleSourceTests case drives the
deny flip with a real ClusterEvent.MemberUp on the EventStream (how the
cluster's domain-event publisher reaches a Cluster.Subscribe subscriber)
instead of a direct RepublishPrimaryVerdict tell; it fails when the
Cluster.Subscribe(IMemberEvent) call is deleted.

MINOR-2: the published verdict MIRRORS the live gate; before this node has
joined its member count is 0 and both allow (seconds; up to the bootstrap
guard's 25 s probe), so the Sparkplug primary-host claim can be held on
both nodes in that window (native ack is unaffected - view and route
agree). Behaviour deliberately unchanged: denying until self is Up would
change the live gate too and delay writes/acks on a single-node cluster's
own join. Comments + AlarmTracking.md / Redundancy.md / Mqtt.md corrected.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:33:52 -04:00
Joseph Doherty e3dcb180c3 fix(runtime): Primary verdict published from host start, cache boot never adopts malformed bytes, strict drain-gate predicate, HostAbortedException passthrough, seed only Up members
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
  cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
  Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
  re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
  SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:34 -04:00
Joseph Doherty 53247a056a fix(runtime): malformed artifact bytes on boot/restore are no answer (#601); evict telemetry snapshots on driver removal (#609)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:53:45 -04:00
Joseph Doherty e280db3753 fix(runtime): comms_latched_refs meter tag key 'driver' -> 'driver_instance' — the R3-04 bare-role-literal guard caught the R3-02 tag key at merge (cross-branch collision; neither branch saw the other) 2026-08-20 11:55:16 -04:00
Joseph Doherty 157d01c2c2 Merge branch 'fix/r3-02-stale-good-family' — R3-02: one quality contract at four seams — comms loss marks raw+UNS value variables BadNoCommunication (WriteStatuses sink member, recovery audit), Calculation + VT hold on Bad / worst-of sub-code / (value,status) dedup, MQTT initial push + ConnectionLost stale, Sparkplug birth replay (archreview 03/S15, 05/STAB-19, 05/STAB-18, 02/S14) 2026-08-20 11:49:06 -04:00
Joseph Doherty f1b976f5e2 Merge branch 'fix/r3-04-one-role-authority' — R3-04: EffectiveRoles is the one role authority (Cluster:Roles wins over OTOPCUA_ROLES, normalised, validated, fail-start on unknown/overlay-changed roles); Program.cs + all four validators consume it; case-insensitive gossip predicates; source-scan guard (archreview 01/S-12 = 03/C5, 01/C-9) 2026-08-20 11:05:09 -04:00
Joseph Doherty 1d251cc037 Merge branch 'master' into fix/r3-02-stale-good-family
# Conflicts:
#	src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
#	tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Harness/V3ArtifactSeeder.cs
2026-08-20 11:03:02 -04:00
Joseph Doherty 0793b0cba8 fix(runtime): rate-limit the supervision reprovision — deferred, not dropped (review F3)
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.

Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.

Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
2026-08-20 10:39:13 -04:00
Joseph Doherty 2b7100cf19 fix(runtime): route a reprovision by incarnation, not driver id (review F2)
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.

Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
2026-08-20 10:27:02 -04:00
Joseph Doherty 7d98ef9332 feat(runtime): audit refs still BadNoCommunication after a reconnect (dedup-trap tripwire) 2026-08-20 10:16:01 -04:00
Joseph Doherty 97f4ea0c9e fix(runtime): comms loss marks raw + UNS value nodes BadNoCommunication, value retained (archreview 03/S15) 2026-08-20 10:07:12 -04:00
Joseph Doherty e46c1db5e5 fix(cluster): gossiped-role predicates compare case-insensitively — a miscased peer no longer un-elects the Primary 2026-08-20 10:04:16 -04:00
Joseph Doherty 2b9a397b2d fix(runtime): re-provision a driver child after a supervision restart (archreview 03/S6, S16) 2026-08-20 09:56:05 -04:00
Joseph Doherty d97e441979 test(runtime): operator restart re-subscribes the respawned child (archreview 03/S16, U6)
Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
2026-08-20 09:53:38 -04:00
Joseph Doherty daaeb854d6 fix(runtime): every spawn hands the child its desired subscriptions (archreview 03/S16) 2026-08-20 09:36:52 -04:00
Joseph Doherty bdc81a1372 refactor(runtime): one desired-subscription payload authority (_desiredByDriver) 2026-08-20 09:34:49 -04:00
Joseph Doherty 7c5a843ace docs(code): close-out comment fixes from the follow-ups review round 2026-08-17 04:32:53 -04:00
Joseph Doherty dba5f806d3 fix(runtime): explicit 30s command timeout on the Direct-apply artifact read 2026-08-17 03:49:27 -04:00
Joseph Doherty 5b92eb0ff6 perf(runtime): direct-mode artifact read piped off the host mailbox 2026-08-16 04:48:02 -04:00
Joseph Doherty b6275f439c perf(runtime): per-driver index for connectivity fans; comms-loss is one batch 2026-08-16 04:27:20 -04:00
Joseph Doherty 0dbbb5ac6b fix(runtime): stash dedup + applying-wedge watchdog; cache-failure diagnostics 2026-08-15 17:43:15 -04:00
Joseph Doherty 5f2e60278c perf(runtime): apply-path cache write off-mailbox; stash deferred dispatches 2026-08-15 17:12:01 -04:00
Joseph Doherty 2fd60ecce1 perf(runtime): apply path parses the artifact once per actor
Both apply handlers (DriverHostActor.ApplyAndAck + HandleFetchedForApply)
and OpcUaPublishActor.HandleRebuild used to rebuild a JsonDocument over the
same artifact bytes for every question they asked of it — driver instances,
composition, cluster scope, data-plane authority. Each now opens ONE
ParsedDeploymentArtifact per apply and threads it through:

  - ReconcileDriversForApply / ReconcileDriversFromParsed return the blob
    plus the parse the rest of the apply reads; the handler owns it and
    disposes it in its finally.
  - PushDesiredSubscriptionsFromParsed and CacheAppliedArtifact take that
    parse instead of re-parsing the bytes.
  - HandleRebuild opens once after the #485 guard and feeds both the
    data-plane authority and the composition from it.

The blob-based variants stay as the lenient wrappers the boot paths need
(BootstrapFromCache, RestoreApplied), so the three remaining static Parse*
call sites are boot-only — none on an apply path.

The RebuildAddressSpace message still carries raw BYTES, deliberately: the
publish actor is a different actor with its own mailbox and
ParsedDeploymentArtifact is a disposable, non-thread-safe object owned by
the sending handler. It opens its own single parse on the far side.

FeedDataPlaneAuthority still SHA-256s the RAW bytes for the revision id — a
re-serialisation of the parsed document would be a different byte sequence
and would not match the deployment's RevisionHash.

Behaviour note: Open() throws JsonException on bytes the static wrappers
swallowed to an empty result. That is routed into each path's EXISTING
failure handling — ApplyAndAck's catch (Failed ack, children left running)
and HandleRebuild's catch (Error log, last-known-good still served) — not
as a new outcome. An empty composition is not a benign no-op there: the
planner diffs it as a PureRemove and tears the address space down, so the
throw is the honest reading of unusable bytes.
2026-08-15 17:00:05 -04:00