Commit Graph

351 Commits

Author SHA1 Message Date
Joseph Doherty e7dd742541 Merge b647-focas-alarms: FOCAS native alarms reach Part 9 conditions (ALARM: address) + respawn keeps acks host-side (#647)
v2-ci / build (push) Successful in 4m57s
v2-ci / unit-tests (push) Failing after 21m6s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:46:13 -04:00
Joseph Doherty bac93744ce fix(alarms): #647 re-review — re-statement needs the same message; CNC refusal is not comms loss; worst-of health
R1: NativeAlarmProjector treats Raise-on-Active as a re-statement only when the message matches what
the condition shows; a different alarm appearing during a respawn un-acks and writes its alerts row.
FOCAS condition messages are a deterministic function of the matched set (tested across fresh driver
instances and reordered lists); Runtime E2E now covers both the same alarm and a new one during a
respawn. Residual (source message determinism) documented in AlarmTracking.md.

R2: a non-OK cnc_rdalmmsg2 return is thrown as FocasWireException{Rc} and classified as a CNC
refusal: conditions get a quality-only BadNotSupported once, one Warning, polling continues, and
driver health is untouched (no 90 s watchdog reconnect loop on a production control). Only transport
faults degrade (falsification control in the test).

R3: alarm-path health is tracked apart from the value path and combined in GetHealth: worst state,
later LastSuccessfulRead. A good value read can no longer mask a failing alarm read (no per-poll
/hosts flip) and a good alarm read cannot mask a value Degraded.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:45:32 -04:00
Joseph Doherty beccfc6404 test(#606): native-ack routing waits for the driver child to be Connected; widen a starved poll
DriverHostActorNativeAlarmAckRoutingTests: Applied does not mean the child DriverInstanceActor is
Connected, and an ack reaching a still-Connecting child is dropped by design — the single
positive-control ack was itself dropped (4/20 under DOTNET_PROCESSOR_COUNT=1 x8, still failing
with the presence budget dilated to 45 s, so not a budget problem). SpawnHostAndApply now waits
for SubscribeAlarmsAsync, which only the Connected state calls. ScriptedAlarmEngine
OnEvent_emission_happens_outside_evalGate: 2 s blocking poll -> 15 s (5/20 under starvation).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:37:15 -04:00
Joseph Doherty 89961af6e6 fix(alarms): #647 review — Raise on an active condition is a re-statement; FOCAS outage quality + health
MAJOR-1: a deploy that changes a driver's config respawns it, and the fresh instance's empty told
record re-sent Raise for every still-active alarm, un-acking acknowledged conditions and writing a
duplicate "Activated" alerts row. NativeAlarmProjector now treats Raise on an already-active
condition as a re-statement (keeps Acked; ForwardNativeAlarm skips the alerts row); only Retrigger
(or inactive->active) un-acks. Sources whose upstream ack state is authoritative send Retrigger for
an unacked re-statement: OpcUaClient on first sight (no prior told), Galaxy for an unacked active
snapshot entry. Runtime E2E drives a FRESH FOCAS driver after a redeploy and asserts the ack survives.

MAJOR-2: a failed FOCAS alarm read annotates each subscribed condition BadNoCommunication once per
outage (#626 quality-only path) with one Warning (MINOR-6), lifted to Good on recovery; failed alarm
reads degrade driver health with LastSuccessfulRead frozen (successful ones advance it), so an
alarm-only driver trips the #527 comms-loss watchdog.

MINOR-3 CLAUDE.md #630 bullet; MINOR-4 split line; MINOR-5 a non-empty cnc_rdalmmsg2 payload that
fits neither stride throws; MINOR-7 value-tag + parsed-address tables are built and swapped, never
cleared in place under the retained poll engine. Docs corrected (FOCAS.md, AlarmTracking.md, CLAUDE.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:18:03 -04:00
Joseph Doherty 482bafdfac feat(focas): CNC alarms reach Part 9 conditions via ALARM: tags (#647)
An ALARM: address (*, <Type>, <Type>/<n>, <Type>/<lo>-<hi>, T<code>, OTHER) on a raw tag that
also carries the cross-driver alarm object is ONE condition with ConditionId == RawPath and a
Kind (Raise / Retrigger / Clear), so DriverHostActor.ForwardNativeAlarm routes it and
NativeAlarmProjector activates it. Fixes all seven triage defects:

1. SubscribeAlarmsAsync treated the host's RawPaths as device hosts -> nothing polled.
2. Decorated "{host}#{Type}:{Number}" ConditionId -> every event dropped.
3. No Kind -> projector kept prior state.
4. WireFocasClient.ReadAlarmsAsync answered [] on failure -> mass clear + un-acked re-raise.
   It now throws; a failed device read keeps every condition's state.
5. Acks lost on reconnect -> told-record dedup (MTConnect #626 pattern) held for the driver's
   lifetime; same-set re-subscribe / re-init re-raise nothing; changed set prunes leavers only.
6. Subscribe threw while AlarmProjection.Enabled=false -> the tag is the opt-in; flag is a no-op.
7. Alarm tags could not be authored -> ALARM: grammar (FocasAlarmSelector, Contracts) backs the
   editor, the address picker, the deploy-time inspector and driver init.

Severity: authored alarm.severity -> exact OpcUaSeverity; else the 16i-era type bucket. The type
table stays the only one (moved to Contracts, documented 16i-era; 30i codes not invented; raw
T<code> selector + raw code in every message). Read-only: ack is server-side. Also clears the
value table on a re-initialise (#630 gap that let a removed/moved tag keep reading).

E2E: Runtime.Tests FocasNativeAlarmEndToEndTests drives the real driver through the real
DriverHostActor and asserts the condition goes Active then inactive.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 19:46:48 -04:00
Joseph Doherty 55ae230fa4 ci(#606): Gitea-compatible trx upload + deterministic fixes for the CI-load test flakes
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and
  v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3
  on the test step.
- Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR
  (TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget.
- Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests
  RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process
  (abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s.
- Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start
  state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes
  host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while
  Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick).
- Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps
  (wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log.
- Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on
  Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped.
- infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 19:44:31 -04:00
Joseph Doherty 8801631739 Merge b640-601: honest deploy acks — Applied only after the address-space outcome, degraded flag + reason (#601)
v2-ci / build (push) Successful in 4m42s
v2-ci / unit-tests (push) Failing after 20m19s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

# Conflicts:
#	src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs
2026-10-08 18:00:23 -04:00
Joseph Doherty 0a821417ea fix(deploy): derive the apply-outcome timeout from central's apply deadline (105 s) and name a timeout reason distinctly (#601 review)
60 s labelled a slow-but-clean large rebuild degraded. The deadline moves to a
shared Commons DeployTimings constant that ConfigPublishCoordinator reads, and
the node's timeout is deadline - 15 s margin, so the held ack still lands in
time. A timed-out ack's reason now says the outcome was not reported in time,
distinct from a real sink failure.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:54:09 -04:00
Joseph Doherty 345f40550b fix(deploy): a node with a ConfigDb records its degraded apply on its own NodeDeploymentState row (#601 review)
The apply writes the node's row Applied/null before the address-space outcome
is known; central adds the reason only when it ACCEPTS the ack, so a discarded
(stale/timed-out) ack left a fused node's row reading clean. The prefix moves
to ApplyAck.DegradedReasonPrefix so both writers spell the row identically.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:46:18 -04:00
Joseph Doherty fe7603ec6e fix(opcua): an identical re-deploy after a degraded REBUILD repairs instead of reporting clean (#601 review)
A degraded rebuild advances _lastApplied (deliberately), so re-deploying the
same revision diffed to an empty plan and reported a CLEAN outcome - the
#601 're-deploy re-applies' repair path sealed green over an address space
still known partial. The publish actor now remembers that the last advance
was degraded and, on an empty plan, re-ensures the whole composition
(idempotent, no rebuild), reporting clean only when that pass is clean.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:44:12 -04:00
Joseph Doherty 90d7f6dfca fix(deploy): hold the Applied ack until the address-space apply outcome is known (#601)
DriverHostActor no longer acks Applied the moment the revision advances. It
asks OpcUaPublishActor for the outcome (RebuildAddressSpace.ReportOutcome) and
acks Applied+Degraded(reason) when the apply was degraded; a missing outcome
times out (60 s, inside the coordinator's 2 min deadline) to a degraded ack,
never a clean one. Re-dispatching a revision whose apply was degraded or is
still unconfirmed re-applies instead of taking the clean idempotent
short-circuit - that re-apply is also what repairs the degraded nodes.
Tests with a bare publish probe use a clean-outcome auto-pilot probe.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:29:45 -04:00
Joseph Doherty 5af9aae2d4 Merge b640-641: native alarm state survives rebuild; projector pruned; AbCip ConditionId=RawPath (#641)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:26:47 -04:00
Joseph Doherty 521e1eb89b feat(deploy): additive degraded flag on ApplyAck + publish-actor apply outcome reply (#601)
ApplyAck gains trailing Degraded/DegradedReason (defaulted, so an old node's
DPS JSON or gRPC ack decodes exactly as before). mesh_transport.proto adds
ApplyAckMessage.degraded = 6 / optional degraded_reason = 7. OpcUaPublishActor
replies RebuildCompleted(Degraded, Reason) on every handler path when the
RebuildAddressSpace carries ReportOutcome.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:19:57 -04:00
Joseph Doherty 25f02e4102 fix(runtime): the native-alarm projector learns a routed client acknowledgement (#641 review)
#641 keeps the projector's prior state across a deploy instead of resetting it to the
acked default. The projector never saw a client's ack (the SDK applies it to the node and
the host only routes it), so a Clear after ack+deploy now projected Acked=false and
un-acknowledged the operator's ack - previously masked by the per-deploy reset.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:19:24 -04:00
Joseph Doherty 4128d9c313 test(runtime): de-flake Rediscovery_flag_persists (vacuous AwaitAssert, no publish trigger)
The AwaitAssert lambdas returned bool into an Action, so they never waited;
the test raced the rediscovery publish (5-22% failure on base too). Make the
asserts real and drive the republish with a short health poll interval.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:10:55 -04:00
Joseph Doherty 64d2514e2a fix(abcip): report ALMD transitions with ConditionId = RawPath and a Kind (#641)
Found while verifying item 4 of #641. The host routes a native alarm transition
on an exact ConditionId == RawPath match (DriverHostActor.ForwardNativeAlarm),
but AbCipAlarmProjection emitted "{ref}#active" and no Kind, so every ALMD
raise/clear was dropped as "no alarm condition" — and even if routed, an
Unspecified kind projects no state change. It now reports the subscribed
reference (the RawPath) and Raise/Clear. A host test pins that a decorated id
is dropped.

FOCAS has the same id shape plus a device filter keyed by host address (a
RawPath subscription polls nothing); it needs a mapping decision and is
documented in docs/drivers/FOCAS.md rather than patched here.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:08:47 -04:00
Joseph Doherty 11f13f6326 fix(runtime): prune the native-alarm projector on deploy instead of clearing it (#641)
DriverHostActor cleared NativeAlarmProjector's per-condition prior state on
EVERY apply, so after any deploy a driver Acknowledge of a still-active alarm
projected Active=false (writing it inactive) and a Clear of an unacknowledged
alarm projected Acked=true (silently acknowledging it). The projector is now
pruned to the conditions the new composition still carries (RetainOnly).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:05:42 -04:00
Joseph Doherty ddd7678269 fix(opcua): carry native alarm condition state across a full rebuild (#641)
A full RebuildAddressSpace recreated every native Part 9 condition inactive +
acknowledged, and when the alarm-tag set was unchanged (e.g. one
UnsTagReference added to an alarm-bearing raw tag) nothing re-told the new node
its state: the host does not re-subscribe for an unchanged set, and Galaxy /
OpcUaClient have no snapshot to replay anyway. Active alarms read normal until
their next device transition.

OtOpcUaNodeManager now captures each NATIVE condition's live state (from the
node, so a client-applied ack is kept) before the rebuild clears, restores it
when the same id is re-materialised native in the same apply, and reports ONE
condition event per restored off-normal condition at the new
IOpcUaAddressSpaceSink.CompleteAddressSpaceApply (after notifier wiring).
Unclaimed captures are discarded there, so a later re-add is fresh. Quality is
carried silently for normal conditions too (it is only re-annotated on a
connectivity edge). Scripted conditions are untouched (#487 re-asserts them).
OpcUaPublishActor calls CompleteAddressSpaceApply at the end of every apply.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:02:12 -04:00
Joseph Doherty c2bdb7d5de feat(hosts): driver diagnostics channel on /hosts, S7 negotiated PDU size first (#640)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 16:57:38 -04:00
Joseph Doherty 9c8e3da5da fix(mtconnect): review rulings for CONDITION alarms (#626)
- Unavailable keeps an ACTIVE condition active: quality-only BadNoCommunication; on data
  return same statement => quality-only Good, Normal => Clear, different => Retrigger.
- A changed alarm reference set (alarm-bearing delta = full rebuild) clears the whole
  last-told record so every active condition is re-asserted; the same set keeps it.
- Quality-only events go through the host's AlarmQualityUpdate path, never a projected
  snapshot (which could ack a retained unacked condition or rewrite message/severity).
- Deterministic tie-break between equal-level conditions (ordinal nativeCode, then type).
- Dedup key is level+nativeCode+nativeSeverity+qualifier; a text-only change updates nothing.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 15:38:52 -04:00
Joseph Doherty cb85a455c9 feat(alarms): exact OPC UA severity + per-condition quality on native alarm events (#626)
AlarmEventArgs gains optional OpcUaSeverity (supersedes the four-bucket mapping, clamped
1..1000) and QualityStatusCode (projected onto the condition's Quality). An Unspecified-kind
event carrying a quality is a quality-only annotation: the host updates the condition but
publishes no alerts row. Both default null, so every existing native source is unchanged.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 14:55:46 -04:00
Joseph Doherty 4a10ea1e1c fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.

- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
  Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
  survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
  re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
  fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
  real invoker's bracket across the removal and signals when its late writes are done (positive
  control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
  (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
  host test.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 09:07:11 -04:00
Joseph Doherty 3be8a1217e fix(runtime): the passive half of a LocalDb pair stands its alarm drain down for its mesh partner (review R6 MINOR-4)
The drain gate's 'is my queue-sharing peer Primary?' used only the host of
LocalDb:Replication:PeerAddress, which the passive half (ApiKey +
SyncListenPort) does not set - so as Secondary it kept draining beside its
Primary partner and every alarm event reached the historian twice.
DriverHostActor takes a sharesAlarmQueueWithLocal predicate used when no
peer host is known; WithOtOpcUaRuntimeActors builds it
(BuildAlarmQueuePeerPredicate) for a SharesRowsWithPeer node with no
PeerAddress as 'a member of my own cluster-{ClusterId} mesh other than
me'. A node sharing no rows, or with no mesh role, gets none and keeps
draining. AlarmHistorian.md / Redundancy.md / pair-replication runbook
updated.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:43:06 -04:00
Joseph Doherty e93d115cbe fix(runtime): pin the Primary-verdict member-event subscription with a test; document the pre-join allow window (review R6 MINOR-2/3)
MINOR-3: new DriverHostActorPrimaryRoleSourceTests case drives the
deny flip with a real ClusterEvent.MemberUp on the EventStream (how the
cluster's domain-event publisher reaches a Cluster.Subscribe subscriber)
instead of a direct RepublishPrimaryVerdict tell; it fails when the
Cluster.Subscribe(IMemberEvent) call is deleted.

MINOR-2: the published verdict MIRRORS the live gate; before this node has
joined its member count is 0 and both allow (seconds; up to the bootstrap
guard's 25 s probe), so the Sparkplug primary-host claim can be held on
both nodes in that window (native ack is unaffected - view and route
agree). Behaviour deliberately unchanged: denying until self is Up would
change the live gate too and delay writes/acks on a single-node cluster's
own join. Comments + AlarmTracking.md / Redundancy.md / Mqtt.md corrected.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:33:52 -04:00
Joseph Doherty 52708bff52 fix(address-space): numeric array writes convert (checked) to the node element type; array-to-scalar refusal only for Modbus/S7 (#610 review R6)
MAJOR-1: R4's exact element-type match refused ScadaBridge's CLR-list
writes (its element types are only Boolean/Int32/Float/Double/String/
DateTime) to every Int16/UInt16/Byte/UInt32/Int64/UInt64 array node and a
Double list to a Float array. EvaluateEquipmentWriteStructure now converts
a numeric array of another numeric element type element-wise, CHECKED, into
the node's element type and hands the converted array back through
OnEquipmentTagWrite's ref value, so the node stores and the driver receives
the node's own type. Any out-of-range, fractional-to-integer, NaN/Inf-to-
integer or Float-overflowing element refuses the whole write BadTypeMismatch
(nothing dispatched) - so the UInt32 -> Int32 bit reinterpretation stays
closed. Non-numeric mismatches stay refused. The wrong 'no client relies on
a looser rule' comment is corrected.

MINOR-1: the array-value-to-scalar-node refusal broke writes back to
browse-committed array tags (committed scalar, isArray never set) on
pass-through drivers. It now applies only when the owning driver's scalar
encoder cannot take an array (DriverArrayWriteSupport.
RefusesArrayValueOnScalarTag: Modbus, S7), carried RawTagPlan.
ScalarRefusesArrayValue / UnsReferenceVariable.BackingScalarRefusesArrayValue
-> EnsureVariable(refuseArrayOnScalar) -> a per-node set in the node
manager. OpcUaClient/Galaxy/others keep pre-#610 dispatch. Not on the
classifier whitelist (driver-type edit rebuilds); a mirror whose flag
changes is recreated, not updated in place.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:31:46 -04:00
Joseph Doherty b99aea46b2 Merge review R1 fixes: removed driver's resilience tracker released; SDK timed-unshelve no-op (engine owns expiry); commented re-ack stored as AddComment; native-ack Primary verdict published from start; drain gate SharesRowsWithPeer; malformed cache bytes don't adopt the revision; HostAbortedException passthrough; ClusterRoleInfo seed filtered
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:57 -04:00
Joseph Doherty e3dcb180c3 fix(runtime): Primary verdict published from host start, cache boot never adopts malformed bytes, strict drain-gate predicate, HostAbortedException passthrough, seed only Up members
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
  cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
  Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
  re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
  SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:34 -04:00
Joseph Doherty 8a9bf23048 fix(telemetry): release a removed driver's resilience tracker state so the publisher stops re-emitting it (#609)
Review R1 M1: StopChild evicted the hub cache, but DriverResilienceStatusPublisherService re-emits the
tracker every tick. IDriverCapabilityInvokerFactory.Release (default no-op) now invalidates the pipeline
builder + status tracker; the host calls it before Evict. Test runs a publisher tick after removal, with a
neighbour instance as positive control.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:13 -04:00
Joseph Doherty 18f8eb5d2f feat(address-space): server-side array writes gated per driver capability (#610)
Array nodes were forced read-only for every driver. A new single authority,
Core.Abstractions DriverArrayWriteSupport, says which (driver, element type)
pairs encode a whole-array write: Modbus (Coils / numeric HoldingRegisters),
S7 (Byte/Int16/UInt16/Int32/UInt32/Float32), OpcUaClient (pass-through) and
Galaxy (Boolean/Int32/Int64/Float/Double/String/DateTime). Everything else
stays read-only, so the server never advertises a write the driver refuses.

- Composer derives RawTagPlan.ArrayWritable + UnsReferenceVariable
  .BackingArrayWritable at compose time (both seams, byte-parity); the
  applier's one writability rule is Writable && (!IsArray || ArrayWritable)
  for materialise and surgical paths, raw and UNS alike.
- ArrayWritable is on the classifier's surgical raw-tag whitelist and
  mirrored on the UNS record (UnsReferenceBackingParityTests updated).
- Node manager refuses a wrong-length / non-array value on an array node
  (BadTypeMismatch) and any IndexRange write (BadWriteNotSupported) inline,
  before dispatch; WriteOperate/ACL gates and the revert are unchanged.
- Per-driver parity tests pin the table against the real Modbus write path,
  S7 EncodeArrayBlock and Galaxy MxValueEncoder (UInt32/UInt64 excluded:
  array covariance reinterprets their bits).
- Over-the-wire proof in OpcUaServer.IntegrationTests (ArrayWriteWireTests).
- ValueRank > 1 documented as out of scope by decision (docs/Uns.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:39:05 -04:00
Joseph Doherty e9e35baa70 Merge #600.1/#603: provision changed-to-historized tags; retire dormant VirtualTagEngine; cache ScriptSandbox references; one LocalDb replication predicate for registration and drain gate
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:01:41 -04:00
Joseph Doherty 4f6231fa40 refactor(localdb): one replication predicate shared by table registration and the alarm drain gate (#603)
The drain gate counted SyncListenPort; a port with no ApiKey shares nothing (fail-closed
passive endpoint), and an ApiKey-only passive node shares its queue but was never gated.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:56:00 -04:00
Joseph Doherty 53247a056a fix(runtime): malformed artifact bytes on boot/restore are no answer (#601); evict telemetry snapshots on driver removal (#609)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:53:45 -04:00
Joseph Doherty e280db3753 fix(runtime): comms_latched_refs meter tag key 'driver' -> 'driver_instance' — the R3-04 bare-role-literal guard caught the R3-02 tag key at merge (cross-branch collision; neither branch saw the other) 2026-08-20 11:55:16 -04:00
Joseph Doherty 886082fec7 fix(review): calc ReadOne reports an inputs-Bad publish; VT inputs-bad counter ticks only on a real report 2026-08-20 11:48:41 -04:00
Joseph Doherty 62c5bc3aad fix(runtime): a stopping driver child announces comms loss so restart/respawn latches its nodes Bad 2026-08-20 11:07:02 -04:00
Joseph Doherty 1d251cc037 Merge branch 'master' into fix/r3-02-stale-good-family
# Conflicts:
#	src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
#	tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Harness/V3ArtifactSeeder.cs
2026-08-20 11:03:02 -04:00
Joseph Doherty e3a72dc385 fix(runtime): virtual tags hold on Bad inputs and inherit the worst input status (archreview 02/S14) 2026-08-20 10:53:26 -04:00
Joseph Doherty 0793b0cba8 fix(runtime): rate-limit the supervision reprovision — deferred, not dropped (review F3)
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.

Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.

Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
2026-08-20 10:39:13 -04:00
Joseph Doherty 3f8d5815b3 fix(runtime): mux adapter forwards the dependency's real status code (05/STAB-19) 2026-08-20 10:30:14 -04:00
Joseph Doherty ae97b32c82 test(runtime): comms-loss → identical-value recovery contract + silent-resubscribe control (03/S15) 2026-08-20 10:27:48 -04:00
Joseph Doherty 2b7100cf19 fix(runtime): route a reprovision by incarnation, not driver id (review F2)
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.

Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
2026-08-20 10:27:02 -04:00
Joseph Doherty edd01fcc34 fix(runtime): a changed alarm set re-subscribes instead of latching (review F1)
The alarm subscribe gated on 'a handle exists', which is correct for an idempotent
re-push and wrong for a deploy that widens/narrows the set. Combined with the
spawn-time desired set, a respawning deploy could latch the PREVIOUS revision's
alarm refs for the life of the connection; sourceNodeIds-filtering drivers
(OpcUaClient/AbCip/Focas) would never deliver the new alarm.

Seen RED: LastSubscribedAlarmRefs stayed [alarmX] after a deploy widening to
[alarmX, alarmY].
2026-08-20 10:24:52 -04:00
Joseph Doherty 7d98ef9332 feat(runtime): audit refs still BadNoCommunication after a reconnect (dedup-trap tripwire) 2026-08-20 10:16:01 -04:00
Joseph Doherty 97f4ea0c9e fix(runtime): comms loss marks raw + UNS value nodes BadNoCommunication, value retained (archreview 03/S15) 2026-08-20 10:07:12 -04:00
Joseph Doherty 5c7e5725a5 test(runtime): first supervision-restart test — child re-initialises + re-subscribes (archreview 03/U6)
Crash seam: the planned null-Snapshot OnDataChange (no SupervisorStrategy fallback needed).
Seen RED with T6 reverted: InitializeCount 1, not 2 — with a preceding ShutdownCount>=1
assertion proving the restart really happened, so the red is not ambiguous with 'never crashed'.
2026-08-20 10:05:45 -04:00
Joseph Doherty 04019e9414 feat(runtime): publish actor routes status-only batches to the sink 2026-08-20 10:03:25 -04:00
Joseph Doherty d97e441979 test(runtime): operator restart re-subscribes the respawned child (archreview 03/S16, U6)
Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
2026-08-20 09:53:38 -04:00
Joseph Doherty b3a71f6655 test(runtime): recording subscribable driver factory + enabled-driver v3 seed 2026-08-20 09:42:33 -04:00
Joseph Doherty a801e2aefb fix(runtime): comms-loss grace is per connection — max(LastSuccessfulRead, connectedSince) (archreview 03/S17) 2026-08-20 09:32:30 -04:00
Joseph Doherty df53391646 test(localdb): deregister preserves data + partial-flip fails loudly (review round)
Review round on 3e62f161. Two coverage gaps and three wording/comment nits.

IMPORTANT 1 — deregistration is now proven DATA-PRESERVING. The self-heal test
inserted a deployment_pointer row and rebooted unconfigured but never looked at
the row again, so a library regression that touched base-table data during
cleanup would have gone unnoticed — on the very tables that ARE the
boot-from-cache backstop, and on the default-OFF nodes least likely to notice.
The new DisablingReplication_PrunesBookkeepingButNeverTouchesTheCachedDataItself
asserts the row's CONTENT survives (not just its count — a count passes against
a cleanup that blanks columns) while __localdb_row_version and __localdb_oplog,
the bookkeeping deregistration does own, go to zero.

IMPORTANT 2 — mid-loop partial failure is documented and pinned. Both loops flip
four tables with no try/catch, so a throw on the third leaves two flipped and
propagates out of the AddZbLocalDb singleton factory, failing host start. That
is correct — a node that came up anyway would be silently capturing on an
arbitrary subset — and both library calls are idempotent, so the retried boot
completes the flip. Written into OnReady's remarks and pinned by
AFailureMidwayThroughRegistration_FailsBootLoudly_AndTheRetriedBootConverges,
which drives OnReady through a delegating ILocalDb that throws on the Nth
RegisterReplicated, asserts the throw propagates, asserts the wrapper actually
reached that call, then re-runs OnReady unwrapped and asserts all four converge.

MINOR 3 — the Boot helper now records that OnReady's null-logger fallback lands
on Serilog's static Log.Logger, which this assembly never assigns; assertions
about logging pass an explicit logger.
MINOR 5 — log wording: "are both unset" -> "are both absent — unset or blank",
matching IsNullOrWhiteSpace.
MINOR 4 — LocalDbDeploymentArtifactCacheTests said its setup "mirrors
LocalDbSetup.OnReady"; it reproduces the two tables only and registers them
unconditionally without baselining. Softened to say so and to name the
divergence as intentional and inert there.

Falsified both new pins. Wrapper set to failOnCall: 99 (never throws) => the
partial-flip pin is the SOLE failure, so it genuinely exercises the Nth-call
throw path rather than some incidental exception. Predicate forced constant-true
=> five pins red including the new data-preservation one. Both mutations
reverted.

LocalDbSetupTests 15/15. Host.IntegrationTests 226 passed / 0 failed / 5
skipped. Runtime.Tests LocalDbDeploymentArtifactCacheTests 12/12.
2026-08-18 07:42:06 -04:00