Commit Graph

205 Commits

Author SHA1 Message Date
Joseph Doherty b39bb6794d fix(alarms): native-alarm re-statements no longer duplicate history or hide acked alarms (#656)
- AlarmTransitionKind.ActiveAcknowledged (appended): active AND acked applied as ONE projection.
  Galaxy ActiveAcked snapshot entries and OpcUaClient inactive->(active,acked) use it; the
  projector writes 'Activated' only if not already active, 'Acknowledged' only if not acked.
- AlarmEventArgs.IsRestatement (optional, default false): set only by Galaxy snapshot entries
  and OpcUaClient first sight. A flagged Retrigger on an Active+unacked condition with the same
  message and severity, or a flagged Clear on an inactive condition, writes no /alerts row.
- NativeAlarmProjector reports the history rows per transition; ForwardNativeAlarm writes one
  row per entry. Unflagged sources (FOCAS/MTConnect/AbCip, live Galaxy) are unchanged.
- Tests: projector rules each with a must-write control; OpcUaClient transition table;
  Galaxy feed/driver mapping; Galaxy + OpcUaClient end-to-end through DriverHostActor.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 23:35:50 -04:00
Joseph Doherty e7dd742541 Merge b647-focas-alarms: FOCAS native alarms reach Part 9 conditions (ALARM: address) + respawn keeps acks host-side (#647)
v2-ci / build (push) Successful in 4m57s
v2-ci / unit-tests (push) Failing after 21m6s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:46:13 -04:00
Joseph Doherty bac93744ce fix(alarms): #647 re-review — re-statement needs the same message; CNC refusal is not comms loss; worst-of health
R1: NativeAlarmProjector treats Raise-on-Active as a re-statement only when the message matches what
the condition shows; a different alarm appearing during a respawn un-acks and writes its alerts row.
FOCAS condition messages are a deterministic function of the matched set (tested across fresh driver
instances and reordered lists); Runtime E2E now covers both the same alarm and a new one during a
respawn. Residual (source message determinism) documented in AlarmTracking.md.

R2: a non-OK cnc_rdalmmsg2 return is thrown as FocasWireException{Rc} and classified as a CNC
refusal: conditions get a quality-only BadNotSupported once, one Warning, polling continues, and
driver health is untouched (no 90 s watchdog reconnect loop on a production control). Only transport
faults degrade (falsification control in the test).

R3: alarm-path health is tracked apart from the value path and combined in GetHealth: worst state,
later LastSuccessfulRead. A good value read can no longer mask a failing alarm read (no per-poll
/hosts flip) and a good alarm read cannot mask a value Degraded.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:45:32 -04:00
Joseph Doherty beccfc6404 test(#606): native-ack routing waits for the driver child to be Connected; widen a starved poll
DriverHostActorNativeAlarmAckRoutingTests: Applied does not mean the child DriverInstanceActor is
Connected, and an ack reaching a still-Connecting child is dropped by design — the single
positive-control ack was itself dropped (4/20 under DOTNET_PROCESSOR_COUNT=1 x8, still failing
with the presence budget dilated to 45 s, so not a budget problem). SpawnHostAndApply now waits
for SubscribeAlarmsAsync, which only the Connected state calls. ScriptedAlarmEngine
OnEvent_emission_happens_outside_evalGate: 2 s blocking poll -> 15 s (5/20 under starvation).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:37:15 -04:00
Joseph Doherty 89961af6e6 fix(alarms): #647 review — Raise on an active condition is a re-statement; FOCAS outage quality + health
MAJOR-1: a deploy that changes a driver's config respawns it, and the fresh instance's empty told
record re-sent Raise for every still-active alarm, un-acking acknowledged conditions and writing a
duplicate "Activated" alerts row. NativeAlarmProjector now treats Raise on an already-active
condition as a re-statement (keeps Acked; ForwardNativeAlarm skips the alerts row); only Retrigger
(or inactive->active) un-acks. Sources whose upstream ack state is authoritative send Retrigger for
an unacked re-statement: OpcUaClient on first sight (no prior told), Galaxy for an unacked active
snapshot entry. Runtime E2E drives a FRESH FOCAS driver after a redeploy and asserts the ack survives.

MAJOR-2: a failed FOCAS alarm read annotates each subscribed condition BadNoCommunication once per
outage (#626 quality-only path) with one Warning (MINOR-6), lifted to Good on recovery; failed alarm
reads degrade driver health with LastSuccessfulRead frozen (successful ones advance it), so an
alarm-only driver trips the #527 comms-loss watchdog.

MINOR-3 CLAUDE.md #630 bullet; MINOR-4 split line; MINOR-5 a non-empty cnc_rdalmmsg2 payload that
fits neither stride throws; MINOR-7 value-tag + parsed-address tables are built and swapped, never
cleared in place under the retained poll engine. Docs corrected (FOCAS.md, AlarmTracking.md, CLAUDE.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 20:18:03 -04:00
Joseph Doherty 482bafdfac feat(focas): CNC alarms reach Part 9 conditions via ALARM: tags (#647)
An ALARM: address (*, <Type>, <Type>/<n>, <Type>/<lo>-<hi>, T<code>, OTHER) on a raw tag that
also carries the cross-driver alarm object is ONE condition with ConditionId == RawPath and a
Kind (Raise / Retrigger / Clear), so DriverHostActor.ForwardNativeAlarm routes it and
NativeAlarmProjector activates it. Fixes all seven triage defects:

1. SubscribeAlarmsAsync treated the host's RawPaths as device hosts -> nothing polled.
2. Decorated "{host}#{Type}:{Number}" ConditionId -> every event dropped.
3. No Kind -> projector kept prior state.
4. WireFocasClient.ReadAlarmsAsync answered [] on failure -> mass clear + un-acked re-raise.
   It now throws; a failed device read keeps every condition's state.
5. Acks lost on reconnect -> told-record dedup (MTConnect #626 pattern) held for the driver's
   lifetime; same-set re-subscribe / re-init re-raise nothing; changed set prunes leavers only.
6. Subscribe threw while AlarmProjection.Enabled=false -> the tag is the opt-in; flag is a no-op.
7. Alarm tags could not be authored -> ALARM: grammar (FocasAlarmSelector, Contracts) backs the
   editor, the address picker, the deploy-time inspector and driver init.

Severity: authored alarm.severity -> exact OpcUaSeverity; else the 16i-era type bucket. The type
table stays the only one (moved to Contracts, documented 16i-era; 30i codes not invented; raw
T<code> selector + raw code in every message). Read-only: ack is server-side. Also clears the
value table on a re-initialise (#630 gap that let a removed/moved tag keep reading).

E2E: Runtime.Tests FocasNativeAlarmEndToEndTests drives the real driver through the real
DriverHostActor and asserts the condition goes Active then inactive.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 19:46:48 -04:00
Joseph Doherty 55ae230fa4 ci(#606): Gitea-compatible trx upload + deterministic fixes for the CI-load test flakes
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and
  v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3
  on the test step.
- Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR
  (TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget.
- Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests
  RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process
  (abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s.
- Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start
  state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes
  host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while
  Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick).
- Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps
  (wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log.
- Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on
  Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped.
- infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 19:44:31 -04:00
Joseph Doherty 8801631739 Merge b640-601: honest deploy acks — Applied only after the address-space outcome, degraded flag + reason (#601)
v2-ci / build (push) Successful in 4m42s
v2-ci / unit-tests (push) Failing after 20m19s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

# Conflicts:
#	src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs
2026-10-08 18:00:23 -04:00
Joseph Doherty 0a821417ea fix(deploy): derive the apply-outcome timeout from central's apply deadline (105 s) and name a timeout reason distinctly (#601 review)
60 s labelled a slow-but-clean large rebuild degraded. The deadline moves to a
shared Commons DeployTimings constant that ConfigPublishCoordinator reads, and
the node's timeout is deadline - 15 s margin, so the held ack still lands in
time. A timed-out ack's reason now says the outcome was not reported in time,
distinct from a real sink failure.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:54:09 -04:00
Joseph Doherty 345f40550b fix(deploy): a node with a ConfigDb records its degraded apply on its own NodeDeploymentState row (#601 review)
The apply writes the node's row Applied/null before the address-space outcome
is known; central adds the reason only when it ACCEPTS the ack, so a discarded
(stale/timed-out) ack left a fused node's row reading clean. The prefix moves
to ApplyAck.DegradedReasonPrefix so both writers spell the row identically.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:46:18 -04:00
Joseph Doherty 90d7f6dfca fix(deploy): hold the Applied ack until the address-space apply outcome is known (#601)
DriverHostActor no longer acks Applied the moment the revision advances. It
asks OpcUaPublishActor for the outcome (RebuildAddressSpace.ReportOutcome) and
acks Applied+Degraded(reason) when the apply was degraded; a missing outcome
times out (60 s, inside the coordinator's 2 min deadline) to a degraded ack,
never a clean one. Re-dispatching a revision whose apply was degraded or is
still unconfirmed re-applies instead of taking the clean idempotent
short-circuit - that re-apply is also what repairs the degraded nodes.
Tests with a bare publish probe use a clean-outcome auto-pilot probe.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:29:45 -04:00
Joseph Doherty 5af9aae2d4 Merge b640-641: native alarm state survives rebuild; projector pruned; AbCip ConditionId=RawPath (#641)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:26:47 -04:00
Joseph Doherty 25f02e4102 fix(runtime): the native-alarm projector learns a routed client acknowledgement (#641 review)
#641 keeps the projector's prior state across a deploy instead of resetting it to the
acked default. The projector never saw a client's ack (the SDK applies it to the node and
the host only routes it), so a Clear after ack+deploy now projected Acked=false and
un-acknowledged the operator's ack - previously masked by the per-deploy reset.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:19:24 -04:00
Joseph Doherty 4128d9c313 test(runtime): de-flake Rediscovery_flag_persists (vacuous AwaitAssert, no publish trigger)
The AwaitAssert lambdas returned bool into an Action, so they never waited;
the test raced the rediscovery publish (5-22% failure on base too). Make the
asserts real and drive the republish with a short health poll interval.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:10:55 -04:00
Joseph Doherty 64d2514e2a fix(abcip): report ALMD transitions with ConditionId = RawPath and a Kind (#641)
Found while verifying item 4 of #641. The host routes a native alarm transition
on an exact ConditionId == RawPath match (DriverHostActor.ForwardNativeAlarm),
but AbCipAlarmProjection emitted "{ref}#active" and no Kind, so every ALMD
raise/clear was dropped as "no alarm condition" — and even if routed, an
Unspecified kind projects no state change. It now reports the subscribed
reference (the RawPath) and Raise/Clear. A host test pins that a decorated id
is dropped.

FOCAS has the same id shape plus a device filter keyed by host address (a
RawPath subscription polls nothing); it needs a mapping decision and is
documented in docs/drivers/FOCAS.md rather than patched here.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:08:47 -04:00
Joseph Doherty 11f13f6326 fix(runtime): prune the native-alarm projector on deploy instead of clearing it (#641)
DriverHostActor cleared NativeAlarmProjector's per-condition prior state on
EVERY apply, so after any deploy a driver Acknowledge of a still-active alarm
projected Active=false (writing it inactive) and a Clear of an unacknowledged
alarm projected Acked=true (silently acknowledging it). The projector is now
pruned to the conditions the new composition still carries (RetainOnly).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:05:42 -04:00
Joseph Doherty c2bdb7d5de feat(hosts): driver diagnostics channel on /hosts, S7 negotiated PDU size first (#640)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 16:57:38 -04:00
Joseph Doherty 9c8e3da5da fix(mtconnect): review rulings for CONDITION alarms (#626)
- Unavailable keeps an ACTIVE condition active: quality-only BadNoCommunication; on data
  return same statement => quality-only Good, Normal => Clear, different => Retrigger.
- A changed alarm reference set (alarm-bearing delta = full rebuild) clears the whole
  last-told record so every active condition is re-asserted; the same set keeps it.
- Quality-only events go through the host's AlarmQualityUpdate path, never a projected
  snapshot (which could ack a retained unacked condition or rewrite message/severity).
- Deterministic tie-break between equal-level conditions (ordinal nativeCode, then type).
- Dedup key is level+nativeCode+nativeSeverity+qualifier; a text-only change updates nothing.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 15:38:52 -04:00
Joseph Doherty cb85a455c9 feat(alarms): exact OPC UA severity + per-condition quality on native alarm events (#626)
AlarmEventArgs gains optional OpcUaSeverity (supersedes the four-bucket mapping, clamped
1..1000) and QualityStatusCode (projected onto the condition's Quality). An Unspecified-kind
event carrying a quality is a quality-only annotation: the host updates the condition but
publishes no alerts row. Both default null, so every existing native source is unchanged.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 14:55:46 -04:00
Joseph Doherty 4a10ea1e1c fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.

- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
  Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
  survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
  re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
  fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
  real invoker's bracket across the removal and signals when its late writes are done (positive
  control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
  (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
  host test.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 09:07:11 -04:00
Joseph Doherty 3be8a1217e fix(runtime): the passive half of a LocalDb pair stands its alarm drain down for its mesh partner (review R6 MINOR-4)
The drain gate's 'is my queue-sharing peer Primary?' used only the host of
LocalDb:Replication:PeerAddress, which the passive half (ApiKey +
SyncListenPort) does not set - so as Secondary it kept draining beside its
Primary partner and every alarm event reached the historian twice.
DriverHostActor takes a sharesAlarmQueueWithLocal predicate used when no
peer host is known; WithOtOpcUaRuntimeActors builds it
(BuildAlarmQueuePeerPredicate) for a SharesRowsWithPeer node with no
PeerAddress as 'a member of my own cluster-{ClusterId} mesh other than
me'. A node sharing no rows, or with no mesh role, gets none and keeps
draining. AlarmHistorian.md / Redundancy.md / pair-replication runbook
updated.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:43:06 -04:00
Joseph Doherty e93d115cbe fix(runtime): pin the Primary-verdict member-event subscription with a test; document the pre-join allow window (review R6 MINOR-2/3)
MINOR-3: new DriverHostActorPrimaryRoleSourceTests case drives the
deny flip with a real ClusterEvent.MemberUp on the EventStream (how the
cluster's domain-event publisher reaches a Cluster.Subscribe subscriber)
instead of a direct RepublishPrimaryVerdict tell; it fails when the
Cluster.Subscribe(IMemberEvent) call is deleted.

MINOR-2: the published verdict MIRRORS the live gate; before this node has
joined its member count is 0 and both allow (seconds; up to the bootstrap
guard's 25 s probe), so the Sparkplug primary-host claim can be held on
both nodes in that window (native ack is unaffected - view and route
agree). Behaviour deliberately unchanged: denying until self is Up would
change the live gate too and delay writes/acks on a single-node cluster's
own join. Comments + AlarmTracking.md / Redundancy.md / Mqtt.md corrected.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 08:33:52 -04:00
Joseph Doherty b99aea46b2 Merge review R1 fixes: removed driver's resilience tracker released; SDK timed-unshelve no-op (engine owns expiry); commented re-ack stored as AddComment; native-ack Primary verdict published from start; drain gate SharesRowsWithPeer; malformed cache bytes don't adopt the revision; HostAbortedException passthrough; ClusterRoleInfo seed filtered
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:57 -04:00
Joseph Doherty e3dcb180c3 fix(runtime): Primary verdict published from host start, cache boot never adopts malformed bytes, strict drain-gate predicate, HostAbortedException passthrough, seed only Up members
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
  cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
  Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
  re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
  SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:34 -04:00
Joseph Doherty 8a9bf23048 fix(telemetry): release a removed driver's resilience tracker state so the publisher stops re-emitting it (#609)
Review R1 M1: StopChild evicted the hub cache, but DriverResilienceStatusPublisherService re-emits the
tracker every tick. IDriverCapabilityInvokerFactory.Release (default no-op) now invalidates the pipeline
builder + status tracker; the host calls it before Evict. Test runs a publisher tick after removal, with a
neighbour instance as positive control.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:47:13 -04:00
Joseph Doherty 18f8eb5d2f feat(address-space): server-side array writes gated per driver capability (#610)
Array nodes were forced read-only for every driver. A new single authority,
Core.Abstractions DriverArrayWriteSupport, says which (driver, element type)
pairs encode a whole-array write: Modbus (Coils / numeric HoldingRegisters),
S7 (Byte/Int16/UInt16/Int32/UInt32/Float32), OpcUaClient (pass-through) and
Galaxy (Boolean/Int32/Int64/Float/Double/String/DateTime). Everything else
stays read-only, so the server never advertises a write the driver refuses.

- Composer derives RawTagPlan.ArrayWritable + UnsReferenceVariable
  .BackingArrayWritable at compose time (both seams, byte-parity); the
  applier's one writability rule is Writable && (!IsArray || ArrayWritable)
  for materialise and surgical paths, raw and UNS alike.
- ArrayWritable is on the classifier's surgical raw-tag whitelist and
  mirrored on the UNS record (UnsReferenceBackingParityTests updated).
- Node manager refuses a wrong-length / non-array value on an array node
  (BadTypeMismatch) and any IndexRange write (BadWriteNotSupported) inline,
  before dispatch; WriteOperate/ACL gates and the revert are unchanged.
- Per-driver parity tests pin the table against the real Modbus write path,
  S7 EncodeArrayBlock and Galaxy MxValueEncoder (UInt32/UInt64 excluded:
  array covariance reinterprets their bits).
- Over-the-wire proof in OpcUaServer.IntegrationTests (ArrayWriteWireTests).
- ValueRank > 1 documented as out of scope by decision (docs/Uns.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:39:05 -04:00
Joseph Doherty 53247a056a fix(runtime): malformed artifact bytes on boot/restore are no answer (#601); evict telemetry snapshots on driver removal (#609)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:53:45 -04:00
Joseph Doherty e280db3753 fix(runtime): comms_latched_refs meter tag key 'driver' -> 'driver_instance' — the R3-04 bare-role-literal guard caught the R3-02 tag key at merge (cross-branch collision; neither branch saw the other) 2026-08-20 11:55:16 -04:00
Joseph Doherty 62c5bc3aad fix(runtime): a stopping driver child announces comms loss so restart/respawn latches its nodes Bad 2026-08-20 11:07:02 -04:00
Joseph Doherty 1d251cc037 Merge branch 'master' into fix/r3-02-stale-good-family
# Conflicts:
#	src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
#	tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Harness/V3ArtifactSeeder.cs
2026-08-20 11:03:02 -04:00
Joseph Doherty 0793b0cba8 fix(runtime): rate-limit the supervision reprovision — deferred, not dropped (review F3)
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.

Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.

Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
2026-08-20 10:39:13 -04:00
Joseph Doherty ae97b32c82 test(runtime): comms-loss → identical-value recovery contract + silent-resubscribe control (03/S15) 2026-08-20 10:27:48 -04:00
Joseph Doherty 2b7100cf19 fix(runtime): route a reprovision by incarnation, not driver id (review F2)
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.

Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
2026-08-20 10:27:02 -04:00
Joseph Doherty edd01fcc34 fix(runtime): a changed alarm set re-subscribes instead of latching (review F1)
The alarm subscribe gated on 'a handle exists', which is correct for an idempotent
re-push and wrong for a deploy that widens/narrows the set. Combined with the
spawn-time desired set, a respawning deploy could latch the PREVIOUS revision's
alarm refs for the life of the connection; sourceNodeIds-filtering drivers
(OpcUaClient/AbCip/Focas) would never deliver the new alarm.

Seen RED: LastSubscribedAlarmRefs stayed [alarmX] after a deploy widening to
[alarmX, alarmY].
2026-08-20 10:24:52 -04:00
Joseph Doherty 7d98ef9332 feat(runtime): audit refs still BadNoCommunication after a reconnect (dedup-trap tripwire) 2026-08-20 10:16:01 -04:00
Joseph Doherty 97f4ea0c9e fix(runtime): comms loss marks raw + UNS value nodes BadNoCommunication, value retained (archreview 03/S15) 2026-08-20 10:07:12 -04:00
Joseph Doherty 5c7e5725a5 test(runtime): first supervision-restart test — child re-initialises + re-subscribes (archreview 03/U6)
Crash seam: the planned null-Snapshot OnDataChange (no SupervisorStrategy fallback needed).
Seen RED with T6 reverted: InitializeCount 1, not 2 — with a preceding ShutdownCount>=1
assertion proving the restart really happened, so the red is not ambiguous with 'never crashed'.
2026-08-20 10:05:45 -04:00
Joseph Doherty d97e441979 test(runtime): operator restart re-subscribes the respawned child (archreview 03/S16, U6)
Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
2026-08-20 09:53:38 -04:00
Joseph Doherty b3a71f6655 test(runtime): recording subscribable driver factory + enabled-driver v3 seed 2026-08-20 09:42:33 -04:00
Joseph Doherty a801e2aefb fix(runtime): comms-loss grace is per connection — max(LastSuccessfulRead, connectedSince) (archreview 03/S17) 2026-08-20 09:32:30 -04:00
Joseph Doherty dba5f806d3 fix(runtime): explicit 30s command timeout on the Direct-apply artifact read 2026-08-17 03:49:27 -04:00
Joseph Doherty dbc9223206 docs(code): correct comments stale after the perf remainder 2026-08-16 05:09:00 -04:00
Joseph Doherty 5b92eb0ff6 perf(runtime): direct-mode artifact read piped off the host mailbox 2026-08-16 04:48:02 -04:00
Joseph Doherty b6275f439c perf(runtime): per-driver index for connectivity fans; comms-loss is one batch 2026-08-16 04:27:20 -04:00
Joseph Doherty 0dbbb5ac6b fix(runtime): stash dedup + applying-wedge watchdog; cache-failure diagnostics 2026-08-15 17:43:15 -04:00
Joseph Doherty 14c31941c7 fix(runtime): parallel-safe decompression limit; tighten ParsedDeploymentArtifact surface 2026-08-15 17:16:09 -04:00
Joseph Doherty 5f2e60278c perf(runtime): apply-path cache write off-mailbox; stash deferred dispatches 2026-08-15 17:12:01 -04:00
Joseph Doherty 875ec9fe47 fix(opcuaserver): UNS reference nodes repair on backing-tag historian/shape edits
Adversarial review of 9f68f272 (classifier: non-alarm raw-tag deltas became
surgical) found stale UNS inheritance after a surgical raw edit. Editing a raw
tag's IsHistorized / HistorianTagname / IsArray / ArrayLength left every
referencing UNS node STALE: none of those fields were on UnsReferenceVariable,
so no coincident UnsReferenceDelta fired; the applier updated the raw node in
place; and EnsureVariable early-returns on an existing node. Consequences:
HistoryRead through the UNS NodeId served a different series than through the
raw NodeId (breaking the documented dual-registration invariant), a
de-historized tag kept serving history via UNS, and a scalar->array flip left
the UNS mirror scalar AND writable (bypassing the array read-only write gate).

UnsReferenceVariable now carries Backing{IsHistorized,HistorianTagname,IsArray,
ArrayLength}, copied from the backing tag's TagConfigIntent at compose time, so
record equality makes the reference Changed in the SAME plan. Both deltas are
non-rebuild kinds, so the existing machinery repairs the mirror surgically:
remove-previous in Apply, recreate in MaterialiseUnsReferences. That pass now
reads the four values from the RECORD (the single truth the planner diffed)
instead of a rawByPath back-lookup, preserving historian resolution exactly
(blank override => BackingRawPath, which is the backing tag's NodeId).

The artifact-decode seam needs no change: DeploymentArtifact.BuildRawAndUnsSubtrees
drives AddressSpaceComposer.Compose verbatim, which is what guarantees byte-parity.
The classifier needs no change either (a UnsReferenceDelta never forces Rebuild);
its comment now records why.

Also from the same review: a three-place-contract comment on RawTagPlan's
hand-written Equals naming its two downstream consumers, plus classifier pins for
the .Any(!eligible) quantifier, an ArrayLength-only delta, and an
alarm-content-only delta.
2026-08-15 17:11:03 -04:00
Joseph Doherty 9dbd5e96f7 perf(controlplane): gzip artifact blob inside the hashed bytes (~10-20x smaller)
ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest)
immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so
the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob,
the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a
FetchAndCache node verifies against. The invariant the deploy path rests on
— RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what
leaves the fetcher, the cache and the serve path untouched: nothing between
the composer and DeploymentArtifact.Open ever looks inside the blob.

Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start
with 0x1F, so no framing of our own is needed) and inflates through a size
guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can
be exercised without inflating half a gigabyte per test run. Pre-upgrade
uncompressed blobs take the identical path they always did.

Every decompression failure — corrupt stream, truncated stream, over-limit
inflation — is reported as JsonException, the same exception family the
malformed-JSON path already owns. So the lenient static wrappers still
degrade to empty results and the apply paths still fail the apply and keep
last-known-good (#485); no caller's decision changes because the reason the
bytes are unusable changed.

gzip output is not guaranteed byte-stable across .NET releases. Harmless
here: the hash is computed FROM the emitted bytes at compose time and never
re-derived from the JSON, so the worst an implementation change costs is
one spurious "changed" revision after a runtime upgrade. Snapshot
determinism is unaffected — compression wraps the final bytes and reorders
nothing.

Tests: gzip/uncompressed parity through both the instance parses and the
statics, corrupt-gzip leniency through the wrappers, the size guard (with a
falsifiability control proving the same bytes inflate fine under the
production limit), and composer-side blob-starts-with-1F8B +
RevisionHash == SHA256(blob) with a control proving the hash is NOT over
the JSON.
2026-08-15 17:00:19 -04:00
Joseph Doherty 59202017cd refactor(runtime): ParsedDeploymentArtifact — one JsonDocument per apply
Every public entry point on DeploymentArtifact took ReadOnlySpan<byte> and did
JsonDocument.Parse(blob.ToArray()) — a full copy plus a full document build per
call — so one apply paid ~8-11 full parses of the same bytes (the scoped
ParseComposition alone parsed three times: scope, composition, BuildClusterSets).

DeploymentArtifact.Open(byte[]) now parses once into ParsedDeploymentArtifact,
which owns the JsonDocument and exposes an instance method mirroring each static
entry point. The moved bodies operate on Root; BuildClusterSets takes JsonElement
instead of re-parsing the blob; ResolveClusterScope is memoized per node id, since
every consumer of one apply asks about the same node.

Every static is now a thin wrapper (Open + delegate + dispose) keeping its
ReadOnlySpan<byte> signature and its lenient empty/malformed contract — the
wrapper pays the one copy and owns the JsonException catch, so output shapes are
unchanged. Hot callers migrate to Open() separately.

ParsedDeploymentArtifactTests pins same-bytes-same-output through both doors for
all six entry points over a rich multi-cluster artifact (raw chain + UNS refs +
scripts + keys + ACLs), with falsifiability controls proving the fixture actually
scopes, plus memoization and after-dispose-use coverage.
2026-08-15 13:23:05 -04:00