Commit Graph

40 Commits

Author SHA1 Message Date
Joseph Doherty bd63810645 fix(alarms): native ack refused off-Primary, commented re-ack reported, real UnshelveTime (#597)
- Native-condition Acknowledge on a non-Primary node returns BadInvalidState (nothing
  routed/applied/reported) via NativeAlarmAckPrimaryGate, fed from
  IDriverPrimaryRoleSource by OtOpcUaServerHostedService.
- A re-ack carrying a comment reports one comment event (Acked/Confirmed unchanged).
- TimedShelve applies its requested duration and the scripted projection carries
  UnshelveAtUtc, so ShelvingState/UnshelveTime counts down instead of ~24.8 days.
- Native re-ack still routed to the driver (deliberate: documented retry path), pinned.
- AdminUI /alerts client ack/shelve path test; docs/AlarmTracking.md decisions incl.
  accepted SDK EventsMonitored residual (item 6).
- Also #599/03/S20: RemoveEquipmentSubtree severs inverse Organizes edges.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 07:12:29 -04:00
Joseph Doherty e46c1db5e5 fix(cluster): gossiped-role predicates compare case-insensitively — a miscased peer no longer un-elects the Primary 2026-08-20 10:04:16 -04:00
Joseph Doherty b9e9df4081 docs(code): honest rationales — Akka restart semantics, per-key shards, EF literal inlining; retire or annotate orphaned members 2026-08-16 05:33:33 -04:00
Joseph Doherty 384568b47a perf(historian): batch ingest — one group commit per value burst 2026-08-16 04:33:35 -04:00
Joseph Doherty 140bf0c131 perf(adminui): pooled DbContext factory; role-grants reads per-call, not per-circuit 2026-08-16 04:05:34 -04:00
Joseph Doherty 4d2dfb5b50 test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:

  1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
     point, kept as a test rather than a live-gate note: each cluster stays single-member and
     none ever holds another's address as a member.
  2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
     site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
     would be refused at the far side.
  3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
     it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
     process did". The same node then accepts the right key, so the silence is the auth gate
     and not a dead endpoint.

All three run fully offline — no Docker fixture — and pass.

Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.

The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
2026-08-06 15:58:36 -04:00
Joseph Doherty c81cef39f5 feat(alarms): bridge driver comms-loss into scripted-alarm input quality (#481)
Layer 4 of the condition-Quality chain (#477 -> #478). #478 gave a scripted
condition the worst quality of its input tags, but only for inputs whose driver
PUBLISHES a Bad data change. A poll driver whose device goes unreachable
publishes nothing — it raises ConnectivityChanged(false) and goes silent — so
the engine held its last Good value, WorstInputStatus stayed Good, and worse,
AreInputsReady stayed true, so the predicate went on evaluating a stale value
after the source had died.

`DriverHostActor.FanCommsLossToDependencyMux` pushes a synthetic
BadNoCommunication with a null value into the dependency mux for every value ref
the disconnected driver owns.

The issue asked for a driver -> all-tag-refs map to be built for this. It
already exists: `_nodeIdByDriverRef`, keyed (DriverInstanceId, RawPath) and
rebuilt from the composition on every apply. Alarm-bearing tags live in the
separate `_alarmNodeIdByDriverRef` and are deliberately excluded — a native
condition is not a mux value and takes the Part 9 quality path instead.

Nothing is pushed on reconnect, deliberately: publishing Good would assert a
value quality this actor has not observed. Recovery rides the driver's own
republish, and the mechanism was verified rather than assumed — the
Connected -> Reconnecting edge calls DetachSubscription, which nulls the
subscription handle, so the reconnect's ResubscribeDesired builds a FRESH
subscription whose PollGroupEngine initial push runs with forceRaise: true.
Without that detach the poll engine would compare against its retained
LastValues, see an unchanged value, raise nothing, and strand these refs Bad
forever — the value-dedup trap that bit the v3 Batch 4 virtual-tag reassert.

Closes #478 Finding 3, which the above depends on. WorstInputStatus skipped
every input whose VALUE was null as "no data yet" — and Bad-with-no-value is
exactly the shape a comms-lost driver produces, so the skip was hiding the case
that matters most; without this half the fan-out above would be silently inert.
The upstream source now stamps a never-pushed path BadWaitingForInitialData —
the honest Part 4 code for that state — and the engine skips that specific code
rather than every null value.

Deleting the null-skip outright is the obvious fix and is wrong in the other
direction: it makes every not-yet-published input drag its condition Bad at
deploy, flooding the quality path with load-time annotations, which is what the
skip existed to prevent. Both mistakes are pinned by a test pair that goes red
under the opposite error, and both were falsified by reverting the production
code:

  old null-skip restored     -> A_pushed_Bad_carrying_no_value      RED
  null-skip deleted outright -> An_input_that_was_never_pushed      RED
  fan-out call removed       -> the three DriverHostActor tests     RED (1 of 7)

The never-pushed test was vacuous on first writing and is worth recording: the
quality bucket only emits on CHANGE, so the load-time evaluation is the one that
flashes Bad, and subscribing to OnEvent after LoadAsync made the assertion
unfalsifiable — it passed against the broken behaviour. It now subscribes first.

`FakeUpstream.ReadTag` returned 0x80340000 (BadNotConnected) for an unset path,
a different Bad code from production's placeholder. Harmless while any Bad
behaved alike; load-bearing now, so it is aligned and pinned by a reflection
guard asserting the engine's private constant agrees with what the production
source stamps.

Behaviour change to weigh, flagged in the issue and confirmed: virtual tags
computed from a comms-lost input now go Bad too. That follows from using the
shared mux and reads as correct, but it is broader than scripted alarms.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
2026-07-31 09:21:11 -04:00
Joseph Doherty 2d20aeb779 fix(opcua,drivers): carry driver status sub-codes to clients (#519) + health-write ordering (#517)
#519 — a driver's OPC UA status code now reaches a client with its sub-code intact.
DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum
and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication,
BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every
driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in
logs only. Resolved by option (b): the raw uint rides AttributeValuePublished ->
AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and
severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad).

The surface was wider than the issue costed (41 files, not "the sink and 3
implementations"): OpcUaQuality had non-driver producers with no uint to carry.
Each got its natural code — BadInternalError for a failed virtual-tag script,
BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of
mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the
engine already held. AddressSpaceApplier's removal writes deliberately keep the
generic Bad: not losing a code the driver had is this issue; inventing one for the
deploy path is a separate call.

Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change
rather than a severity-bucket change. While the annotation was 3-state those were the
same predicate; now a move between two Bad reasons changes what a subscriber reads.

New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and
StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so
it can only see a named constant, never an inline literal.

#517 — the symptom is real but the issue named the wrong path and its fix was unsafe.
ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the
mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth,
has no production call site.) InitializeAsync can: it runs on a fire-and-forget
Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a
catch which disposes the client before writing Faulted.

Moving the write earlier, as suggested, inverts the terminal state — teardown drains
in-flight loops that write health as they unwind, so a late loop could leave Healthy at
rest with every connection closed. The fix is a bracket (conservative before, terminal
after), applied to MTConnect's init-failure path only; shutdown keeps its single
trailing write, and the rule plus the reachability table are documented on
IDriver.ShutdownAsync.

CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing —
Healthy before initialize, after shutdown and after dispose. Now derived from _running.
Its test, named Health_is_always_Connected, asserted the defect and is replaced.

Every new guard was falsified by breaking the production code and watching it go red.
The two sub-code tests each assert two different Bad codes, since a single-code
assertion could pass against a hard-coded constant under the old projection.

Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in
Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in
Host.IntegrationTests.

Closes #517
Closes #519

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
2026-07-31 05:25:27 -04:00
Joseph Doherty 549c656489 fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions`
and re-materialises every condition node fresh — inactive, acked, confirmed,
unshelved. The engine reloads its persisted state and re-derives Active from the
predicate, so it ends up correct; but there is no transition to report, so
`LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing
writes the node.

Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that
had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild
reporting Acknowledged with Retain=false — so it disappeared from
ConditionRefresh entirely, while the engine still held Unacknowledged. An
operator's outstanding alarm silently vanishes from the alarm list on deploy.

`ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it
reads the new `ScriptedAlarmEngine.GetProjections()` and sends one
`AlarmStateUpdate` per alarm NOT in the Part 9 no-event position.

Load-bearing properties:

- Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts`
  topic, never the telemetry hub. `alerts` is the historization path, so a row
  per alarm per deploy would append a duplicate historian/AVEVA record every
  time anyone deploys. This is why it reads a projection rather than re-emitting:
  `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for
  the emission machinery — or a future refactor — to mistake for a transition.
  The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone
  would have clobbered the node's severity and message, so the projection also
  carries the definition's severity, the resolved message template, and the
  last-observed worst-input quality.
- Only non-normal alarms. One in the no-event position already matches what the
  materialise built, so a never-fired alarm stays byte-identical to a deploy
  without this pass (including its Message, which the materialise seeds with the
  display name), and an all-normal deploy writes nothing at all.
- Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant.
- No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against
  the node's CURRENT state, so a re-assert onto a freshly materialised node is a
  genuine delta (fires once — correct, the rebuild reset what clients see) while
  one onto a surgically-preserved node is suppressed.

Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by
disabling the call — 4 of the 6 host tests go red; the other 2 are absence
guards against an over-broad fix and pass either way by design.

Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image):
- pre-fix: condition absent from ConditionRefresh after a rebuild;
- post-fix: present, Unacknowledged, Retain=True, stamped with its own
  LastTransitionUtc rather than the restart instant;
- /alerts stayed empty across two re-asserts, with the panel proven live by a
  real ACTIVATED transition immediately afterwards — no duplicate history.
2026-07-26 09:06:39 -04:00
Joseph Doherty 8e5090edf3 feat(mesh-phase5): tap the 4 telemetry publish seams into the local hub (DPS intact)
Each of the four telemetry publish seams now also emits its DPS payload into the
node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub
publish untouched:

- driver-health   AkkaDriverHealthPublisher            -> TelemetryItem.Health
- resilience      DriverResilienceStatusPublisherService -> TelemetryItem.Resilience
- alerts          ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm
- script-logs     VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script

DI services take the hub as a required ctor param; actors take an optional
nullable hub threaded through their Props and the real spawn sites
(DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor
<- DriverHostActor), so existing test Props constructions keep compiling and simply
do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe
on every node.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 15:27:29 -04:00
Joseph Doherty 3a590a0cb7 refactor(mesh-phase4): retire EfAlarmConditionStateStore + drop dead ScriptedAlarmState table (Task 9)
v2-ci / build (push) Successful in 4m3s
v2-ci / unit-tests (push) Failing after 13m59s
Scripted-alarm condition state lives in LocalDb (Phase 4); the ConfigDb-backed
Ef store + ScriptedAlarmState table are now dead. Removes the test-harness Ef
fallback (a DB-backed node with no store now skips the alarm host), deletes the
store + entity + model config, and drops the table via migration.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 14:33:11 -04:00
Joseph Doherty 4f4cdd05ec feat(mesh-phase4): LocalDb alarm-condition-state store (replicated, pair-local)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 12:19:44 -04:00
Joseph Doherty 5cb0e72166 feat(mesh): node subscribers accept EventStream commands; _coordinatorOverride -> _ackRouter
Phase 2 Task 5. DriverHostActor and ScriptedAlarmHostActor now subscribe to the
node-local EventStream alongside their existing DPS topic subscriptions, so the
transport flag lives ENTIRELY on the publishing side -- exactly one of the two
ever publishes, and the switch can be flipped or reverted without touching a
single subscriber.

Note the alarm case is not symmetric: the DPS subscribe there still carries the
node-LOCAL publisher (OtOpcUaServerHostedService's OPC UA Part 9 method calls),
which never crosses the boundary and stays on DPS in both modes. Only the
central AdminUI leg moves.

Renamed DriverHostActor._coordinatorOverride to _ackRouter. Under ClusterClient
mode that ref is the NodeCommunicationActor, emphatically not a coordinator,
and the old name would send the next reader looking for one.

Adds two wiring tests, because the unit tests either side of this seam BOTH pass
with the subscription deleted -- the comm actor's test asserts a probe receives
the message, and the host tests drive the actor by direct Tell. Neither notices
a missing PreStart subscribe. Sabotage-verified: deleting either subscription
reddens exactly its own wiring test and nothing else.

Three things went wrong while writing those tests, all worth keeping:

- The first version raced. ActorOf returns before PreStart runs, and an
  EventStream.Publish with no subscriber is dropped silently -- no buffering, no
  dead letter. Fixed with a warm-up whose ack proves PreStart completed; the
  comment says why it is not ceremony.
- The alarm version was VACUOUS: it told the host directly INSIDE the assertion
  window alongside the relay, so the direct Tell alone produced the log the
  assertion waited for. Split into warm-up (outside) and relay (inside).
- The alarm test needs its own ActorSystem: the shared harness pins
  loglevel = WARNING and the only signal an unowned alarm gives is a Debug line.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 12:19:29 -04:00
Joseph Doherty 8c5e2be92e feat(alarms): scripted condition Quality from worst-of-input tag quality (#478)
v2-ci / build (pull_request) Successful in 3m48s
v2-ci / unit-tests (pull_request) Failing after 11m0s
Layer 3 of #477: a scripted alarm's condition Quality now reflects the WORST
quality across its input tags, mirroring the native OT semantic (#477 L2).

Plumbing (quality was silently discarded twice on the live path):
- VirtualTagActor.DependencyValueChanged gains Quality (defaulted Good); the
  DependencyMuxActor forwards the published AttributeValuePublished.Quality it
  already carried; ScriptedAlarmHostActor.OnDependencyChanged pushes the real
  quality into the engine (was hardcoded 0u/Good).

Engine (Core.ScriptedAlarms):
- ScriptedAlarmEngine computes worst-of-input quality each eval (skipping
  not-yet-published inputs, which are a readiness concern, not a quality signal)
  and carries it on ScriptedAlarmEvent.WorstInputStatusCode.
- A real transition carries the current worst quality so ToSnapshot's full
  snapshot doesn't clobber quality back to Good (e.g. transition while Uncertain).
- A Bad input freezes the condition (no transition), like a comms-lost native
  driver; a quality-bucket change with no transition emits the new
  EmissionKind.QualityChanged, routed to the existing #477-L2
  AlarmQualityUpdate -> WriteAlarmQuality node path (quality only, no /alerts
  row, no historian write). ScriptedAlarmSource skips QualityChanged so it never
  fabricates a phantom IAlarmSource event.

Host: ToSnapshot maps WorstInputStatusCode -> OpcUaQuality; OnEngineEmission
routes QualityChanged out of band.

Tests (TDD, RED-first): engine worst-carry + Bad/restore QualityChanged +
unchanged-bucket-no-emit; source swallows QualityChanged; mux forwards quality;
host Bad-dep -> AlarmQualityUpdate(no alerts) + transition snapshot carries worst.
Docs: AlarmTracking.md Layer-3 section + design doc.

Closes #478
2026-07-17 15:56:06 -04:00
Joseph Doherty db751d12a5 feat(alarms): condition Quality tracks source connectivity (#477)
v2-ci / build (pull_request) Successful in 4m49s
v2-ci / unit-tests (pull_request) Failing after 3h11m25s
Part 9 ConditionType.Quality was never assigned; default(StatusCode)==Good
so every native + scripted condition reported Good unconditionally — a
comms-lost device still showed a healthy, inactive, Good condition (a
wrong-VALUE bug, distinct from the null-value #473/#475). Clients (and HMIs
bucketing on IsGood) could not tell "genuinely inactive" from "lost contact".

Layer 1 — make Quality a real, plumbed field:
- AlarmConditionSnapshot gains OpcUaQuality Quality (default Good).
- MaterialiseAlarmCondition sets it (native BadWaitingForInitialData, scripted Good).
- WriteAlarmCondition projects snapshot.Quality; the delta-gate gains a Quality
  member so a quality-bucket change fires a Part 9 event.

Layer 2 — drive native quality from driver connectivity (a comms-lost driver
emits no alarm transitions, and an alarm-bearing raw tag has no value variable,
so quality can't come from either existing channel):
- DriverInstanceActor Tells parent ConnectivityChanged on Connected/Reconnecting.
- DriverHostActor fans it to every native condition the driver owns as
  OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
- New dedicated IOpcUaAddressSpaceSink.WriteAlarmQuality sets ONLY Quality and
  fires only on a bucket change — never touches Active/Acked/Retain (an active
  alarm that loses comms stays active). Not a full-snapshot re-projection, so it
  can't clobber severity/message and works for a never-fired condition.
  Forwarded through DeferredAddressSpaceSink (F10b trap; auto-verified by the
  reflection forwarding guard). Ungated by redundancy role; no /alerts row.

Scripted conditions stay Good; worst-of-input quality deferred to #478 (Layer 3).

Tests: node-level (materialise/project/no-clobber/unknown-node no-op),
NativeAlarmProjector, DriverInstanceActor connectivity emission, DriverHostActor
fan-out, OpcUaPublishActor routing, and the wire-level guard
(Condition_event_Quality_tracks_source_connectivity_on_the_wire) — RED-verified
against a simulated pre-fix always-Good server. Existing DriverInstanceActor
parent probes ignore the new ConnectivityChanged.

Docs: docs/AlarmTracking.md §"Condition source-data Quality (#477)";
design doc docs/plans/2026-07-17-alarm-condition-quality-477-design.md.
2026-07-17 15:10:04 -04:00
Joseph Doherty 2e0743ad25 fix(v3-batch4-wp3): realm-qualified write routing + dormant discovery guard + self-correction/byte-parity tests (Wave B review H1/M1/M2/L1/L3)
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only.
A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare
strings; the bare-only key let a colliding raw+UNS pair route to the WRONG
driver ref (last-writer-wins). The realm the node manager resolves (RealmOf)
is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite ->
_driverRefByNodeId keyed by (realm, bareId). New regression test:
Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm.

M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes
hard-short-circuits (single enforcement point; _discoveredByDriver never
populates so the re-inject tail is inert too), with a clear log pointing at the
/raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3;
the 16+2 v2 injection scenarios re-pointed to an accurate skip reason
(DiscoveryInjectionDormantV3).

M2 (MEDIUM): realm-qualified dual-node self-correction tests —
Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched +
Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped).

L1: removed the = AddressSpaceRealm.Uns defaults from the consequential
node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/
EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/
RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the
AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit
realm; read-only accessors + internal builders retain their defaults.

L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the
retired EquipmentNodeIds.Variable).

Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity
round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition.

Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
2026-07-16 11:30:13 -04:00
Joseph Doherty 1647942414 fix(r2-04): scripted-alarm alerts emit uses PrimaryGatePolicy (consistency with 03/S4) 2026-07-13 11:10:43 -04:00
Joseph Doherty 9cad9ed0fc docs(xmldoc): fill missing XML docs + strip tracking-ID comments across src
v2-ci / build (push) Failing after 41s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (push) Has been skipped
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
2026-07-07 12:38:39 -04:00
Joseph Doherty de6ce147fc fix(runtime): drop OpcUaProbeResult in redundancy-topic subscribers (no dead-letter) 2026-06-19 00:32:03 -04:00
Joseph Doherty 56750e110f fix(alarms): historize the real operator for shelve/unshelve/enable/disable transitions 2026-06-11 13:14:00 -04:00
Joseph Doherty 8012509584 feat(historian): honor per-alarm HistorizeToAveva opt-out at the durable write 2026-06-11 12:48:13 -04:00
Joseph Doherty f64f7ce669 fix(alarms): historize the operator (not 'system') for CommentAdded transitions (review)
v2-ci / build (push) Failing after 50s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (push) Has been skipped
2026-06-11 11:42:56 -04:00
Joseph Doherty 8ac3ac5be9 feat(alarms): carry AlarmTypeName + operator Comment on AlarmTransitionEvent (historian feed prep) 2026-06-11 11:03:00 -04:00
Joseph Doherty 06c415598c feat(redundancy): gate scripted-alarm alerts publish on Primary (A1) 2026-06-11 08:44:44 -04:00
Joseph Doherty 370a2b7b48 feat(alerts): AdminUI alarm ack/shelve via AdminOperationsActor singleton
T21: add an AdminUI path for acknowledging/shelving alarms that routes
through the admin-pinned AdminOperationsActor cluster singleton, which
republishes onto the same 'alarm-commands' DPS topic the OPC UA method
path (T18) and the engine subscriber (T19) use. The broadcast + the
ScriptedAlarmHostActor ownership filter handle cross-node routing, so
the singleton needs no knowledge of which node owns the alarm.

- Commons: AcknowledgeAlarmCommand/ShelveAlarmCommand (+ result records)
  and a shared AlarmCommandsTopic const; ScriptedAlarmHostActor now
  re-exports that const (mirrors the DriverControlTopic pattern).
- AdminOperationsActor: two handlers map the control-plane messages to
  AlarmCommand (Acknowledge / OneShotShelve / TimedShelve / Unshelve,
  threading User/Comment/UnshelveAtUtc) and publish via the DPS mediator.
- IAdminOperationsClient + AdminOperationsClient: typed Acknowledge/Shelve
  ask wrappers mirroring StartDeploymentAsync.
- Alerts.razor: per-row DriverOperator-gated Ack/Shelve/Unshelve controls;
  operator name from AuthenticationState. Timed-shelve datetime UI deferred.
- 5 TestKit tests (mediator-probe subscribed to alarm-commands) verifying
  each kind's mapping + reply; 56/56 ControlPlane tests green.
2026-06-11 06:44:27 -04:00
Joseph Doherty 1d7e2a0f8b fix(runtime): reject empty AddComment instead of silently swallowing it
Validate AddComment up-front (IsNullOrWhiteSpace guard + Warning log) so
a blank-comment command is cleanly rejected before reaching the engine
rather than faulting inside ApplyAddComment and being silently swallowed
by the outer catch.  Mirrors the existing TimedShelve missing-UnshelveAtUtc
pattern.

Also fix two stale inline comments: the "async void crash" note on
TimedShelve now correctly says "fault escaping async Task → supervision
restart", and the ownership-filter now documents the benign race with a
concurrent LoadAsync clearing the loaded set.

Tests: AlarmCommand_add_comment_empty_text_is_rejected_not_driven (Theory
— empty string + whitespace) and AlarmCommand_add_comment_nonempty_drives_engine
(positive path, asserts CommentAdded transition on alerts topic).
2026-06-11 06:32:53 -04:00
Joseph Doherty 4f7999eac2 feat(alarms): consume alarm-commands topic in ScriptedAlarmHostActor (T19)
Subscribe the host to the cluster alarm-commands DPS topic in PreStart and
drive the matching ScriptedAlarmEngine op per inbound AlarmCommand. An
ownership filter (engine.LoadedAlarmIds) ignores commands for alarms this
node does not own; TimedShelve without UnshelveAtUtc and unknown operations
are logged + rejected (never thrown); op failures are caught + logged so a
faulting op can't fault the actor. Re-projection is left to the engine's
existing OnEvent -> OnEngineEmission path.

Handler is a Task-returning ReceiveAsync (the project's AK2003 analyzer
forbids an async-void Receive delegate), giving ordered awaited async on the
actor thread. Adds 3 TestKit tests: ack drives the engine with mapped args,
unowned command ignored, missing-UnshelveAtUtc TimedShelve rejected not
thrown.
2026-06-11 06:23:08 -04:00
Joseph Doherty 63289d377c feat(alarms): route inbound Part 9 alarm methods through AlarmAck gate (T18)
Wire the materialised AlarmConditionState method handlers so a client calling
Acknowledge/Confirm/Shelve/AddComment is gated on the AlarmAck data-plane role
and, when allowed, routed back to the scripted-alarm engine via a new
`alarm-commands` DistributedPubSub topic.

- Commons: new AlarmCommand DTO (AlarmId/Operation/User/Comment/UnshelveAtUtc).
- ScriptedAlarmHostActor: add AlarmCommandsTopic const.
- OtOpcUaNodeManager: settable AlarmCommandRouter + wire OnAcknowledge/OnConfirm/
  OnAddComment/OnShelve/OnTimedUnshelve. Each resolves the principal off
  ISessionOperationContext.UserIdentity as RoleCarryingUserIdentity, fails closed
  (BadUserAccessDenied) when the AlarmAck role is absent or no identity, else maps
  + routes an AlarmCommand and returns Good. OnShelve discriminates OneShotShelve/
  TimedShelve/Unshelve from the SDK flags; TimedShelve expiry = UtcNow + ms.
  No Akka/IActorRef handle — only the Action<AlarmCommand> delegate. T20 de-dup
  note left; WriteAlarmCondition untouched.
- OpcUaServer.Security: OpcUaDataPlaneRoles.AlarmAck shared const (the role was a
  bare string everywhere; introduced one symbol for the gate + tests).
- OtOpcUaSdkServer: SetAlarmCommandRouter pass-through.
- Host: boot wiring publishes each command via mediator.Tell(Publish(...)) using a
  lazy ActorSystem accessor (mirrors DpsScriptLogPublisher).
- Tests: 11 new gate + mapping tests (OpcUaServer.Tests 88->99, all green).
2026-06-11 06:05:39 -04:00
Joseph Doherty 4eb1d65e2b feat(scripted-alarms): richer AlarmConditionState bridge to the OPC UA node (T15) 2026-06-10 19:41:16 -04:00
Joseph Doherty fc0d43a3dc refactor(scripted-alarms): retire orphaned ScriptedAlarmActor + F9b evaluator (T11) 2026-06-10 15:22:26 -04:00
Joseph Doherty dafaf2faec fix(scripted-alarms): ScriptedAlarmHostActor review fixes — load-gen guard, quiet cancel, parse guard (T9 review) 2026-06-10 15:08:54 -04:00
Joseph Doherty 3b418a54f1 feat(scripted-alarms): ScriptedAlarmHostActor — engine runtime host (T9) 2026-06-10 14:57:42 -04:00
Joseph Doherty 55101baaa4 refactor(scripted-alarms): review-fix polish for T5/T7/T8 (observer isolation, warning hoist, doc) 2026-06-10 14:32:49 -04:00
Joseph Doherty 1c96fe0be0 feat(scripted-alarms): EfAlarmConditionStateStore (T8) 2026-06-10 14:21:19 -04:00
Joseph Doherty 945ccd0b85 feat(scripted-alarms): DependencyMuxTagUpstreamSource (T7)
Concrete ITagUpstreamSource the scripted-alarm host actor pushes
DependencyValueChanged values into and ScriptedAlarmEngine reads/subscribes
from. Thread-safe: ConcurrentDictionary value cache + per-path ImmutableList
observer lists with atomic add/remove and capture-then-invoke fan-out.
ReadTag of an unknown path returns a Bad-quality (0x80000000) snapshot stamped
via the injected clock. Adds the Core.ScriptedAlarms project reference Runtime
needs to see the interface.
2026-06-10 14:20:02 -04:00
Joseph Doherty 64e3fbe035 docs: backfill XML documentation across 756 files
v2-ci / build (push) Failing after 1m43s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (push) Has been skipped
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
2026-05-28 08:10:17 -04:00
Joseph Doherty 52997ee164 feat(observability): F13d Prometheus + OpenTelemetry instrumentation
v2-ci / build (push) Failing after 38s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (push) Has been skipped
OtOpcUaTelemetry (Commons/Observability) centralizes the project's Meter
+ ActivitySource so all instrumentation points emit through a single
named surface. Counters cover the hot paths:

  otopcua.deploy.applied               (outcome=ack|reject)
  otopcua.deploy.apply.duration        (s, histogram)
  otopcua.driver.lifecycle             (event=spawn|spawn_stub|stop|fault)
  otopcua.virtualtag.eval              (outcome=ok|fail|skip)
  otopcua.scriptedalarm.transition     (state=activated|acknowledged|cleared)
  otopcua.opcua.sink.write             (kind=value|alarm|rebuild)
  otopcua.redundancy.service_level_change (level=byte)

Plus two ActivitySource spans:

  otopcua.deploy.apply                 wraps DriverHostActor.ApplyAndAck
  otopcua.opcua.address_space_rebuild  wraps OpcUaPublishActor.HandleRebuild

Instruments are no-op until a listener attaches, so tests + dev hosts
pay nothing for unread telemetry.

Host Program.cs gains AddOtOpcUaObservability() (binds the OtOpcUa Meter
+ ActivitySource to OpenTelemetry, attaches a Prometheus exporter) and
MapOtOpcUaMetrics() (mounts /metrics scrape endpoint). Driver-side
internals + ASP.NET request metrics deliberately stay off — the scrape
payload is scoped to OtOpcUa signals only.

Tests use MeterListener + ActivityListener to verify
VirtualTagActor.eval, OpcUaPublishActor.AttributeValueUpdate, and
RebuildAddressSpace actually emit on the central instruments. Runtime
suite is 72 / 72 green (+3).

Closes #105. Path A (F13b/c/d) complete; next batch options: #85 UNS
folder hierarchy in SDK, or F8b/F9b production engine bindings.
2026-05-26 10:29:40 -04:00
Joseph Doherty f427dc4f26 feat(runtime): #112 ScriptedAlarmActor state persistence via IAlarmActorStateStore
v2-ci / build (push) Failing after 42s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (push) Has been skipped
ScriptedAlarmActor now survives actor restart: PreStart loads from
the configured store + restores in-memory state; every Transition()
fires a fire-and-forget save. ActiveState still re-derives from the
evaluator on first tick (Phase 7 decision #14), but Acked state +
lastAckUser persist verbatim so operators don't re-ack across an
outage.

Three pieces:
- IAlarmActorStateStore seam in Commons.Engines, with the
  AlarmActorStateSnapshot record (alarmId / state / lastTransitionUtc
  / lastAckUser) and NullAlarmActorStateStore default.
- EfAlarmActorStateStore in Runtime.ScriptedAlarms — production
  adapter over the existing ScriptedAlarmState table in ConfigDb.
  Maps the actor's 3-state enum to the table's AckedState column
  (Active⇒Unacknowledged, Acknowledged⇒Acknowledged, Inactive⇒
  Acknowledged). Concurrency conflicts are logged + dropped — the
  next transition writes again.
- ScriptedAlarmActor PreStart load (async, piped back as
  StateRestored) + Transition save. New Props overload takes the
  store; default is NullAlarmActorStateStore so tests stay quiet.

Tests: Runtime 52 -> 57 (+5):
- Transition writes Active then Acknowledged snapshots with
  lastAckUser populated
- PreStart with persisted Active state restores so a subsequent
  AcknowledgeAlarm fires (not ignored as it would be from Inactive)
- Empty store boots Inactive (AcknowledgeAlarm correctly ignored)
- EfAlarmActorStateStore Save + Load round-trips via in-memory EF
- Load for unknown alarmId returns null

All 6 v2 test suites green: 157 tests passing.

Closes #112. F9 (#80) remaining residual is predicate binding to
Core.ScriptedAlarms.ScriptedAlarmEngine — split as F9b in tasks JSON.
2026-05-26 09:34:37 -04:00
Joseph Doherty 14fb2b05ed feat(runtime): F8/F9 engine evaluator seams + DPS fan-out
VirtualTagActor and ScriptedAlarmActor now route through pluggable
evaluator interfaces and fan out to the cluster's live-tail topics
shipped in F15.3:

- IVirtualTagEvaluator + NullVirtualTagEvaluator in Commons.Engines.
  VirtualTagActor calls evaluator on every DependencyValueChanged,
  dedupes unchanged values, forwards EvaluationResult to its parent,
  and publishes ScriptLogEntry Warning to the script-logs DPS topic
  whenever the evaluator fails.

- IScriptedAlarmEvaluator + NullScriptedAlarmEvaluator. ScriptedAlarmActor
  takes an AlarmConfig (id/name/equipment-path/severity/predicate) and
  publishes both an AlarmTransitionEvent (alerts topic) and a
  ScriptLogEntry (script-logs topic) at every transition. Manual
  ConditionMet/Acknowledge/Cleared still flow through the same
  Transition() so callers without engine bindings still drive the
  state machine; the legacy single-string Props() overload routes
  through a default AlarmConfig.

The Null* defaults keep the actors safe when no engine is bound —
unconfigured nodes never spuriously alarm. Production binding to
Core.VirtualTags.VirtualTagEngine and Core.ScriptedAlarms is the
remaining residual (F8b/F9b — split in tasks JSON).

Tests: Runtime 34 -> 40 (+6):
- VirtualTagActorTests x3 (evaluator drives EvaluationResult,
  unchanged-value dedup, failure publishes Warning ScriptLogEntry)
- ScriptedAlarmActorTests x3 (engine threshold drives Activated +
  Cleared on alerts topic, manual Acknowledge attribution).

All 6 v2 test suites green: 126 tests passing.
2026-05-26 09:05:04 -04:00
Joseph Doherty 95ef533822 feat(runtime): ScriptedAlarmActor state machine (engine wiring tracked as F9) 2026-05-26 05:09:03 -04:00