b8d2fc37ce93b7a4e69fda377e90488e8bd7bb63
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
f54922f94c |
fix(alarms): revert a native ack that reaches no driver; refuse native acks while the host is Stale (lmxopcua#765)
Residuals of #760: 1. An ack routed to a disabled / not-yet-spawned driver, or to a stubbed child (driver does not run on this node, or its factory rejected the config), was dropped and left the condition Acknowledged. DriverHostActor now reverts it through the same NativeAlarmProjector.TryRevertAcknowledge path as a refused ack. 2. A native ack arriving while the host is Stale was dropped before the projector recorded it, so it could never be reverted. Stale now marks the whole NativeAlarmAckPreflightView unavailable (SetHostUnavailable), so the node manager refuses every native ack synchronously; BecomeSteady clears it. 3. (keyed sessions without a Galaxy identity) is not addressed. Tests: DriverHostActorNativeAlarmTests' client-ack test and the FOCAS respawn theory now start the host's own driver (AckingAlarmDriver; a factory building a real FocasDriver per spawn over per-instance fake CNCs, with a real config-change respawn) - both fail against the fix in their old shape. New: ack to a disabled driver reverts, ack to a stubbed driver reverts, a Stale host refuses through the pre-check view - all three fail with the fix mutated out. Runtime.Tests 943/0/3. Docs: AlarmTracking.md, .claude/rules/alarms.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
8e8d4cdf13 |
fix(alarms): surface a refused native alarm ack synchronously, revert one that fails later (lmxopcua#760)
The native-ack path is fire-and-forget after the Part 9 Acknowledge returns, so a Galaxy ack refused for lack of a credential (#749), or rejected by MXAccess, left the client with Good and the condition showing Acknowledged while Galaxy was never acked. Both options from the issue, layered: 1. Synchronous refusal for what the driver can decide without I/O. New optional IAlarmAcknowledgePreflight (Core.Abstractions); GalaxyDriver refuses an anonymous session when the driver has no AlarmAck fallback account. DriverHostActor publishes a condition NodeId -> owning-driver snapshot into NativeAlarmAckPreflightView (lock-free, volatile); the node manager's new NativeAlarmAckPreflight seam, consulted after the authz and Primary gates, returns BadUserAccessDenied with the reason - nothing routed, applied or reported. A keyed session is not pre-checked (its identity lives in the Secrets store, which must not be read under the SDK lock). 2. Revert for everything else. RouteAlarmAck carries the condition NodeId; DriverInstanceActor replies AlarmAckFailed on a refusal, a backend failure, a driver without an alarm source, or an ack dropped while connecting / reconnecting. DriverHostActor re-projects the condition unacknowledged via NativeAlarmProjector.TryRevertAcknowledge, but only while the local ack is still pending - any driver transition projected since then wins. A disabled or unspawned driver is not reverted (unchanged). GatewayGalaxyAlarmAcknowledger now throws DriverOperationRefusedException on a non-zero native hresult (wrong password, no ack permission) instead of only warning, so that case reverts too; as a refusal it neither retries nor trips the breaker. Tests (each fails with its fix mutated out): projector revert/supersede, DriverHostActor refused + failed ack -> unacked AlarmStateUpdate (successful ack control), preflight view snapshot, node-manager preflight unit test, over-the-wire ConditionMethodEventTests (BadUserAccessDenied, no route, no event; positive control), Galaxy preflight + hresult refusal. Docs: Galaxy.md, AlarmTracking.md, ScriptedAlarms.md, .claude/rules/alarms.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
dc0a24f224 |
feat(galaxy): operator-authenticated alarm ack — send Galaxy credential to mxaccessgw (#749)
v2-ci / unit-tests (push) Failing after 1h8m43s
mxaccessgw 488649c+ acknowledges by AuthenticateUser as the operator and
writing <alarm>.AckMsg, so AcknowledgeAlarmRequest now needs a real Galaxy
operator_user + operator_domain + operator_password. The OPC UA data plane is
API-key/anonymous and carries no Galaxy password, so the driver resolves the
account per ack:
1. the session's API key's Galaxy ack identity, set on AdminUI /apikeys and
held ONLY in the encrypted Secrets store
(otopcua/apikeys/<keyId>/galaxy-ack-{user,password}); a key with a user but
no password secret is refused, never silently downgraded;
2. else the driver instance's AlarmAck fallback account (DriverConfig
FallbackUser/FallbackDomain/FallbackPasswordSecretRef; secret:/${secret:}/
env:/file:/dev: only — a literal is refused);
3. else DriverOperationRefusedException: nothing is sent, never an empty
operator_user. Refusals are excluded from the resilience retry/breaker.
- Pins ZB.MOM.WW.MxGateway.Client/Contracts 0.2.1 -> 0.3.0 (vendored into
nuget-packages/ from the Gitea feed, per the local-mxgw source mapping).
- The session ApiKeyId (not a secret) rides NativeAlarmAck ->
RouteNativeAlarmAck -> RouteAlarmAck -> AlarmAcknowledgeRequest; the password
is resolved in the driver and never travels through actor messages.
- The OPC UA principal + key id are appended to the AckMsg comment
("[OPC UA: <principal>, key <id>]") and logged, so a shared fallback
account does not lose who acknowledged.
- Password never logged: GalaxyAckCredential.ToString omits it, the
acknowledger scrubs it from every gateway message it echoes and withholds
any exception that echoes it; audit rows carry only the key id.
- AdminUI: /apikeys "Ack identity" editor (set/remove; delete-key clears it),
Galaxy driver form gains the fallback section.
- Synchronous OPC UA status for a refusal is follow-up #760 (the native ack
path is fire-and-forget after the Part 9 method returns).
Tests: request mapping (password populated), password absent from logs and
exception text (mutation-checked), key-mapped / fallback / refused paths at
resolver and driver level, breaker non-trip, key-id propagation through node
manager and DriverHostActor, AdminUI service + form round-trip.
Closes #749
|
||
|
Joseph Doherty
|
beccfc6404 |
test(#606): native-ack routing waits for the driver child to be Connected; widen a starved poll
DriverHostActorNativeAlarmAckRoutingTests: Applied does not mean the child DriverInstanceActor is Connected, and an ack reaching a still-Connecting child is dropped by design — the single positive-control ack was itself dropped (4/20 under DOTNET_PROCESSOR_COUNT=1 x8, still failing with the presence budget dilated to 45 s, so not a budget problem). SpawnHostAndApply now waits for SubscribeAlarmsAsync, which only the Connected state calls. ScriptedAlarmEngine OnEvent_emission_happens_outside_evalGate: 2 s blocking poll -> 15 s (5/20 under starvation). Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
Joseph Doherty
|
55ae230fa4 |
ci(#606): Gitea-compatible trx upload + deterministic fixes for the CI-load test flakes
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3 on the test step. - Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR (TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget. - Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process (abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s. - Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick). - Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps (wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log. - Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped. - infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe |
||
|
Joseph Doherty
|
b815446940 |
refactor(drivers): delete GetMemoryFootprint / FlushOptionalCachesAsync (#525)
Both `IDriver` members had no production call site. Their only consumer was `MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and `GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which had not been true since v3, even before #522 removed the poller. Owner decision on #525 was (a) delete rather than (b) build the process-wide watchdog that would consume them. The driver-side implementations are in git history and were mostly reusable; what was always missing is the trigger, and a member nobody calls is how the misleading doc-comment survived this long. Removed from `IDriver`, all 12 driver implementations, and every test stub — plus the support code that existed only to feed them, which is the part a member-only deletion would have left behind as dead weight: - MTConnect's three per-entry byte-weight constants, its cached `ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run on every cache fill to populate a number with no remaining reader. - OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it. - MQTT's `ApproxBytesPerAuthoredTag`. Two behavioural notes, neither a change in what the software does: - `AbCipTemplateCache` now has no production reset path. Its only caller was AB CIP's flush, which itself had no caller — so nothing changes except how visible the fact is. Recorded on the type, along with the correction that its comment's claim about a device `ReinitializeAsync` also flushing it was never true. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt. - MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an Agent restart. That is now `internal` rather than `private` so the tests which used the flush purely to empty the cache drive the real path instead — they now cover a case that actually occurs in production. Several unrelated doc comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are repointed to it. Tests: the ones that existed only to assert on the removed members are deleted (3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a footprint of 0 before the flush and 0 after with no discovery in between, so it would have passed against a flush that did nothing at all. Full suite green except 7 failures verified identical on master before and after (4 AbLegacy integration with the compose profile down, 3 Host.IntegrationTests), checked by name rather than assumed. Docs corrected where they described this as live: `DriverLifecycle.md` (which carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and `v2/driver-stability.md` — whose "polls this every 30 s" + per-driver `DriverConfig` memory budget were never built as written and are now banner-ed as design intent rather than description. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
240d7aa8aa |
fix(test): give the three absence assertions a positive control (#501)
`AwaitAssert` returns on its FIRST success, so an assertion that is already true at t=0 passes instantly and spends none of its duration. All three sites asserted an absence against a collection that starts empty, so none of them ever gave the handler the time their comments claimed. Replaced with mailbox-ordering positive controls rather than sleeps: - `RouteNativeAlarmAck_unknown_node_is_dropped` — follow the unmapped ack with a MAPPED one. The host forwards via `Tell` and the child handles `RouteAlarmAck` with `ReceiveAsync` (which suspends its own mailbox), so once the control reaches the driver the unmapped one has provably been handled. - `RouteNativeAlarmAck_on_non_primary_is_dropped` — the role itself is the control: return the node to Primary and re-send. Comments (`gated` / `control`) are the discriminator, so a leaked ack is visible rather than merged into a count. - `PeerProbeSupervisorTests.Single_node_snapshot_spawns_no_children` — a FOURTH site the issue did not list. It says the other `ChildCount.ShouldBe(0)` waits are each preceded by a `ShouldBe(1)`; that is true of three of the four, but not this one, which asserts the initial state. Now followed by a two-node snapshot: the count reaching 1 proves the single-node snapshot was processed, and a local-node child would have made it 2. The issue warned these may turn red, which would be a real defect surfacing. They did not — the routing is correct. Proven by breaking it deliberately instead: disabling the Primary gate reddens the Secondary test, and routing unmapped ids to an arbitrary mapping reddens the unknown-node test. Under the old form both breaks passed. |
||
|
Joseph Doherty
|
50b55ee4b9 |
fix(redundancy): elect the Primary by cluster age, not by address
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
8ebc712eff |
feat(v3-batch4-wp4): multi-notifier native alarms (single ReportEvent → raw + equipment notifiers) + teardown symmetry
Materialize each native alarm ONCE at the raw tag (ConditionId = RawPath, Raw realm); wire the single condition as an SDK event notifier of each referencing equipment's UNS folder so one ReportEvent fans to every root without re-reporting per root (which would break Server-object dedup + Part 9 ack correlation). - New sink method WireAlarmNotifiers(alarmNodeId, alarmRealm, notifierFolderNodeIds, notifierFolderRealm) on IOpcUaAddressSpaceSink, forwarded through DeferredAddressSpaceSink + SdkAddressSpaceSink + NullOpcUaAddressSpaceSink (the forwarding-trap guard); auto-covered by the DeferredSinkForwardingReflectionTests realm + forwarding guards + a hand-written forward test. - OtOpcUaNodeManager: the normative AddNotifier(isInverse) pair + idempotent EnsureFolderIsEventNotifier per equipment folder; tracked per condition in _alarmNotifierWiring. Teardown symmetry: RemoveNotifier(bidirectional:true) on RebuildAddressSpace, RemoveAlarmConditionNode, and RemoveEquipmentSubtree so no inverse-notifier entry leaks across redeploys. - AddressSpaceApplier.MaterialiseRawSubtree wires notifiers for each native alarm tag, resolving its ReferencingEquipmentPaths (Area/Line/Equipment) to the EquipmentId folder NodeIds via BuildEquipmentIdByFolderPath. - AlarmTransitionEvent gains ReferencingEquipmentPaths (empty default); /alerts renders the referencing-equipment list as display metadata. - Un-skipped + rewrote the native-alarm dark tests (DriverHostActorNativeAlarmTests x6, DriverHostActorNativeAlarmAckRoutingTests x1) for the v3 raw-condition model; new NodeManagerMultiNotifierAlarmTests proves multi-notifier wiring + teardown symmetry (no leaked duplicates after a re-trip) + applier wiring test. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
329144b1aa |
v3 B1-WP6: migrate Core + Server test projects to greenfield schema
Fix TESTS only (production already green) so the four in-scope test projects compile + pass under the v3 dark address space: - Core.Abstractions.Tests: drop AllowedNamespaceKinds/NamespaceKindCompatibility from DriverTypeMetadata; rewrite EquipmentTagRefResolver tests to the single-func RawPath-lookup contract (no parseRef/transient cache). - Core.Tests: rewrite EquipmentNodeWalkerTests to EquipmentNamespaceContent(Areas, Lines,Equipment,VirtualTags?,ScriptedAlarms?) — folders + VirtualTags + ScriptedAlarms only; raw-tag variables dark (skipped Batch-4 placeholder). Drop retired Equipment.DriverInstanceId. - Runtime.Tests: rewrite golden corpus + TagConfigCorpusParityTests to the v3 RawPath/RawTagEntry round-trip (byte + TagConfigIntent parity; EquipmentTags dark). Salvage VirtualTag + ScriptedAlarm artifact parity to the new Compose signature. Retire the equipment-tag-materialization parity files (Array/Historize/Alias) and the equipment-device-binding DeviceHost parity to documented skipped placeholders. Skip 34 dark equipment-tag routing/value/alarm/write/discovery actor tests with a shared DarkAddressSpaceReasons constant. Re-key cluster-scoped tests to v3 UNS line->area attribution. - ControlPlane.Tests: ConfigComposer round-trips re-keyed to RawFolder/Device (no Namespaces); tag-config gate resolves driver via Device; deploy-gate collision test re-keyed to UnsEffectiveNameCollision; drop retired BadCrossClusterNamespaceBinding case. |
||
|
Joseph Doherty
|
93d9160dae | feat(alarms): DriverHostActor routes native-condition acks to the owning driver [H6d] |