From 90331b622cf22969ddb99dd67be49c14ad33f27b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 18 Aug 2026 05:40:11 -0400 Subject: [PATCH] =?UTF-8?q?docs(alarms):=20second=20wnwrap=20probe=20attem?= =?UTF-8?q?pt=20=E2=80=94=20secured-write=20verb=20unblocked=20the=20rig,?= =?UTF-8?q?=20answering=20@COUNT=20and=20the=20re-raise=20GUID=20leg?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AuthenticateUser("Administrator", "") + WriteSecured raises the alarm UDAs that plain Write could not touch (SecurityError detail=1008), so the 2026-08-17 blocker was the verb, exactly as that run's own Unblocking list predicted. Two of the three open questions are now observed rather than assumed: - ALARM_RECORDS/@COUNT reports the records in the reply, not the total active count. With three alarms active it read 1 at cap 1 and 2 at cap 2. There is no exact truncation signal to switch to, so IsTruncatedFetch's conservative rule is the design rather than a placeholder — behaviour unchanged, only the comments. - Clear-then-re-raise mints a new GUID; the ALM->RTN leg keeps its GUID (reconfirming the 2026-05-01 capture). ComputeTransitions already reads the re-raise correctly as one instance ending and another beginning. The acknowledge leg stays unobserved for a narrower reason: every wnwrap ack surface is inert on this rig. AlarmAckByName returns 0 from the ack-only consumer and -55 from the SetXmlAlarmQuery-applied one, for both the 6-arg and 8-arg forms, and neither the snapshot STATE, OPERATOR_NAME, nor the extension's own .Acked attribute moves. That corrects AlarmClientDiscovery.md, which read the zero return as a working ack. Comment- and prose-only; no behaviour change. The three throwaway probes ran from the windev CI clone and were deleted; that clone is a clean tree at ab3ff16. --- docs/AlarmClientDiscovery.md | 11 + docs/AlarmProbeFindings.md | 208 +++++++++++++++--- docs/DesignDecisions.md | 15 +- gateway.md | 7 +- .../MxAccess/WnWrapAlarmConsumer.cs | 39 ++-- 5 files changed, 224 insertions(+), 56 deletions(-) diff --git a/docs/AlarmClientDiscovery.md b/docs/AlarmClientDiscovery.md index f43fff8..411a500 100644 --- a/docs/AlarmClientDiscovery.md +++ b/docs/AlarmClientDiscovery.md @@ -764,6 +764,17 @@ The v2 8-arg method returns -55 on this AVEVA build regardless of operator-identity inputs — looks like a stub. The v1 6-arg method works. Production `WnWrapAlarmConsumer.AcknowledgeByName` calls the 6-arg overload and discards the proto's `domain` + `full_name` fields. + +**Correction, 2026-08-18** (`docs/AlarmProbeFindings.md`). Both halves +of the paragraph above are narrower than they read. The -55 tracks the +*consumer*, not the overload: against the `SetXmlAlarmQuery`-applied +reader both the 6-arg and 8-arg forms return -55, and against the +ack-only consumer both return 0. And "works" means "returns 0" only — +a probe that raised a real alarm and then acked it six ways watched the +snapshot stay `UNACK_ALM`, `OPERATOR_NAME` stay empty, and the alarm +extension's own `.Acked` attribute stay `False` for 16 s after each +call. On this rig the wnwrap ack is accepted and then inert, so a zero +return code must not be read as an applied acknowledgement. The proto contract keeps the 8 fields for forward compatibility if AVEVA fixes the v2 method later. diff --git a/docs/AlarmProbeFindings.md b/docs/AlarmProbeFindings.md index f67e360..26f74d9 100644 --- a/docs/AlarmProbeFindings.md +++ b/docs/AlarmProbeFindings.md @@ -1,7 +1,8 @@ # Alarm Probe Findings -`WnWrapAlarmConsumer` rests on two assumptions that no unit test can settle, because both -are properties of AVEVA's alarm provider rather than of our code: +`WnWrapAlarmConsumer` rested on two assumptions that no unit test can settle, because both +are properties of AVEVA's alarm provider rather than of our code. The second is now settled +and one half of the first is; the questions are stated here as they were asked: 1. **GUID identity.** The snapshot diff in `ComputeTransitions` keys on the alarm record's `GUID`. If wnwrap mints a fresh GUID when an alarm changes state, a single @@ -13,23 +14,27 @@ are properties of AVEVA's alarm provider rather than of our code: rather than the records-in-reply count, truncation detection can become exact instead of conservative, and the bounded staleness `ApplySnapshotUpdate` accepts goes away. -This document records what a live probe run against the dev rig (`DESKTOP-6JL3KKO`, -2026-08-17) could and could not establish, so the next attempt starts from the blocker -rather than rediscovering it. +This document records what live probe runs against the dev rig (`DESKTOP-6JL3KKO`, +2026-08-17 and 2026-08-18) could and could not establish, so the next attempt starts from +the blocker rather than rediscovering it. ## Outcome | Question | Status | |---|---| -| GUID stable across polls and `ALM → RTN` | Answered — yes, by the 2026-05-01 capture in `AlarmClientDiscovery.md` | -| GUID stable across `UNACK → ACK`, and across clear-then-re-raise | **Open** | -| `COUNT` = total active vs records-in-reply under a capped fetch | **Open** | +| GUID stable across polls and `ALM → RTN` | Answered — yes (2026-05-01 capture in `AlarmClientDiscovery.md`, reconfirmed 2026-08-18) | +| GUID stable across clear-then-re-raise | Answered 2026-08-18 — **no**, a re-raise mints a new GUID | +| GUID stable across `UNACK → ACK` | **Open** — the rig cannot be driven into an acknowledged state at all | +| `COUNT` = total active vs records-in-reply under a capped fetch | Answered 2026-08-18 — **records in the reply** | -Both open questions are blocked by the same thing: the rig has no active alarm and cannot -be driven into one over MXAccess, so there is no alarm instance whose GUID can be followed -through an acknowledge and no population large enough to overflow a capped fetch. +The 2026-08-17 run below is kept because it is the record of the wrong-verb blocker. The +2026-08-18 run cleared that blocker with `AuthenticateUser` + `WriteSecured` and answered +two of the three questions; the acknowledge leg is now blocked on something narrower and +different, described in "Second attempt". -## Why The Rig Cannot Raise An Alarm +## First attempt (2026-08-17): plain `Write` + +### Why The Rig Could Not Raise An Alarm The rig is otherwise healthy, which is what makes the blocker specific rather than a general "nothing works": @@ -64,7 +69,8 @@ for the whole probe session. ### Unblocking -Any one of these makes both questions answerable, in rough order of cost: +Any one of these makes both questions answerable, in rough order of cost. The second one +is what the 2026-08-18 run did, and it worked: - Re-enable the System Platform script that flips `TestMachine_001.TestAlarm001` (referenced throughout `AlarmClientDiscovery.md`). It writes from inside the engine, so @@ -78,7 +84,7 @@ Three separate objects are wired to the same alarm UDA name, so once writes land `maxAlmCnt` of 1 or 2 forces truncation against three active alarms and answers the `COUNT` question in the same run. -## Evidence +### Evidence Snapshot payload, identical at every cap (1, 2, and 1024) and at every poll across the ~100-second session: @@ -98,26 +104,164 @@ The probe used for the run was a throwaway file in the windev CI clone harness it was modelled on is `src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/WnWrapConsumerProbeTests.cs`. -## Implications +## Second attempt (2026-08-18): secured-write verb -### Transition identity +The findings above named the fix in their own "Unblocking" list, and it holds: +`AuthenticateUser` + `WriteSecured` raises the alarms that plain `Write` could not touch. +The rig's alarm attributes are not unreachable — they are secured, and the 2026-08-17 probe +used the wrong verb. -`ComputeTransitions` keying on GUID is safe for the raise and clear legs, which is the -evidence `AlarmClientDiscovery.md` already carries. The acknowledge leg — the one where a -re-minted GUID would corrupt the feed, because an ack is the state change most likely to -create a new record in a provider that models acknowledgement as a separate event — is -still assumed rather than observed. Nothing here justifies changing the diff, but the -assumption should not be described in code as established. +### Method -### Truncation detection +Three throwaway probes in the windev CI clone (`C:\build\mxaccessgw-ci`, detached at +`ab3ff16`), each an x86 net48 xUnit fact running on a dedicated STA with a +`PeekMessage`/`DispatchMessage` pump so MXAccess events deliver: -`IsTruncatedFetch` stays as written. Tightening it to an exact test requires knowing that -`COUNT` reports the total, and this run cannot show that. The conservative rule keeps its -justification: at the cap, treating a complete fetch as truncated costs one poll of -staleness, while treating a truncated fetch as complete broadcasts clears for every alarm -past the cap. +1. secured write + both GUID legs + the capped-fetch question, +2. the acknowledge surface, exhaustively, +3. the acknowledge question re-asked through an independent observer. -The one substantive correction is to the phrasing rather than the logic. The reply is not -featureless — it carries a `COUNT` attribute the parser currently ignores. Whether that -attribute is a usable "more available" signal is unverified, not absent, and the comments -in `WnWrapAlarmConsumer` now say so. +All three used `LMXProxyServerClass` directly for the write path and a +`wwAlarmConsumerClass` pair — a reader with `SetXmlAlarmQuery` applied and an ack-only +consumer without it — mirroring `WnWrapAlarmConsumer`'s own two-consumer arrangement. The +subscription was `\\DESKTOP-6JL3KKO\Galaxy!TestArea`, matching the 2026-08-17 run. +`MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER` / `_PASSWORD` are set at neither machine nor +user scope on the box, so the probes used the default rig identity, `Administrator` with an +empty password. All three files were deleted afterwards and the clone is a clean tree. + +### The secured write lands + +Same session, same item, one after the other — the control reproduces 2026-08-17 exactly +and the secured verb succeeds: + +``` +Write 'TestMachine_001.TestAlarm001' <- True +WRITE-COMPLETE hLMX=1 hItem=1 statuses=[success=0 category=SecurityError detectedBy=RespondingAutomationObject detail=1008 text=] + observed 'TestMachine_001.TestAlarm001' = False + +AuthenticateUser -> userId=1 +WriteSecured 'TestMachine_001.TestAlarm001' <- True currentUserId=1 verifierUserId=0 +DATA-CHANGE hItem=1 value=True quality=192 +WRITE-COMPLETE hLMX=1 hItem=1 statuses=[success=-1 category=Ok detectedBy=RespondingAutomationObject detail=0 text=] + observed 'TestMachine_001.TestAlarm001' = True +``` + +`AuthenticateUser("Administrator", "")` resolves to user id 1, matching the value +`WorkerLiveMxAccessSmokeTests` records. No verifier is needed: `verifierUserId=0` is +accepted. All six subsequent raise/clear writes across the three objects behaved the same +way, so the path is reliable rather than a one-off. + +### Q2 — `ALARM_RECORDS/@COUNT` reports the reply, not the total + +With all three `TestMachine_00{1,2,3}.TestAlarm001` alarms active, the same subscription +fetched at three caps back to back: + +``` +FETCH three-active cap=1024 ALARM_RECORDS/@COUNT=3 elementCount=3 +FETCH three-active cap=1 ALARM_RECORDS/@COUNT=1 elementCount=1 +FETCH three-active cap=2 ALARM_RECORDS/@COUNT=2 elementCount=2 +``` + +`@COUNT` tracked the cap, not the population. It equals the element count in every reply, +including the two that are provably truncated — the galaxy held three active alarms while +`@COUNT` read 1 and 2. The attribute therefore carries no "more available" information, and +the hypothesis that it might report the total is refuted rather than merely untested. + +`IsTruncatedFetch` stays exactly as written, and this is now settled rather than deferred: +there is no exact truncation signal to switch to. Tightening it is **not** a future task. +The conservative rule keeps its original justification — at the cap, treating a complete +fetch as truncated costs one poll of staleness, while treating a truncated fetch as +complete broadcasts clears for every alarm past the cap. + +### Q1 — clear-then-re-raise mints a new GUID + +`TestMachine_001.TestAlarm001` was raised, cleared, and re-raised through `WriteSecured`, +with a full snapshot fetch after each leg: + +``` +after-raise TAG=TestMachine_001.TestAlarm001 GUID=72B84A6E7BA74D42B611EB5393626F79 STATE=UNACK_ALM VALUE=true +after-clear-001 TAG=TestMachine_001.TestAlarm001 GUID=72B84A6E7BA74D42B611EB5393626F79 STATE=UNACK_RTN VALUE=false +after-reraise TAG=TestMachine_001.TestAlarm001 GUID=2394FEAA94774154AAA97C3AC29CA51E STATE=UNACK_ALM VALUE=true +``` + +Two facts, one confirming and one new. The `ALM → RTN` leg holds its GUID, reconfirming the +2026-05-01 capture against a write-driven rather than script-driven transition. The re-raise +does not: the returned record keeps the old GUID and the new active condition arrives as a +separate record under a fresh one. + +That is the correct reading for `ComputeTransitions` rather than a problem for it. The old +GUID leaving the active set is a real Clear and the new GUID is a real Raise, because they +are two alarm instances. The diff needed no change. + +### Q1 — the acknowledge leg is still unobserved, for a new reason + +The blocker moved. The rig can now be driven into an alarm; it cannot be driven out of +`UNACK_ALM`. Every acknowledge surface `wwAlarmConsumerClass` exposes was tried against a +freshly raised alarm, each followed by 16 seconds of polling: + +| Attempt | Result | +|---|---| +| `AlarmAckByName` 6-arg on the ack-only consumer, node = machine | `rc=0`, state unchanged | +| same, node empty | `rc=0`, state unchanged | +| same, name as `Galaxy!TestArea.TestMachine_001.TestAlarm001` | `rc=0`, state unchanged | +| same, operator name `Administrator` | `rc=0`, state unchanged | +| `AlarmAckByName` 6-arg on the reader consumer (`SetXmlAlarmQuery` applied) | `rc=-55`, state unchanged | +| `AlarmAckByName` 8-arg (v2) on the ack-only consumer | `rc=0`, state unchanged | + +The `rc=-55` on the `SetXmlAlarmQuery`-applied consumer reproduces the 2026-05-01 finding +that motivated the two-consumer split, so the ack-only consumer was correctly provisioned; +the `rc=0` returns are the ones that go nowhere. + +A third probe removed the last doubt by watching the alarm extension's own `.Acked` +attribute over MXAccess — an observer entirely independent of the wnwrap snapshot: + +``` +VALUES after-raise: 'TestMachine_001.TestAlarm001.Acked' = False +AlarmAckByName -> rc=0 +VALUES after-wnwrap-ack: 'TestMachine_001.TestAlarm001.Acked' = False +XML after-wnwrap-ack: STATE=UNACK_ALM OPRNAME= OPRNODE= +``` + +Nothing moves: not the extension attribute, not the snapshot `STATE`, not `OPERATOR_NAME`. +`rc=0` from wnwrap means the call was accepted, not that an acknowledgement was applied. + +Acknowledging by writing the extension attribute directly is not an alternative — the +attribute is not writable, and says so with an operational rather than a security failure, +for both verbs: + +``` +WriteSecured 'TestMachine_001.TestAlarm001.Acked' <- true +WRITE-COMPLETE statuses=[success=0 category=MxCategoryOperationalError detectedBy=MxSourceRespondingAutomationObject detail=1007] +Write 'TestMachine_001.TestAlarm001.Acked' <- true +WRITE-COMPLETE statuses=[success=0 category=MxCategoryOperationalError detectedBy=MxSourceRespondingAutomationObject detail=1007] +``` + +`detail=1007` from the responding automation object, unchanged by authentication, is a +read-only attribute rather than a permission refusal — the `1008` `SecurityError` the alarm +UDA itself returns is what a permission refusal looks like on this rig. So the ack has no +MXAccess-side entry point and the wnwrap-side entry point is inert. + +#### Remaining unblock paths for the acknowledge leg + +- Acknowledge from the System Platform side — the IDE's alarm client, InTouch, or an + ArchestrA graphic bound to the alarm — and watch the snapshot from a running probe. This + proves whether wnwrap's `STATE` ever reports `ACK_ALM` at all, which is the actual + question; the ack API being inert may be a wnwrap defect layered on top of a snapshot + that would report the state correctly. +- Check whether `alarmmgr` on this rig is configured with an alarm-acknowledgement security + requirement that the wnwrap consumer, which passes an operator *name* string and no + authenticated identity, cannot meet. If so, ack over wnwrap is not merely untested here + but unavailable by configuration, and the gateway's `AcknowledgeByName` path needs the + same treatment on any customer galaxy configured that way. +- If neither lands, the acknowledge leg stays assumed. It is worth restating that this is a + documentation gap, not a correctness one: a re-minted GUID on acknowledge would produce a + spurious Clear plus a spurious Raise, which is the same shape the now-observed re-raise + behaviour produces and which `ComputeTransitions` already handles as two instances. + +### Rig state left behind + +The three `TestMachine_00{1,2,3}.TestAlarm001` UDAs are back to `false` and their +`.InAlarm` subtags read `false`, but each leaves a `UNACK_RTN` record in the wnwrap +snapshot, since nothing can acknowledge them away. `SnapshotActiveAlarms` counts only +`UNACK_ALM` and `ACK_ALM` as active, so these are inert for the gateway; they will clear on +the next `alarmmgr` restart. diff --git a/docs/DesignDecisions.md b/docs/DesignDecisions.md index 32055df..9e9848f 100644 --- a/docs/DesignDecisions.md +++ b/docs/DesignDecisions.md @@ -226,13 +226,14 @@ payload carries the flag as well because a prefix filter (or an empty galaxy) ca leave zero records, and a truncated fetch with nothing to report still has to say so. -The **detection heuristic is unchanged**: `IsTruncatedFetch` remains -`fetchedRecordCount >= maxAlarmsPerFetch`. The live probe run for this work could -not verify whether `ALARM_RECORDS/@COUNT` reports the total active count or only -the records in the reply (`docs/AlarmProbeFindings.md`), and an exact-looking -signal derived from an unverified attribute is worse than an honest heuristic — -it would read as precise while being wrong in the one direction that matters. -Switching to `@COUNT` stays blocked on probe evidence. +The **detection heuristic is unchanged, and now permanently**: `IsTruncatedFetch` +remains `fetchedRecordCount >= maxAlarmsPerFetch`. The 2026-08-18 probe settled +what the earlier run could not: with three alarms active, `ALARM_RECORDS/@COUNT` +read 1 at a cap of 1 and 2 at a cap of 2, so it reports the records in the reply +rather than the total active count (`docs/AlarmProbeFindings.md`). There is no +exact signal to switch to — `@COUNT` cannot distinguish a capped fetch from a +complete one — so the heuristic is the design rather than a placeholder waiting +on evidence. The flag is **not latched**. It is replaced by each fetch's verdict, so the first sub-cap fetch clears it, and `GatewayAlarmMonitor.ClearCache` drops it with the diff --git a/gateway.md b/gateway.md index 610904a..c46b102 100644 --- a/gateway.md +++ b/gateway.md @@ -265,9 +265,10 @@ means "this active set may be incomplete", not "this record is unreliable" — it is independent of the subtag-fallback `degraded` field above. It is not latched: the first fetch that comes back under the cap is complete, restores absence authority, and clears it. Detection remains the record-count heuristic; -the reply's `ALARM_RECORDS/@COUNT` attribute would make the test exact only if -it reported the total active count rather than the records in the reply, which -a live probe could not discriminate (see `docs/AlarmProbeFindings.md`). +the reply's `ALARM_RECORDS/@COUNT` attribute would have made the test exact only +if it reported the total active count, and the 2026-08-18 live probe observed it +reporting the records in the reply instead (see `docs/AlarmProbeFindings.md`), so +there is no exact alternative. Forced modes are available via `MxGateway:Alarms:Fallback:Mode`: `ForceAlarmManager` disables failover; `ForceSubtag` forces the standby diff --git a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs index 8280e13..072916a 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs @@ -429,10 +429,12 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer // have exactly that many active alarms. Treat the ambiguous case as // truncated: the false-positive cost is a snapshot that stays stale for // one poll, the false-negative cost is every alarm past the cap reading - // as cleared. (The reply's ALARM_RECORDS/@COUNT attribute is a - // candidate exact signal, but only if it reports the total rather than - // the records in the reply — untested on a live rig, see - // docs/AlarmProbeFindings.md.) + // as cleared. (The reply's ALARM_RECORDS/@COUNT attribute was the + // candidate exact signal; the 2026-08-18 live probe settled it — with + // three alarms active, @COUNT read 3, 1, and 2 at caps 1024, 1, and 2, + // matching the records in the reply every time. It reports the reply, + // not the total, so it cannot tell a capped fetch from a complete one. + // See docs/AlarmProbeFindings.md.) bool truncated = IsTruncatedFetch(fetchedRecordCount, maxAlarmsPerFetch); IReadOnlyList transitions = @@ -491,12 +493,14 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer /// offers no confirmed "more available" flag, so a reply at exactly the /// cap is indistinguishable from a galaxy that happens to hold exactly /// that many active alarms; both are treated as truncated. The reply - /// root carries an ALARM_RECORDS/@COUNT attribute that would make - /// the test exact if it reported the total active count rather than the - /// records in this reply; a live probe could not discriminate the two - /// (see docs/AlarmProbeFindings.md), so the count is deliberately - /// not trusted here. Exposed as internal static so the rule is - /// unit-testable without the wnwrapConsumer COM object. + /// root carries an ALARM_RECORDS/@COUNT attribute that would have + /// made the test exact had it reported the total active count; the + /// 2026-08-18 live probe observed it tracking the records in the reply + /// instead (three alarms active returned @COUNT 1 at cap 1 and 2 at cap + /// 2), so it carries no "more available" information and is deliberately + /// not trusted here (see docs/AlarmProbeFindings.md). Exposed as + /// internal static so the rule is unit-testable without the + /// wnwrapConsumer COM object. /// /// ALARM records the reply carried. /// The cap that was passed to the fetch. @@ -660,10 +664,17 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer /// instance rather than the state it is in: a re-minted /// GUID would read as the old alarm vanishing and a new one /// appearing, i.e. a spurious Clear plus a spurious Raise. Live - /// capture confirms stability across the active→returned leg only - /// (docs/AlarmClientDiscovery.md); the acknowledge leg and - /// re-raise-after-clear are assumed, not observed, because the dev - /// rig's alarm attributes reject unauthenticated writes — see + /// capture confirms stability across the active→returned leg + /// (docs/AlarmClientDiscovery.md, reconfirmed 2026-08-18). + /// Re-raise-after-clear was observed on 2026-08-18 and does + /// not reuse the GUID: clearing left the returned record in + /// place under its original GUID and the re-raise replaced it with a + /// freshly minted one. That is the intended reading — the old + /// instance ended and a new one began — so the diff emits a Clear + /// plus a Raise, which is what happened. The acknowledge leg remains + /// assumed rather than observed: on the dev rig every ack surface is + /// inert (AlarmAckByName returns 0 and nothing moves), so no + /// record can be driven into an acknowledged state — see /// docs/AlarmProbeFindings.md. /// ///