docs(alarms): second wnwrap probe attempt — secured-write verb unblocked the rig, answering @COUNT and the re-raise GUID leg
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m25s
ci / portable (push) Successful in 24m49s

AuthenticateUser("Administrator", "") + WriteSecured raises the alarm UDAs that
plain Write could not touch (SecurityError detail=1008), so the 2026-08-17 blocker
was the verb, exactly as that run's own Unblocking list predicted.

Two of the three open questions are now observed rather than assumed:

- ALARM_RECORDS/@COUNT reports the records in the reply, not the total active
  count. With three alarms active it read 1 at cap 1 and 2 at cap 2. There is no
  exact truncation signal to switch to, so IsTruncatedFetch's conservative rule is
  the design rather than a placeholder — behaviour unchanged, only the comments.
- Clear-then-re-raise mints a new GUID; the ALM->RTN leg keeps its GUID
  (reconfirming the 2026-05-01 capture). ComputeTransitions already reads the
  re-raise correctly as one instance ending and another beginning.

The acknowledge leg stays unobserved for a narrower reason: every wnwrap ack
surface is inert on this rig. AlarmAckByName returns 0 from the ack-only consumer
and -55 from the SetXmlAlarmQuery-applied one, for both the 6-arg and 8-arg forms,
and neither the snapshot STATE, OPERATOR_NAME, nor the extension's own .Acked
attribute moves. That corrects AlarmClientDiscovery.md, which read the zero return
as a working ack.

Comment- and prose-only; no behaviour change. The three throwaway probes ran from
the windev CI clone and were deleted; that clone is a clean tree at ab3ff16.
This commit is contained in:
Joseph Doherty
2026-08-18 05:40:11 -04:00
parent ad30adca2f
commit 90331b622c
5 changed files with 224 additions and 56 deletions
+11
View File
@@ -764,6 +764,17 @@ The v2 8-arg method returns -55 on this AVEVA build regardless of
operator-identity inputs — looks like a stub. The v1 6-arg method
works. Production `WnWrapAlarmConsumer.AcknowledgeByName` calls the
6-arg overload and discards the proto's `domain` + `full_name` fields.
**Correction, 2026-08-18** (`docs/AlarmProbeFindings.md`). Both halves
of the paragraph above are narrower than they read. The -55 tracks the
*consumer*, not the overload: against the `SetXmlAlarmQuery`-applied
reader both the 6-arg and 8-arg forms return -55, and against the
ack-only consumer both return 0. And "works" means "returns 0" only —
a probe that raised a real alarm and then acked it six ways watched the
snapshot stay `UNACK_ALM`, `OPERATOR_NAME` stay empty, and the alarm
extension's own `.Acked` attribute stay `False` for 16 s after each
call. On this rig the wnwrap ack is accepted and then inert, so a zero
return code must not be read as an applied acknowledgement.
The proto contract keeps the 8 fields for forward compatibility if
AVEVA fixes the v2 method later.
+176 -32
View File
@@ -1,7 +1,8 @@
# Alarm Probe Findings
`WnWrapAlarmConsumer` rests on two assumptions that no unit test can settle, because both
are properties of AVEVA's alarm provider rather than of our code:
`WnWrapAlarmConsumer` rested on two assumptions that no unit test can settle, because both
are properties of AVEVA's alarm provider rather than of our code. The second is now settled
and one half of the first is; the questions are stated here as they were asked:
1. **GUID identity.** The snapshot diff in `ComputeTransitions` keys on the alarm record's
`GUID`. If wnwrap mints a fresh GUID when an alarm changes state, a single
@@ -13,23 +14,27 @@ are properties of AVEVA's alarm provider rather than of our code:
rather than the records-in-reply count, truncation detection can become exact instead of
conservative, and the bounded staleness `ApplySnapshotUpdate` accepts goes away.
This document records what a live probe run against the dev rig (`DESKTOP-6JL3KKO`,
2026-08-17) could and could not establish, so the next attempt starts from the blocker
rather than rediscovering it.
This document records what live probe runs against the dev rig (`DESKTOP-6JL3KKO`,
2026-08-17 and 2026-08-18) could and could not establish, so the next attempt starts from
the blocker rather than rediscovering it.
## Outcome
| Question | Status |
|---|---|
| GUID stable across polls and `ALM → RTN` | Answered — yes, by the 2026-05-01 capture in `AlarmClientDiscovery.md` |
| GUID stable across `UNACK → ACK`, and across clear-then-re-raise | **Open** |
| `COUNT` = total active vs records-in-reply under a capped fetch | **Open** |
| GUID stable across polls and `ALM → RTN` | Answered — yes (2026-05-01 capture in `AlarmClientDiscovery.md`, reconfirmed 2026-08-18) |
| GUID stable across clear-then-re-raise | Answered 2026-08-18 — **no**, a re-raise mints a new GUID |
| GUID stable across `UNACK → ACK` | **Open** — the rig cannot be driven into an acknowledged state at all |
| `COUNT` = total active vs records-in-reply under a capped fetch | Answered 2026-08-18 — **records in the reply** |
Both open questions are blocked by the same thing: the rig has no active alarm and cannot
be driven into one over MXAccess, so there is no alarm instance whose GUID can be followed
through an acknowledge and no population large enough to overflow a capped fetch.
The 2026-08-17 run below is kept because it is the record of the wrong-verb blocker. The
2026-08-18 run cleared that blocker with `AuthenticateUser` + `WriteSecured` and answered
two of the three questions; the acknowledge leg is now blocked on something narrower and
different, described in "Second attempt".
## Why The Rig Cannot Raise An Alarm
## First attempt (2026-08-17): plain `Write`
### Why The Rig Could Not Raise An Alarm
The rig is otherwise healthy, which is what makes the blocker specific rather than a
general "nothing works":
@@ -64,7 +69,8 @@ for the whole probe session.
### Unblocking
Any one of these makes both questions answerable, in rough order of cost:
Any one of these makes both questions answerable, in rough order of cost. The second one
is what the 2026-08-18 run did, and it worked:
- Re-enable the System Platform script that flips `TestMachine_001.TestAlarm001`
(referenced throughout `AlarmClientDiscovery.md`). It writes from inside the engine, so
@@ -78,7 +84,7 @@ Three separate objects are wired to the same alarm UDA name, so once writes land
`maxAlmCnt` of 1 or 2 forces truncation against three active alarms and answers the `COUNT`
question in the same run.
## Evidence
### Evidence
Snapshot payload, identical at every cap (1, 2, and 1024) and at every poll across the
~100-second session:
@@ -98,26 +104,164 @@ The probe used for the run was a throwaway file in the windev CI clone
harness it was modelled on is
`src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/WnWrapConsumerProbeTests.cs`.
## Implications
## Second attempt (2026-08-18): secured-write verb
### Transition identity
The findings above named the fix in their own "Unblocking" list, and it holds:
`AuthenticateUser` + `WriteSecured` raises the alarms that plain `Write` could not touch.
The rig's alarm attributes are not unreachable — they are secured, and the 2026-08-17 probe
used the wrong verb.
`ComputeTransitions` keying on GUID is safe for the raise and clear legs, which is the
evidence `AlarmClientDiscovery.md` already carries. The acknowledge leg — the one where a
re-minted GUID would corrupt the feed, because an ack is the state change most likely to
create a new record in a provider that models acknowledgement as a separate event — is
still assumed rather than observed. Nothing here justifies changing the diff, but the
assumption should not be described in code as established.
### Method
### Truncation detection
Three throwaway probes in the windev CI clone (`C:\build\mxaccessgw-ci`, detached at
`ab3ff16`), each an x86 net48 xUnit fact running on a dedicated STA with a
`PeekMessage`/`DispatchMessage` pump so MXAccess events deliver:
`IsTruncatedFetch` stays as written. Tightening it to an exact test requires knowing that
`COUNT` reports the total, and this run cannot show that. The conservative rule keeps its
justification: at the cap, treating a complete fetch as truncated costs one poll of
staleness, while treating a truncated fetch as complete broadcasts clears for every alarm
past the cap.
1. secured write + both GUID legs + the capped-fetch question,
2. the acknowledge surface, exhaustively,
3. the acknowledge question re-asked through an independent observer.
The one substantive correction is to the phrasing rather than the logic. The reply is not
featureless — it carries a `COUNT` attribute the parser currently ignores. Whether that
attribute is a usable "more available" signal is unverified, not absent, and the comments
in `WnWrapAlarmConsumer` now say so.
All three used `LMXProxyServerClass` directly for the write path and a
`wwAlarmConsumerClass` pair — a reader with `SetXmlAlarmQuery` applied and an ack-only
consumer without it — mirroring `WnWrapAlarmConsumer`'s own two-consumer arrangement. The
subscription was `\\DESKTOP-6JL3KKO\Galaxy!TestArea`, matching the 2026-08-17 run.
`MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER` / `_PASSWORD` are set at neither machine nor
user scope on the box, so the probes used the default rig identity, `Administrator` with an
empty password. All three files were deleted afterwards and the clone is a clean tree.
### The secured write lands
Same session, same item, one after the other — the control reproduces 2026-08-17 exactly
and the secured verb succeeds:
```
Write 'TestMachine_001.TestAlarm001' <- True
WRITE-COMPLETE hLMX=1 hItem=1 statuses=[success=0 category=SecurityError detectedBy=RespondingAutomationObject detail=1008 text=]
observed 'TestMachine_001.TestAlarm001' = False
AuthenticateUser -> userId=1
WriteSecured 'TestMachine_001.TestAlarm001' <- True currentUserId=1 verifierUserId=0
DATA-CHANGE hItem=1 value=True quality=192
WRITE-COMPLETE hLMX=1 hItem=1 statuses=[success=-1 category=Ok detectedBy=RespondingAutomationObject detail=0 text=]
observed 'TestMachine_001.TestAlarm001' = True
```
`AuthenticateUser("Administrator", "")` resolves to user id 1, matching the value
`WorkerLiveMxAccessSmokeTests` records. No verifier is needed: `verifierUserId=0` is
accepted. All six subsequent raise/clear writes across the three objects behaved the same
way, so the path is reliable rather than a one-off.
### Q2 — `ALARM_RECORDS/@COUNT` reports the reply, not the total
With all three `TestMachine_00{1,2,3}.TestAlarm001` alarms active, the same subscription
fetched at three caps back to back:
```
FETCH three-active cap=1024 ALARM_RECORDS/@COUNT=3 elementCount=3
FETCH three-active cap=1 ALARM_RECORDS/@COUNT=1 elementCount=1
FETCH three-active cap=2 ALARM_RECORDS/@COUNT=2 elementCount=2
```
`@COUNT` tracked the cap, not the population. It equals the element count in every reply,
including the two that are provably truncated — the galaxy held three active alarms while
`@COUNT` read 1 and 2. The attribute therefore carries no "more available" information, and
the hypothesis that it might report the total is refuted rather than merely untested.
`IsTruncatedFetch` stays exactly as written, and this is now settled rather than deferred:
there is no exact truncation signal to switch to. Tightening it is **not** a future task.
The conservative rule keeps its original justification — at the cap, treating a complete
fetch as truncated costs one poll of staleness, while treating a truncated fetch as
complete broadcasts clears for every alarm past the cap.
### Q1 — clear-then-re-raise mints a new GUID
`TestMachine_001.TestAlarm001` was raised, cleared, and re-raised through `WriteSecured`,
with a full snapshot fetch after each leg:
```
after-raise TAG=TestMachine_001.TestAlarm001 GUID=72B84A6E7BA74D42B611EB5393626F79 STATE=UNACK_ALM VALUE=true
after-clear-001 TAG=TestMachine_001.TestAlarm001 GUID=72B84A6E7BA74D42B611EB5393626F79 STATE=UNACK_RTN VALUE=false
after-reraise TAG=TestMachine_001.TestAlarm001 GUID=2394FEAA94774154AAA97C3AC29CA51E STATE=UNACK_ALM VALUE=true
```
Two facts, one confirming and one new. The `ALM → RTN` leg holds its GUID, reconfirming the
2026-05-01 capture against a write-driven rather than script-driven transition. The re-raise
does not: the returned record keeps the old GUID and the new active condition arrives as a
separate record under a fresh one.
That is the correct reading for `ComputeTransitions` rather than a problem for it. The old
GUID leaving the active set is a real Clear and the new GUID is a real Raise, because they
are two alarm instances. The diff needed no change.
### Q1 — the acknowledge leg is still unobserved, for a new reason
The blocker moved. The rig can now be driven into an alarm; it cannot be driven out of
`UNACK_ALM`. Every acknowledge surface `wwAlarmConsumerClass` exposes was tried against a
freshly raised alarm, each followed by 16 seconds of polling:
| Attempt | Result |
|---|---|
| `AlarmAckByName` 6-arg on the ack-only consumer, node = machine | `rc=0`, state unchanged |
| same, node empty | `rc=0`, state unchanged |
| same, name as `Galaxy!TestArea.TestMachine_001.TestAlarm001` | `rc=0`, state unchanged |
| same, operator name `Administrator` | `rc=0`, state unchanged |
| `AlarmAckByName` 6-arg on the reader consumer (`SetXmlAlarmQuery` applied) | `rc=-55`, state unchanged |
| `AlarmAckByName` 8-arg (v2) on the ack-only consumer | `rc=0`, state unchanged |
The `rc=-55` on the `SetXmlAlarmQuery`-applied consumer reproduces the 2026-05-01 finding
that motivated the two-consumer split, so the ack-only consumer was correctly provisioned;
the `rc=0` returns are the ones that go nowhere.
A third probe removed the last doubt by watching the alarm extension's own `.Acked`
attribute over MXAccess — an observer entirely independent of the wnwrap snapshot:
```
VALUES after-raise: 'TestMachine_001.TestAlarm001.Acked' = False
AlarmAckByName -> rc=0
VALUES after-wnwrap-ack: 'TestMachine_001.TestAlarm001.Acked' = False
XML after-wnwrap-ack: STATE=UNACK_ALM OPRNAME= OPRNODE=
```
Nothing moves: not the extension attribute, not the snapshot `STATE`, not `OPERATOR_NAME`.
`rc=0` from wnwrap means the call was accepted, not that an acknowledgement was applied.
Acknowledging by writing the extension attribute directly is not an alternative — the
attribute is not writable, and says so with an operational rather than a security failure,
for both verbs:
```
WriteSecured 'TestMachine_001.TestAlarm001.Acked' <- true
WRITE-COMPLETE statuses=[success=0 category=MxCategoryOperationalError detectedBy=MxSourceRespondingAutomationObject detail=1007]
Write 'TestMachine_001.TestAlarm001.Acked' <- true
WRITE-COMPLETE statuses=[success=0 category=MxCategoryOperationalError detectedBy=MxSourceRespondingAutomationObject detail=1007]
```
`detail=1007` from the responding automation object, unchanged by authentication, is a
read-only attribute rather than a permission refusal — the `1008` `SecurityError` the alarm
UDA itself returns is what a permission refusal looks like on this rig. So the ack has no
MXAccess-side entry point and the wnwrap-side entry point is inert.
#### Remaining unblock paths for the acknowledge leg
- Acknowledge from the System Platform side — the IDE's alarm client, InTouch, or an
ArchestrA graphic bound to the alarm — and watch the snapshot from a running probe. This
proves whether wnwrap's `STATE` ever reports `ACK_ALM` at all, which is the actual
question; the ack API being inert may be a wnwrap defect layered on top of a snapshot
that would report the state correctly.
- Check whether `alarmmgr` on this rig is configured with an alarm-acknowledgement security
requirement that the wnwrap consumer, which passes an operator *name* string and no
authenticated identity, cannot meet. If so, ack over wnwrap is not merely untested here
but unavailable by configuration, and the gateway's `AcknowledgeByName` path needs the
same treatment on any customer galaxy configured that way.
- If neither lands, the acknowledge leg stays assumed. It is worth restating that this is a
documentation gap, not a correctness one: a re-minted GUID on acknowledge would produce a
spurious Clear plus a spurious Raise, which is the same shape the now-observed re-raise
behaviour produces and which `ComputeTransitions` already handles as two instances.
### Rig state left behind
The three `TestMachine_00{1,2,3}.TestAlarm001` UDAs are back to `false` and their
`.InAlarm` subtags read `false`, but each leaves a `UNACK_RTN` record in the wnwrap
snapshot, since nothing can acknowledge them away. `SnapshotActiveAlarms` counts only
`UNACK_ALM` and `ACK_ALM` as active, so these are inert for the gateway; they will clear on
the next `alarmmgr` restart.
+8 -7
View File
@@ -226,13 +226,14 @@ payload carries the flag as well because a prefix filter (or an empty galaxy) ca
leave zero records, and a truncated fetch with nothing to report still has to say
so.
The **detection heuristic is unchanged**: `IsTruncatedFetch` remains
`fetchedRecordCount >= maxAlarmsPerFetch`. The live probe run for this work could
not verify whether `ALARM_RECORDS/@COUNT` reports the total active count or only
the records in the reply (`docs/AlarmProbeFindings.md`), and an exact-looking
signal derived from an unverified attribute is worse than an honest heuristic —
it would read as precise while being wrong in the one direction that matters.
Switching to `@COUNT` stays blocked on probe evidence.
The **detection heuristic is unchanged, and now permanently**: `IsTruncatedFetch`
remains `fetchedRecordCount >= maxAlarmsPerFetch`. The 2026-08-18 probe settled
what the earlier run could not: with three alarms active, `ALARM_RECORDS/@COUNT`
read 1 at a cap of 1 and 2 at a cap of 2, so it reports the records in the reply
rather than the total active count (`docs/AlarmProbeFindings.md`). There is no
exact signal to switch to — `@COUNT` cannot distinguish a capped fetch from a
complete one — so the heuristic is the design rather than a placeholder waiting
on evidence.
The flag is **not latched**. It is replaced by each fetch's verdict, so the first
sub-cap fetch clears it, and `GatewayAlarmMonitor.ClearCache` drops it with the
+4 -3
View File
@@ -265,9 +265,10 @@ means "this active set may be incomplete", not "this record is unreliable" —
it is independent of the subtag-fallback `degraded` field above. It is not
latched: the first fetch that comes back under the cap is complete, restores
absence authority, and clears it. Detection remains the record-count heuristic;
the reply's `ALARM_RECORDS/@COUNT` attribute would make the test exact only if
it reported the total active count rather than the records in the reply, which
a live probe could not discriminate (see `docs/AlarmProbeFindings.md`).
the reply's `ALARM_RECORDS/@COUNT` attribute would have made the test exact only
if it reported the total active count, and the 2026-08-18 live probe observed it
reporting the records in the reply instead (see `docs/AlarmProbeFindings.md`), so
there is no exact alternative.
Forced modes are available via `MxGateway:Alarms:Fallback:Mode`:
`ForceAlarmManager` disables failover; `ForceSubtag` forces the standby
@@ -429,10 +429,12 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
// have exactly that many active alarms. Treat the ambiguous case as
// truncated: the false-positive cost is a snapshot that stays stale for
// one poll, the false-negative cost is every alarm past the cap reading
// as cleared. (The reply's ALARM_RECORDS/@COUNT attribute is a
// candidate exact signal, but only if it reports the total rather than
// the records in the reply — untested on a live rig, see
// docs/AlarmProbeFindings.md.)
// as cleared. (The reply's ALARM_RECORDS/@COUNT attribute was the
// candidate exact signal; the 2026-08-18 live probe settled it — with
// three alarms active, @COUNT read 3, 1, and 2 at caps 1024, 1, and 2,
// matching the records in the reply every time. It reports the reply,
// not the total, so it cannot tell a capped fetch from a complete one.
// See docs/AlarmProbeFindings.md.)
bool truncated = IsTruncatedFetch(fetchedRecordCount, maxAlarmsPerFetch);
IReadOnlyList<MxAlarmTransitionEvent> transitions =
@@ -491,12 +493,14 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
/// offers no confirmed "more available" flag, so a reply at exactly the
/// cap is indistinguishable from a galaxy that happens to hold exactly
/// that many active alarms; both are treated as truncated. The reply
/// root carries an <c>ALARM_RECORDS/@COUNT</c> attribute that would make
/// the test exact if it reported the total active count rather than the
/// records in this reply; a live probe could not discriminate the two
/// (see <c>docs/AlarmProbeFindings.md</c>), so the count is deliberately
/// not trusted here. Exposed as <c>internal static</c> so the rule is
/// unit-testable without the wnwrapConsumer COM object.
/// root carries an <c>ALARM_RECORDS/@COUNT</c> attribute that would have
/// made the test exact had it reported the total active count; the
/// 2026-08-18 live probe observed it tracking the records in the reply
/// instead (three alarms active returned @COUNT 1 at cap 1 and 2 at cap
/// 2), so it carries no "more available" information and is deliberately
/// not trusted here (see <c>docs/AlarmProbeFindings.md</c>). Exposed as
/// <c>internal static</c> so the rule is unit-testable without the
/// wnwrapConsumer COM object.
/// </summary>
/// <param name="fetchedRecordCount">ALARM records the reply carried.</param>
/// <param name="maxAlarmsPerFetch">The cap that was passed to the fetch.</param>
@@ -660,10 +664,17 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
/// <em>instance</em> rather than the state it is in: a re-minted
/// GUID would read as the old alarm vanishing and a new one
/// appearing, i.e. a spurious Clear plus a spurious Raise. Live
/// capture confirms stability across the active→returned leg only
/// (<c>docs/AlarmClientDiscovery.md</c>); the acknowledge leg and
/// re-raise-after-clear are assumed, not observed, because the dev
/// rig's alarm attributes reject unauthenticated writes — see
/// capture confirms stability across the active→returned leg
/// (<c>docs/AlarmClientDiscovery.md</c>, reconfirmed 2026-08-18).
/// Re-raise-after-clear was observed on 2026-08-18 and does
/// <em>not</em> reuse the GUID: clearing left the returned record in
/// place under its original GUID and the re-raise replaced it with a
/// freshly minted one. That is the intended reading — the old
/// instance ended and a new one began — so the diff emits a Clear
/// plus a Raise, which is what happened. The acknowledge leg remains
/// assumed rather than observed: on the dev rig every ack surface is
/// inert (<c>AlarmAckByName</c> returns 0 and nothing moves), so no
/// record can be driven into an acknowledged state — see
/// <c>docs/AlarmProbeFindings.md</c>.
/// </para>
/// </remarks>