mxaccessgw/docs/AlarmProbeFindings.md

# Alarm Probe Findings

`WnWrapAlarmConsumer` rested on two assumptions that no unit test can settle, because both
are properties of AVEVA's alarm provider rather than of our code. The second is now settled
and one half of the first is; the questions are stated here as they were asked:

1. **GUID identity.** The snapshot diff in `ComputeTransitions` keys on the alarm record's
   `GUID`. If wnwrap mints a fresh GUID when an alarm changes state, a single
   `UNACK_ALM → ACK_ALM` transition reads as one alarm disappearing and a different one
   appearing — a spurious clear plus a spurious raise on every acknowledge.
2. **`ALARM_RECORDS/@COUNT` semantics.** `IsTruncatedFetch` treats a reply holding exactly
   `maxAlmCnt` records as truncated, because `GetXmlCurrentAlarms2` exposes no explicit
   "more available" flag. If the reply's `COUNT` attribute carries the *total* active count
   rather than the records-in-reply count, truncation detection can become exact instead of
   conservative, and the bounded staleness `ApplySnapshotUpdate` accepts goes away.

This document records what live probe runs against the dev rig (`DESKTOP-6JL3KKO`,
2026-08-17 and 2026-08-18) could and could not establish, so the next attempt starts from
the blocker rather than rediscovering it.

## Outcome

| Question | Status |
|---|---|
| GUID stable across polls and `ALM → RTN` | Answered — yes (2026-05-01 capture in `AlarmClientDiscovery.md`, reconfirmed 2026-08-18) |
| GUID stable across clear-then-re-raise | Answered 2026-08-18 — **no**, a re-raise mints a new GUID |
| GUID stable across `UNACK → ACK` | **Open** — the rig cannot be driven into an acknowledged state; 2026-08-18 (third attempt) identifies the reason as the test attribute's `MxSecurityOperate` classification, which no non-interactive ack surface on the rig can satisfy |
| `COUNT` = total active vs records-in-reply under a capped fetch | Answered 2026-08-18 — **records in the reply** |

The 2026-08-17 run below is kept because it is the record of the wrong-verb blocker. The
2026-08-18 run cleared that blocker with `AuthenticateUser` + `WriteSecured` and answered
two of the three questions; the acknowledge leg is now blocked on something narrower and
different, described in "Second attempt" and diagnosed in "Third attempt".

## First attempt (2026-08-17): plain `Write`

### Why The Rig Could Not Raise An Alarm

The rig is otherwise healthy, which is what makes the blocker specific rather than a
general "nothing works":

- `aaEngine`, `alarmmgr`, `NmxSvc`, and `wnwrapServerEx` are all running.
- `TestArea` (area of `TestMachine_001`…`_003`) and the objects themselves are deployed
  (`deployed_version` non-null in the `ZB` Galaxy Repository) and on scan — the probe's
  advised `ScanState` subtags report true, and every advised alarm attribute delivers an
  initial value, so the MXAccess read path is live.
- The wnwrap consumer subscribes cleanly: `InitializeConsumer`, `RegisterConsumer`,
  `Subscribe(\\DESKTOP-6JL3KKO\Galaxy!TestArea)`, and `SetXmlAlarmQuery` all return 0, and
  `GetXmlCurrentAlarms2` returns well-formed XML on every poll.

What fails is the *write* that would set the alarm condition. Every `Write` to the alarm
UDAs completes with a security failure:

```
WRITE-COMPLETE hLMX=1 hItem=1 statuses=[success=0 category=SecurityError detectedBy=RespondingAutomationObject detail=1008 text=]
```

The status comes back from the responding automation object, not from the proxy, so the
request reaches the engine and the engine refuses it. The advised value confirms the
refusal is total rather than transient: neither the alarm UDA nor its `.InAlarm` /`.Acked`
subtags report any change after a write attempt, across six write attempts in one session
(raise, clear, re-raise, cleanup). The attributes carry a security classification that a
plain `Write` cannot satisfy.

The 2026-05-01 capture that answered the `ALM → RTN` leg did not hit this, because the
alarm condition was driven from *inside* the engine by a System Platform script rather than
from an external MXAccess client. That script is not running now, and the values sat idle
for the whole probe session.

### Unblocking

Any one of these makes both questions answerable, in rough order of cost. The second one
is what the 2026-08-18 run did, and it worked:

- Re-enable the System Platform script that flips `TestMachine_001.TestAlarm001`
  (referenced throughout `AlarmClientDiscovery.md`). It writes from inside the engine, so
  the attribute's security classification does not apply.
- Drive the write through `AuthenticateUser` + `WriteSecured` with a Galaxy account
  permitted on that classification. The worker already implements both verbs; the probe
  used plain `Write`, which is the wrong verb for a secured attribute.
- Reclassify the test UDAs to free access in the IDE and redeploy `TestMachine_001`…`_003`.

Three separate objects are wired to the same alarm UDA name, so once writes land, a
`maxAlmCnt` of 1 or 2 forces truncation against three active alarms and answers the `COUNT`
question in the same run.

### Evidence

Snapshot payload, identical at every cap (1, 2, and 1024) and at every poll across the
~100-second session:

```xml
<?xml version="1.0"?><ALARM_RECORDS COUNT="0"></ALARM_RECORDS>
```

Two things follow from the empty case alone. `COUNT` is present on the root element in
every reply, so the attribute exists as a candidate signal rather than something wnwrap
omits. And `COUNT` agrees with the element count here — but trivially, since both are zero,
which is exactly the case that cannot discriminate the two hypotheses.

The probe used for the run was a throwaway file in the windev CI clone
(`C:\build\mxaccessgw-ci`), deleted afterwards; the clone is back to a clean tree at
`origin/main`. Nothing in this repository changed to run it. The reusable, Skip-gated
harness it was modelled on is
`src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/WnWrapConsumerProbeTests.cs`.

## Second attempt (2026-08-18): secured-write verb

The findings above named the fix in their own "Unblocking" list, and it holds:
`AuthenticateUser` + `WriteSecured` raises the alarms that plain `Write` could not touch.
The rig's alarm attributes are not unreachable — they are secured, and the 2026-08-17 probe
used the wrong verb.

### Method

Three throwaway probes in the windev CI clone (`C:\build\mxaccessgw-ci`, detached at
`ab3ff16`), each an x86 net48 xUnit fact running on a dedicated STA with a
`PeekMessage`/`DispatchMessage` pump so MXAccess events deliver:

1. secured write + both GUID legs + the capped-fetch question,
2. the acknowledge surface, exhaustively,
3. the acknowledge question re-asked through an independent observer.

All three used `LMXProxyServerClass` directly for the write path and a
`wwAlarmConsumerClass` pair — a reader with `SetXmlAlarmQuery` applied and an ack-only
consumer without it — mirroring `WnWrapAlarmConsumer`'s own two-consumer arrangement. The
subscription was `\\DESKTOP-6JL3KKO\Galaxy!TestArea`, matching the 2026-08-17 run.
`MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER` / `_PASSWORD` are set at neither machine nor
user scope on the box, so the probes used the default rig identity, `Administrator` with an
empty password. All three files were deleted afterwards and the clone is a clean tree.

### The secured write lands

Same session, same item, one after the other — the control reproduces 2026-08-17 exactly
and the secured verb succeeds:

```
Write 'TestMachine_001.TestAlarm001' <- True
WRITE-COMPLETE hLMX=1 hItem=1 statuses=[success=0 category=SecurityError detectedBy=RespondingAutomationObject detail=1008 text=]
  observed 'TestMachine_001.TestAlarm001' = False

AuthenticateUser -> userId=1
WriteSecured 'TestMachine_001.TestAlarm001' <- True currentUserId=1 verifierUserId=0
DATA-CHANGE hItem=1 value=True quality=192
WRITE-COMPLETE hLMX=1 hItem=1 statuses=[success=-1 category=Ok detectedBy=RespondingAutomationObject detail=0 text=]
  observed 'TestMachine_001.TestAlarm001' = True
```

`AuthenticateUser("Administrator", "")` resolves to user id 1, matching the value
`WorkerLiveMxAccessSmokeTests` records. No verifier is needed: `verifierUserId=0` is
accepted. All six subsequent raise/clear writes across the three objects behaved the same
way, so the path is reliable rather than a one-off.

### Q2 — `ALARM_RECORDS/@COUNT` reports the reply, not the total

With all three `TestMachine_00{1,2,3}.TestAlarm001` alarms active, the same subscription
fetched at three caps back to back:

```
FETCH three-active cap=1024   ALARM_RECORDS/@COUNT=3  elementCount=3
FETCH three-active cap=1      ALARM_RECORDS/@COUNT=1  elementCount=1
FETCH three-active cap=2      ALARM_RECORDS/@COUNT=2  elementCount=2
```

`@COUNT` tracked the cap, not the population. It equals the element count in every reply,
including the two that are provably truncated — the galaxy held three active alarms while
`@COUNT` read 1 and 2. The attribute therefore carries no "more available" information, and
the hypothesis that it might report the total is refuted rather than merely untested.

`IsTruncatedFetch` stays exactly as written, and this is now settled rather than deferred:
there is no exact truncation signal to switch to. Tightening it is **not** a future task.
The conservative rule keeps its original justification — at the cap, treating a complete
fetch as truncated costs one poll of staleness, while treating a truncated fetch as
complete broadcasts clears for every alarm past the cap.

### Q1 — clear-then-re-raise mints a new GUID

`TestMachine_001.TestAlarm001` was raised, cleared, and re-raised through `WriteSecured`,
with a full snapshot fetch after each leg:

```
after-raise      TAG=TestMachine_001.TestAlarm001 GUID=72B84A6E7BA74D42B611EB5393626F79 STATE=UNACK_ALM  VALUE=true
after-clear-001  TAG=TestMachine_001.TestAlarm001 GUID=72B84A6E7BA74D42B611EB5393626F79 STATE=UNACK_RTN  VALUE=false
after-reraise    TAG=TestMachine_001.TestAlarm001 GUID=2394FEAA94774154AAA97C3AC29CA51E STATE=UNACK_ALM  VALUE=true
```

Two facts, one confirming and one new. The `ALM → RTN` leg holds its GUID, reconfirming the
2026-05-01 capture against a write-driven rather than script-driven transition. The re-raise
does not: the record comes back under a GUID that has never been seen before.

The two records do **not** coexist — the snapshot carries one record per tag, and the
re-raise replaces the returned record rather than joining it. Both fetches above are the
full `cap=1024` reply, byte-identical in length (1613) at `@COUNT=3`/`elementCount=3`, the
three records being one per `TestMachine_00{1,2,3}`; `72B84A6E…` is simply absent from the
later one. So a single poll spanning the re-raise sees the old GUID disappear and the new
one appear together.

That is the correct reading for `ComputeTransitions` rather than a problem for it. The old
GUID leaving the active set is a real Clear and the new GUID is a real Raise, because they
are two alarm instances. The diff needed no change.

### Q1 — the acknowledge leg is still unobserved, for a new reason

The blocker moved. The rig can now be driven into an alarm; it cannot be driven out of
`UNACK_ALM`. Every acknowledge surface `wwAlarmConsumerClass` exposes was tried against a
freshly raised alarm, each followed by 16 seconds of polling:

| Attempt | Result |
|---|---|
| `AlarmAckByName` 6-arg on the ack-only consumer, node = machine | `rc=0`, state unchanged |
| same, node empty | `rc=0`, state unchanged |
| same, name as `Galaxy!TestArea.TestMachine_001.TestAlarm001` | `rc=0`, state unchanged |
| same, operator name `Administrator` | `rc=0`, state unchanged |
| `AlarmAckByName` 6-arg on the reader consumer (`SetXmlAlarmQuery` applied) | `rc=-55`, state unchanged |
| `AlarmAckByName` 8-arg (v2) on the ack-only consumer | `rc=0`, state unchanged |

The `rc=-55` on the `SetXmlAlarmQuery`-applied consumer reproduces the 2026-05-01 finding
that motivated the two-consumer split, so the ack-only consumer was correctly provisioned;
the `rc=0` returns are the ones that go nowhere.

A third probe removed the last doubt by watching the alarm extension's own `.Acked`
attribute over MXAccess — an observer entirely independent of the wnwrap snapshot:

```
VALUES after-raise:        'TestMachine_001.TestAlarm001.Acked' = False
AlarmAckByName -> rc=0
VALUES after-wnwrap-ack:   'TestMachine_001.TestAlarm001.Acked' = False
XML   after-wnwrap-ack:    STATE=UNACK_ALM OPRNAME= OPRNODE=
```

Nothing moves: not the extension attribute, not the snapshot `STATE`, not `OPERATOR_NAME`.
`rc=0` from wnwrap means the call was accepted, not that an acknowledgement was applied.

Acknowledging by writing the extension attribute directly is not an alternative — the
attribute is not writable, and says so with an operational rather than a security failure,
for both verbs:

```
WriteSecured 'TestMachine_001.TestAlarm001.Acked' <- true
WRITE-COMPLETE statuses=[success=0 category=MxCategoryOperationalError detectedBy=MxSourceRespondingAutomationObject detail=1007]
Write        'TestMachine_001.TestAlarm001.Acked' <- true
WRITE-COMPLETE statuses=[success=0 category=MxCategoryOperationalError detectedBy=MxSourceRespondingAutomationObject detail=1007]
```

`detail=1007` from the responding automation object, unchanged by authentication, is a
read-only attribute rather than a permission refusal — the `1008` `SecurityError` the alarm
UDA itself returns is what a permission refusal looks like on this rig. So the ack has no
MXAccess-side entry point and the wnwrap-side entry point is inert.

#### Remaining unblock paths for the acknowledge leg

- Acknowledge from the System Platform side — the IDE's alarm client, InTouch, or an
  ArchestrA graphic bound to the alarm — and watch the snapshot from a running probe. This
  proves whether wnwrap's `STATE` ever reports `ACK_ALM` at all, which is the actual
  question; the ack API being inert may be a wnwrap defect layered on top of a snapshot
  that would report the state correctly.
- Check whether `alarmmgr` on this rig is configured with an alarm-acknowledgement security
  requirement that the wnwrap consumer, which passes an operator *name* string and no
  authenticated identity, cannot meet. If so, ack over wnwrap is not merely untested here
  but unavailable by configuration, and the gateway's `AcknowledgeByName` path needs the
  same treatment on any customer galaxy configured that way. (Answered by the Third attempt
  below — enforced by the alarm attribute's `MxSecurityOperate` security classification, an
  engine-level write-security setting, not a separate `alarmmgr`-side ack policy.)
- If neither lands, the acknowledge leg stays assumed. It is worth restating that this is a
  documentation gap, not a correctness one: a re-minted GUID on acknowledge would produce a
  spurious Clear plus a spurious Raise, which is the same shape the now-observed re-raise
  behaviour produces and which `ComputeTransitions` already handles as two instances.

## Third attempt (2026-08-18): why the acknowledge is refused

The second attempt left two candidate explanations for `AlarmAckByName` returning `rc=0`
and changing nothing: the rig enforces an acknowledgement security requirement the wnwrap
consumer cannot meet, or wnwrap's ack is simply broken here. This attempt was read-only —
no writes, no alarms raised, no configuration touched — and settles the first question:
the requirement is **enforced**, and the alarm attribute's security classification is what
enforces it.

### Method

Read-only inspection of the `ZB` Galaxy Repository over `sqlcmd -S localhost -d ZB -E`,
plus the already-built `mxa` CLI (`C:\Users\dohertj2\Desktop\wwtools\mxaccesscli\src\MxAccess.Cli\bin\x86\Release\net48\mxa.exe`)
for runtime reads. Nothing was written and nothing was installed. The `lmxopcua\gr` schema
notes referenced elsewhere in this repo are **not present on this box** (`Test-Path` is
`False`), so the schema was located by querying `sys.tables` / `sys.columns` directly.

### The test attribute is classified `MxSecurityOperate`

UDA security classification lives in `dynamic_attribute.security_classification`, keyed by
`gobject_id`. For the `$TestMachine` template that is `1055`:

```
SELECT gobject_id, tag_name, hierarchical_name FROM gobject WHERE gobject_id=1055;
1055|$TestMachine|$TestMachine
```

```
attribute_name  | security_classification | mx_attribute_category
ProtectedValue  | 2                       | 10
ProtectedValue1 | 3                       | 10
TestAlarm001    | 1                       | 10
TestAlarm002    | 1                       | 10
TestAlarm003    | 1                       | 10
TestChangingInt | 1                       | 10
```

The enum is pinned by two independent sources rather than assumed. `ProtectedValue` and
`ProtectedValue1` are the mxaccess analysis project's documented *secured-write* and
*verified-write* fixtures (`C:\Users\dohertj2\Desktop\mxaccess\docs\galaxy-test-fixtures.md`),
and `docs/NMX-COM-Contracts.md` in the same project records "Galaxy security classification
(`2` for `SecuredWrite`, `3` for VerifiedWrite)" — so `2` and `3` land on exactly the two
attributes that are supposed to carry them. `galaxy-test-fixtures.md` also records the
provisioning verb used for every UDA in that inventory: `--security MxSecurityOperate`.
`TestAlarm001` therefore reads `1` = **`MxSecurityOperate`**.

That is the missing piece from 2026-08-17. `Operate` is not free access: it requires an
authenticated galaxy identity holding Operate permission on the object's security group.
An unauthenticated `Write` is refused with `SecurityError` `1008`, and the same write after
`AuthenticateUser` succeeds — which is precisely the pair of results both prior attempts
recorded. Galaxy security is live on this rig, not disabled.

`wwAlarmConsumerClass.AlarmAckByName` carries an operator *name* string and a comment. It
carries no authenticated user id, no credential, and no token — there is no parameter on
either the 6-arg or the 8-arg overload that could convey one. A consumer calling it cannot
satisfy an `Operate` classification, and `rc=0` followed by no state change is what an ack
dropped downstream of an accepted call looks like.

The one step this stops short of is a direct experiment: the rig's security configuration
is out of scope for a read-only probe, so "an `Operate` alarm cannot be acknowledged
without an authenticated identity" is inferred from the classification plus the observed
`1008`/`rc=0` pattern rather than observed by relaxing the classification and watching the
ack start working. The human step below is exactly that experiment.

### There is no writeable `.Ack` attribute — the second attempt targeted the only one there is

A plausible reading of the second attempt was that it wrote the wrong sub-attribute:
ArchestrA alarm extensions are commonly described as exposing a writeable `.Ack` alongside
the read-only `.Acked`. On this galaxy they do not. Every `Ack`-named attribute the alarm
primitives define:

```
attribute_name        | security_classification | mx_attribute_category
Acked                 | -1                      | 2
AckMsg                | 0                       | 6
Bad.Acked             | -1                      | 2
Bad.AckMsg            | 0                       | 6
TimeAlarmAcked        | -1                      | 2
AlarmAckCnt           | -1                      | 2
AlarmAckErrorsCnt     | -1                      | 2
AlarmMostUrgentAcked  | -1                      | 2
AlarmUnAckedCnt       | -1                      | 2
```

No `Ack`. `Acked` carries `security_classification = -1` — no classification at all, which
is what a non-writeable attribute carries, as distinct from the `1` on the writeable
`TestAlarm001`. That is the configuration-side counterpart of the `detail=1007`
operational refusal the second attempt got when it wrote `.Acked`: not a permission
refusal, but an attribute that has no write path to refuse. The second attempt had already
found the only MXAccess-side candidate, and it is read-only by definition.

### No non-interactive acknowledge surface is installed

The `wwtools` collection on the box (`aalogcli`, `aot`, `graccesscli`, `grdb`, `histdb`,
`mbproxy`, `mxaccesscli`, `secrets`) is the most likely home for a scriptable ack. There
isn't one. `mxa --help` lists `diag`, `info`, `read`, `read-batch`, `subscribe`,
`subscribe-batch`, `write`, `write-batch` — a tag data-plane only, with no alarm surface.
`graccesscli` is a Galaxy Repository configuration tool (`object uda add`, `instance
deploy`), which acts at configure/deploy time and not on live alarm state. Reading the
galaxy's own authentication mode at runtime is also unavailable: `Galaxy.AuthenticationMode`
does not resolve over MXAccess (`Category=4 Detail=6`), and the value is not in
`dynamic_attribute` — only the attribute *definition* names `AuthenticationMode` and
`_AuthenticationModeEnum` exist in `attribute_definition`.

So every remaining acknowledge surface on this rig is interactive: the IDE's alarm client,
InTouch, or an ArchestrA graphic bound to the alarm. Driving those is out of scope.

### Status of the acknowledge leg

**Unavailable by configuration, and the GUID question stays assumed.** The two are separate
statements and both matter:

- The wnwrap ack path is unavailable on this rig as configured, for an identified reason
  rather than an unknown one. This is a real finding for the gateway: `AcknowledgeByName`
  will behave the same way — accepted, inert — on any customer galaxy whose alarmed
  attributes carry a non-free-access security classification (inferred from the mechanism —
  no `AlarmAckByName` overload can carry a credential — not confirmed by relaxing the
  classification and re-testing; see "What a human would need to do"). It is worth noting in
  the alarm client's documentation that a silent `rc=0` is not proof of acknowledgement.
- Whether wnwrap re-mints the record GUID on `UNACK_ALM → ACK_ALM` is still unobserved, and
  after three attempts it stays assumed. As the second attempt already noted, this remains
  a documentation gap rather than a correctness one: a re-minted GUID produces a spurious
  Clear plus a spurious Raise, the same shape the observed re-raise behaviour produces, and
  `ComputeTransitions` already handles that correctly as two instances.

#### What a human would need to do

Either of these answers it; the second is cheaper and also confirms or refutes the
classification hypothesis above, which the read-only probe could only infer.

1. Acknowledge `TestMachine_001.TestAlarm001` from an interactive System Platform client
   (IDE alarm client, InTouch, or an ArchestrA graphic) while a wnwrap probe polls
   `GetXmlCurrentAlarms2` against `\\DESKTOP-6JL3KKO\Galaxy!TestArea`, and record whether
   `STATE` reaches `ACK_ALM` and whether `GUID` survives the transition. Raise the alarm
   first with `AuthenticateUser` + `WriteSecured` as the second attempt did.
2. Reclassify `TestAlarm001` on the `$TestMachine` template to free access and redeploy —
   `graccesscli object uda ... --security MxSecurityFreeAccess` against `$TestMachine`,
   then `instance deploy TestMachine_001` — and re-run the second attempt's ack probe
   unchanged. If `AlarmAckByName` then moves `STATE` to `ACK_ALM`, the classification is
   confirmed as the blocker and the GUID question is answered in the same run. Restore the
   classification to `MxSecurityOperate` afterwards, since the secured-write fixtures in
   `WorkerLiveMxAccessSmokeTests` depend on the alarm UDAs being secured.

### Rig state left behind

The third attempt changed nothing. `TestMachine_001.TestAlarm001` and its `.Acked` subtag
both read `false` at the end of the session, matching the state the second attempt left.

The three `TestMachine_00{1,2,3}.TestAlarm001` UDAs are back to `false` and their
`.InAlarm` subtags read `false`, but each leaves a `UNACK_RTN` record in the wnwrap
snapshot, since nothing can acknowledge them away. `SnapshotActiveAlarms` counts only
`UNACK_ALM` and `ACK_ALM` as active, so these are inert for the gateway; they will clear on
the next `alarmmgr` restart.