docs: alarm acknowledgement writes AckMsg as the authenticated operator

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
This commit is contained in:
Joseph Doherty
2026-10-19 19:44:18 -04:00
parent 2858e42e80
commit 9409488008
12 changed files with 243 additions and 105 deletions
+1 -1
View File
@@ -126,7 +126,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
- **One worker per session** (invariant). Multi-subscriber event fan-out and reconnect-with-replay have shipped and are config-gated: `AllowMultipleEventSubscribers` (default `false`) enables fan-out up to `MaxEventSubscribersPerSession` (default `8`); `DetachGraceSeconds` (default `30`) retains a session after its last subscriber drops so clients can reconnect; `ReplayBufferCapacity` / `ReplayRetentionSeconds` control how much event history the replay ring keeps. Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect. The reconnect protocol is consumable end-to-end: a resuming `StreamEvents` (via `after_worker_sequence`) that predates the retained ring gets a `ReplayGap` sentinel, and all five official clients surface it as a typed signal. Orphan-worker reattach after a gateway restart is **deferred, not planned** — see `docs/DesignDecisions.md` (Session-Resilience Epic Scope, session-resilience epic Phase 5); the invariant on the next line stands. See `docs/DesignDecisions.md` and `docs/Sessions.md`.
- **Gateway restart does not reattach orphan workers.** The first version terminates orphaned workers on startup; do not design code paths that assume reattachment.
- **No Blazor UI component libraries.** Dashboard uses local Bootstrap CSS/JS only — do not introduce MudBlazor, Radzen, FluentUI, etc.
- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, and `AuthenticateUser` credentials must never reach logs. Value logging is opt-in and redacted.
- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, `AuthenticateUser` credentials, and `AcknowledgeAlarm` `operator_password` must never reach logs. Value logging is opt-in and redacted.
- **Generated code** under `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`, `clients/*/generated*/`, `clients/python/src/zb_mom_ww_mxgateway/generated/`, etc., is build output. Don't hand-edit. To regenerate, build the contracts project (`dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj`) or run the per-client generation step in that client's README.
- **Documentation style** (`StyleGuide.md`): PascalCase filenames, no marketing language, present tense, explain *why* not *what*.
- **Update docs in the same change as the source.** When public APIs, contracts, configuration, build steps, security behavior, event shapes, value conversion, status mapping, or lifecycle rules change, the affected docs (`gateway.md`, `docs/`, client READMEs, design docs) must change in the same commit. Don't leave stale prose describing old behavior.
+50 -52
View File
@@ -621,8 +621,9 @@ returned to normal but is unacknowledged — i.e., visible in the
"current alarms" list because operator hasn't acked it yet) and
`UNACK_ALM` (the alarm is currently active and unacknowledged).
The other states from `eAlmState` (`ACK_RTN`, `ACK_ALM`) would
appear when an ack is performed — `wwAlarmConsumerClass.AlarmAckByGUID`
is the method to call.
appear when an ack is performed. The alarm-client ack methods
(`AlarmAckByGUID` and friends) do not produce them headless; an `AckMsg`
write does (`AlarmProbeFindings.md`).
### `GetStatistics` AV — unrelated quirk
@@ -735,25 +736,32 @@ for `SetXmlAlarmQuery`.
the canonical `\\<machine>\Galaxy!<area>` form into the XML's
NODE/PROVIDER/GROUP fields. Mangled or not, the call enables reads.
### 2. Two consumers required: read-side vs. ack-side
### 2. Two consumers required: read-side vs. ack-side (historical)
> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator;
> see `AlarmProbeFindings.md`. The worker no longer provisions an ack-only
> wnwrap consumer. The text below records why the split existed.
`SetXmlAlarmQuery` enables reads but **breaks `AlarmAckByName` on
the same consumer instance**. With SetXml applied, AlarmAckByName
returns -55 even with valid name+provider+group+operator. Without
SetXml, AlarmAckByName succeeds with rc=0.
The production consumer therefore provisions **two** wnwrap COM
The consumer at the time provisioned **two** wnwrap COM
instances:
- Primary consumer (`client`): runs full lifecycle including
`SetXmlAlarmQuery` for `GetXmlCurrentAlarms2` polls.
- Ack-only consumer (`ackClient`): runs Initialize → Register →
Subscribe via the v1-prefixed methods, **no SetXmlAlarmQuery**.
All `AcknowledgeByName` calls dispatch through this instance.
All by-name ack calls dispatched through this instance.
Both consumers subscribe to the same expression. Disposal cleans up
both via a shared `ReleaseConsumerCom` helper.
### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg
### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg (historical)
> Superseded 2026-10-19: the gateway no longer calls `AlarmAckByName`; see
> `AlarmProbeFindings.md`.
`wwAlarmConsumerClass` exposes two `AlarmAckByName` overloads:
- `IwwAlarmConsumer2` v2: 8 args (`name, provider, group, comment,
@@ -762,8 +770,8 @@ both via a shared `ReleaseConsumerCom` helper.
The v2 8-arg method returns -55 on this AVEVA build regardless of
operator-identity inputs — looks like a stub. The v1 6-arg method
works. Production `WnWrapAlarmConsumer.AcknowledgeByName` calls the
6-arg overload and discards the proto's `domain` + `full_name` fields.
works. `WnWrapAlarmConsumer.AcknowledgeByName` called the
6-arg overload and discarded the proto's `domain` + `full_name` fields.
**Correction, 2026-08-18** (`docs/AlarmProbeFindings.md`). Both halves
of the paragraph above are narrower than they read. The -55 tracks the
@@ -775,42 +783,34 @@ snapshot stay `UNACK_ALM`, `OPERATOR_NAME` stay empty, and the alarm
extension's own `.Acked` attribute stay `False` for 16 s after each
call. On this rig the wnwrap ack is accepted and then inert, so a zero
return code must not be read as an applied acknowledgement.
The proto contract keeps the 8 fields for forward compatibility if
AVEVA fixes the v2 method later.
### 4. `AlarmAckByGUID` is not implemented
### 4. `AlarmAckByGUID` is not implemented (historical)
> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator;
> see `AlarmProbeFindings.md`.
The v2 `AlarmAckByGUID(VBGUID, …)` throws `NotImplementedException`
(COM `E_NOTIMPL`) on `wwAlarmConsumerClass` against this AVEVA
build. The reference→GUID lookup that we initially planned to wire
through `AlarmAckByGUID` is therefore not viable on wnwrap; all acks
must go through `AlarmAckByName`.
through `AlarmAckByGUID` is therefore not viable on wnwrap.
The proto `AcknowledgeAlarmCommand` (GUID-based) and the worker's
`MxAccessCommandExecutor.ExecuteAcknowledgeAlarm` switch arm remain
in the codebase for the forward-compat shape, but the gateway-side
`WorkerAlarmRpcDispatcher.AcknowledgeAsync` now always routes through
`AcknowledgeAlarmByName` when the public RPC supplies a recognizable
`Provider!Group.Tag` reference.
#### Managed GUID ack experiment — did not work (live-tested 2026-10-18, removed 2026-10-19)
**Opt-in managed GUID ack — does not work on this rig (live-tested
2026-10-18).** Setting `MXGATEWAY_USE_MANAGED_ALARM_ACK=1` in the
gateway's environment (workers inherit it) makes the worker build an
`AaManagedAlarmAckConsumer`, which drives
`aaAlarmManagedClient.AlarmClient` by reflection (its members are
non-public) and is meant to take GUID-keyed acks in place of the
wnwrap method. The live test shows it cannot do that as built, so leave
the flag off.
An opt-in experiment drove `aaAlarmManagedClient.AlarmClient` by reflection
(its members are non-public) to take GUID-keyed acks in place of the wnwrap
method. The live test showed it could not work, and the experiment and its
enabling flag were removed once the `AckMsg` write replaced it. The findings
are kept because they explain why the alarm-client route was abandoned.
The test raised `TestMachine_001.TestAlarm001` with a secured write,
read its GUID from the wnwrap XML snapshot (the public API exposes no
alarm GUIDs), and acknowledged it through the `AcknowledgeAlarm` RPC
with the flag on:
with the experiment enabled:
| Step | Result |
|---|---|
| RPC ack by GUID, flag off (control) | `NotImplementedException`; `.Acked` stays `False` |
| RPC ack by GUID, flag on | Same `NotImplementedException`; `.Acked` `False`, `.AckMsg` empty |
| RPC ack by GUID, experiment off (control) | `NotImplementedException`; `.Acked` stays `False` |
| RPC ack by GUID, experiment on | Same `NotImplementedException`; `.Acked` `False`, `.AckMsg` empty |
| `AlarmClient.RegisterConsumer` in a process with a registered wnwrap consumer | `-65` — and wnwrap gets `-65` if the managed client registered first; a real `hWnd` does not help |
| Managed client alone in its own process, records loaded | `AlarmAckByGUID` → `-1`, `SelectByGUID` → `-1` for the wnwrap GUID in both byte orders, `AlarmAckSelected` → `-57` |
@@ -820,7 +820,7 @@ Three separate problems follow:
a wnwrap reader, so the managed `RegisterConsumer` fails with `-65`.
The dispatcher is built to drop back to the wnwrap ack path on a
lifecycle failure (so alarm reads keep working), and it does so
silently — which is why the flag-on RPC is indistinguishable from the
silently — which is why the experiment-on RPC was indistinguishable from the
control.
2. **The managed client does not recognise the wnwrap GUID.** `-1` from
`SelectByGUID` is "not found", not a security refusal, even with the
@@ -833,14 +833,8 @@ Three separate problems follow:
strings, so the domain and full name the managed method carries
would not arrive through the public RPC even if 1 and 2 were solved.
A working managed ack would need the client hosted out of process, a way
to resolve the alarm to the identifier the managed client uses, and the
identity fields plumbed through the gateway. Routing rules the worker
already applies stay correct and would carry over: subtag-mode acks
(synthetic GUIDs) stay on the subtag consumer, and by-name acks stay on
wnwrap because `AlarmClient` has no by-name ack. The skip-gated
`AlarmManagedGuidAckLiveSmokeTests` exercises the in-process
arrangement and hits problem 1.
The route was dropped in favour of the authenticated `AckMsg` write
(`AlarmProbeFindings.md`, "Fourth attempt").
### 5. STA / threading — resolved
@@ -996,20 +990,24 @@ AVEVA-internal GUID.
### Acknowledge in subtag mode
`AlarmDispatcher` routes ack calls by active provider mode:
Both provider modes acknowledge through the worker's `AlarmAckWriter`, which
calls `AuthenticateUser` as the requesting operator and writes the comment to
the ack item over MXAccess on its own `LMXProxyServer`:
- **Alarm-manager mode:** `AlarmAckByName` on `wwAlarmConsumerClass` (unchanged).
- **Subtag mode:** `SubtagAlarmConsumer.AcknowledgeByName` resolves the
watch-list entry's `ack_comment_subtag` and issues a `Write(comment)` on
the STA via `LmxSubtagAlarmSource`. Writing the `AckMsg` subtag performs
the acknowledge in AVEVA (`AckMsg` is the confirmed `AlarmExtension` ack-comment
write target).
- **Alarm-manager mode:** the item is `<TagName>.AckMsg` for Galaxy-provider
alarms. An alarm from another provider is not ackable.
- **Subtag mode:** `SubtagAlarmConsumer` resolves the watch-list entry's
`ack_comment_subtag` and the writer writes that item verbatim. Writing the
`AckMsg` subtag performs the acknowledge in AVEVA (`AckMsg` is the confirmed
`AlarmExtension` ack-comment write target).
If the alarm has no writable ack-comment subtag (`AckComment` config key is
empty, or the entry's `ack_comment_subtag` field is empty), the ack call
returns a failure code that the gateway surfaces as `FailedPrecondition`.
`AcknowledgeByGuid` maps the synthetic GUID back to its reference via an
internal dictionary, then calls the same write path.
Because the write is authenticated, the Historian records the operator's name,
domain and full name, and the writer's MXAccess client name
(`MxGateway:Alarms:AckClientName`) as OperatorNode. If the alarm has no ack
target (`AckComment` config key is empty, the entry's `ack_comment_subtag` is
empty, or the reference or synthetic GUID is unknown), the worker reports it
as not found or not ackable and the gateway surfaces `InvalidRequest` with
`hresult` -1.
`SubtagAlarmConsumer.Subscribe` advises the ack-comment subtag alongside the
observed ones (active/acked/priority). This is required: MXAccess rejects a
@@ -1027,7 +1025,7 @@ The subtag path was validated against live MXAccess on the dev rig
`…​.Priority` → `500` (Int32), `…​.AckMsg` → string — confirming the field
names **and** the runtime reference shape `<Object>.<AlarmAttr>.<field>`
with **no** intermediate alarm-condition segment.
- `AcknowledgeByName` (AckMsg write) returned `0` once the ack-comment subtag
- The `AckMsg` write returned `0` once the ack-comment subtag
was advised — confirming the ack-by-comment-write mechanism end to end.
### Fidelity limitations
+71
View File
@@ -25,6 +25,7 @@ the blocker rather than rediscovering it.
| GUID stable across polls and `ALM → RTN` | Answered — yes (2026-05-01 capture in `AlarmClientDiscovery.md`, reconfirmed 2026-08-18) |
| GUID stable across clear-then-re-raise | Answered 2026-08-18 — **no**, a re-raise mints a new GUID |
| GUID stable across `UNACK → ACK` | **Open** — the rig cannot be driven into an acknowledged state; 2026-08-18 (third attempt) identifies the reason as the test attribute's `MxSecurityOperate` classification, which no non-interactive ack surface on the rig can satisfy |
| Acknowledge an alarm headless | Answered 2026-10-19 — **an `AckMsg` write as the authenticated operator**; every alarm-client ack is inert (see "Fourth attempt") |
| `COUNT` = total active vs records-in-reply under a capped fetch | Answered 2026-08-18 — **records in the reply** |
The 2026-08-17 run below is kept because it is the record of the wrong-verb blocker. The
@@ -420,3 +421,73 @@ The second attempt's inventory, unchanged since: the three
snapshot, since nothing can acknowledge them away. `SnapshotActiveAlarms` counts only
`UNACK_ALM` and `ACK_ALM` as active, so these are inert for the gateway; they will clear on
the next `alarmmgr` restart.
## Fourth attempt (2026-10-19): authenticated AckMsg write
The third attempt left the acknowledge leg blocked. This run found a route that
acknowledges and identifies the operator, and ruled out the alarm-client routes.
It drove `TestMachine_001.TestAlarm001`, raised with
`mxa write TestMachine_001.TestAlarm001 true --type bool --secured -u Administrator`.
### Writing `AckMsg` acknowledges
Writing the comment to `TestMachine_001.TestAlarm001.AckMsg` over MXAccess
acknowledges the alarm: `.Acked` becomes `True` and the wnwrap record moves to
`STATE=ACK_ALM` with an `ACK_COMMENT`. `AckMsg` is `MxSecurityFreeAccess` even
though the alarm UDA itself is `MxSecurityOperate`, so no Galaxy change is
needed.
What the Historian records, read from `Runtime.dbo.v_AlarmEventHistory2`:
| Write | Operator | DomainName | UserFullName | OperatorNode | Description |
|---|---|---|---|---|---|
| Unauthenticated `Write` | `NULL` | `NULL` | `NULL` | `NULL` | the comment |
| `AuthenticateUser` + `Write(value, userId)` | `Administrator` | `ArchestrA` | `Administrator` | `mxa` (the MXAccess `Register()` client name) | the comment |
The wnwrap XML also gains `ACK_OPERATOR_NAME`, `ACK_DOMAIN_NAME`,
`ACK_OPERATOR_NODE` and `ACK_COMMENT` on the acknowledged record; the worker's
snapshot parser reads them for Acknowledge transitions. An unauthenticated write
therefore acknowledges but leaves no operator in the Historian, which is why the
gateway requires `operator_user` and authenticates it.
Another actor's `AckMsg` writes on the same day (authenticated as `dohertj2`)
logged `Operator=dohertj2`, `Domain="."` and that machine's name as the node,
which matches the client-name rule. By this date unauthenticated writes to
`MxSecurityOperate` attributes were also accepted Galaxy-wide on the rig, where
August refused them with 1008.
### Alarm-client acks are inert in every variant
All of these returned `rc=0` or `-1` and changed neither the snapshot, `.Acked`
nor the Historian:
- wnwrap `AlarmAckByName` and `AlarmAckByGUID` (`E_NOTIMPL`) on the ack-only and
reader consumers.
- wnwrap `IwwAlarmConsumer2` on a sole consumer with `SetXmlAlarmQuery` applied
(list populated, `GetAlarm(0)` returns the row): `AlarmAck` with ids
0/1/2/3/-1/65536, and `Select(0)` + `AlarmAckSelected`, with operators
`TestOperator`, `Administrator` and `dohertj2`. `SelectByGUID` is `E_NOTIMPL`.
- The same calls with `TestAlarm001` temporarily reclassified to
`MxSecurityFreeAccess` (template and instance, restored afterwards), so the
security classification is not the cause.
- Managed `aaAlarmManagedClient`: `AlarmAckByGUID` and `SelectByGUID` return
`-1` (the client's list never fills headless, even with a real `hWnd` and a
message pump); `AckSelectedTagAlarms_Ex` returns `0` even for a
`NOSUCHPROVIDER` provider and writes no Historian row. The managed and wnwrap
consumers cannot share a process (`RegisterConsumer` returns `-65`).
- A 20 s continuous message pump after `AlarmAck`, and registering as `InTouch`,
`WindowViewer` or `Wonderware InTouch`.
Static reverse engineering matched: the five-string COM acks forward domain and
full name with an empty password, and InTouch's `AlarmViewer.ocx` also sends an
empty password. The user-request call is synchronous request-to-return-code with
no completion channel, so the drop happens provider-side. Hooking the user
request in a live WindowViewer that acknowledges successfully and diffing it
against the headless probe is the one definitive step not taken.
### Consequence
The gateway acknowledges by authenticating the operator and writing `AckMsg`
(`docs/DesignDecisions.md`, "Alarm acknowledgement writes AckMsg as the
authenticated operator"). The managed ack consumer and the second wnwrap ack
consumer were removed.
+1 -1
View File
@@ -125,7 +125,7 @@ public string ResolveRequiredScope(object request)
}
```
The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*` — and `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce.
The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*` — and stays `invoke:write` rather than `invoke:secure` because the request carries the operator's own Galaxy credential, which the worker authenticates before the `AckMsg` write, so the credential, not the API-key scope, authorizes the identity the Historian records — and `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce.
`MxCommandRequest` is special because it multiplexes many MxAccess operations through a single RPC. The resolver inspects the embedded `MxCommandKind` so each operation gets its own scope:
+11
View File
@@ -49,6 +49,17 @@ worker prepares the replacement on its STA before swapping it into the existing
failover consumer and returns a normal `MxCommandReply`; callers must treat only
`protocol_status = OK` as acknowledgement of the new list.
`AcknowledgeAlarmRequest` carries the operator's identity: `operator_user`
(required), `operator_password` (field 6) and `operator_domain` (field 7). The
gateway copies them onto `AcknowledgeAlarmCommand` (`operator_password` = 7) and
`AcknowledgeAlarmByNameCommand` (`operator_password` = 9); the worker
authenticates the operator and writes `AckMsg` as that user. The fields are
additive, so `GatewayProtocolVersion` stays `3`. `operator_password` is a
credential and follows the same never-log rule as
`AuthenticateUserCommand.verify_user_password`: it must not reach logs, metric
labels, `DiagnosticMessage`, `ProtocolStatus.Message`, or exception text. A
not-found or not-ackable alarm is reported with `hresult` -1.
The command model also includes bulk write/read command kinds:
`WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and
`ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway`
+42 -8
View File
@@ -703,14 +703,15 @@ latch in subtag mode.
### Acknowledge via ack-comment write
In subtag mode, `AcknowledgeAlarm` writes the operator comment to the alarm
attribute's ack-comment subtag (`Fallback:Subtags:AckComment`). The write
performs the native ack in AVEVA. This differs from alarmmgr mode, where
`AlarmAckByName` on `wwAlarmConsumerClass` is called directly. The `AckComment`
subtag name is empty by default; configuring it is required for ack to work in
subtag mode. The exact AVEVA subtag names are not hard-coded — the `Subtags`
config block exists precisely so names are not guessed without validation
against the live MXAccess attribute set.
Both provider modes acknowledge by writing the operator comment to `AckMsg` as
the authenticated operator (see "Alarm acknowledgement writes AckMsg as the
authenticated operator" below). In alarm-manager mode the target is
`<TagName>.AckMsg`; in subtag mode it is the watch-list entry's ack-comment
subtag (`Fallback:Subtags:AckComment`, default `AckMsg`). The write performs
the ack in AVEVA. An entry with an empty ack-comment subtag is not ackable and
the RPC reports `InvalidRequest`. The exact AVEVA subtag names are not
hard-coded — the `Subtags` config block exists so names are not guessed
without validation against the live MXAccess attribute set.
### Related documentation
@@ -784,6 +785,39 @@ The following items were previously listed here and have since shipped:
`MxGateway:Sessions:MaxEventSubscribersPerSession`.
See [Event Subscribers](#event-subscribers) above and [Sessions](./Sessions.md).
## Alarm acknowledgement writes AckMsg as the authenticated operator
**Decision (2026-10-19).** `AcknowledgeAlarm` authenticates the requesting
operator with `AuthenticateUser` (`operator_user`, `operator_password`,
optional `operator_domain`) and writes the comment to `<TagName>.AckMsg` over
MXAccess as that user, on a dedicated `LMXProxyServer` registered as
`MxGateway:Alarms:AckClientName`. Both provider modes use this path.
**Why.** Live tests (2026-10-18/19) showed every alarm-client ack is inert on a
headless consumer: wnwrap `AlarmAckByName`/`AlarmAckByGUID`/`AlarmAck`/
`AlarmAckSelected`, and the managed `aaAlarmManagedClient` GUID ack, return
rc=0 or -1 and change nothing, regardless of security classification,
operator name or consumer name. An MXAccess write to `AckMsg` does acknowledge
the alarm, and when the write is made under an authenticated user the AVEVA
runtime historizes that identity (`Operator`, `DomainName`, `UserFullName`),
with the MXAccess client name as `OperatorNode`. The requirement is that the
real operator appears in the Historian, as with InTouch, so a shared gateway
service account was rejected. Operators have Galaxy accounts, so the
per-request credential is workable. See
[Alarm Probe Findings](./AlarmProbeFindings.md).
**Consequences.**
- `operator_user` is required; `operator_password` may be empty.
- `operator_password` is a credential. It is forwarded to the worker over the
pipe but never logged: `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are
credential-bearing commands and any property named like `password` is
redacted.
- `Applied` and `Unconfirmed` (write sent, no `OnWriteComplete` in the bounded
wait) both report `OK`; clients that need certainty watch for the transition.
- The managed ack consumer, the WNAL native ack consumer, the second wnwrap
ack-only consumer and the opt-in managed-ack environment flag are removed.
## Related Documentation
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
+5 -3
View File
@@ -4,7 +4,7 @@ The diagnostics subsystem provides structured logging, credential redaction, and
## Goals
The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly.
The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, `AcknowledgeAlarm` (its `operator_password`), or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly.
A second goal is parity-test diagnosability. Because MXAccess sessions, workers, correlation ids, and command methods are the units of comparison, every log entry produced inside a request scope must carry those identifiers without each call site having to format them.
@@ -70,7 +70,9 @@ private static readonly HashSet<string> SensitiveCommandMethods = new(StringComp
{
"AuthenticateUser",
"WriteSecured",
"WriteSecured2"
"WriteSecured2",
"AcknowledgeAlarm",
"AcknowledgeAlarmByName"
};
public static bool IsCredentialBearingCommand(string? commandMethod)
@@ -80,7 +82,7 @@ public static bool IsCredentialBearingCommand(string? commandMethod)
}
```
The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below.
The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below. `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are in the set because their commands carry the operator's `operator_password`. Independently of the method set, the name-based seam redacts any property whose name contains `password`, so a credential field added to a payload later is redacted without a code change.
### API key redaction
+9 -8
View File
@@ -442,17 +442,18 @@ behavior.
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor re-resolves and, on material change, hot-swaps the subtag fallback watch-list, then reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot. The acknowledged watch-list fingerprint advances only after an OK worker reply. Floored at 5 seconds. |
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. |
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. |
| `MxGateway:Alarms:AckClientName` | `MxGateway` | MXAccess client name the alarm worker's ack writer registers with; the Historian records it as each acknowledgement's OperatorNode. 1-64 printable characters. Conveyed via `MXGATEWAY_ALARM_ACK_CLIENT_NAME`. |
The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
`StreamAlarms` are session-less RPCs served by the monitor.
The worker also reads `MXGATEWAY_USE_MANAGED_ALARM_ACK` (inherited from the
gateway process). Setting it to `1` is meant to route GUID-keyed acknowledges
through AVEVA's managed alarm client, but a live test (2026-10-18) showed the
path cannot work as built: the managed client fails to register alongside the
worker's wnwrap reader and the worker silently falls back to wnwrap. Leave it
unset. It has no `MxGateway:*` option. See
[AlarmClientDiscovery](./AlarmClientDiscovery.md) ("Opt-in managed GUID ack").
`AckClientName` is the MXAccess client name the alarm worker's ack writer
registers its dedicated `LMXProxyServer` with. The Historian records it as the
OperatorNode of each acknowledgement, so it identifies the gateway as the
acknowledging node. The gateway passes it to each worker through the
`MXGATEWAY_ALARM_ACK_CLIENT_NAME` environment variable. Startup validation
rejects an empty value, a value over 64 characters, or one containing control
characters.
### Alarm fallback options
@@ -471,7 +472,7 @@ native wnwrap alarm-manager provider and the subtag-monitoring fallback.
| `MxGateway:Alarms:Fallback:Discovery:ExcludeAttributes` | _(empty)_ | Attribute paths to remove from the merged watch-list (case-insensitive). The exclude is applied after the Repository-derived rows and the explicit `IncludeAttributes` entries are combined, so an exclude that matches an explicit include suppresses it too — excludes win. Ignored when `UseGalaxyRepository` is `false`. |
| `MxGateway:Alarms:Fallback:Subtags:Active` | `InAlarm` | Subtag name for the in-alarm boolean. Confirmed AVEVA `AlarmExtension` field name. |
| `MxGateway:Alarms:Fallback:Subtags:Acked` | `Acked` | Subtag name for the acknowledged boolean. Confirmed AVEVA `AlarmExtension` field name. |
| `MxGateway:Alarms:Fallback:Subtags:AckComment` | `AckMsg` | Subtag name for the acknowledgement comment write target. Writing this subtag performs the acknowledge in AVEVA. Confirmed AVEVA `AlarmExtension` field name. When empty, the ack-comment write path is disabled. |
| `MxGateway:Alarms:Fallback:Subtags:AckComment` | `AckMsg` | Subtag name for the acknowledgement comment write target. Writing this subtag, as the authenticated operator, performs the acknowledge in AVEVA. Confirmed AVEVA `AlarmExtension` field name. When empty, the alarm is not ackable and the RPC returns `InvalidRequest`. |
| `MxGateway:Alarms:Fallback:Subtags:Priority` | `Priority` | Subtag name for the alarm priority / severity value. Confirmed AVEVA `AlarmExtension` field name. |
Validation rules:
+18 -2
View File
@@ -96,9 +96,24 @@ Selected control commands have additional public behavior beyond their protobuf
### `AcknowledgeAlarm`
`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` and `operator_user` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
An `OK` response means the alarm provider accepted the acknowledgement, not that it applied it: the gateway forwards the ack and reports what the provider returned, in keeping with the parity rule. On galaxies whose alarmed attributes carry a non-free-access security classification the by-name ack is accepted and inert — the underlying `AlarmAckByName` conveys an operator name rather than an authenticated identity, observed on the probe rig with the mechanism inferred rather than confirmed — so a client that needs to confirm an acknowledgement should watch for the resulting transition on `StreamAlarms` instead of treating `OK` as proof. See [Alarm Probe Findings](./AlarmProbeFindings.md).
Request fields that identify the acknowledging operator:
- `operator_user` (required): the Galaxy account name. An empty value is rejected with `InvalidArgument` so the real operator is always recorded.
- `operator_password` (field 6): the operator's Galaxy password. May be empty because Galaxy accounts can have empty passwords. It is a credential and is never logged.
- `operator_domain` (field 7): optional. When set, the worker authenticates `domain\user`; when empty it authenticates `user`.
The worker calls `AuthenticateUser` as the operator, then writes `operator_comment` to `<TagName>.AckMsg` (subtag mode: the configured ack-comment subtag) on a dedicated MXAccess server registered as `MxGateway:Alarms:AckClientName`, and waits a bounded time for `OnWriteComplete`. The AVEVA runtime treats that write as the acknowledgement, and the Historian (`Runtime.dbo.v_AlarmEventHistory2`) records the operator name, domain and full name from the authenticated account, with the ack client name as `OperatorNode`. The alarm-client ack calls (`AlarmAckByName`, `AlarmAckByGUID`) are not used because a headless consumer's acks are accepted and dropped; see [Alarm Probe Findings](./AlarmProbeFindings.md).
| Worker outcome | Response status | `hresult` |
|---|---|---|
| Write applied, or sent but no `OnWriteComplete` within the wait (unconfirmed, with a diagnostic) | `OK` | `0` |
| Unknown alarm, or no ack target (non-Galaxy provider, no ack-comment subtag) | `INVALID_REQUEST` | `-1` |
| `AuthenticateUser` failed | `MXACCESS_FAILURE` | the `AuthenticateUser` HRESULT |
| Write rejected (`OnWriteComplete` non-OK) | `MXACCESS_FAILURE` | the status `detail` |
`reply.Statuses` carries the `OnWriteComplete` `MxStatusProxy` rows when they were observed. Because an unconfirmed write also returns `OK`, a client that must be certain should watch for the resulting transition on `StreamAlarms`. `AcknowledgeAlarm` stays under the `invoke:write` scope; see [Authorization](./Authorization.md).
### `StreamAlarms`
@@ -217,6 +232,7 @@ receive this event directly.
| `StreamEvents` | `session_id` must be non-empty. | `InvalidArgument` |
| `Invoke` | `session_id` non-empty, `command` present, `kind` not `Unspecified`, payload oneof must match `kind`. | `InvalidArgument` |
| `AcknowledgeAlarm` | `alarm_full_reference` must be non-empty. Validated inline in the handler, not by `MxAccessGrpcRequestValidator`. | `InvalidArgument` |
| `AcknowledgeAlarm` | `operator_user` must be non-empty. Validated inline in the handler. | `InvalidArgument` |
| `StreamAlarms` | No required fields — `alarm_filter_prefix` is optional. | — |
| `QueryActiveAlarms` | No required fields — `alarm_filter_prefix` is optional. Not routed through `MxAccessGrpcRequestValidator`; the handler only null-checks the request. | — |
+22 -18
View File
@@ -140,8 +140,9 @@ one gateway-owned worker session subscribes the configured AVEVA alarm
provider, caches the active-alarm set (reconciled periodically against the
worker's snapshot), and fans it out to every client through the session-less
`StreamAlarms` RPC — the stream opens with the current active-alarm snapshot,
then streams live transitions. `AcknowledgeAlarm` is session-less and routes
through the monitor. Clients never open a worker session to see alarms, and
then streams live transitions. `AcknowledgeAlarm` is session-less, routes
through the monitor, and carries the operator's credentials (see
"Acknowledge" below). Clients never open a worker session to see alarms, and
alarm monitoring is independent of client lifecycle; the monitor re-opens its
session if the worker faults. Gated by `MxGateway:Alarms:Enabled` — see
`docs/DesignDecisions.md` for why this reverses the v1 single-subscriber rule
@@ -221,20 +222,22 @@ forwards only events the worker emits and never synthesizes transitions
itself. The synthesis rules are documented in
`docs/AlarmClientDiscovery.md`.
**Acknowledge in subtag mode:** the ack-by-name path writes the operator
comment to the alarm attribute's ack-comment subtag. The write performs the
ack. If the attribute has no writable ack-comment subtag configured, the RPC
returns `FailedPrecondition`. In alarm-manager mode, `AlarmAckByName` is
used as before.
**Acceptance is not application:** the ack is forwarded to the provider, and a
successful return means the provider accepted the call, not that the
acknowledgement was applied. On galaxies whose alarmed attributes carry a
non-free-access security classification, `AlarmAckByName` returns `rc=0` and the
alarm stays `UNACK_ALM` — it carries an operator name, not an authenticated
identity. (Observed on the probe rig; the mechanism is inferred.) Confirm an ack
by the resulting transition, never by the return code. See
`docs/AlarmProbeFindings.md`.
**Acknowledge:** the gateway acknowledges an alarm by writing the operator's
comment to the alarm attribute's `AckMsg` (subtag mode: the configured
ack-comment subtag) over MXAccess, after `AuthenticateUser` as the requesting
operator (`operator_user`, `operator_password`, optional `operator_domain`).
The AVEVA runtime treats that write as the acknowledgement and historizes it
with the authenticated account's name, domain and full name; the Historian's
OperatorNode is the ack writer's MXAccess client name
(`MxGateway:Alarms:AckClientName`). The worker waits a bounded time for
`OnWriteComplete`: an OK status is reported as success, a non-OK status as
`MxaccessFailure` with the status detail, and no completion as success with a
diagnostic noting the write was sent but unconfirmed. An unknown alarm or one
with no ack target (non-Galaxy provider, no ack-comment subtag) is reported as
`InvalidRequest` with `hresult` -1. Alarm-client ack calls
(`AlarmAckByName`/`AlarmAckByGUID`, the managed `AlarmClient`, WNAL) are not
used: live tests showed a headless consumer's acks are accepted and silently
dropped (see `docs/AlarmProbeFindings.md`).
**Degraded state visibility:** every subtag-mode transition carries
`degraded = true` and `source_provider = ALARM_PROVIDER_MODE_SUBTAG` on the
@@ -1028,8 +1031,9 @@ Internal worker IPC:
- gateway validates worker executable path and version,
- no secrets in command line when avoidable.
Credential-sensitive commands such as `AuthenticateUser` and `WriteSecured`
must not log passwords or raw credential values.
Credential-sensitive commands such as `AuthenticateUser`, `WriteSecured` and
`AcknowledgeAlarm` (its `operator_password`) must not log passwords or raw
credential values.
## Observability
@@ -88,7 +88,8 @@
"DefaultArea": "",
"ReconcileIntervalSeconds": 30,
"PollIntervalMilliseconds": 500,
"MaxAlarmsPerFetch": 1024
"MaxAlarmsPerFetch": 1024,
"AckClientName": "MxGateway"
}
}
}
@@ -202,14 +202,14 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
// Use the IwwAlarmConsumer (v1) prefix-named methods for the
// lifecycle. Empirically (live dev-rig 2026-05-01, reconfirmed
// 2026-08-18) this is the only path that lets AlarmAckByName
// return rc=0 afterwards. The v2 Initialize/Register/Subscribe
// methods on the class succeed (return 0) but acks against that
// 2026-08-18) the v1 path is the one that leaves the consumer in
// a usable state: the v2 Initialize/Register/Subscribe methods on
// the class succeed (return 0) but later calls against that
// consumer state return -55. The v1 prefix path is what
// WIN-911-style code uses against the same wnwrap library.
// Note rc=0 means the call was
// accepted, not that an acknowledgement was applied — see
// docs/AlarmProbeFindings.md. Acks now go through AlarmAckWriter.
// WIN-911-style code uses against the same wnwrap library. This
// consumer only reads the alarm snapshot; acknowledgements go
// through AlarmAckWriter, not through wnwrap (see
// docs/AlarmProbeFindings.md).
int init = com.IwwAlarmConsumer_InitializeConsumer(DefaultApplicationName);
if (init != 0)
{
@@ -615,10 +615,10 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
/// place under its original GUID and the re-raise replaced it with a
/// freshly minted one. That is the intended reading — the old
/// instance ended and a new one began — so the diff emits a Clear
/// plus a Raise, which is what happened. The acknowledge leg remains
/// assumed rather than observed: on the dev rig every ack surface is
/// inert (<c>AlarmAckByName</c> returns 0 and nothing moves), so no
/// record can be driven into an acknowledged state — see
/// plus a Raise, which is what happened. Every alarm-client ack surface
/// is inert on the dev rig (it returns 0 and nothing moves), so
/// acknowledgements go through the AckMsg writer; an AckMsg write
/// moves the record to <c>ACK_ALM</c> (observed 2026-10-19) — see
/// <c>docs/AlarmProbeFindings.md</c>.
/// </para>
/// </remarks>