diff --git a/CLAUDE.md b/CLAUDE.md index d3c2dfb..f376839 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -126,7 +126,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 - **One worker per session** (invariant). Multi-subscriber event fan-out and reconnect-with-replay have shipped and are config-gated: `AllowMultipleEventSubscribers` (default `false`) enables fan-out up to `MaxEventSubscribersPerSession` (default `8`); `DetachGraceSeconds` (default `30`) retains a session after its last subscriber drops so clients can reconnect; `ReplayBufferCapacity` / `ReplayRetentionSeconds` control how much event history the replay ring keeps. Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect. The reconnect protocol is consumable end-to-end: a resuming `StreamEvents` (via `after_worker_sequence`) that predates the retained ring gets a `ReplayGap` sentinel, and all five official clients surface it as a typed signal. Orphan-worker reattach after a gateway restart is **deferred, not planned** — see `docs/DesignDecisions.md` (Session-Resilience Epic Scope, session-resilience epic Phase 5); the invariant on the next line stands. See `docs/DesignDecisions.md` and `docs/Sessions.md`. - **Gateway restart does not reattach orphan workers.** The first version terminates orphaned workers on startup; do not design code paths that assume reattachment. - **No Blazor UI component libraries.** Dashboard uses local Bootstrap CSS/JS only — do not introduce MudBlazor, Radzen, FluentUI, etc. -- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, and `AuthenticateUser` credentials must never reach logs. Value logging is opt-in and redacted. +- **Don't log secrets or full tag values by default.** API keys, passwords, `WriteSecured` payloads, `AuthenticateUser` credentials, and `AcknowledgeAlarm` `operator_password` must never reach logs. Value logging is opt-in and redacted. - **Generated code** under `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`, `clients/*/generated*/`, `clients/python/src/zb_mom_ww_mxgateway/generated/`, etc., is build output. Don't hand-edit. To regenerate, build the contracts project (`dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj`) or run the per-client generation step in that client's README. - **Documentation style** (`StyleGuide.md`): PascalCase filenames, no marketing language, present tense, explain *why* not *what*. - **Update docs in the same change as the source.** When public APIs, contracts, configuration, build steps, security behavior, event shapes, value conversion, status mapping, or lifecycle rules change, the affected docs (`gateway.md`, `docs/`, client READMEs, design docs) must change in the same commit. Don't leave stale prose describing old behavior. diff --git a/docs/AlarmClientDiscovery.md b/docs/AlarmClientDiscovery.md index 1c040ce..51090de 100644 --- a/docs/AlarmClientDiscovery.md +++ b/docs/AlarmClientDiscovery.md @@ -621,8 +621,9 @@ returned to normal but is unacknowledged — i.e., visible in the "current alarms" list because operator hasn't acked it yet) and `UNACK_ALM` (the alarm is currently active and unacknowledged). The other states from `eAlmState` (`ACK_RTN`, `ACK_ALM`) would -appear when an ack is performed — `wwAlarmConsumerClass.AlarmAckByGUID` -is the method to call. +appear when an ack is performed. The alarm-client ack methods +(`AlarmAckByGUID` and friends) do not produce them headless; an `AckMsg` +write does (`AlarmProbeFindings.md`). ### `GetStatistics` AV — unrelated quirk @@ -735,25 +736,32 @@ for `SetXmlAlarmQuery`. the canonical `\\\Galaxy!` form into the XML's NODE/PROVIDER/GROUP fields. Mangled or not, the call enables reads. -### 2. Two consumers required: read-side vs. ack-side +### 2. Two consumers required: read-side vs. ack-side (historical) + +> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator; +> see `AlarmProbeFindings.md`. The worker no longer provisions an ack-only +> wnwrap consumer. The text below records why the split existed. `SetXmlAlarmQuery` enables reads but **breaks `AlarmAckByName` on the same consumer instance**. With SetXml applied, AlarmAckByName returns -55 even with valid name+provider+group+operator. Without SetXml, AlarmAckByName succeeds with rc=0. -The production consumer therefore provisions **two** wnwrap COM +The consumer at the time provisioned **two** wnwrap COM instances: - Primary consumer (`client`): runs full lifecycle including `SetXmlAlarmQuery` for `GetXmlCurrentAlarms2` polls. - Ack-only consumer (`ackClient`): runs Initialize → Register → Subscribe via the v1-prefixed methods, **no SetXmlAlarmQuery**. - All `AcknowledgeByName` calls dispatch through this instance. + All by-name ack calls dispatched through this instance. Both consumers subscribe to the same expression. Disposal cleans up both via a shared `ReleaseConsumerCom` helper. -### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg +### 3. `AlarmAckByName` v2 8-arg vs. v1 6-arg (historical) + +> Superseded 2026-10-19: the gateway no longer calls `AlarmAckByName`; see +> `AlarmProbeFindings.md`. `wwAlarmConsumerClass` exposes two `AlarmAckByName` overloads: - `IwwAlarmConsumer2` v2: 8 args (`name, provider, group, comment, @@ -762,8 +770,8 @@ both via a shared `ReleaseConsumerCom` helper. The v2 8-arg method returns -55 on this AVEVA build regardless of operator-identity inputs — looks like a stub. The v1 6-arg method -works. Production `WnWrapAlarmConsumer.AcknowledgeByName` calls the -6-arg overload and discards the proto's `domain` + `full_name` fields. +works. `WnWrapAlarmConsumer.AcknowledgeByName` called the +6-arg overload and discarded the proto's `domain` + `full_name` fields. **Correction, 2026-08-18** (`docs/AlarmProbeFindings.md`). Both halves of the paragraph above are narrower than they read. The -55 tracks the @@ -775,42 +783,34 @@ snapshot stay `UNACK_ALM`, `OPERATOR_NAME` stay empty, and the alarm extension's own `.Acked` attribute stay `False` for 16 s after each call. On this rig the wnwrap ack is accepted and then inert, so a zero return code must not be read as an applied acknowledgement. -The proto contract keeps the 8 fields for forward compatibility if -AVEVA fixes the v2 method later. -### 4. `AlarmAckByGUID` is not implemented +### 4. `AlarmAckByGUID` is not implemented (historical) + +> Superseded 2026-10-19: acks now write `AckMsg` as the authenticated operator; +> see `AlarmProbeFindings.md`. The v2 `AlarmAckByGUID(VBGUID, …)` throws `NotImplementedException` (COM `E_NOTIMPL`) on `wwAlarmConsumerClass` against this AVEVA build. The reference→GUID lookup that we initially planned to wire -through `AlarmAckByGUID` is therefore not viable on wnwrap; all acks -must go through `AlarmAckByName`. +through `AlarmAckByGUID` is therefore not viable on wnwrap. -The proto `AcknowledgeAlarmCommand` (GUID-based) and the worker's -`MxAccessCommandExecutor.ExecuteAcknowledgeAlarm` switch arm remain -in the codebase for the forward-compat shape, but the gateway-side -`WorkerAlarmRpcDispatcher.AcknowledgeAsync` now always routes through -`AcknowledgeAlarmByName` when the public RPC supplies a recognizable -`Provider!Group.Tag` reference. +#### Managed GUID ack experiment — did not work (live-tested 2026-10-18, removed 2026-10-19) -**Opt-in managed GUID ack — does not work on this rig (live-tested -2026-10-18).** Setting `MXGATEWAY_USE_MANAGED_ALARM_ACK=1` in the -gateway's environment (workers inherit it) makes the worker build an -`AaManagedAlarmAckConsumer`, which drives -`aaAlarmManagedClient.AlarmClient` by reflection (its members are -non-public) and is meant to take GUID-keyed acks in place of the -wnwrap method. The live test shows it cannot do that as built, so leave -the flag off. +An opt-in experiment drove `aaAlarmManagedClient.AlarmClient` by reflection +(its members are non-public) to take GUID-keyed acks in place of the wnwrap +method. The live test showed it could not work, and the experiment and its +enabling flag were removed once the `AckMsg` write replaced it. The findings +are kept because they explain why the alarm-client route was abandoned. The test raised `TestMachine_001.TestAlarm001` with a secured write, read its GUID from the wnwrap XML snapshot (the public API exposes no alarm GUIDs), and acknowledged it through the `AcknowledgeAlarm` RPC -with the flag on: +with the experiment enabled: | Step | Result | |---|---| -| RPC ack by GUID, flag off (control) | `NotImplementedException`; `.Acked` stays `False` | -| RPC ack by GUID, flag on | Same `NotImplementedException`; `.Acked` `False`, `.AckMsg` empty | +| RPC ack by GUID, experiment off (control) | `NotImplementedException`; `.Acked` stays `False` | +| RPC ack by GUID, experiment on | Same `NotImplementedException`; `.Acked` `False`, `.AckMsg` empty | | `AlarmClient.RegisterConsumer` in a process with a registered wnwrap consumer | `-65` — and wnwrap gets `-65` if the managed client registered first; a real `hWnd` does not help | | Managed client alone in its own process, records loaded | `AlarmAckByGUID` → `-1`, `SelectByGUID` → `-1` for the wnwrap GUID in both byte orders, `AlarmAckSelected` → `-57` | @@ -820,7 +820,7 @@ Three separate problems follow: a wnwrap reader, so the managed `RegisterConsumer` fails with `-65`. The dispatcher is built to drop back to the wnwrap ack path on a lifecycle failure (so alarm reads keep working), and it does so - silently — which is why the flag-on RPC is indistinguishable from the + silently — which is why the experiment-on RPC was indistinguishable from the control. 2. **The managed client does not recognise the wnwrap GUID.** `-1` from `SelectByGUID` is "not found", not a security refusal, even with the @@ -833,14 +833,8 @@ Three separate problems follow: strings, so the domain and full name the managed method carries would not arrive through the public RPC even if 1 and 2 were solved. -A working managed ack would need the client hosted out of process, a way -to resolve the alarm to the identifier the managed client uses, and the -identity fields plumbed through the gateway. Routing rules the worker -already applies stay correct and would carry over: subtag-mode acks -(synthetic GUIDs) stay on the subtag consumer, and by-name acks stay on -wnwrap because `AlarmClient` has no by-name ack. The skip-gated -`AlarmManagedGuidAckLiveSmokeTests` exercises the in-process -arrangement and hits problem 1. +The route was dropped in favour of the authenticated `AckMsg` write +(`AlarmProbeFindings.md`, "Fourth attempt"). ### 5. STA / threading — resolved @@ -996,20 +990,24 @@ AVEVA-internal GUID. ### Acknowledge in subtag mode -`AlarmDispatcher` routes ack calls by active provider mode: +Both provider modes acknowledge through the worker's `AlarmAckWriter`, which +calls `AuthenticateUser` as the requesting operator and writes the comment to +the ack item over MXAccess on its own `LMXProxyServer`: -- **Alarm-manager mode:** `AlarmAckByName` on `wwAlarmConsumerClass` (unchanged). -- **Subtag mode:** `SubtagAlarmConsumer.AcknowledgeByName` resolves the - watch-list entry's `ack_comment_subtag` and issues a `Write(comment)` on - the STA via `LmxSubtagAlarmSource`. Writing the `AckMsg` subtag performs - the acknowledge in AVEVA (`AckMsg` is the confirmed `AlarmExtension` ack-comment - write target). +- **Alarm-manager mode:** the item is `.AckMsg` for Galaxy-provider + alarms. An alarm from another provider is not ackable. +- **Subtag mode:** `SubtagAlarmConsumer` resolves the watch-list entry's + `ack_comment_subtag` and the writer writes that item verbatim. Writing the + `AckMsg` subtag performs the acknowledge in AVEVA (`AckMsg` is the confirmed + `AlarmExtension` ack-comment write target). -If the alarm has no writable ack-comment subtag (`AckComment` config key is -empty, or the entry's `ack_comment_subtag` field is empty), the ack call -returns a failure code that the gateway surfaces as `FailedPrecondition`. -`AcknowledgeByGuid` maps the synthetic GUID back to its reference via an -internal dictionary, then calls the same write path. +Because the write is authenticated, the Historian records the operator's name, +domain and full name, and the writer's MXAccess client name +(`MxGateway:Alarms:AckClientName`) as OperatorNode. If the alarm has no ack +target (`AckComment` config key is empty, the entry's `ack_comment_subtag` is +empty, or the reference or synthetic GUID is unknown), the worker reports it +as not found or not ackable and the gateway surfaces `InvalidRequest` with +`hresult` -1. `SubtagAlarmConsumer.Subscribe` advises the ack-comment subtag alongside the observed ones (active/acked/priority). This is required: MXAccess rejects a @@ -1027,7 +1025,7 @@ The subtag path was validated against live MXAccess on the dev rig `…​.Priority` → `500` (Int32), `…​.AckMsg` → string — confirming the field names **and** the runtime reference shape `..` with **no** intermediate alarm-condition segment. -- `AcknowledgeByName` (AckMsg write) returned `0` once the ack-comment subtag +- The `AckMsg` write returned `0` once the ack-comment subtag was advised — confirming the ack-by-comment-write mechanism end to end. ### Fidelity limitations diff --git a/docs/AlarmProbeFindings.md b/docs/AlarmProbeFindings.md index f47f7e8..6ef466d 100644 --- a/docs/AlarmProbeFindings.md +++ b/docs/AlarmProbeFindings.md @@ -25,6 +25,7 @@ the blocker rather than rediscovering it. | GUID stable across polls and `ALM → RTN` | Answered — yes (2026-05-01 capture in `AlarmClientDiscovery.md`, reconfirmed 2026-08-18) | | GUID stable across clear-then-re-raise | Answered 2026-08-18 — **no**, a re-raise mints a new GUID | | GUID stable across `UNACK → ACK` | **Open** — the rig cannot be driven into an acknowledged state; 2026-08-18 (third attempt) identifies the reason as the test attribute's `MxSecurityOperate` classification, which no non-interactive ack surface on the rig can satisfy | +| Acknowledge an alarm headless | Answered 2026-10-19 — **an `AckMsg` write as the authenticated operator**; every alarm-client ack is inert (see "Fourth attempt") | | `COUNT` = total active vs records-in-reply under a capped fetch | Answered 2026-08-18 — **records in the reply** | The 2026-08-17 run below is kept because it is the record of the wrong-verb blocker. The @@ -420,3 +421,73 @@ The second attempt's inventory, unchanged since: the three snapshot, since nothing can acknowledge them away. `SnapshotActiveAlarms` counts only `UNACK_ALM` and `ACK_ALM` as active, so these are inert for the gateway; they will clear on the next `alarmmgr` restart. + +## Fourth attempt (2026-10-19): authenticated AckMsg write + +The third attempt left the acknowledge leg blocked. This run found a route that +acknowledges and identifies the operator, and ruled out the alarm-client routes. +It drove `TestMachine_001.TestAlarm001`, raised with +`mxa write TestMachine_001.TestAlarm001 true --type bool --secured -u Administrator`. + +### Writing `AckMsg` acknowledges + +Writing the comment to `TestMachine_001.TestAlarm001.AckMsg` over MXAccess +acknowledges the alarm: `.Acked` becomes `True` and the wnwrap record moves to +`STATE=ACK_ALM` with an `ACK_COMMENT`. `AckMsg` is `MxSecurityFreeAccess` even +though the alarm UDA itself is `MxSecurityOperate`, so no Galaxy change is +needed. + +What the Historian records, read from `Runtime.dbo.v_AlarmEventHistory2`: + +| Write | Operator | DomainName | UserFullName | OperatorNode | Description | +|---|---|---|---|---|---| +| Unauthenticated `Write` | `NULL` | `NULL` | `NULL` | `NULL` | the comment | +| `AuthenticateUser` + `Write(value, userId)` | `Administrator` | `ArchestrA` | `Administrator` | `mxa` (the MXAccess `Register()` client name) | the comment | + +The wnwrap XML also gains `ACK_OPERATOR_NAME`, `ACK_DOMAIN_NAME`, +`ACK_OPERATOR_NODE` and `ACK_COMMENT` on the acknowledged record; the worker's +snapshot parser reads them for Acknowledge transitions. An unauthenticated write +therefore acknowledges but leaves no operator in the Historian, which is why the +gateway requires `operator_user` and authenticates it. + +Another actor's `AckMsg` writes on the same day (authenticated as `dohertj2`) +logged `Operator=dohertj2`, `Domain="."` and that machine's name as the node, +which matches the client-name rule. By this date unauthenticated writes to +`MxSecurityOperate` attributes were also accepted Galaxy-wide on the rig, where +August refused them with 1008. + +### Alarm-client acks are inert in every variant + +All of these returned `rc=0` or `-1` and changed neither the snapshot, `.Acked` +nor the Historian: + +- wnwrap `AlarmAckByName` and `AlarmAckByGUID` (`E_NOTIMPL`) on the ack-only and + reader consumers. +- wnwrap `IwwAlarmConsumer2` on a sole consumer with `SetXmlAlarmQuery` applied + (list populated, `GetAlarm(0)` returns the row): `AlarmAck` with ids + 0/1/2/3/-1/65536, and `Select(0)` + `AlarmAckSelected`, with operators + `TestOperator`, `Administrator` and `dohertj2`. `SelectByGUID` is `E_NOTIMPL`. +- The same calls with `TestAlarm001` temporarily reclassified to + `MxSecurityFreeAccess` (template and instance, restored afterwards), so the + security classification is not the cause. +- Managed `aaAlarmManagedClient`: `AlarmAckByGUID` and `SelectByGUID` return + `-1` (the client's list never fills headless, even with a real `hWnd` and a + message pump); `AckSelectedTagAlarms_Ex` returns `0` even for a + `NOSUCHPROVIDER` provider and writes no Historian row. The managed and wnwrap + consumers cannot share a process (`RegisterConsumer` returns `-65`). +- A 20 s continuous message pump after `AlarmAck`, and registering as `InTouch`, + `WindowViewer` or `Wonderware InTouch`. + +Static reverse engineering matched: the five-string COM acks forward domain and +full name with an empty password, and InTouch's `AlarmViewer.ocx` also sends an +empty password. The user-request call is synchronous request-to-return-code with +no completion channel, so the drop happens provider-side. Hooking the user +request in a live WindowViewer that acknowledges successfully and diffing it +against the headless probe is the one definitive step not taken. + +### Consequence + +The gateway acknowledges by authenticating the operator and writing `AckMsg` +(`docs/DesignDecisions.md`, "Alarm acknowledgement writes AckMsg as the +authenticated operator"). The managed ack consumer and the second wnwrap ack +consumer were removed. diff --git a/docs/Authorization.md b/docs/Authorization.md index 12c393d..21e60e7 100644 --- a/docs/Authorization.md +++ b/docs/Authorization.md @@ -125,7 +125,7 @@ public string ResolveRequiredScope(object request) } ``` -The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*` — and `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce. +The `_ => GatewayScopes.Admin` fallback is intentional: any future request type that the resolver does not recognize fails closed, requiring the strongest scope until the resolver is updated. `AcknowledgeAlarm` is treated as a write — it mutates alarm state, mirroring `MxCommandKind.Write*` — and stays `invoke:write` rather than `invoke:secure` because the request carries the operator's own Galaxy credential, which the worker authenticates before the `AckMsg` write, so the credential, not the API-key scope, authorizes the identity the Historian records — and `StreamAlarms` and `QueryActiveAlarms` share the alarm/event surface with `StreamEvents` and `MxCommandKind.DrainEvents`, so they carry `events:read` (the active-alarm snapshot is the same data reachable through the event surface). All three alarm RPCs are session-less: the scope check is the only authorization gate, since there is no per-session ownership to enforce. `MxCommandRequest` is special because it multiplexes many MxAccess operations through a single RPC. The resolver inspects the embedded `MxCommandKind` so each operation gets its own scope: diff --git a/docs/Contracts.md b/docs/Contracts.md index dfa9093..d784ddb 100644 --- a/docs/Contracts.md +++ b/docs/Contracts.md @@ -49,6 +49,17 @@ worker prepares the replacement on its STA before swapping it into the existing failover consumer and returns a normal `MxCommandReply`; callers must treat only `protocol_status = OK` as acknowledgement of the new list. +`AcknowledgeAlarmRequest` carries the operator's identity: `operator_user` +(required), `operator_password` (field 6) and `operator_domain` (field 7). The +gateway copies them onto `AcknowledgeAlarmCommand` (`operator_password` = 7) and +`AcknowledgeAlarmByNameCommand` (`operator_password` = 9); the worker +authenticates the operator and writes `AckMsg` as that user. The fields are +additive, so `GatewayProtocolVersion` stays `3`. `operator_password` is a +credential and follows the same never-log rule as +`AuthenticateUserCommand.verify_user_password`: it must not reach logs, metric +labels, `DiagnosticMessage`, `ProtocolStatus.Message`, or exception text. A +not-found or not-ackable alarm is reported with `hresult` -1. + The command model also includes bulk write/read command kinds: `WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and `ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway` diff --git a/docs/DesignDecisions.md b/docs/DesignDecisions.md index 1ea08f5..66ff30b 100644 --- a/docs/DesignDecisions.md +++ b/docs/DesignDecisions.md @@ -703,14 +703,15 @@ latch in subtag mode. ### Acknowledge via ack-comment write -In subtag mode, `AcknowledgeAlarm` writes the operator comment to the alarm -attribute's ack-comment subtag (`Fallback:Subtags:AckComment`). The write -performs the native ack in AVEVA. This differs from alarmmgr mode, where -`AlarmAckByName` on `wwAlarmConsumerClass` is called directly. The `AckComment` -subtag name is empty by default; configuring it is required for ack to work in -subtag mode. The exact AVEVA subtag names are not hard-coded — the `Subtags` -config block exists precisely so names are not guessed without validation -against the live MXAccess attribute set. +Both provider modes acknowledge by writing the operator comment to `AckMsg` as +the authenticated operator (see "Alarm acknowledgement writes AckMsg as the +authenticated operator" below). In alarm-manager mode the target is +`.AckMsg`; in subtag mode it is the watch-list entry's ack-comment +subtag (`Fallback:Subtags:AckComment`, default `AckMsg`). The write performs +the ack in AVEVA. An entry with an empty ack-comment subtag is not ackable and +the RPC reports `InvalidRequest`. The exact AVEVA subtag names are not +hard-coded — the `Subtags` config block exists so names are not guessed +without validation against the live MXAccess attribute set. ### Related documentation @@ -784,6 +785,39 @@ The following items were previously listed here and have since shipped: `MxGateway:Sessions:MaxEventSubscribersPerSession`. See [Event Subscribers](#event-subscribers) above and [Sessions](./Sessions.md). +## Alarm acknowledgement writes AckMsg as the authenticated operator + +**Decision (2026-10-19).** `AcknowledgeAlarm` authenticates the requesting +operator with `AuthenticateUser` (`operator_user`, `operator_password`, +optional `operator_domain`) and writes the comment to `.AckMsg` over +MXAccess as that user, on a dedicated `LMXProxyServer` registered as +`MxGateway:Alarms:AckClientName`. Both provider modes use this path. + +**Why.** Live tests (2026-10-18/19) showed every alarm-client ack is inert on a +headless consumer: wnwrap `AlarmAckByName`/`AlarmAckByGUID`/`AlarmAck`/ +`AlarmAckSelected`, and the managed `aaAlarmManagedClient` GUID ack, return +rc=0 or -1 and change nothing, regardless of security classification, +operator name or consumer name. An MXAccess write to `AckMsg` does acknowledge +the alarm, and when the write is made under an authenticated user the AVEVA +runtime historizes that identity (`Operator`, `DomainName`, `UserFullName`), +with the MXAccess client name as `OperatorNode`. The requirement is that the +real operator appears in the Historian, as with InTouch, so a shared gateway +service account was rejected. Operators have Galaxy accounts, so the +per-request credential is workable. See +[Alarm Probe Findings](./AlarmProbeFindings.md). + +**Consequences.** + +- `operator_user` is required; `operator_password` may be empty. +- `operator_password` is a credential. It is forwarded to the worker over the + pipe but never logged: `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are + credential-bearing commands and any property named like `password` is + redacted. +- `Applied` and `Unconfirmed` (write sent, no `OnWriteComplete` in the bounded + wait) both report `OK`; clients that need certainty watch for the transition. +- The managed ack consumer, the WNAL native ack consumer, the second wnwrap + ack-only consumer and the opt-in managed-ack environment flag are removed. + ## Related Documentation - [Gateway Process Detailed Design](./GatewayProcessDesign.md) diff --git a/docs/Diagnostics.md b/docs/Diagnostics.md index 0e886b0..f061e11 100644 --- a/docs/Diagnostics.md +++ b/docs/Diagnostics.md @@ -4,7 +4,7 @@ The diagnostics subsystem provides structured logging, credential redaction, and ## Goals -The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly. +The subsystem exists to satisfy two security rules from `gateway.md`: never log passwords or raw credential values for `AuthenticateUser`, `WriteSecured`, `AcknowledgeAlarm` (its `operator_password`), or related secured operations, and never log full MXAccess values by default. Code paths that touch credentials or tag values must therefore route through `GatewayLogRedactor` rather than emitting them directly. A second goal is parity-test diagnosability. Because MXAccess sessions, workers, correlation ids, and command methods are the units of comparison, every log entry produced inside a request scope must carry those identifiers without each call site having to format them. @@ -70,7 +70,9 @@ private static readonly HashSet SensitiveCommandMethods = new(StringComp { "AuthenticateUser", "WriteSecured", - "WriteSecured2" + "WriteSecured2", + "AcknowledgeAlarm", + "AcknowledgeAlarmByName" }; public static bool IsCredentialBearingCommand(string? commandMethod) @@ -80,7 +82,7 @@ public static bool IsCredentialBearingCommand(string? commandMethod) } ``` -The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below. +The names match the MXAccess command list in `gateway.md` exactly. `Write` and `Write2` are not in the set because their payloads are tag values, not credentials, and are governed by the `valueLoggingEnabled` flag described below. `AcknowledgeAlarm` and `AcknowledgeAlarmByName` are in the set because their commands carry the operator's `operator_password`. Independently of the method set, the name-based seam redacts any property whose name contains `password`, so a credential field added to a payload later is redacted without a code change. ### API key redaction diff --git a/docs/GatewayConfiguration.md b/docs/GatewayConfiguration.md index 98b8535..b9c8e69 100644 --- a/docs/GatewayConfiguration.md +++ b/docs/GatewayConfiguration.md @@ -442,17 +442,18 @@ behavior. | `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor re-resolves and, on material change, hot-swaps the subtag fallback watch-list, then reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot. The acknowledged watch-list fingerprint advances only after an OK worker reply. Floored at 5 seconds. | | `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. | | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. | +| `MxGateway:Alarms:AckClientName` | `MxGateway` | MXAccess client name the alarm worker's ack writer registers with; the Historian records it as each acknowledgement's OperatorNode. 1-64 printable characters. Conveyed via `MXGATEWAY_ALARM_ACK_CLIENT_NAME`. | The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and `StreamAlarms` are session-less RPCs served by the monitor. -The worker also reads `MXGATEWAY_USE_MANAGED_ALARM_ACK` (inherited from the -gateway process). Setting it to `1` is meant to route GUID-keyed acknowledges -through AVEVA's managed alarm client, but a live test (2026-10-18) showed the -path cannot work as built: the managed client fails to register alongside the -worker's wnwrap reader and the worker silently falls back to wnwrap. Leave it -unset. It has no `MxGateway:*` option. See -[AlarmClientDiscovery](./AlarmClientDiscovery.md) ("Opt-in managed GUID ack"). +`AckClientName` is the MXAccess client name the alarm worker's ack writer +registers its dedicated `LMXProxyServer` with. The Historian records it as the +OperatorNode of each acknowledgement, so it identifies the gateway as the +acknowledging node. The gateway passes it to each worker through the +`MXGATEWAY_ALARM_ACK_CLIENT_NAME` environment variable. Startup validation +rejects an empty value, a value over 64 characters, or one containing control +characters. ### Alarm fallback options @@ -471,7 +472,7 @@ native wnwrap alarm-manager provider and the subtag-monitoring fallback. | `MxGateway:Alarms:Fallback:Discovery:ExcludeAttributes` | _(empty)_ | Attribute paths to remove from the merged watch-list (case-insensitive). The exclude is applied after the Repository-derived rows and the explicit `IncludeAttributes` entries are combined, so an exclude that matches an explicit include suppresses it too — excludes win. Ignored when `UseGalaxyRepository` is `false`. | | `MxGateway:Alarms:Fallback:Subtags:Active` | `InAlarm` | Subtag name for the in-alarm boolean. Confirmed AVEVA `AlarmExtension` field name. | | `MxGateway:Alarms:Fallback:Subtags:Acked` | `Acked` | Subtag name for the acknowledged boolean. Confirmed AVEVA `AlarmExtension` field name. | -| `MxGateway:Alarms:Fallback:Subtags:AckComment` | `AckMsg` | Subtag name for the acknowledgement comment write target. Writing this subtag performs the acknowledge in AVEVA. Confirmed AVEVA `AlarmExtension` field name. When empty, the ack-comment write path is disabled. | +| `MxGateway:Alarms:Fallback:Subtags:AckComment` | `AckMsg` | Subtag name for the acknowledgement comment write target. Writing this subtag, as the authenticated operator, performs the acknowledge in AVEVA. Confirmed AVEVA `AlarmExtension` field name. When empty, the alarm is not ackable and the RPC returns `InvalidRequest`. | | `MxGateway:Alarms:Fallback:Subtags:Priority` | `Priority` | Subtag name for the alarm priority / severity value. Confirmed AVEVA `AlarmExtension` field name. | Validation rules: diff --git a/docs/Grpc.md b/docs/Grpc.md index 32bb4ff..11a2edd 100644 --- a/docs/Grpc.md +++ b/docs/Grpc.md @@ -96,9 +96,24 @@ Selected control commands have additional public behavior beyond their protobuf ### `AcknowledgeAlarm` -`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`. +`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` and `operator_user` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`. -An `OK` response means the alarm provider accepted the acknowledgement, not that it applied it: the gateway forwards the ack and reports what the provider returned, in keeping with the parity rule. On galaxies whose alarmed attributes carry a non-free-access security classification the by-name ack is accepted and inert — the underlying `AlarmAckByName` conveys an operator name rather than an authenticated identity, observed on the probe rig with the mechanism inferred rather than confirmed — so a client that needs to confirm an acknowledgement should watch for the resulting transition on `StreamAlarms` instead of treating `OK` as proof. See [Alarm Probe Findings](./AlarmProbeFindings.md). +Request fields that identify the acknowledging operator: + +- `operator_user` (required): the Galaxy account name. An empty value is rejected with `InvalidArgument` so the real operator is always recorded. +- `operator_password` (field 6): the operator's Galaxy password. May be empty because Galaxy accounts can have empty passwords. It is a credential and is never logged. +- `operator_domain` (field 7): optional. When set, the worker authenticates `domain\user`; when empty it authenticates `user`. + +The worker calls `AuthenticateUser` as the operator, then writes `operator_comment` to `.AckMsg` (subtag mode: the configured ack-comment subtag) on a dedicated MXAccess server registered as `MxGateway:Alarms:AckClientName`, and waits a bounded time for `OnWriteComplete`. The AVEVA runtime treats that write as the acknowledgement, and the Historian (`Runtime.dbo.v_AlarmEventHistory2`) records the operator name, domain and full name from the authenticated account, with the ack client name as `OperatorNode`. The alarm-client ack calls (`AlarmAckByName`, `AlarmAckByGUID`) are not used because a headless consumer's acks are accepted and dropped; see [Alarm Probe Findings](./AlarmProbeFindings.md). + +| Worker outcome | Response status | `hresult` | +|---|---|---| +| Write applied, or sent but no `OnWriteComplete` within the wait (unconfirmed, with a diagnostic) | `OK` | `0` | +| Unknown alarm, or no ack target (non-Galaxy provider, no ack-comment subtag) | `INVALID_REQUEST` | `-1` | +| `AuthenticateUser` failed | `MXACCESS_FAILURE` | the `AuthenticateUser` HRESULT | +| Write rejected (`OnWriteComplete` non-OK) | `MXACCESS_FAILURE` | the status `detail` | + +`reply.Statuses` carries the `OnWriteComplete` `MxStatusProxy` rows when they were observed. Because an unconfirmed write also returns `OK`, a client that must be certain should watch for the resulting transition on `StreamAlarms`. `AcknowledgeAlarm` stays under the `invoke:write` scope; see [Authorization](./Authorization.md). ### `StreamAlarms` @@ -217,6 +232,7 @@ receive this event directly. | `StreamEvents` | `session_id` must be non-empty. | `InvalidArgument` | | `Invoke` | `session_id` non-empty, `command` present, `kind` not `Unspecified`, payload oneof must match `kind`. | `InvalidArgument` | | `AcknowledgeAlarm` | `alarm_full_reference` must be non-empty. Validated inline in the handler, not by `MxAccessGrpcRequestValidator`. | `InvalidArgument` | +| `AcknowledgeAlarm` | `operator_user` must be non-empty. Validated inline in the handler. | `InvalidArgument` | | `StreamAlarms` | No required fields — `alarm_filter_prefix` is optional. | — | | `QueryActiveAlarms` | No required fields — `alarm_filter_prefix` is optional. Not routed through `MxAccessGrpcRequestValidator`; the handler only null-checks the request. | — | diff --git a/gateway.md b/gateway.md index f0c5791..c5ae6b4 100644 --- a/gateway.md +++ b/gateway.md @@ -140,8 +140,9 @@ one gateway-owned worker session subscribes the configured AVEVA alarm provider, caches the active-alarm set (reconciled periodically against the worker's snapshot), and fans it out to every client through the session-less `StreamAlarms` RPC — the stream opens with the current active-alarm snapshot, -then streams live transitions. `AcknowledgeAlarm` is session-less and routes -through the monitor. Clients never open a worker session to see alarms, and +then streams live transitions. `AcknowledgeAlarm` is session-less, routes +through the monitor, and carries the operator's credentials (see +"Acknowledge" below). Clients never open a worker session to see alarms, and alarm monitoring is independent of client lifecycle; the monitor re-opens its session if the worker faults. Gated by `MxGateway:Alarms:Enabled` — see `docs/DesignDecisions.md` for why this reverses the v1 single-subscriber rule @@ -221,20 +222,22 @@ forwards only events the worker emits and never synthesizes transitions itself. The synthesis rules are documented in `docs/AlarmClientDiscovery.md`. -**Acknowledge in subtag mode:** the ack-by-name path writes the operator -comment to the alarm attribute's ack-comment subtag. The write performs the -ack. If the attribute has no writable ack-comment subtag configured, the RPC -returns `FailedPrecondition`. In alarm-manager mode, `AlarmAckByName` is -used as before. - -**Acceptance is not application:** the ack is forwarded to the provider, and a -successful return means the provider accepted the call, not that the -acknowledgement was applied. On galaxies whose alarmed attributes carry a -non-free-access security classification, `AlarmAckByName` returns `rc=0` and the -alarm stays `UNACK_ALM` — it carries an operator name, not an authenticated -identity. (Observed on the probe rig; the mechanism is inferred.) Confirm an ack -by the resulting transition, never by the return code. See -`docs/AlarmProbeFindings.md`. +**Acknowledge:** the gateway acknowledges an alarm by writing the operator's +comment to the alarm attribute's `AckMsg` (subtag mode: the configured +ack-comment subtag) over MXAccess, after `AuthenticateUser` as the requesting +operator (`operator_user`, `operator_password`, optional `operator_domain`). +The AVEVA runtime treats that write as the acknowledgement and historizes it +with the authenticated account's name, domain and full name; the Historian's +OperatorNode is the ack writer's MXAccess client name +(`MxGateway:Alarms:AckClientName`). The worker waits a bounded time for +`OnWriteComplete`: an OK status is reported as success, a non-OK status as +`MxaccessFailure` with the status detail, and no completion as success with a +diagnostic noting the write was sent but unconfirmed. An unknown alarm or one +with no ack target (non-Galaxy provider, no ack-comment subtag) is reported as +`InvalidRequest` with `hresult` -1. Alarm-client ack calls +(`AlarmAckByName`/`AlarmAckByGUID`, the managed `AlarmClient`, WNAL) are not +used: live tests showed a headless consumer's acks are accepted and silently +dropped (see `docs/AlarmProbeFindings.md`). **Degraded state visibility:** every subtag-mode transition carries `degraded = true` and `source_provider = ALARM_PROVIDER_MODE_SUBTAG` on the @@ -1028,8 +1031,9 @@ Internal worker IPC: - gateway validates worker executable path and version, - no secrets in command line when avoidable. -Credential-sensitive commands such as `AuthenticateUser` and `WriteSecured` -must not log passwords or raw credential values. +Credential-sensitive commands such as `AuthenticateUser`, `WriteSecured` and +`AcknowledgeAlarm` (its `operator_password`) must not log passwords or raw +credential values. ## Observability diff --git a/src/ZB.MOM.WW.MxGateway.Server/appsettings.json b/src/ZB.MOM.WW.MxGateway.Server/appsettings.json index ad80bcf..1c6dc8f 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/appsettings.json +++ b/src/ZB.MOM.WW.MxGateway.Server/appsettings.json @@ -88,7 +88,8 @@ "DefaultArea": "", "ReconcileIntervalSeconds": 30, "PollIntervalMilliseconds": 500, - "MaxAlarmsPerFetch": 1024 + "MaxAlarmsPerFetch": 1024, + "AckClientName": "MxGateway" } } } diff --git a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs index 85089ea..ed998d9 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs @@ -202,14 +202,14 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer // Use the IwwAlarmConsumer (v1) prefix-named methods for the // lifecycle. Empirically (live dev-rig 2026-05-01, reconfirmed - // 2026-08-18) this is the only path that lets AlarmAckByName - // return rc=0 afterwards. The v2 Initialize/Register/Subscribe - // methods on the class succeed (return 0) but acks against that + // 2026-08-18) the v1 path is the one that leaves the consumer in + // a usable state: the v2 Initialize/Register/Subscribe methods on + // the class succeed (return 0) but later calls against that // consumer state return -55. The v1 prefix path is what - // WIN-911-style code uses against the same wnwrap library. - // Note rc=0 means the call was - // accepted, not that an acknowledgement was applied — see - // docs/AlarmProbeFindings.md. Acks now go through AlarmAckWriter. + // WIN-911-style code uses against the same wnwrap library. This + // consumer only reads the alarm snapshot; acknowledgements go + // through AlarmAckWriter, not through wnwrap (see + // docs/AlarmProbeFindings.md). int init = com.IwwAlarmConsumer_InitializeConsumer(DefaultApplicationName); if (init != 0) { @@ -615,10 +615,10 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer /// place under its original GUID and the re-raise replaced it with a /// freshly minted one. That is the intended reading — the old /// instance ended and a new one began — so the diff emits a Clear - /// plus a Raise, which is what happened. The acknowledge leg remains - /// assumed rather than observed: on the dev rig every ack surface is - /// inert (AlarmAckByName returns 0 and nothing moves), so no - /// record can be driven into an acknowledged state — see + /// plus a Raise, which is what happened. Every alarm-client ack surface + /// is inert on the dev rig (it returns 0 and nothing moves), so + /// acknowledgements go through the AckMsg writer; an AckMsg write + /// moves the record to ACK_ALM (observed 2026-10-19) — see /// docs/AlarmProbeFindings.md. /// ///