From 9e4be2580745dcefc51790dbae731ade192e8c59 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 23:34:46 -0500 Subject: [PATCH] test(live): capture OperationComplete trigger, subtag Clear and status rows on windev (#210, #211) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live MXAccess captures against the windev DEV Galaxy, recorded in docs/LiveCaptures-2026-11-13.md: - OperationComplete follows a Suspend/Activate whose synchronous MxStatus is Pending (advised, resolved item); one Ok row from RespondingLmx. Pinned by the new OperationCompleteLiveProbeTests live fact, with an unresolved-reference negative control. (#211 §3.1) - No public COM event carries completion-only status bytes. All 71 completions were typed MXSTATUS_PROXY[], so they stay raw permanently. (#211 §3.3) - Subtag-mode Clear is drivable: the TestAlarm UDAs are MxSecurityOperate, so AuthenticateUser + WriteSecured toggles them. The new self-driven LiveSubtag_SelfDrivenRaiseThenClear_SynthesizesClear fact asserts the UnackAlm -> UnackRtn Clear (mutation-checked). (#210 §3.7) - The B8 buffered "NoData bootstrap" is a ConfigurationError resolution failure (detail 6 / 1002). The DEV Galaxy has no buffered attribute, so a multi-sample capture needs IDE work. provider_switches was not driven. (#210) - Status-row catalogue for lmxopcua#872: no Warning row seen, and Pending only on the synchronous Suspend/Activate return. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- docs/DesignDecisions.md | 24 +- docs/LiveCaptures-2026-11-13.md | 157 ++++++++++++ gateway.md | 7 +- .../WorkerLiveMxAccessSmokeTests.cs | 25 +- .../Probes/AlarmSubtagLiveSmokeTests.cs | 157 +++++++++++- .../Probes/OperationCompleteLiveProbeTests.cs | 230 ++++++++++++++++++ stillpending.md | 8 +- 7 files changed, 584 insertions(+), 24 deletions(-) create mode 100644 docs/LiveCaptures-2026-11-13.md create mode 100644 src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/OperationCompleteLiveProbeTests.cs diff --git a/docs/DesignDecisions.md b/docs/DesignDecisions.md index bd840763..45d8fbd2 100644 --- a/docs/DesignDecisions.md +++ b/docs/DesignDecisions.md @@ -576,9 +576,15 @@ Decision: model and forward `OperationComplete` only when native MXAccess fires it. Do not synthesize `OperationComplete` from writes, command replies, ASB completion queues, or other status frames. -Rationale: the event signature is known, but the MXAccess analysis has not yet -captured the runtime condition that triggers the public event. Synthesizing it -would risk breaking parity. +Rationale: synthesizing it would risk breaking parity. + +Trigger (captured live 2026-11-13, `docs/LiveCaptures-2026-11-13.md` §1, pinned by +`OperationCompleteLiveProbeTests`): MXAccess raises `OperationComplete` after a +`Suspend` or `Activate` whose synchronous `MxStatus` is Pending (raw category 1, +detail 0). That happens only for an advised item whose reference resolved. The event +carries the same server/item handles and one Ok row detected by RespondingLmx. A +`Suspend`/`Activate` that returns Ok synchronously (already in that state, or an +unresolved reference) raises nothing, and writes never raise it. ## Buffered Data Change @@ -589,6 +595,12 @@ The event signature and native path are known. A live buffered sample batch has not yet been observed. Until then, preserve raw value, quality, timestamp, data type, and status metadata whenever conversion is incomplete. +The empty `NoData` event that the rig delivers after `Advise` is a resolution +failure, not a bootstrap: its status row is ConfigurationError, detail 6 for the +`(item, context)` form and detail 1002 `InvalidPropertyId` for a full reference to an +attribute without a buffer (`docs/LiveCaptures-2026-11-13.md` §5). A multi-sample +capture needs a deployed field attribute or I/O extension with *Buffer* enabled. + ## Completion-Only Status Mapping Decision: preserve completion-only operation-status bytes as raw diagnostic @@ -598,6 +610,12 @@ proves an exact `MXSTATUS_PROXY[]` mapping. Do not guess status category/source/detail values for frames that MXAccess does not expose through its public COM events. +Settled 2026-11-13 (`docs/LiveCaptures-2026-11-13.md` §2): every captured +`OnWriteComplete` and `OperationComplete` carried a typed `MXSTATUS_PROXY[]`, and no +public COM event delivers completion-only bytes. The mapping is not needed, and any +such bytes stay raw permanently. `PreserveCompletionOnlyStatusBytes` has no production +caller. + ## API Key Administration Decision: v1 API key management is a local administrative CLI/tool, not a diff --git a/docs/LiveCaptures-2026-11-13.md b/docs/LiveCaptures-2026-11-13.md new file mode 100644 index 00000000..1d5d25ac --- /dev/null +++ b/docs/LiveCaptures-2026-11-13.md @@ -0,0 +1,157 @@ +# Live MXAccess captures, 2026-11-13 (mxaccessgw#210, #211; lmxopcua#872) + +Captured on windev (`DESKTOP-6JL3KKO`, TEST Galaxy `DEV`, Galaxy Repository DB `ZB`) +between 04:10 and 04:35 UTC on 2026-11-13 (the box clock runs about 9 minutes behind +UTC, so worker timestamps in raw logs read about 04:01 to 04:26). + +Two capture paths were used: + +- **Through the running gateway.** A throwaway .NET client (built on + `ZB.MOM.WW.MxGateway.Client`) talked to the `MxAccessGw` service on + `http://10.100.0.48:5120` (build `447070a5`) with a temporary API key, revoked + afterwards. It opened sessions, added/advised items, ran commands and recorded every + `StreamEvents` event and every command reply as JSON. +- **Worker.Tests live facts on windev** (direct COM, no gateway): the two facts added + with this document, `OperationCompleteLiveProbeTests` and + `AlarmSubtagLiveSmokeTests.LiveSubtag_SelfDrivenRaiseThenClear_SynthesizesClear`. + +The service was never restarted or reconfigured. Writes went to `TestMachine_037`, +`_038` and `_039`; every value a probe changed was written back afterwards. + +## 1. `OperationComplete` trigger (§3.1, #211): captured + +`OperationComplete` follows a `Suspend` or `Activate` whose synchronous status is +**Pending**. That happens only for an item that is advised (plain or supervisory) +and whose reference resolved. + +| Call | Synchronous `MxStatus` | `OperationComplete` | +|---|---|---| +| `Suspend`, advised resolved item | Pending (raw 1), success -1, detail 0, RequestingLmx | yes, 110-220 ms later: one row Ok, detail 0, **RespondingLmx (raw 1)**, same server/item handles | +| `Activate` of that suspended item | Pending (raw 1), as above | yes, as above | +| `Suspend` again on an already-suspended item | Ok | no | +| `Activate` on an already-active item | Ok | no | +| `Suspend`/`Activate`, advised but unresolved reference (`InvalidReference`) | Ok | no | +| `Suspend`/`Activate`, added but not advised | `E_INVALIDARG` (0x80070057) | no | +| `Suspend`/`Activate`, supervisory-advised resolved item | Pending | yes | +| Any `Write`/`WriteSecured`/`Write2`, success or failure | n/a | never (only `OnWriteComplete`) | + +The item stops delivering `OnDataChange` between the Suspend's completion and the +Activate; the Activate's completion arrives together with a fresh `OnDataChange`. + +The gateway already forwards the native event untouched (no synthesis), and the +mapping is unchanged. `OperationCompleteLiveProbeTests` pins it, including the +unresolved-reference negative control. + +## 2. Completion-only status bytes (§3.3, #211): resolved, nothing to map + +Of the 71 completion events captured (63 `OnWriteComplete`, 8 `OperationComplete`), +every one carried a typed `MXSTATUS_PROXY[]` that the converter unpacked. None set +`raw_status`. The public COM events have no completion-only byte payload. +`MxStatusProxyConverter.PreserveCompletionOnlyStatusBytes` has no production caller. +Any completion-only bytes exist only in the private ASB/NMX frames described by the +MXAccess analysis, which the worker never sees. So no `MXSTATUS_PROXY[]` mapping is +needed, and if such bytes ever reach the worker they stay raw. + +## 3. Status rows observed (input for lmxopcua#872) + +Every status-row shape these probes produced. **No `Warning` category (raw 2) was +observed anywhere, and `Pending` (raw 1) appeared only as the synchronous return of +`Suspend`/`Activate`.** It never appeared on an `OnDataChange`, `OnWriteComplete`, +`OperationComplete` or write-reply row. Every `Pending` had detail 0 and success -1. + +| Category (raw) | Detail | Detected by (raw) | Produced by | +|---|---|---|---| +| Ok (0) | 0 | RequestingLmx (0) | every good `OnDataChange` | +| Ok (0) | 0 | RespondingAutomationObject (5) | every successful write completion | +| Ok (0) | 0 | RespondingLmx (1) | `OperationComplete` | +| **Pending (1)** | **0** | RequestingLmx (0) | `Suspend`/`Activate` synchronous return (success = -1) | +| ConfigurationError (4) | 6 `InvalidReference` | RequestingLmx (0) | advise/write of a nonexistent object or attribute; `AddBufferedItem(item, context)` form | +| ConfigurationError (4) | 8 `InvalidObjectId` | RespondingLmx (1) | advise of an attribute on an undeployed object (`BCD_T1`, `OtOpcUaParityTest_001`) | +| ConfigurationError (4) | 1002 `InvalidPropertyId` | RespondingAutomationObject (5) | `AddBufferedItem` on a UDA without a buffer | +| ConfigurationError (4) | 1003 `IndexOutOfRange` | RespondingAutomationObject (5) | advise/write `TestIntArray[99]` (dim 50) | +| ConfigurationError (4) | 8001 (AO-specific) | RespondingAutomationObject (5) | write rejected for type conversion: string to double/duration/datetime/bool, int to datetime, 1e300 to float, string[] to bool[] | +| OperationalError (5) | 1004 `DataOutOfRange` | RespondingAutomationObject (5) | alarm `Priority` 5000 or -3, `AlarmModeCmd` 99, scalar written to an array | +| OperationalError (5) | 1007 `NotWriteable` | RespondingAutomationObject (5) | write to `.Acked`, `ScanState`, `Tagname` | +| SecurityError (6) | 1008 `WriteAccessDenied` | RespondingAutomationObject (5) | plain `Write` to an Operate/Secured attribute through a plain advise; `WriteSecured` of a verified-write attribute with no verifier | +| SecurityError (6) | 8018 (AO-specific) | RespondingAutomationObject (5) | `WriteSecured` of a verified-write attribute (`ProtectedValue1`) with verifier = the writing user | + +Writes accepted silently (Ok): NaN, +Inf, a numeric string into a double, int64 +overflow / 3.7 / true into an int, float precision loss, a 3-element array into a +50-element int array, and 60 strings into a 50-element string array. The last two +**resize the array**. A plain `Write` through a *supervisory* advise succeeds where +the same write through a plain advise is refused with 1008. Off-scan objects +(`ScanStateCmd=false`) keep accepting writes, and their UDAs stay quality 192. +Unconnected DI items (`OI_MBTCP.PLC.40001`) report quality 32 and then 4, with an +Ok status row. + +Conditions tried that produced no Warning: out-of-range, type mismatch, NaN/Inf, +array resize, off-scan object, undeployed object, unconnected I/O, read-only, +secured/verified writes, alarm-config writes, Suspend/Activate edge cases. + +## 4. Subtag-mode Clear (§3.7, #210): captured + +The rig's `TestAlarm00x` UDAs are `MxSecurityOperate`. A plain `Write` is refused +with 1008, and that refusal is why the 2026-06 runs concluded the alarms were +object-driven. `AuthenticateUser("Administrator", "")` followed by +`WriteSecured` drives them both ways. With the subtag pipeline +(`LmxSubtagAlarmSource` + `SubtagAlarmConsumer`, the production classes) watching +`TestMachine_037.TestAlarm001`: + +``` +Driving condition = true (RAISE) ... +raw ValueChanged: TestMachine_037.TestAlarm001.InAlarm = True +Transition emitted: Kind=Raise PreviousState=Unspecified State=UnackAlm ... Degraded=True Guid=9b60979f-826d-2842-e00b-2bf1234a7809 +Driving condition = false (CLEAR) ... +raw ValueChanged: TestMachine_037.TestAlarm001.InAlarm = False +Transition emitted: Kind=Clear PreviousState=UnackAlm State=UnackRtn ... Degraded=True Guid=9b60979f-826d-2842-e00b-2bf1234a7809 +CONFIRMED: subtag Clear synthesized (UnackAlm -> UnackRtn, Guid=9b60979f-826d-2842-e00b-2bf1234a7809). +``` + +Mutation check: with the clear write removed, the fact fails on `Assert.NotNull` +after its 30 s wait. The finally-block restore write then produces the Clear. + +The same raise/clear through the running gateway's alarm feed (alarmmgr mode) +emitted `RAISE` and then `CLEAR` for `Galaxy!TestArea3.TestMachine_037.TestAlarm001` +(type `DSC`, severity 500, `sourceProvider` ALARMMGR). + +## 5. Multi-sample `OnBufferedDataChange` (§3.2, #210): not capturable on this Galaxy + +`AddBufferedItem` + `SetBufferedUpdateInterval` + `Advise` all return OK, but the +first and only `OnBufferedDataChange` carries a **ConfigurationError** status row: + +- the `(item, context)` form (`"TestDouble"`, `"TestMachine_039"`) gives detail 6 + `InvalidReference` (RequestingLmx); +- the full-reference form (`"TestMachine_039.TestFloat"`, context `""`) gives detail + 1002 `InvalidPropertyId` from the automation object. + +The empty `NoData` payload recorded as a "bootstrap" in B8 is therefore a +resolution failure, not a parity bootstrap. Buffering is a property of a field +attribute or I/O extension with *Buffer* enabled (the `_HasBuffer` property of the +`AnalogUtilities`/`BooleanUtilities` primitives). No deployed object in this Galaxy +has one. Rapid writes did not change this (36 secured writes at 150 ms to `TestDouble` +and `TestFloat`, 3 s interval). + +**What a capture needs (IDE work, not done):** an object with an analog or discrete +field attribute (or I/O extension) that has *Buffer* enabled, bound to a source that +changes faster than the buffered update interval (for example an OI server item with +a reachable device or simulator), deployed. Then `AddBufferedItem(".", "")` +should deliver arrays of length > 1. + +## 6. `provider_switches{from,to,reason}` (§1.3, #210): not driven + +A real alarmmgr to subtag failover needs the monitor worker's +`WnWrapAlarmConsumer.PollOnce` to throw `Threshold` times in a row. The only ways +to make that happen on this box are to kill or stop `alarmmgr.exe` (the rig's +Galaxy-wide alarm manager, which every engine's alarm provider registers with) or to +change the gateway's `Alarms:Fallback:Mode` and restart it. The first can disrupt the +rig's alarm registration for everyone using it, and its recovery is unproven. The +second was ruled out because it restarts the service. Neither was done. The probe +needs a maintenance window. Then either: + +1. with the service running and `/metrics` scraped, stop `alarmmgr.exe` and wait for + `mxgateway_alarms_provider_mode` to go to subtag and a `provider_switches` sample + with `reason` set. Restart alarmmgr (or the platform) and watch the failback + switch; or +2. restart the gateway with `MxGateway__Alarms__Fallback__Mode=ForceSubtag`. This + proves only the forced-start path (`AlarmProviderReasons.ForcedSubtag`), not a + failover `reason`. diff --git a/gateway.md b/gateway.md index ff05894b..11bbc1e1 100644 --- a/gateway.md +++ b/gateway.md @@ -793,7 +793,12 @@ Codes observed live in write-completion correlation: `1007` (`MX_E_WriteAccessDenied` — e.g. a write through a plain-advised handle that lacks supervisory access). `1012`/`1013` mark attributes classified for secured/verified writes; `1016`/`1017` are the secured-write credential -failures. +failures. The 2026-11-13 capture (`docs/LiveCaptures-2026-11-13.md` §3) also saw +`6`, `8`, `1002`, `1003` and `1004`, plus two automation-object-specific codes +outside this table: `8001` (ConfigurationError, write rejected for type conversion) +and `8018` (SecurityError, verified write with the writer as verifier). No Warning +row was seen, and Pending appeared only as the synchronous `Suspend`/`Activate` +status. For command replies, return: diff --git a/src/ZB.MOM.WW.MxGateway.IntegrationTests/WorkerLiveMxAccessSmokeTests.cs b/src/ZB.MOM.WW.MxGateway.IntegrationTests/WorkerLiveMxAccessSmokeTests.cs index 9e732563..392bf85f 100644 --- a/src/ZB.MOM.WW.MxGateway.IntegrationTests/WorkerLiveMxAccessSmokeTests.cs +++ b/src/ZB.MOM.WW.MxGateway.IntegrationTests/WorkerLiveMxAccessSmokeTests.cs @@ -1040,15 +1040,16 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output) // Best-effort capture of a SAMPLE-BEARING buffered batch. // - // Live observation (B8): immediately after Advise the provider delivers an - // initial OnBufferedDataChange with data_type=NoData / raw_data_type=0 and zero - // quality+timestamp samples — the buffered analogue of the bad-quality/ - // registration-state bootstrap event the OnDataChange tests skip with their - // family-match predicate. That empty bootstrap is parity, NOT a dropped payload: - // the converter ran without crashing and there were simply no samples to carry. - // We therefore match only a batch that actually carries samples, so a real - // multi-sample conversion can be validated and the empty bootstrap is skipped - // rather than mistaken for a defect. + // Live observation (B8, re-read 2026-11-13): immediately after Advise the provider + // delivers one OnBufferedDataChange with data_type=NoData and zero quality and + // timestamp samples. Its status row is ConfigurationError: detail 6 + // (InvalidReference) for the (item, context) form used here, and detail 1002 + // (InvalidPropertyId) for a full reference to an attribute without a buffer. It is + // a resolution failure, not a bootstrap. The DEV Galaxy has no buffered attribute, + // so no sample-bearing batch can arrive (docs/LiveCaptures-2026-11-13.md §5). The + // converter handling it without crashing is still worth keeping. We match only a + // batch that carries samples, so the test validates a multi-sample conversion if + // the item is pointed at a buffered attribute. MxEvent? bufferedBatch = null; try { @@ -1082,9 +1083,9 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output) { // RESIDUAL (documented): the command path (AddBufferedItem + // SetBufferedUpdateInterval + Advise) is proven and the buffered EVENT plumbing - // is live (the empty NoData bootstrap arrives and converts without crashing), - // but the rig did not drive a sample-bearing buffered batch within the timeout - // — the same environmental limitation as the externally-undrivable alarm rig. + // is live (the NoData resolution-failure event arrives and converts without + // crashing), but the rig has no buffered attribute to deliver a sample-bearing + // batch (docs/LiveCaptures-2026-11-13.md §5). // The §3.2 OnBufferedDataChange MULTI-SAMPLE conversion therefore remains // unverified live. This is environmental, not a defect — let the test pass. output.WriteLine( diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/AlarmSubtagLiveSmokeTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/AlarmSubtagLiveSmokeTests.cs index 4c046716..d0d39e7a 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/AlarmSubtagLiveSmokeTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/AlarmSubtagLiveSmokeTests.cs @@ -18,10 +18,12 @@ // 2. Remove (or set to null) the Skip parameter on the [Fact] below. // 3. Drive a TestMachine alarm so its Active/Acked subtags toggle — either an // alarm flip script (same one used by AlarmsLiveSmokeTests, ~10 s cadence) -// or a manual operator/IDE toggle of the alarm attribute. The rig's -// TestAlarm attributes are object-driven, so an external MXAccess Write -// cannot toggle them (confirmed live 2026-06-14 by toggling TestAlarm002 -// from the IDE). +// or a manual operator/IDE toggle of the alarm attribute. The 2026-06-14 +// conclusion that the rig's TestAlarm attributes are object-driven was +// wrong: they are MxSecurityOperate, so a plain Write is refused with 1008, +// and AuthenticateUser + WriteSecured toggles them. +// LiveSubtag_SelfDrivenRaiseThenClear_SynthesizesClear drives its own alarm +// that way and needs no flip script (docs/LiveCaptures-2026-11-13.md §4). // // net48/x86 constraints: // - No init-only properties, records, index/range operators, C# 8+ pattern @@ -205,6 +207,153 @@ public sealed class AlarmSubtagLiveSmokeTests } } + /// + /// Self-driven subtag lifecycle (mxaccessgw#210, stillpending §3.7): raises the + /// alarm with a secured write of its condition attribute, waits for the + /// subtag state machine's Raise, clears it with a second secured write and + /// asserts the synthesized Clear (UnackAlm -> UnackRtn, Degraded, same + /// synthetic GUID). The rig's TestAlarm UDAs are MxSecurityOperate: + /// a plain Write is refused with SecurityError 1008, which is why the + /// 2026-06 runs found them "object-driven". WriteSecured after + /// AuthenticateUser drives them (docs/AlarmProbeFindings.md). Runs only + /// when MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1. Object defaults to + /// TestMachine_001; override with MXGATEWAY_LIVE_SUBTAG_ALARM_OBJECT. + /// The write identity is MXGATEWAY_PROBE_ACK_USER/_PASSWORD + /// (default: the rig's Administrator with an empty password). + /// The alarm is left cleared (UNACK_RTN). + /// + [LiveMxAccessFact] + public void LiveSubtag_SelfDrivenRaiseThenClear_SynthesizesClear() + { + Exception? threadException = null; + ManualResetEventSlim done = new ManualResetEventSlim(false); + Thread thread = new Thread(() => + { + try { RunSelfDrivenClear(); } + catch (Exception ex) { threadException = ex; } + finally { done.Set(); } + }); + thread.IsBackground = true; + thread.SetApartmentState(ApartmentState.STA); + thread.Start(); + Assert.True(done.Wait(LiveProbeTimeout), "Live probe thread did not finish within " + LiveProbeTimeout + "."); + thread.Join(); + + List lines = new List(); + string? line; + while (log.TryDequeue(out line)) + { + output.WriteLine(line); + lines.Add(line); + } + + string logPath = Environment.GetEnvironmentVariable("PROBE_LOG") + ?? System.IO.Path.Combine(System.IO.Path.GetTempPath(), "alarm-subtag-clear-probe.log"); + try { System.IO.File.WriteAllLines(logPath, lines); } + catch (Exception writeEx) { output.WriteLine("probe-log write failed: " + writeEx.Message); } + + if (threadException != null) + { + throw threadException; + } + } + + private void RunSelfDrivenClear() + { + string? objectName = Environment.GetEnvironmentVariable("MXGATEWAY_LIVE_SUBTAG_ALARM_OBJECT"); + if (string.IsNullOrWhiteSpace(objectName)) + { + objectName = "TestMachine_001"; + } + + string alarmTag = objectName + ".TestAlarm001"; + string fullReference = AlarmRecordTransitionMapper.ComposeFullReference(Provider, Group, alarmTag); + Log("=== Self-driven subtag Raise -> Clear ==="); + Log("AlarmFullReference: " + fullReference); + + AlarmSubtagTarget target = new AlarmSubtagTarget + { + AlarmFullReference = fullReference, + SourceObjectReference = alarmTag, + ActiveSubtag = alarmTag + ".InAlarm", + AckedSubtag = alarmTag + ".Acked", + AckCommentSubtag = alarmTag + ".AckMsg", + PrioritySubtag = alarmTag + ".Priority", + }; + + MxAccessComObjectFactory factory = new MxAccessComObjectFactory(); + LmxSubtagAlarmSource source = new LmxSubtagAlarmSource(factory, clientName: null); + using SubtagAlarmConsumer consumer = new SubtagAlarmConsumer(source, new List { target }); + ConcurrentQueue transitions = new ConcurrentQueue(); + consumer.AlarmTransitionEmitted += (_, e) => + { + Log("Transition emitted: " + DescribeTransition(e)); + transitions.Enqueue(e); + }; + source.ValueChanged += (_, c) => Log(string.Format("raw ValueChanged: {0} = {1}", c.ItemAddress, c.Value)); + + // A separate registration drives the condition attribute, as an operator client would. + IMxAccessServer writer = new MxAccessComServer(factory.Create()); + int writerHandle = writer.Register("mxgw-subtag-clear-probe"); + int conditionItem = writer.AddItem(writerHandle, alarmTag); + writer.Advise(writerHandle, conditionItem); // MXAccess refuses writes to unadvised items + string user = Environment.GetEnvironmentVariable("MXGATEWAY_PROBE_ACK_USER") ?? "Administrator"; + string password = Environment.GetEnvironmentVariable("MXGATEWAY_PROBE_ACK_PASSWORD") ?? string.Empty; + int userId = writer.AuthenticateUser(writerHandle, user, password); + Log(string.Format("AuthenticateUser({0}) -> userId={1}", user, userId)); + Assert.NotEqual(0, userId); + + try + { + consumer.Subscribe(string.Format(@"\\{0}\Galaxy!{1}", Environment.MachineName, Group)); + PumpFor(TimeSpan.FromSeconds(4)); + + Log("Driving condition = false (baseline) ..."); + writer.WriteSecured(writerHandle, conditionItem, userId, 0, false); + PumpFor(TimeSpan.FromSeconds(4)); + MxAlarmTransitionEvent? drained; + while (transitions.TryDequeue(out drained)) + { + Log("Drained baseline transition: " + DescribeTransition(drained!)); + } + + Log("Driving condition = true (RAISE) ..."); + writer.WriteSecured(writerHandle, conditionItem, userId, 0, true); + MxAlarmTransitionEvent? raise = WaitForTransitionKind(transitions, AlarmTransitionKind.Raise, RaiseWaitTimeout, "Raise"); + Assert.NotNull(raise); + Assert.True(raise!.Record.Degraded); + Assert.Equal(MxAlarmStateKind.UnackAlm, raise.Record.State); + + Log("Driving condition = false (CLEAR) ..."); + writer.WriteSecured(writerHandle, conditionItem, userId, 0, false); + MxAlarmTransitionEvent? clear = WaitForTransitionKind(transitions, AlarmTransitionKind.Clear, ClearWaitTimeout, "Clear"); + Assert.NotNull(clear); + Assert.True(clear!.Record.Degraded, "Subtag-synthesized Clear must be Degraded."); + Assert.Equal(MxAlarmStateKind.UnackAlm, clear.PreviousState); + Assert.Equal(MxAlarmStateKind.UnackRtn, clear.Record.State); + Assert.Equal(raise.Record.AlarmGuid, clear.Record.AlarmGuid); + Log(string.Format("CONFIRMED: subtag Clear synthesized ({0} -> {1}, Guid={2}).", + clear.PreviousState, clear.Record.State, clear.Record.AlarmGuid)); + } + finally + { + try { writer.WriteSecured(writerHandle, conditionItem, userId, 0, false); PumpFor(TimeSpan.FromSeconds(1)); } + catch (Exception ex) { Log("restore write failed: " + ex.Message); } + try { writer.Unregister(writerHandle); } + catch (Exception ex) { Log("writer unregister failed: " + ex.Message); } + } + } + + private static void PumpFor(TimeSpan duration) + { + DateTime deadline = DateTime.UtcNow + duration; + while (DateTime.UtcNow < deadline) + { + PumpMessages(); + Thread.Sleep(100); + } + } + private void RunProbe() { Log("=== Subtag runtime-path resolution probe ==="); diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/OperationCompleteLiveProbeTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/OperationCompleteLiveProbeTests.cs new file mode 100644 index 00000000..698c37dd --- /dev/null +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/OperationCompleteLiveProbeTests.cs @@ -0,0 +1,230 @@ +// OperationCompleteLiveProbeTests.cs +// +// Live fact for mxaccessgw#211 (stillpending §3.1): what makes native MXAccess raise +// the public OperationComplete COM event. +// +// RESULT (windev, DEV Galaxy, 2026-11-13): Suspend and Activate on an ADVISED item +// whose reference RESOLVED return a synchronous MxStatus of category Pending +// (raw category 1, success -1, detail 0, detected by RequestingLmx). About 110-220 ms +// later MXAccess raises OperationComplete for the same (server, item) handle pair +// with one MXSTATUS_PROXY row: category Ok, detected by RespondingLmx (raw 1), +// detail 0. Every call that does not return Pending raises no OperationComplete: +// Suspend/Activate on an advised but unresolved reference (synchronous Ok), Activate +// on an already-active item and a second Suspend on a suspended item (both +// synchronous Ok), and an added-but-unadvised item (E_INVALIDARG). Plain and +// supervisory advises behave the same. Writes never raise it: their outcome is +// OnWriteComplete only. Recorded in docs/LiveCaptures-2026-11-13.md. +// +// The probe suspends and re-activates one item on its own registration. It never +// writes. Runs only when MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1. The item defaults to +// TestMachine_001.TestChangingInt; override with MXGATEWAY_LIVE_OPERATION_COMPLETE_ITEM. + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Runtime.InteropServices; +using System.Threading; +using ArchestrA.MxAccess; +using MxStatusCategory = ZB.MOM.WW.MxGateway.Contracts.Proto.MxStatusCategory; +using MxStatusProxy = ZB.MOM.WW.MxGateway.Contracts.Proto.MxStatusProxy; +using MxStatusSource = ZB.MOM.WW.MxGateway.Contracts.Proto.MxStatusSource; +using ZB.MOM.WW.MxGateway.Worker.Conversion; +using ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport; +using Xunit.Abstractions; + +namespace ZB.MOM.WW.MxGateway.Worker.Tests.Probes; + +/// +/// Live probe that pins the native trigger of the MXAccess OperationComplete +/// event: a Suspend or Activate whose synchronous status is Pending. +/// +public sealed class OperationCompleteLiveProbeTests +{ + private static readonly TimeSpan LiveProbeTimeout = TimeSpan.FromMinutes(3); + private static readonly TimeSpan EventWait = TimeSpan.FromSeconds(5); + + private readonly ITestOutputHelper output; + private readonly ConcurrentQueue log = new ConcurrentQueue(); + + /// Initializes a new instance of the OperationCompleteLiveProbeTests class. + /// Test output helper for logging. + public OperationCompleteLiveProbeTests(ITestOutputHelper output) + { + this.output = output; + } + + /// + /// Suspend then Activate an advised, resolved item: each returns Pending and is + /// followed by one OperationComplete (Ok, RespondingLmx) for that item. Suspend on + /// an unresolved reference returns Ok and raises nothing (negative control). + /// + [LiveMxAccessFact] + public void LiveSuspendActivate_PendingStatus_IsFollowedByOperationComplete() + { + Exception? threadException = null; + ManualResetEventSlim done = new ManualResetEventSlim(false); + Thread thread = new Thread(() => + { + try { Run(); } + catch (Exception ex) { threadException = ex; } + finally { done.Set(); } + }); + thread.IsBackground = true; + thread.SetApartmentState(ApartmentState.STA); + thread.Start(); + Assert.True(done.Wait(LiveProbeTimeout), "Live probe thread did not finish within " + LiveProbeTimeout + "."); + thread.Join(); + + string? line; + while (log.TryDequeue(out line)) + { + output.WriteLine(line); + } + + if (threadException != null) + { + throw threadException; + } + } + + private void Run() + { + string? item = Environment.GetEnvironmentVariable("MXGATEWAY_LIVE_OPERATION_COMPLETE_ITEM"); + if (string.IsNullOrWhiteSpace(item)) + { + item = "TestMachine_001.TestChangingInt"; + } + + MxStatusProxyConverter converter = new MxStatusProxyConverter(); + List>> completions = new List>>(); + LMXProxyServerClass com = new LMXProxyServerClass(); + com.OperationComplete += (int hServer, int hItem, ref MXSTATUS_PROXY[] statuses) => + { + IReadOnlyList rows = converter.ConvertMany(statuses); + completions.Add(Tuple.Create(hServer, hItem, rows)); + Log(string.Format("OperationComplete hServer={0} hItem={1} rows=[{2}]", hServer, hItem, Describe(rows))); + }; + + int serverHandle = com.Register("mxgw-operation-complete-probe"); + try + { + int resolved = com.AddItem(serverHandle, item); + com.Advise(serverHandle, resolved); + int unresolved = com.AddItem(serverHandle, "TestMachine_001.NoSuchAttribute_OperationCompleteProbe"); + com.Advise(serverHandle, unresolved); + PumpFor(TimeSpan.FromSeconds(3)); + + ExpectPendingThenCompletion("Suspend", serverHandle, resolved, completions, converter, + () => { ((ILMXProxyServer4)com).Suspend(serverHandle, resolved, out MxStatus s); return s; }); + ExpectPendingThenCompletion("Activate", serverHandle, resolved, completions, converter, + () => { ((ILMXProxyServer4)com).Activate(serverHandle, resolved, out MxStatus s); return s; }); + + // Negative control: an unresolved reference completes synchronously and raises nothing. + completions.Clear(); + ((ILMXProxyServer4)com).Suspend(serverHandle, unresolved, out MxStatus unresolvedStatus); + MxStatusProxy unresolvedRow = converter.Convert(unresolvedStatus); + Log("Suspend(unresolved) -> " + Describe(new[] { unresolvedRow })); + Assert.Equal(MxStatusCategory.Ok, unresolvedRow.Category); + PumpFor(EventWait); + Assert.Empty(completions); + ((ILMXProxyServer4)com).Activate(serverHandle, unresolved, out MxStatus _); + } + finally + { + try { com.Unregister(serverHandle); } + catch (COMException ex) { Log("Unregister failed: 0x" + ex.HResult.ToString("X8")); } + } + } + + private void ExpectPendingThenCompletion( + string verb, + int serverHandle, + int itemHandle, + List>> completions, + MxStatusProxyConverter converter, + Func call) + { + completions.Clear(); + MxStatusProxy sync = converter.Convert(call()); + Log(string.Format("{0} -> {1}", verb, Describe(new[] { sync }))); + Assert.Equal(MxStatusCategory.Pending, sync.Category); + Assert.Equal(1, sync.RawCategory); + Assert.Equal(0, sync.Detail); + + DateTime deadline = DateTime.UtcNow + EventWait; + while (completions.Count == 0 && DateTime.UtcNow < deadline) + { + PumpFor(TimeSpan.FromMilliseconds(100)); + } + + Assert.True(completions.Count == 1, verb + " must be followed by exactly one OperationComplete, saw " + completions.Count + "."); + Tuple> completion = completions[0]; + Assert.Equal(serverHandle, completion.Item1); + Assert.Equal(itemHandle, completion.Item2); + Assert.Single(completion.Item3); + Assert.Equal(MxStatusCategory.Ok, completion.Item3[0].Category); + Assert.Equal(MxStatusSource.RespondingLmx, completion.Item3[0].DetectedBy); + } + + private static string Describe(IEnumerable rows) + { + List parts = new List(); + foreach (MxStatusProxy r in rows) + { + parts.Add(string.Format( + "success={0} category={1} rawCategory={2} detectedBy={3} rawDetectedBy={4} detail={5}", + r.Success, r.Category, r.RawCategory, r.DetectedBy, r.RawDetectedBy, r.Detail)); + } + + return string.Join("; ", parts); + } + + private void Log(string line) + { + log.Enqueue(string.Format("{0:HH:mm:ss.fff} {1}", DateTime.UtcNow, line)); + } + + private static void PumpFor(TimeSpan duration) + { + DateTime deadline = DateTime.UtcNow + duration; + while (DateTime.UtcNow < deadline) + { + NativeMethods.MSG msg; + while (NativeMethods.PeekMessage(out msg, IntPtr.Zero, 0, 0, NativeMethods.PM_REMOVE)) + { + NativeMethods.TranslateMessage(ref msg); + NativeMethods.DispatchMessage(ref msg); + } + + Thread.Sleep(20); + } + } + + private static class NativeMethods + { + internal const uint PM_REMOVE = 0x0001; + + [StructLayout(LayoutKind.Sequential)] + internal struct MSG + { + internal IntPtr hwnd; + internal uint message; + internal IntPtr wParam; + internal IntPtr lParam; + internal uint time; + internal int ptX; + internal int ptY; + } + + [DllImport("user32.dll")] + [return: MarshalAs(UnmanagedType.Bool)] + internal static extern bool PeekMessage(out MSG lpMsg, IntPtr hWnd, uint wMsgFilterMin, uint wMsgFilterMax, uint wRemoveMsg); + + [DllImport("user32.dll")] + [return: MarshalAs(UnmanagedType.Bool)] + internal static extern bool TranslateMessage(ref MSG lpMsg); + + [DllImport("user32.dll")] + internal static extern IntPtr DispatchMessage(ref MSG lpmsg); + } +} diff --git a/stillpending.md b/stillpending.md index 77fb1c37..773f64d2 100644 --- a/stillpending.md +++ b/stillpending.md @@ -75,13 +75,13 @@ These are documented, deliberate, and mostly enforced. Listed so the deferred su Intentionally incomplete, awaiting a live capture or an AVEVA fix; raw payload/metadata is preserved in the meantime (no synthesis). -- 🟡 **3.1 `OperationComplete` native trigger condition unknown** → #211 — modeled and emitted only from the real event (no synthesis), but the runtime condition that fires it isn't captured. `docs/DesignDecisions.md:280-289`, `gateway.md:1094`, `docs/MxAccessWorkerInstanceDesign.md:341,366`. -- 🟡 **3.2 `OnBufferedDataChange` multi-sample conversion unvalidated — STILL OPEN (residual after B8)** → #210 — `AddBufferedItem`/`SetBufferedUpdateInterval` are now implemented and live-confirmed (§1.1), and the live B8 test (`f7ada90`) confirms the worker receives and cleanly converts the empty `NoData` bootstrap `OnBufferedDataChange` (no crash, no dropped payload). But the rig's object logic does not drive a multi-sample buffered batch on demand (same limitation as the alarm rig), so a real parallel quality/timestamp sample array (length > 1) has never been observed live — it is exercised only by the B-bundle unit tests against a fake `IMxAccessServer`. Re-run `GatewaySession_WithLiveWorker_BufferedItem_*` against a fast-changing simulated tag to close this. `docs/DesignDecisions.md:291-297`. -- 🟡 **3.3 Completion-only status → `MXSTATUS_PROXY[]` mapping unproven** → #211 — completion-only operation-status bytes are kept as raw diagnostic metadata until the analysis proves an exact mapping. `docs/DesignDecisions.md:299-306`. +- ✅ **3.1 `OperationComplete` native trigger — CAPTURED 2026-11-13** (#211): it follows a `Suspend`/`Activate` whose synchronous status is Pending (advised, resolved item); see `docs/LiveCaptures-2026-11-13.md` §1 and `OperationCompleteLiveProbeTests`. Original entry: modeled and emitted only from the real event (no synthesis), but the runtime condition that fires it isn't captured. `docs/DesignDecisions.md:280-289`, `gateway.md:1094`, `docs/MxAccessWorkerInstanceDesign.md:341,366`. +- 🟡 **3.2 `OnBufferedDataChange` multi-sample conversion unvalidated — STILL OPEN (residual after B8)** → #210 — **2026-11-13:** the B8 "empty NoData bootstrap" is a ConfigurationError resolution failure (detail 6 / 1002), and no deployed attribute in the DEV Galaxy has a buffer, so a capture needs a buffered field attribute deployed (`docs/LiveCaptures-2026-11-13.md` §5). Original entry: `AddBufferedItem`/`SetBufferedUpdateInterval` are now implemented and live-confirmed (§1.1), and the live B8 test (`f7ada90`) confirms the worker receives and cleanly converts the empty `NoData` bootstrap `OnBufferedDataChange` (no crash, no dropped payload). But the rig's object logic does not drive a multi-sample buffered batch on demand (same limitation as the alarm rig), so a real parallel quality/timestamp sample array (length > 1) has never been observed live — it is exercised only by the B-bundle unit tests against a fake `IMxAccessServer`. Re-run `GatewaySession_WithLiveWorker_BufferedItem_*` against a fast-changing simulated tag to close this. `docs/DesignDecisions.md:291-297`. +- ✅ **3.3 Completion-only status — SETTLED 2026-11-13** (#211): no public COM event delivers completion-only bytes. All 71 captured completions were typed `MXSTATUS_PROXY[]`, so the bytes stay raw permanently (`docs/LiveCaptures-2026-11-13.md` §2). Original entry: completion-only operation-status bytes are kept as raw diagnostic metadata until the analysis proves an exact mapping. `docs/DesignDecisions.md:299-306`. - 🟡 **3.4 `AlarmAckByGUID` is `E_NOTIMPL` on this AVEVA build (historical; note corrected 2026-10-22)** — it throws `NotImplementedException`, but the worker no longer calls it. `MxCommandKind.AcknowledgeAlarm` has live dispatch (`Worker/MxAccess/MxAccessCommandExecutor.cs` `ExecuteAcknowledgeAlarm`, ~895-936): it parses `alarm_guid` and calls `alarmCommandHandler.Acknowledge(guid, comment, AlarmAckOperator)`, which acknowledges by writing `AckMsg` as the authenticated operator. A GUID ack therefore works; only the vendor `AlarmAckByGUID` call is dead. `docs/AlarmClientDiscovery.md:750-763`. - 🟡 **3.5 8-arg `AlarmAckByName` v2 is a vendor stub (returns -55)** — no longer on the ack path (acks write `AckMsg`); `operator_full_name` is carried for forward-compat only and is not propagated (see 1.4). `docs/AlarmClientDiscovery.md:743-748`. - 🟡 **3.6 Subtag degraded-mode fidelity limits** — `category`, `description`, `alarm_type_name`, operator fields, and `retrigger` are not populated/synthesized in subtag fallback (no subtag exists for them). Documented, by design. `docs/AlarmClientDiscovery.md:913-931`, `docs/plans/2026-06-13-alarm-subtag-fallback-design.md:292-298`. -- 🟡 **3.7 Subtag `Clear` transition unvalidated live** → #210 — Raise/Ack/AckMsg are live-confirmed; Clear is externally undrivable on the rig (object logic owns alarm state). Environmental, not code. (`project_alarm_subtag_fallback`, `project_rig_alarms_object_driven`.) +- ✅ **3.7 Subtag `Clear` transition — CAPTURED 2026-11-13** (#210): driven with `WriteSecured` and pinned by `AlarmSubtagLiveSmokeTests.LiveSubtag_SelfDrivenRaiseThenClear_SynthesizesClear` (`docs/LiveCaptures-2026-11-13.md` §4). Original entry: Raise/Ack/AckMsg are live-confirmed; Clear is externally undrivable on the rig (object logic owns alarm state). Environmental, not code. (`project_alarm_subtag_fallback`, `project_rig_alarms_object_driven`.) ---