Compare commits

...

2 Commits

Author SHA1 Message Date
Joseph Doherty aaeb86b132 fix(worker-tests): decouple FakeRuntimeSession dispatch safety-net from the inspection window
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m13s
ci / java (push) Successful in 2m26s
ci / portable (push) Failing after 4m56s
The BlockDispatch branch waited 5 real seconds and then proceeded regardless — it does not
branch on the wait's result. The long-in-flight test's inspection loop is bounded by
elapsed time and a frame floor, so on a loaded box (the documented 4-5x slowdown class) it
can plausibly outrun that 5 s. When it does, the reply is emitted mid-window,
AssertNotWorkerFault waves it past, and the reply leg then waits for a reply already gone
by — failing at the 20 s cancellation with no message, on exactly the loaded-box run the
widened windows exist to survive.

The wait is a pure safety net: nothing asserts on it firing, and every test that blocks
dispatch releases it explicitly (ReleaseDispatch, or a WorkerShutdown envelope, both of
which Set the event) — none reaches the timeout on a healthy run. Named it
BlockedDispatchSafetyNet and raised it to 30 s, above any window a test opens and above the
20 s cancellation those tests arm, so a wedged test always fails on its own token with its
own message. Dispose still releases the wait, so teardown never waits on it either.

Inline rationale in the test now states the decoupling and what a close pairing would cost,
rather than asserting the window stays inside a 5 s ceiling. Nothing else changed.
2026-08-18 07:08:47 -04:00
Joseph Doherty bc22792a36 docs(probe): close superseded unblock bullet, hedge the summary, evidence gobject_id, ack-caveat in RPC docs 2026-08-18 07:03:20 -04:00
5 changed files with 41 additions and 8 deletions
+13 -4
View File
@@ -258,7 +258,9 @@ MXAccess-side entry point and the wnwrap-side entry point is inert.
requirement that the wnwrap consumer, which passes an operator *name* string and no
authenticated identity, cannot meet. If so, ack over wnwrap is not merely untested here
but unavailable by configuration, and the gateway's `AcknowledgeByName` path needs the
same treatment on any customer galaxy configured that way.
same treatment on any customer galaxy configured that way. (Answered by the Third attempt
below — enforced by the alarm attribute's `MxSecurityOperate` security classification, an
engine-level write-security setting, not a separate `alarmmgr`-side ack policy.)
- If neither lands, the acknowledge leg stays assumed. It is worth restating that this is a
documentation gap, not a correctness one: a re-minted GUID on acknowledge would produce a
spurious Clear plus a spurious Raise, which is the same shape the now-observed re-raise
@@ -284,7 +286,12 @@ notes referenced elsewhere in this repo are **not present on this box** (`Test-P
### The test attribute is classified `MxSecurityOperate`
UDA security classification lives in `dynamic_attribute.security_classification`, keyed by
`gobject_id` — for the `$TestMachine` template that is `1055`:
`gobject_id`. For the `$TestMachine` template that is `1055`:
```
SELECT gobject_id, tag_name, hierarchical_name FROM gobject WHERE gobject_id=1055;
1055|$TestMachine|$TestMachine
```
```
attribute_name | security_classification | mx_attribute_category
@@ -374,8 +381,10 @@ statements and both matter:
- The wnwrap ack path is unavailable on this rig as configured, for an identified reason
rather than an unknown one. This is a real finding for the gateway: `AcknowledgeByName`
will behave the same way — accepted, inert — on any customer galaxy whose alarmed
attributes carry a non-free-access security classification. It is worth noting in the
alarm client's documentation that a silent `rc=0` is not proof of acknowledgement.
attributes carry a non-free-access security classification (inferred from the mechanism —
no `AlarmAckByName` overload can carry a credential — not confirmed by relaxing the
classification and re-testing; see "What a human would need to do"). It is worth noting in
the alarm client's documentation that a silent `rc=0` is not proof of acknowledgement.
- Whether wnwrap re-mints the record GUID on `UNACK_ALM → ACK_ALM` is still unobserved, and
after three attempts it stays assumed. As the second attempt already noted, this remains
a documentation gap rather than a correctness one: a re-minted GUID produces a spurious
+2
View File
@@ -92,6 +92,8 @@ An accepted gRPC command payload can still be too large for the worker pipe: the
`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
An `OK` response means the alarm provider accepted the acknowledgement, not that it applied it: the gateway forwards the ack and reports what the provider returned, in keeping with the parity rule. On galaxies whose alarmed attributes carry a non-free-access security classification the by-name ack is accepted and inert — the underlying `AlarmAckByName` conveys an operator name rather than an authenticated identity — so a client that needs to confirm an acknowledgement should watch for the resulting transition on `StreamAlarms` instead of treating `OK` as proof. See [Alarm Probe Findings](./AlarmProbeFindings.md).
### `StreamAlarms`
`StreamAlarms` is a server-streaming, **session-less** RPC that attaches to the gateway's central alarm feed. The handler delegates to `IGatewayAlarmService.StreamAsync`. The stream opens with a `provider_status` and a `snapshot_status` `AlarmFeedMessage` (the current provider mode and snapshot-completeness verdict), then one `AlarmFeedMessage` carrying an `active_alarm` per currently-active alarm (the ConditionRefresh snapshot), then a single `snapshot_complete`, then a `transition` for every subsequent raise / acknowledge / clear — interleaved with a further `provider_status` on each failover/failback and a further `snapshot_status` on each change of the truncation verdict. It is served by the always-on `GatewayAlarmMonitor`, which owns a single gateway-managed worker session and fans out to every attached client — clients no longer open a session of their own. `alarm_filter_prefix`, when set, scopes the stream to a sub-tree.
+8
View File
@@ -227,6 +227,14 @@ ack. If the attribute has no writable ack-comment subtag configured, the RPC
returns `FailedPrecondition`. In alarm-manager mode, `AlarmAckByName` is
used as before.
**Acceptance is not application:** the ack is forwarded to the provider, and a
successful return means the provider accepted the call, not that the
acknowledgement was applied. On galaxies whose alarmed attributes carry a
non-free-access security classification, `AlarmAckByName` returns `rc=0` and the
alarm stays `UNACK_ALM` — it carries an operator name, not an authenticated
identity. Confirm an ack by the resulting transition, never by the return code.
See `docs/AlarmProbeFindings.md`.
**Degraded state visibility:** every subtag-mode transition carries
`degraded = true` and `source_provider = ALARM_PROVIDER_MODE_SUBTAG` on the
`OnAlarmTransitionEvent` and `ActiveAlarmSnapshot` proto fields. The
@@ -1525,9 +1525,12 @@ public sealed class WorkerPipeSessionTests
// activity timestamp stopped advancing would accumulate staleness past the ceiling and
// fault — and require the beats to have actually flowed while it ran, so an inspection
// that saw a couple of frames and timed out cannot pass for a clean window. None may be a
// WorkerFault while activity is continuously refreshed. The window stays well inside
// FakeRuntimeSession's 5 s dispatch-block ceiling, so the command is still in flight
// throughout.
// WorkerFault while activity is continuously refreshed. Nothing here is racing
// FakeRuntimeSession's blocked-dispatch backstop: that wait is a safety net sized far above
// any window a test opens (and above this test's own cancellation), so the command stays in
// flight for however long a loaded box stretches the loop. Were the two close together, a
// slow run would take the reply mid-window and then fail waiting for a reply already gone
// by — a cancellation at teardown, naming nothing.
TimeSpan inspectionWindow = TimeSpan.FromSeconds(2);
const int minimumFramesInspected = 30;
Stopwatch inspection = Stopwatch.StartNew();
@@ -20,6 +20,17 @@ namespace ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport;
/// </summary>
internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
{
/// <summary>
/// Backstop on the <see cref="BlockDispatch"/> wait so a test that never releases leaves no
/// thread parked forever. It is a safety net, never a scenario's timing budget: nothing
/// asserts on it firing, and a test whose blocked window outruns it silently gets its reply
/// mid-window, which then fails as an opaque cancellation somewhere later. Kept far above
/// any test's window — and above the 20 s cancellation those tests arm — so the test's own
/// token always fails first, with its own message. <see cref="Dispose"/> releases the wait
/// regardless, so teardown never depends on this elapsing.
/// </summary>
private static readonly TimeSpan BlockedDispatchSafetyNet = TimeSpan.FromSeconds(30);
private readonly ManualResetEventSlim releaseDispatch = new(false);
private readonly object gate = new();
private readonly Queue<WorkerEvent> events = new();
@@ -91,7 +102,7 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
if (BlockDispatch)
{
releaseDispatch.Wait(TimeSpan.FromSeconds(5));
releaseDispatch.Wait(BlockedDispatchSafetyNet);
}
SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(