fix(alarms): fetch/poll ceilings; truncation-semantics docs; log-format conformance

This commit is contained in:
Joseph Doherty
2026-08-15 17:19:41 -04:00
parent 7c9add3d73
commit b5ea6bb461
11 changed files with 254 additions and 40 deletions
+20
View File
@@ -140,6 +140,26 @@ Two viable A.2 designs given the probe data:
poll period; modest CPU floor because the call is cheap. Matches poll period; modest CPU floor because the call is cheap. Matches
the heartbeat-style WM 0xC275 semantics — AVEVA itself runs a the heartbeat-style WM 0xC275 semantics — AVEVA itself runs a
poll loop internally. poll loop internally.
As shipped, this is the chosen design, and the cadence is **no
longer fixed at 500 ms**: it is the 500 ms *default* of
`MxGateway:Alarms:PollIntervalMilliseconds` (range 100 ms 1 h),
which the gateway hands the worker through the
`MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable. The
per-fetch cap is likewise configurable
(`MxGateway:Alarms:MaxAlarmsPerFetch`, default 1024).
One snapshot rule matters when reading the capture below: a fetch
that returns exactly the cap is treated as **truncated**, and the
worker *merges* it into the retained snapshot instead of replacing
it. `GetXmlCurrentAlarms2` caps its reply with no "more available"
flag, so a capped reply is authoritative about presence only —
alarms it had no room to mention are retained rather than allowed
to vanish, because their disappearance is what the gateway's
reconcile pass reads as a clear. Only a sub-cap fetch replaces the
snapshot wholesale and can therefore clear alarms. See
`docs/DesignDecisions.md`, "Alarms — a capped snapshot fetch never
implies a clear".
2. **Hook AVEVA's internal window.** Discover AVEVA's own window 2. **Hook AVEVA's internal window.** Discover AVEVA's own window
(`hwnd=0x18032E` in the probe), `SetWindowsHookEx` or (`hwnd=0x18032E` in the probe), `SetWindowsHookEx` or
`SetWindowSubclass` on it, and intercept WM 0xC275 on AVEVA's `SetWindowSubclass` on it, and intercept WM 0xC275 on AVEVA's
+24 -4
View File
@@ -172,6 +172,19 @@ Consequences, and how this sits with the existing failover/reconcile design:
happened is broadcast to every `StreamAlarms` subscriber and cannot be taken happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
back. Consumers already apply transitions as "set this alarm to this state", back. Consumers already apply transitions as "set this alarm to this state",
so a repeated or delayed Clear is absorbed. so a repeated or delayed Clear is absorbed.
- **Under *sustained* truncation, some intermediate history is lost — end state
is not.** For an alarm that stays outside the fetch window, a full
clear→re-raise cycle that begins and ends between two sightings emits **no
transitions at all**: the retained record is identical before and after, so
the diff sees nothing to report. Consumers that render current state are
correct; consumers that *count occurrences* lose an event. Likewise, an
operator acknowledgement of an out-of-window alarm does not reach the feed
until that alarm re-enters a fetch window, at which point the reconcile
repairs the acked state. This is a strictly better failure than the
pre-guard behaviour (which fabricated a Clear for every out-of-window alarm
on every poll), but it is not lossless, and it is another reason a
persistently truncating deployment is a configuration defect to fix rather
than a mode to run in.
- **It does not synthesize anything.** Suppressing an inference is the opposite - **It does not synthesize anything.** Suppressing an inference is the opposite
of inventing an event; no transition is fabricated on a truncated poll. of inventing an event; no transition is fabricated on a truncated poll.
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which - **Failover is unaffected.** `FailoverAlarmConsumer` selects which
@@ -179,10 +192,17 @@ Consequences, and how this sits with the existing failover/reconcile design:
consumer's own snapshot bookkeeping and changes neither the failure counting consumer's own snapshot bookkeeping and changes neither the failure counting
that triggers failover nor the subtag standby's snapshot, which is built from that triggers failover nor the subtag standby's snapshot, which is built from
a bounded watch-list and has no per-fetch cap to hit. a bounded watch-list and has no per-fetch cap to hit.
- **Operators get told.** A truncated poll logs a rate-limited (once per - **Operators get told, weakly.** A truncated poll logs a rate-limited (once
minute) `AlarmSnapshotTruncated` warning carrying the cap, the record counts, per minute) `AlarmSnapshotTruncated` warning carrying the cap, the record
and the running truncated-fetch total — identifiers and counts only, never counts, and the running truncated-fetch total — identifiers and counts only,
tag names, values, limits, or comments. A galaxy that truncates persistently never tag names, values, limits, or comments. Be honest about its reach: it
goes to the worker's console/stderr, which is captured on dev hosts but is
not a metric, not a dashboard tile, and not part of any session-status or
alarm-feed payload, so a production deployment can truncate indefinitely
without anyone noticing. Surfacing truncation as a **structural** degraded
status (a field on the alarm-provider mode/status surface the dashboard and
`StreamAlarms` consumers already read) is filed as a follow-up; until it
lands, the log line is the only signal. A galaxy that truncates persistently
is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`. is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.
## Session-Resilience Epic Scope ## Session-Resilience Epic Scope
+2 -2
View File
@@ -418,8 +418,8 @@ behavior.
| `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. | | `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. |
| `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. | | `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. |
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. | | `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. |
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be `>= 100`: every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or unusable value leaves the worker on the 500 ms default rather than failing the session. | | `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. |
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be `>= 64`. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning (identifiers and counts only). Raise this on galaxies whose steady-state active-alarm count approaches the cap — a galaxy permanently above it holds stale entries in the snapshot until a sub-cap poll. Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or unusable value leaves the worker on the 1024 default. | | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. |
The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
`StreamAlarms` are session-less RPCs served by the monitor. `StreamAlarms` are session-less RPCs served by the monitor.
@@ -49,22 +49,26 @@ public sealed class AlarmsOptions
/// <summary> /// <summary>
/// Cadence at which the worker's STA polls the AVEVA alarm consumer /// Cadence at which the worker's STA polls the AVEVA alarm consumer
/// (<c>GetXmlCurrentAlarms2</c>) for the current active-alarm snapshot. /// (<c>GetXmlCurrentAlarms2</c>) for the current active-alarm snapshot.
/// Default 500 ms; must be at least 100 ms. Every poll is a COM call /// Default 500 ms; must be between 100 ms and 3,600,000 ms (one hour).
/// plus an XML parse on the STA that also serves reads and writes, so /// Every poll is a COM call plus an XML parse on the STA that also
/// driving it below 100 ms starves the command path. Conveyed to the /// serves reads and writes, so driving it below 100 ms starves the
/// worker through the <c>MXGATEWAY_ALARM_POLL_INTERVAL_MS</c> /// command path; above an hour the cadence stops being a cadence and
/// environment variable. /// silently disables alarm polling. Conveyed to the worker through the
/// <c>MXGATEWAY_ALARM_POLL_INTERVAL_MS</c> environment variable.
/// </summary> /// </summary>
public int PollIntervalMilliseconds { get; init; } = 500; public int PollIntervalMilliseconds { get; init; } = 500;
/// <summary> /// <summary>
/// Cap the worker passes to <c>GetXmlCurrentAlarms2</c>'s /// Cap the worker passes to <c>GetXmlCurrentAlarms2</c>'s
/// <c>maxAlmCnt</c> argument. Default 1024; must be at least 64. A /// <c>maxAlmCnt</c> argument. Default 1024; must be between 64 and
/// fetch that comes back holding exactly this many records is treated /// 65,536 — the worker is a 32-bit process that materializes each
/// as truncated: the worker keeps the alarms the capped fetch could /// fetch as one BSTR plus a full XmlDocument, so an unbounded cap
/// not mention in its snapshot rather than letting their absence read /// faults the STA rather than merely slowing it. A fetch that comes
/// as a clear. Raise it on galaxies whose steady-state active-alarm /// back holding exactly this many records is treated as truncated: the
/// count approaches the cap. Conveyed to the worker through the /// worker keeps the alarms the capped fetch could not mention in its
/// snapshot rather than letting their absence read as a clear. Raise it
/// on galaxies whose steady-state active-alarm count approaches the
/// cap. Conveyed to the worker through the
/// <c>MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH</c> environment variable. /// <c>MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH</c> environment variable.
/// </summary> /// </summary>
public int MaxAlarmsPerFetch { get; init; } = 1024; public int MaxAlarmsPerFetch { get; init; } = 1024;
@@ -427,23 +427,35 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
private static readonly string[] ValidAlarmFallbackModes = ["Auto", "ForceAlarmManager", "ForceSubtag"]; private static readonly string[] ValidAlarmFallbackModes = ["Auto", "ForceAlarmManager", "ForceSubtag"];
private const int MinimumAlarmPollIntervalMilliseconds = 100; private const int MinimumAlarmPollIntervalMilliseconds = 100;
// One hour. Above this the cadence stops being a cadence: int.MaxValue
// milliseconds is ~24 days, which silently disables alarm polling instead
// of reporting the misconfiguration.
private const int MaximumAlarmPollIntervalMilliseconds = 3_600_000;
private const int MinimumMaxAlarmsPerFetch = 64; private const int MinimumMaxAlarmsPerFetch = 64;
// The worker is a 32-bit process and materializes each fetch as one BSTR
// plus a full XmlDocument over it, so an unbounded cap is an out-of-memory
// fault on the STA rather than a slow poll.
private const int MaximumMaxAlarmsPerFetch = 65_536;
private static void ValidateAlarms(AlarmsOptions options, ValidationBuilder builder) private static void ValidateAlarms(AlarmsOptions options, ValidationBuilder builder)
{ {
// Validated regardless of Enabled: both values are stamped onto every // Validated regardless of Enabled: both values are stamped onto every
// worker launch environment, so a bad value is a misconfiguration even // worker launch environment, so a bad value is a misconfiguration even
// before the central monitor is switched on. // before the central monitor is switched on.
if (options.PollIntervalMilliseconds < MinimumAlarmPollIntervalMilliseconds) if (options.PollIntervalMilliseconds is < MinimumAlarmPollIntervalMilliseconds
or > MaximumAlarmPollIntervalMilliseconds)
{ {
builder.Add( builder.Add(
$"MxGateway:Alarms:PollIntervalMilliseconds must be greater than or equal to {MinimumAlarmPollIntervalMilliseconds}."); $"MxGateway:Alarms:PollIntervalMilliseconds must be between {MinimumAlarmPollIntervalMilliseconds} and {MaximumAlarmPollIntervalMilliseconds}.");
} }
if (options.MaxAlarmsPerFetch < MinimumMaxAlarmsPerFetch) if (options.MaxAlarmsPerFetch is < MinimumMaxAlarmsPerFetch or > MaximumMaxAlarmsPerFetch)
{ {
builder.Add( builder.Add(
$"MxGateway:Alarms:MaxAlarmsPerFetch must be greater than or equal to {MinimumMaxAlarmsPerFetch}."); $"MxGateway:Alarms:MaxAlarmsPerFetch must be between {MinimumMaxAlarmsPerFetch} and {MaximumMaxAlarmsPerFetch}.");
} }
if (!options.Enabled) if (!options.Enabled)
@@ -86,7 +86,9 @@
"Enabled": true, "Enabled": true,
"SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV", "SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV",
"DefaultArea": "", "DefaultArea": "",
"ReconcileIntervalSeconds": 30 "ReconcileIntervalSeconds": 30,
"PollIntervalMilliseconds": 500,
"MaxAlarmsPerFetch": 1024
} }
} }
} }
@@ -177,9 +177,10 @@ public sealed class GatewayOptionsValidatorTests
} }
/// <summary> /// <summary>
/// A poll cadence below the 100 ms floor must fail validation. Both /// A poll cadence outside the 100 ms 1 h range must fail validation.
/// values are stamped onto every worker launch environment, so they /// Both values are stamped onto every worker launch environment, so
/// are validated whether or not the central alarm monitor is enabled. /// they are validated whether or not the central alarm monitor is
/// enabled.
/// </summary> /// </summary>
/// <param name="pollIntervalMilliseconds">Cadence under test.</param> /// <param name="pollIntervalMilliseconds">Cadence under test.</param>
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param> /// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
@@ -188,7 +189,12 @@ public sealed class GatewayOptionsValidatorTests
[InlineData(0, false)] [InlineData(0, false)]
[InlineData(-1, false)] [InlineData(-1, false)]
[InlineData(99, true)] [InlineData(99, true)]
public void Validate_Fails_WhenAlarmPollIntervalBelowFloor( // Above the one-hour ceiling the cadence stops being a cadence: int.MaxValue
// milliseconds is ~24 days, which silently disables alarm polling.
[InlineData(3_600_001, false)]
[InlineData(int.MaxValue, false)]
[InlineData(int.MaxValue, true)]
public void Validate_Fails_WhenAlarmPollIntervalOutOfRange(
int pollIntervalMilliseconds, int pollIntervalMilliseconds,
bool alarmsEnabled) bool alarmsEnabled)
{ {
@@ -210,9 +216,12 @@ public sealed class GatewayOptionsValidatorTests
} }
/// <summary> /// <summary>
/// A per-fetch cap below the 64-record floor must fail validation. The /// A per-fetch cap outside the 64 65,536 range must fail validation.
/// cap doubles as the truncation-detection threshold in the worker, so /// The cap doubles as the truncation-detection threshold in the worker,
/// a tiny cap would make almost every fetch read as truncated. /// so a tiny cap would make almost every fetch read as truncated; and
/// the worker is a 32-bit process that materializes each reply as one
/// BSTR plus a full XmlDocument, so an unbounded cap is an
/// out-of-memory fault on the STA rather than a slow poll.
/// </summary> /// </summary>
/// <param name="maxAlarmsPerFetch">Cap under test.</param> /// <param name="maxAlarmsPerFetch">Cap under test.</param>
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param> /// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
@@ -221,7 +230,10 @@ public sealed class GatewayOptionsValidatorTests
[InlineData(0, false)] [InlineData(0, false)]
[InlineData(-1, false)] [InlineData(-1, false)]
[InlineData(63, true)] [InlineData(63, true)]
public void Validate_Fails_WhenMaxAlarmsPerFetchBelowFloor( [InlineData(65_537, false)]
[InlineData(int.MaxValue, false)]
[InlineData(int.MaxValue, true)]
public void Validate_Fails_WhenMaxAlarmsPerFetchOutOfRange(
int maxAlarmsPerFetch, int maxAlarmsPerFetch,
bool alarmsEnabled) bool alarmsEnabled)
{ {
@@ -242,9 +254,15 @@ public sealed class GatewayOptionsValidatorTests
f => f.Contains("MxGateway:Alarms:MaxAlarmsPerFetch", StringComparison.Ordinal)); f => f.Contains("MxGateway:Alarms:MaxAlarmsPerFetch", StringComparison.Ordinal));
} }
/// <summary>Verifies the floor values themselves are accepted.</summary> /// <summary>Verifies the boundary values themselves are accepted at both ends.</summary>
[Fact] /// <param name="pollIntervalMilliseconds">Cadence under test.</param>
public void Validate_Succeeds_AtAlarmPollCadenceAndFetchCapFloors() /// <param name="maxAlarmsPerFetch">Cap under test.</param>
[Theory]
[InlineData(100, 64)] // floors
[InlineData(3_600_000, 65_536)] // ceilings
public void Validate_Succeeds_AtAlarmPollCadenceAndFetchCapBoundaries(
int pollIntervalMilliseconds,
int maxAlarmsPerFetch)
{ {
GatewayOptions options = CloneWithAlarms( GatewayOptions options = CloneWithAlarms(
ValidOptions(), ValidOptions(),
@@ -252,8 +270,8 @@ public sealed class GatewayOptionsValidatorTests
{ {
Enabled = true, Enabled = true,
DefaultArea = "Galaxy", DefaultArea = "Galaxy",
PollIntervalMilliseconds = 100, PollIntervalMilliseconds = pollIntervalMilliseconds,
MaxAlarmsPerFetch = 64, MaxAlarmsPerFetch = maxAlarmsPerFetch,
}); });
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, options); ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, options);
@@ -57,6 +57,51 @@ public sealed class MxAccessStaSessionTests
} }
} }
/// <summary>
/// The alarm poll cadence comes from the launcher-set environment
/// variable; a missing, unparseable, or out-of-range value must fall
/// back to the 500 ms default rather than throw. The ceiling matters as
/// much as the floor: <see cref="int.MaxValue"/> milliseconds is ~24
/// days, which silently disables alarm polling altogether.
/// </summary>
/// <param name="environmentValue">Raw environment value under test.</param>
/// <param name="expectedMilliseconds">Expected resolved cadence, in milliseconds.</param>
[Theory]
[InlineData(null, 500)]
[InlineData("", 500)]
[InlineData("not-a-number", 500)]
[InlineData("0", 500)]
[InlineData("-1", 500)]
[InlineData("99", 500)]
[InlineData("100", 100)]
[InlineData("250", 250)]
[InlineData("3600000", 3600000)]
[InlineData("3600001", 500)]
[InlineData("2147483647", 500)]
public void ResolveAlarmPollInterval_WithEnvironmentValue_FallsBackToDefaultWhenOutOfRange(
string? environmentValue,
int expectedMilliseconds)
{
string? original = Environment.GetEnvironmentVariable(
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName);
try
{
Environment.SetEnvironmentVariable(
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName,
environmentValue);
Assert.Equal(
TimeSpan.FromMilliseconds(expectedMilliseconds),
MxAccessStaSession.ResolveAlarmPollInterval());
}
finally
{
Environment.SetEnvironmentVariable(
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName,
original);
}
}
/// <summary> /// <summary>
/// Verifies that StartAsync creates the MXAccess COM object and attaches the event sink on the STA thread. /// Verifies that StartAsync creates the MXAccess COM object and attaches the event sink on the STA thread.
/// </summary> /// </summary>
@@ -496,6 +496,11 @@ public sealed class WnWrapAlarmConsumerXmlTests
[InlineData("63", 1024)] [InlineData("63", 1024)]
[InlineData("64", 64)] [InlineData("64", 64)]
[InlineData("4096", 4096)] [InlineData("4096", 4096)]
[InlineData("65536", 65536)]
// Above the ceiling: the x86 worker materializes the whole reply as one
// BSTR plus an XmlDocument, so an unbounded cap is an OOM on the STA.
[InlineData("65537", 1024)]
[InlineData("2147483647", 1024)]
public void ResolveMaxAlarmsPerFetch_WithEnvironmentValue_FallsBackToDefaultWhenUnusable( public void ResolveMaxAlarmsPerFetch_WithEnvironmentValue_FallsBackToDefaultWhenUnusable(
string? environmentValue, string? environmentValue,
int expected) int expected)
@@ -518,6 +523,32 @@ public sealed class WnWrapAlarmConsumerXmlTests
} }
} }
/// <summary>
/// A galaxy parked above the cap truncates on every poll, so the
/// warning must be throttled: two truncated polls inside one interval
/// produce exactly one line, and the next one only after the full
/// interval has elapsed.
/// </summary>
[Fact]
public void ShouldWarnTruncation_ThrottlesConsecutiveTruncatedPollsToOneWarningPerInterval()
{
// Seeded so the very first truncated poll always warns.
const long NeverWarned = -60_000;
// Poll 1 at t=0: warns, and records t=0 as the last warning.
Assert.True(WnWrapAlarmConsumer.ShouldWarnTruncation(0, NeverWarned));
// Poll 2 half a second later (the default cadence): suppressed.
Assert.False(WnWrapAlarmConsumer.ShouldWarnTruncation(500, 0));
// Still suppressed just shy of the interval...
Assert.False(WnWrapAlarmConsumer.ShouldWarnTruncation(59_999, 0));
// ...and allowed again exactly on it.
Assert.True(WnWrapAlarmConsumer.ShouldWarnTruncation(60_000, 0));
Assert.True(WnWrapAlarmConsumer.ShouldWarnTruncation(120_000, 60_000));
}
/// <summary> /// <summary>
/// Builds a well-formed ALARM_RECORDS payload with /// Builds a well-formed ALARM_RECORDS payload with
/// <paramref name="count"/> distinct alarms. GUIDs are the dashless /// <paramref name="count"/> distinct alarms. GUIDs are the dashless
@@ -35,6 +35,15 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
/// </summary> /// </summary>
internal static readonly TimeSpan MinimumAlarmPollInterval = TimeSpan.FromMilliseconds(100); internal static readonly TimeSpan MinimumAlarmPollInterval = TimeSpan.FromMilliseconds(100);
/// <summary>
/// Ceiling on the resolved alarm poll cadence (one hour). Mirrors the
/// gateway-side MxGateway:Alarms:PollIntervalMilliseconds maximum.
/// Without it, int.MaxValue milliseconds — which passed startup
/// validation before this existed — silently disables alarm polling
/// for ~24 days rather than reporting a misconfiguration.
/// </summary>
internal static readonly TimeSpan MaximumAlarmPollInterval = TimeSpan.FromMilliseconds(3_600_000);
/// <summary>Default alarm poll cadence when the environment says nothing usable.</summary> /// <summary>Default alarm poll cadence when the environment says nothing usable.</summary>
internal static readonly TimeSpan DefaultAlarmPollInterval = TimeSpan.FromMilliseconds(500); internal static readonly TimeSpan DefaultAlarmPollInterval = TimeSpan.FromMilliseconds(500);
@@ -235,7 +244,9 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
} }
TimeSpan resolved = TimeSpan.FromMilliseconds(milliseconds); TimeSpan resolved = TimeSpan.FromMilliseconds(milliseconds);
return resolved < MinimumAlarmPollInterval ? DefaultAlarmPollInterval : resolved; return resolved < MinimumAlarmPollInterval || resolved > MaximumAlarmPollInterval
? DefaultAlarmPollInterval
: resolved;
} }
/// <summary> /// <summary>
@@ -58,10 +58,20 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
/// </summary> /// </summary>
internal const int MinimumMaxAlarmsPerFetch = 64; internal const int MinimumMaxAlarmsPerFetch = 64;
/// <summary>
/// Ceiling on the resolved per-fetch cap. Mirrors the gateway-side
/// <c>MxGateway:Alarms:MaxAlarmsPerFetch</c> maximum. The worker is a
/// 32-bit process: every poll materializes the whole reply as one BSTR
/// and then a full <see cref="XmlDocument"/> over it, so an unbounded
/// cap (<see cref="int.MaxValue"/> passed startup validation before
/// this existed) is an out-of-memory fault on the STA, not a slow poll.
/// </summary>
internal const int MaximumMaxAlarmsPerFetch = 65_536;
/// <summary> /// <summary>
/// Environment variable the gateway's <c>WorkerProcessLauncher</c> /// Environment variable the gateway's <c>WorkerProcessLauncher</c>
/// sets from <c>MxGateway:Alarms:MaxAlarmsPerFetch</c>. A missing, /// sets from <c>MxGateway:Alarms:MaxAlarmsPerFetch</c>. A missing,
/// unparseable, or below-floor value falls back to /// unparseable, or out-of-range value falls back to
/// <see cref="DefaultMaxAlarmsPerFetch"/> — a bad environment value /// <see cref="DefaultMaxAlarmsPerFetch"/> — a bad environment value
/// must never keep the alarm consumer from starting. /// must never keep the alarm consumer from starting.
/// </summary> /// </summary>
@@ -117,10 +127,13 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
/// <summary> /// <summary>
/// Resolves the per-fetch cap from the launcher-provided environment /// Resolves the per-fetch cap from the launcher-provided environment
/// variable. A missing, unparseable, or below-floor value falls back /// variable. A missing, unparseable, or out-of-range value falls back
/// to <see cref="DefaultMaxAlarmsPerFetch"/> rather than throwing: /// to <see cref="DefaultMaxAlarmsPerFetch"/> rather than throwing:
/// polling at the default cap is always safe, and a session that /// polling at the default cap is always safe, and a session that
/// refuses to subscribe over a mistyped environment variable is not. /// refuses to subscribe over a mistyped environment variable is not.
/// The ceiling is enforced here as well as in the gateway validator so
/// a hand-set environment (or a future launcher bug) cannot hand the
/// x86 worker a cap that faults it on the first poll.
/// </summary> /// </summary>
/// <returns>The cap passed to <c>GetXmlCurrentAlarms2</c>.</returns> /// <returns>The cap passed to <c>GetXmlCurrentAlarms2</c>.</returns>
internal static int ResolveMaxAlarmsPerFetch() internal static int ResolveMaxAlarmsPerFetch()
@@ -132,6 +145,7 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
CultureInfo.InvariantCulture, CultureInfo.InvariantCulture,
out int cap) out int cap)
&& cap >= MinimumMaxAlarmsPerFetch && cap >= MinimumMaxAlarmsPerFetch
&& cap <= MaximumMaxAlarmsPerFetch
? cap ? cap
: DefaultMaxAlarmsPerFetch; : DefaultMaxAlarmsPerFetch;
} }
@@ -498,7 +512,7 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
{ {
total = ++truncatedFetchCount; total = ++truncatedFetchCount;
elapsed = truncationWarningClock.ElapsedMilliseconds; elapsed = truncationWarningClock.ElapsedMilliseconds;
if (elapsed - lastTruncationWarningMilliseconds < TruncationWarningIntervalMilliseconds) if (!ShouldWarnTruncation(elapsed, lastTruncationWarningMilliseconds))
{ {
return; return;
} }
@@ -506,13 +520,24 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
lastTruncationWarningMilliseconds = elapsed; lastTruncationWarningMilliseconds = elapsed;
} }
// Format matches WorkerConsoleLogger's "level=<L> event=<E> k=v k=v"
// line shape so the worker's stderr parses uniformly. Deviation, on
// purpose: IWorkerLogger exposes only Information and Error, so
// "Warning" is a third level string no other worker line emits. A
// truncated snapshot is not an error (the poll succeeded and the
// snapshot is safe) but it is not routine either, so downgrading it to
// Information would bury it. Revisit if the truncation signal becomes
// structural — see docs/DesignDecisions.md.
//
// Identifiers and counts only — no tag names, values, limits, or // Identifiers and counts only — no tag names, values, limits, or
// comments, per the gateway's "don't log tag values by default" rule. // comments, per the gateway's "don't log tag values by default" rule.
// The trailing note stays a single bare k=v token (no spaces, no
// semicolons); remediation prose lives in docs/GatewayConfiguration.md.
string message = string.Format( string message = string.Format(
CultureInfo.InvariantCulture, CultureInfo.InvariantCulture,
"level=Warning event=AlarmSnapshotTruncated maxAlarmsPerFetch={0} fetchedRecords={1} " "level=Warning event=AlarmSnapshotTruncated maxAlarmsPerFetch={0} fetchedRecords={1} "
+ "parsedRecords={2} retainedSnapshotSize={3} truncatedFetchesSinceStart={4} " + "parsedRecords={2} retainedSnapshotSize={3} truncatedFetchesSinceStart={4} "
+ "note=absence-implies-clear suppressed for this poll; raise MxGateway:Alarms:MaxAlarmsPerFetch", + "note=truncated-snapshot-retained",
maxAlarmsPerFetch, maxAlarmsPerFetch,
fetchedRecordCount, fetchedRecordCount,
parsedRecordCount, parsedRecordCount,
@@ -523,6 +548,25 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
sink(message); sink(message);
} }
/// <summary>
/// Rate-limit decision for the truncated-fetch warning: emit only when
/// a full <see cref="TruncationWarningIntervalMilliseconds"/> has
/// elapsed since the last one. A galaxy parked above the cap truncates
/// on every poll, so consecutive truncated polls inside one interval
/// must produce exactly one line. Exposed as <c>internal static</c>
/// so the throttle is unit-testable without the wnwrapConsumer COM
/// object.
/// </summary>
/// <param name="elapsedMilliseconds">Consumer-lifetime clock reading for this poll.</param>
/// <param name="lastWarningMilliseconds">Clock reading when the last warning was emitted.</param>
/// <returns><see langword="true"/> when this poll may emit a warning.</returns>
internal static bool ShouldWarnTruncation(
long elapsedMilliseconds,
long lastWarningMilliseconds)
{
return elapsedMilliseconds - lastWarningMilliseconds >= TruncationWarningIntervalMilliseconds;
}
/// <summary> /// <summary>
/// Pure snapshot-to-transitions diff. Compares the previous polled /// Pure snapshot-to-transitions diff. Compares the previous polled
/// snapshot to the next snapshot and produces one /// snapshot to the next snapshot and produces one
@@ -641,6 +685,13 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
// first-match-wins (null means "not seen yet"), and a single // first-match-wins (null means "not seen yet"), and a single
// coalesce to string.Empty at the end — which keeps the absent and // coalesce to string.Empty at the end — which keeps the absent and
// empty cases indistinguishable exactly as before. // empty cases indistinguishable exactly as before.
//
// One theoretical divergence: XmlNode.Name is the QName, so a child
// carrying its own default-namespace declaration (<GUID xmlns="...">)
// matches here where the unprefixed XPath name test would not. It is
// theoretical — wnwrap emits no namespaces, and a namespaced ALARM
// wrapper makes the outer SelectNodes return nothing either way — and
// it errs permissive (field populated rather than silently dropped).
string? guidHex = null; string? guidHex = null;
string? xmlDate = null; string? xmlDate = null;
string? xmlTime = null; string? xmlTime = null;