fix(alarms): fetch/poll ceilings; truncation-semantics docs; log-format conformance
This commit is contained in:
@@ -140,6 +140,26 @@ Two viable A.2 designs given the probe data:
|
|||||||
poll period; modest CPU floor because the call is cheap. Matches
|
poll period; modest CPU floor because the call is cheap. Matches
|
||||||
the heartbeat-style WM 0xC275 semantics — AVEVA itself runs a
|
the heartbeat-style WM 0xC275 semantics — AVEVA itself runs a
|
||||||
poll loop internally.
|
poll loop internally.
|
||||||
|
|
||||||
|
As shipped, this is the chosen design, and the cadence is **no
|
||||||
|
longer fixed at 500 ms**: it is the 500 ms *default* of
|
||||||
|
`MxGateway:Alarms:PollIntervalMilliseconds` (range 100 ms – 1 h),
|
||||||
|
which the gateway hands the worker through the
|
||||||
|
`MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable. The
|
||||||
|
per-fetch cap is likewise configurable
|
||||||
|
(`MxGateway:Alarms:MaxAlarmsPerFetch`, default 1024).
|
||||||
|
|
||||||
|
One snapshot rule matters when reading the capture below: a fetch
|
||||||
|
that returns exactly the cap is treated as **truncated**, and the
|
||||||
|
worker *merges* it into the retained snapshot instead of replacing
|
||||||
|
it. `GetXmlCurrentAlarms2` caps its reply with no "more available"
|
||||||
|
flag, so a capped reply is authoritative about presence only —
|
||||||
|
alarms it had no room to mention are retained rather than allowed
|
||||||
|
to vanish, because their disappearance is what the gateway's
|
||||||
|
reconcile pass reads as a clear. Only a sub-cap fetch replaces the
|
||||||
|
snapshot wholesale and can therefore clear alarms. See
|
||||||
|
`docs/DesignDecisions.md`, "Alarms — a capped snapshot fetch never
|
||||||
|
implies a clear".
|
||||||
2. **Hook AVEVA's internal window.** Discover AVEVA's own window
|
2. **Hook AVEVA's internal window.** Discover AVEVA's own window
|
||||||
(`hwnd=0x18032E` in the probe), `SetWindowsHookEx` or
|
(`hwnd=0x18032E` in the probe), `SetWindowsHookEx` or
|
||||||
`SetWindowSubclass` on it, and intercept WM 0xC275 on AVEVA's
|
`SetWindowSubclass` on it, and intercept WM 0xC275 on AVEVA's
|
||||||
|
|||||||
+24
-4
@@ -172,6 +172,19 @@ Consequences, and how this sits with the existing failover/reconcile design:
|
|||||||
happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
|
happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
|
||||||
back. Consumers already apply transitions as "set this alarm to this state",
|
back. Consumers already apply transitions as "set this alarm to this state",
|
||||||
so a repeated or delayed Clear is absorbed.
|
so a repeated or delayed Clear is absorbed.
|
||||||
|
- **Under *sustained* truncation, some intermediate history is lost — end state
|
||||||
|
is not.** For an alarm that stays outside the fetch window, a full
|
||||||
|
clear→re-raise cycle that begins and ends between two sightings emits **no
|
||||||
|
transitions at all**: the retained record is identical before and after, so
|
||||||
|
the diff sees nothing to report. Consumers that render current state are
|
||||||
|
correct; consumers that *count occurrences* lose an event. Likewise, an
|
||||||
|
operator acknowledgement of an out-of-window alarm does not reach the feed
|
||||||
|
until that alarm re-enters a fetch window, at which point the reconcile
|
||||||
|
repairs the acked state. This is a strictly better failure than the
|
||||||
|
pre-guard behaviour (which fabricated a Clear for every out-of-window alarm
|
||||||
|
on every poll), but it is not lossless, and it is another reason a
|
||||||
|
persistently truncating deployment is a configuration defect to fix rather
|
||||||
|
than a mode to run in.
|
||||||
- **It does not synthesize anything.** Suppressing an inference is the opposite
|
- **It does not synthesize anything.** Suppressing an inference is the opposite
|
||||||
of inventing an event; no transition is fabricated on a truncated poll.
|
of inventing an event; no transition is fabricated on a truncated poll.
|
||||||
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which
|
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which
|
||||||
@@ -179,10 +192,17 @@ Consequences, and how this sits with the existing failover/reconcile design:
|
|||||||
consumer's own snapshot bookkeeping and changes neither the failure counting
|
consumer's own snapshot bookkeeping and changes neither the failure counting
|
||||||
that triggers failover nor the subtag standby's snapshot, which is built from
|
that triggers failover nor the subtag standby's snapshot, which is built from
|
||||||
a bounded watch-list and has no per-fetch cap to hit.
|
a bounded watch-list and has no per-fetch cap to hit.
|
||||||
- **Operators get told.** A truncated poll logs a rate-limited (once per
|
- **Operators get told, weakly.** A truncated poll logs a rate-limited (once
|
||||||
minute) `AlarmSnapshotTruncated` warning carrying the cap, the record counts,
|
per minute) `AlarmSnapshotTruncated` warning carrying the cap, the record
|
||||||
and the running truncated-fetch total — identifiers and counts only, never
|
counts, and the running truncated-fetch total — identifiers and counts only,
|
||||||
tag names, values, limits, or comments. A galaxy that truncates persistently
|
never tag names, values, limits, or comments. Be honest about its reach: it
|
||||||
|
goes to the worker's console/stderr, which is captured on dev hosts but is
|
||||||
|
not a metric, not a dashboard tile, and not part of any session-status or
|
||||||
|
alarm-feed payload, so a production deployment can truncate indefinitely
|
||||||
|
without anyone noticing. Surfacing truncation as a **structural** degraded
|
||||||
|
status (a field on the alarm-provider mode/status surface the dashboard and
|
||||||
|
`StreamAlarms` consumers already read) is filed as a follow-up; until it
|
||||||
|
lands, the log line is the only signal. A galaxy that truncates persistently
|
||||||
is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.
|
is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.
|
||||||
|
|
||||||
## Session-Resilience Epic Scope
|
## Session-Resilience Epic Scope
|
||||||
|
|||||||
@@ -418,8 +418,8 @@ behavior.
|
|||||||
| `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. |
|
| `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. |
|
||||||
| `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. |
|
| `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. |
|
||||||
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. |
|
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. |
|
||||||
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be `>= 100`: every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or unusable value leaves the worker on the 500 ms default rather than failing the session. |
|
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. |
|
||||||
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be `>= 64`. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning (identifiers and counts only). Raise this on galaxies whose steady-state active-alarm count approaches the cap — a galaxy permanently above it holds stale entries in the snapshot until a sub-cap poll. Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or unusable value leaves the worker on the 1024 default. |
|
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. |
|
||||||
|
|
||||||
The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
|
The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
|
||||||
`StreamAlarms` are session-less RPCs served by the monitor.
|
`StreamAlarms` are session-less RPCs served by the monitor.
|
||||||
|
|||||||
@@ -49,22 +49,26 @@ public sealed class AlarmsOptions
|
|||||||
/// <summary>
|
/// <summary>
|
||||||
/// Cadence at which the worker's STA polls the AVEVA alarm consumer
|
/// Cadence at which the worker's STA polls the AVEVA alarm consumer
|
||||||
/// (<c>GetXmlCurrentAlarms2</c>) for the current active-alarm snapshot.
|
/// (<c>GetXmlCurrentAlarms2</c>) for the current active-alarm snapshot.
|
||||||
/// Default 500 ms; must be at least 100 ms. Every poll is a COM call
|
/// Default 500 ms; must be between 100 ms and 3,600,000 ms (one hour).
|
||||||
/// plus an XML parse on the STA that also serves reads and writes, so
|
/// Every poll is a COM call plus an XML parse on the STA that also
|
||||||
/// driving it below 100 ms starves the command path. Conveyed to the
|
/// serves reads and writes, so driving it below 100 ms starves the
|
||||||
/// worker through the <c>MXGATEWAY_ALARM_POLL_INTERVAL_MS</c>
|
/// command path; above an hour the cadence stops being a cadence and
|
||||||
/// environment variable.
|
/// silently disables alarm polling. Conveyed to the worker through the
|
||||||
|
/// <c>MXGATEWAY_ALARM_POLL_INTERVAL_MS</c> environment variable.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public int PollIntervalMilliseconds { get; init; } = 500;
|
public int PollIntervalMilliseconds { get; init; } = 500;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Cap the worker passes to <c>GetXmlCurrentAlarms2</c>'s
|
/// Cap the worker passes to <c>GetXmlCurrentAlarms2</c>'s
|
||||||
/// <c>maxAlmCnt</c> argument. Default 1024; must be at least 64. A
|
/// <c>maxAlmCnt</c> argument. Default 1024; must be between 64 and
|
||||||
/// fetch that comes back holding exactly this many records is treated
|
/// 65,536 — the worker is a 32-bit process that materializes each
|
||||||
/// as truncated: the worker keeps the alarms the capped fetch could
|
/// fetch as one BSTR plus a full XmlDocument, so an unbounded cap
|
||||||
/// not mention in its snapshot rather than letting their absence read
|
/// faults the STA rather than merely slowing it. A fetch that comes
|
||||||
/// as a clear. Raise it on galaxies whose steady-state active-alarm
|
/// back holding exactly this many records is treated as truncated: the
|
||||||
/// count approaches the cap. Conveyed to the worker through the
|
/// worker keeps the alarms the capped fetch could not mention in its
|
||||||
|
/// snapshot rather than letting their absence read as a clear. Raise it
|
||||||
|
/// on galaxies whose steady-state active-alarm count approaches the
|
||||||
|
/// cap. Conveyed to the worker through the
|
||||||
/// <c>MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH</c> environment variable.
|
/// <c>MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH</c> environment variable.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public int MaxAlarmsPerFetch { get; init; } = 1024;
|
public int MaxAlarmsPerFetch { get; init; } = 1024;
|
||||||
|
|||||||
@@ -427,23 +427,35 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
|
|||||||
private static readonly string[] ValidAlarmFallbackModes = ["Auto", "ForceAlarmManager", "ForceSubtag"];
|
private static readonly string[] ValidAlarmFallbackModes = ["Auto", "ForceAlarmManager", "ForceSubtag"];
|
||||||
|
|
||||||
private const int MinimumAlarmPollIntervalMilliseconds = 100;
|
private const int MinimumAlarmPollIntervalMilliseconds = 100;
|
||||||
|
|
||||||
|
// One hour. Above this the cadence stops being a cadence: int.MaxValue
|
||||||
|
// milliseconds is ~24 days, which silently disables alarm polling instead
|
||||||
|
// of reporting the misconfiguration.
|
||||||
|
private const int MaximumAlarmPollIntervalMilliseconds = 3_600_000;
|
||||||
|
|
||||||
private const int MinimumMaxAlarmsPerFetch = 64;
|
private const int MinimumMaxAlarmsPerFetch = 64;
|
||||||
|
|
||||||
|
// The worker is a 32-bit process and materializes each fetch as one BSTR
|
||||||
|
// plus a full XmlDocument over it, so an unbounded cap is an out-of-memory
|
||||||
|
// fault on the STA rather than a slow poll.
|
||||||
|
private const int MaximumMaxAlarmsPerFetch = 65_536;
|
||||||
|
|
||||||
private static void ValidateAlarms(AlarmsOptions options, ValidationBuilder builder)
|
private static void ValidateAlarms(AlarmsOptions options, ValidationBuilder builder)
|
||||||
{
|
{
|
||||||
// Validated regardless of Enabled: both values are stamped onto every
|
// Validated regardless of Enabled: both values are stamped onto every
|
||||||
// worker launch environment, so a bad value is a misconfiguration even
|
// worker launch environment, so a bad value is a misconfiguration even
|
||||||
// before the central monitor is switched on.
|
// before the central monitor is switched on.
|
||||||
if (options.PollIntervalMilliseconds < MinimumAlarmPollIntervalMilliseconds)
|
if (options.PollIntervalMilliseconds is < MinimumAlarmPollIntervalMilliseconds
|
||||||
|
or > MaximumAlarmPollIntervalMilliseconds)
|
||||||
{
|
{
|
||||||
builder.Add(
|
builder.Add(
|
||||||
$"MxGateway:Alarms:PollIntervalMilliseconds must be greater than or equal to {MinimumAlarmPollIntervalMilliseconds}.");
|
$"MxGateway:Alarms:PollIntervalMilliseconds must be between {MinimumAlarmPollIntervalMilliseconds} and {MaximumAlarmPollIntervalMilliseconds}.");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (options.MaxAlarmsPerFetch < MinimumMaxAlarmsPerFetch)
|
if (options.MaxAlarmsPerFetch is < MinimumMaxAlarmsPerFetch or > MaximumMaxAlarmsPerFetch)
|
||||||
{
|
{
|
||||||
builder.Add(
|
builder.Add(
|
||||||
$"MxGateway:Alarms:MaxAlarmsPerFetch must be greater than or equal to {MinimumMaxAlarmsPerFetch}.");
|
$"MxGateway:Alarms:MaxAlarmsPerFetch must be between {MinimumMaxAlarmsPerFetch} and {MaximumMaxAlarmsPerFetch}.");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!options.Enabled)
|
if (!options.Enabled)
|
||||||
|
|||||||
@@ -86,7 +86,9 @@
|
|||||||
"Enabled": true,
|
"Enabled": true,
|
||||||
"SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV",
|
"SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV",
|
||||||
"DefaultArea": "",
|
"DefaultArea": "",
|
||||||
"ReconcileIntervalSeconds": 30
|
"ReconcileIntervalSeconds": 30,
|
||||||
|
"PollIntervalMilliseconds": 500,
|
||||||
|
"MaxAlarmsPerFetch": 1024
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -177,9 +177,10 @@ public sealed class GatewayOptionsValidatorTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// A poll cadence below the 100 ms floor must fail validation. Both
|
/// A poll cadence outside the 100 ms – 1 h range must fail validation.
|
||||||
/// values are stamped onto every worker launch environment, so they
|
/// Both values are stamped onto every worker launch environment, so
|
||||||
/// are validated whether or not the central alarm monitor is enabled.
|
/// they are validated whether or not the central alarm monitor is
|
||||||
|
/// enabled.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <param name="pollIntervalMilliseconds">Cadence under test.</param>
|
/// <param name="pollIntervalMilliseconds">Cadence under test.</param>
|
||||||
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
|
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
|
||||||
@@ -188,7 +189,12 @@ public sealed class GatewayOptionsValidatorTests
|
|||||||
[InlineData(0, false)]
|
[InlineData(0, false)]
|
||||||
[InlineData(-1, false)]
|
[InlineData(-1, false)]
|
||||||
[InlineData(99, true)]
|
[InlineData(99, true)]
|
||||||
public void Validate_Fails_WhenAlarmPollIntervalBelowFloor(
|
// Above the one-hour ceiling the cadence stops being a cadence: int.MaxValue
|
||||||
|
// milliseconds is ~24 days, which silently disables alarm polling.
|
||||||
|
[InlineData(3_600_001, false)]
|
||||||
|
[InlineData(int.MaxValue, false)]
|
||||||
|
[InlineData(int.MaxValue, true)]
|
||||||
|
public void Validate_Fails_WhenAlarmPollIntervalOutOfRange(
|
||||||
int pollIntervalMilliseconds,
|
int pollIntervalMilliseconds,
|
||||||
bool alarmsEnabled)
|
bool alarmsEnabled)
|
||||||
{
|
{
|
||||||
@@ -210,9 +216,12 @@ public sealed class GatewayOptionsValidatorTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// A per-fetch cap below the 64-record floor must fail validation. The
|
/// A per-fetch cap outside the 64 – 65,536 range must fail validation.
|
||||||
/// cap doubles as the truncation-detection threshold in the worker, so
|
/// The cap doubles as the truncation-detection threshold in the worker,
|
||||||
/// a tiny cap would make almost every fetch read as truncated.
|
/// so a tiny cap would make almost every fetch read as truncated; and
|
||||||
|
/// the worker is a 32-bit process that materializes each reply as one
|
||||||
|
/// BSTR plus a full XmlDocument, so an unbounded cap is an
|
||||||
|
/// out-of-memory fault on the STA rather than a slow poll.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <param name="maxAlarmsPerFetch">Cap under test.</param>
|
/// <param name="maxAlarmsPerFetch">Cap under test.</param>
|
||||||
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
|
/// <param name="alarmsEnabled">Whether the central alarm monitor is on.</param>
|
||||||
@@ -221,7 +230,10 @@ public sealed class GatewayOptionsValidatorTests
|
|||||||
[InlineData(0, false)]
|
[InlineData(0, false)]
|
||||||
[InlineData(-1, false)]
|
[InlineData(-1, false)]
|
||||||
[InlineData(63, true)]
|
[InlineData(63, true)]
|
||||||
public void Validate_Fails_WhenMaxAlarmsPerFetchBelowFloor(
|
[InlineData(65_537, false)]
|
||||||
|
[InlineData(int.MaxValue, false)]
|
||||||
|
[InlineData(int.MaxValue, true)]
|
||||||
|
public void Validate_Fails_WhenMaxAlarmsPerFetchOutOfRange(
|
||||||
int maxAlarmsPerFetch,
|
int maxAlarmsPerFetch,
|
||||||
bool alarmsEnabled)
|
bool alarmsEnabled)
|
||||||
{
|
{
|
||||||
@@ -242,9 +254,15 @@ public sealed class GatewayOptionsValidatorTests
|
|||||||
f => f.Contains("MxGateway:Alarms:MaxAlarmsPerFetch", StringComparison.Ordinal));
|
f => f.Contains("MxGateway:Alarms:MaxAlarmsPerFetch", StringComparison.Ordinal));
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Verifies the floor values themselves are accepted.</summary>
|
/// <summary>Verifies the boundary values themselves are accepted at both ends.</summary>
|
||||||
[Fact]
|
/// <param name="pollIntervalMilliseconds">Cadence under test.</param>
|
||||||
public void Validate_Succeeds_AtAlarmPollCadenceAndFetchCapFloors()
|
/// <param name="maxAlarmsPerFetch">Cap under test.</param>
|
||||||
|
[Theory]
|
||||||
|
[InlineData(100, 64)] // floors
|
||||||
|
[InlineData(3_600_000, 65_536)] // ceilings
|
||||||
|
public void Validate_Succeeds_AtAlarmPollCadenceAndFetchCapBoundaries(
|
||||||
|
int pollIntervalMilliseconds,
|
||||||
|
int maxAlarmsPerFetch)
|
||||||
{
|
{
|
||||||
GatewayOptions options = CloneWithAlarms(
|
GatewayOptions options = CloneWithAlarms(
|
||||||
ValidOptions(),
|
ValidOptions(),
|
||||||
@@ -252,8 +270,8 @@ public sealed class GatewayOptionsValidatorTests
|
|||||||
{
|
{
|
||||||
Enabled = true,
|
Enabled = true,
|
||||||
DefaultArea = "Galaxy",
|
DefaultArea = "Galaxy",
|
||||||
PollIntervalMilliseconds = 100,
|
PollIntervalMilliseconds = pollIntervalMilliseconds,
|
||||||
MaxAlarmsPerFetch = 64,
|
MaxAlarmsPerFetch = maxAlarmsPerFetch,
|
||||||
});
|
});
|
||||||
|
|
||||||
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, options);
|
ValidateOptionsResult result = new GatewayOptionsValidator().Validate(null, options);
|
||||||
|
|||||||
@@ -57,6 +57,51 @@ public sealed class MxAccessStaSessionTests
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The alarm poll cadence comes from the launcher-set environment
|
||||||
|
/// variable; a missing, unparseable, or out-of-range value must fall
|
||||||
|
/// back to the 500 ms default rather than throw. The ceiling matters as
|
||||||
|
/// much as the floor: <see cref="int.MaxValue"/> milliseconds is ~24
|
||||||
|
/// days, which silently disables alarm polling altogether.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="environmentValue">Raw environment value under test.</param>
|
||||||
|
/// <param name="expectedMilliseconds">Expected resolved cadence, in milliseconds.</param>
|
||||||
|
[Theory]
|
||||||
|
[InlineData(null, 500)]
|
||||||
|
[InlineData("", 500)]
|
||||||
|
[InlineData("not-a-number", 500)]
|
||||||
|
[InlineData("0", 500)]
|
||||||
|
[InlineData("-1", 500)]
|
||||||
|
[InlineData("99", 500)]
|
||||||
|
[InlineData("100", 100)]
|
||||||
|
[InlineData("250", 250)]
|
||||||
|
[InlineData("3600000", 3600000)]
|
||||||
|
[InlineData("3600001", 500)]
|
||||||
|
[InlineData("2147483647", 500)]
|
||||||
|
public void ResolveAlarmPollInterval_WithEnvironmentValue_FallsBackToDefaultWhenOutOfRange(
|
||||||
|
string? environmentValue,
|
||||||
|
int expectedMilliseconds)
|
||||||
|
{
|
||||||
|
string? original = Environment.GetEnvironmentVariable(
|
||||||
|
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
Environment.SetEnvironmentVariable(
|
||||||
|
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName,
|
||||||
|
environmentValue);
|
||||||
|
|
||||||
|
Assert.Equal(
|
||||||
|
TimeSpan.FromMilliseconds(expectedMilliseconds),
|
||||||
|
MxAccessStaSession.ResolveAlarmPollInterval());
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
Environment.SetEnvironmentVariable(
|
||||||
|
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName,
|
||||||
|
original);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Verifies that StartAsync creates the MXAccess COM object and attaches the event sink on the STA thread.
|
/// Verifies that StartAsync creates the MXAccess COM object and attaches the event sink on the STA thread.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
|||||||
@@ -496,6 +496,11 @@ public sealed class WnWrapAlarmConsumerXmlTests
|
|||||||
[InlineData("63", 1024)]
|
[InlineData("63", 1024)]
|
||||||
[InlineData("64", 64)]
|
[InlineData("64", 64)]
|
||||||
[InlineData("4096", 4096)]
|
[InlineData("4096", 4096)]
|
||||||
|
[InlineData("65536", 65536)]
|
||||||
|
// Above the ceiling: the x86 worker materializes the whole reply as one
|
||||||
|
// BSTR plus an XmlDocument, so an unbounded cap is an OOM on the STA.
|
||||||
|
[InlineData("65537", 1024)]
|
||||||
|
[InlineData("2147483647", 1024)]
|
||||||
public void ResolveMaxAlarmsPerFetch_WithEnvironmentValue_FallsBackToDefaultWhenUnusable(
|
public void ResolveMaxAlarmsPerFetch_WithEnvironmentValue_FallsBackToDefaultWhenUnusable(
|
||||||
string? environmentValue,
|
string? environmentValue,
|
||||||
int expected)
|
int expected)
|
||||||
@@ -518,6 +523,32 @@ public sealed class WnWrapAlarmConsumerXmlTests
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A galaxy parked above the cap truncates on every poll, so the
|
||||||
|
/// warning must be throttled: two truncated polls inside one interval
|
||||||
|
/// produce exactly one line, and the next one only after the full
|
||||||
|
/// interval has elapsed.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void ShouldWarnTruncation_ThrottlesConsecutiveTruncatedPollsToOneWarningPerInterval()
|
||||||
|
{
|
||||||
|
// Seeded so the very first truncated poll always warns.
|
||||||
|
const long NeverWarned = -60_000;
|
||||||
|
|
||||||
|
// Poll 1 at t=0: warns, and records t=0 as the last warning.
|
||||||
|
Assert.True(WnWrapAlarmConsumer.ShouldWarnTruncation(0, NeverWarned));
|
||||||
|
|
||||||
|
// Poll 2 half a second later (the default cadence): suppressed.
|
||||||
|
Assert.False(WnWrapAlarmConsumer.ShouldWarnTruncation(500, 0));
|
||||||
|
|
||||||
|
// Still suppressed just shy of the interval...
|
||||||
|
Assert.False(WnWrapAlarmConsumer.ShouldWarnTruncation(59_999, 0));
|
||||||
|
|
||||||
|
// ...and allowed again exactly on it.
|
||||||
|
Assert.True(WnWrapAlarmConsumer.ShouldWarnTruncation(60_000, 0));
|
||||||
|
Assert.True(WnWrapAlarmConsumer.ShouldWarnTruncation(120_000, 60_000));
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Builds a well-formed ALARM_RECORDS payload with
|
/// Builds a well-formed ALARM_RECORDS payload with
|
||||||
/// <paramref name="count"/> distinct alarms. GUIDs are the dashless
|
/// <paramref name="count"/> distinct alarms. GUIDs are the dashless
|
||||||
|
|||||||
@@ -35,6 +35,15 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
/// </summary>
|
/// </summary>
|
||||||
internal static readonly TimeSpan MinimumAlarmPollInterval = TimeSpan.FromMilliseconds(100);
|
internal static readonly TimeSpan MinimumAlarmPollInterval = TimeSpan.FromMilliseconds(100);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Ceiling on the resolved alarm poll cadence (one hour). Mirrors the
|
||||||
|
/// gateway-side MxGateway:Alarms:PollIntervalMilliseconds maximum.
|
||||||
|
/// Without it, int.MaxValue milliseconds — which passed startup
|
||||||
|
/// validation before this existed — silently disables alarm polling
|
||||||
|
/// for ~24 days rather than reporting a misconfiguration.
|
||||||
|
/// </summary>
|
||||||
|
internal static readonly TimeSpan MaximumAlarmPollInterval = TimeSpan.FromMilliseconds(3_600_000);
|
||||||
|
|
||||||
/// <summary>Default alarm poll cadence when the environment says nothing usable.</summary>
|
/// <summary>Default alarm poll cadence when the environment says nothing usable.</summary>
|
||||||
internal static readonly TimeSpan DefaultAlarmPollInterval = TimeSpan.FromMilliseconds(500);
|
internal static readonly TimeSpan DefaultAlarmPollInterval = TimeSpan.FromMilliseconds(500);
|
||||||
|
|
||||||
@@ -235,7 +244,9 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
}
|
}
|
||||||
|
|
||||||
TimeSpan resolved = TimeSpan.FromMilliseconds(milliseconds);
|
TimeSpan resolved = TimeSpan.FromMilliseconds(milliseconds);
|
||||||
return resolved < MinimumAlarmPollInterval ? DefaultAlarmPollInterval : resolved;
|
return resolved < MinimumAlarmPollInterval || resolved > MaximumAlarmPollInterval
|
||||||
|
? DefaultAlarmPollInterval
|
||||||
|
: resolved;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
|
|||||||
@@ -58,10 +58,20 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
|
|||||||
/// </summary>
|
/// </summary>
|
||||||
internal const int MinimumMaxAlarmsPerFetch = 64;
|
internal const int MinimumMaxAlarmsPerFetch = 64;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Ceiling on the resolved per-fetch cap. Mirrors the gateway-side
|
||||||
|
/// <c>MxGateway:Alarms:MaxAlarmsPerFetch</c> maximum. The worker is a
|
||||||
|
/// 32-bit process: every poll materializes the whole reply as one BSTR
|
||||||
|
/// and then a full <see cref="XmlDocument"/> over it, so an unbounded
|
||||||
|
/// cap (<see cref="int.MaxValue"/> passed startup validation before
|
||||||
|
/// this existed) is an out-of-memory fault on the STA, not a slow poll.
|
||||||
|
/// </summary>
|
||||||
|
internal const int MaximumMaxAlarmsPerFetch = 65_536;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Environment variable the gateway's <c>WorkerProcessLauncher</c>
|
/// Environment variable the gateway's <c>WorkerProcessLauncher</c>
|
||||||
/// sets from <c>MxGateway:Alarms:MaxAlarmsPerFetch</c>. A missing,
|
/// sets from <c>MxGateway:Alarms:MaxAlarmsPerFetch</c>. A missing,
|
||||||
/// unparseable, or below-floor value falls back to
|
/// unparseable, or out-of-range value falls back to
|
||||||
/// <see cref="DefaultMaxAlarmsPerFetch"/> — a bad environment value
|
/// <see cref="DefaultMaxAlarmsPerFetch"/> — a bad environment value
|
||||||
/// must never keep the alarm consumer from starting.
|
/// must never keep the alarm consumer from starting.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
@@ -117,10 +127,13 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
|
|||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Resolves the per-fetch cap from the launcher-provided environment
|
/// Resolves the per-fetch cap from the launcher-provided environment
|
||||||
/// variable. A missing, unparseable, or below-floor value falls back
|
/// variable. A missing, unparseable, or out-of-range value falls back
|
||||||
/// to <see cref="DefaultMaxAlarmsPerFetch"/> rather than throwing:
|
/// to <see cref="DefaultMaxAlarmsPerFetch"/> rather than throwing:
|
||||||
/// polling at the default cap is always safe, and a session that
|
/// polling at the default cap is always safe, and a session that
|
||||||
/// refuses to subscribe over a mistyped environment variable is not.
|
/// refuses to subscribe over a mistyped environment variable is not.
|
||||||
|
/// The ceiling is enforced here as well as in the gateway validator so
|
||||||
|
/// a hand-set environment (or a future launcher bug) cannot hand the
|
||||||
|
/// x86 worker a cap that faults it on the first poll.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <returns>The cap passed to <c>GetXmlCurrentAlarms2</c>.</returns>
|
/// <returns>The cap passed to <c>GetXmlCurrentAlarms2</c>.</returns>
|
||||||
internal static int ResolveMaxAlarmsPerFetch()
|
internal static int ResolveMaxAlarmsPerFetch()
|
||||||
@@ -132,6 +145,7 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
|
|||||||
CultureInfo.InvariantCulture,
|
CultureInfo.InvariantCulture,
|
||||||
out int cap)
|
out int cap)
|
||||||
&& cap >= MinimumMaxAlarmsPerFetch
|
&& cap >= MinimumMaxAlarmsPerFetch
|
||||||
|
&& cap <= MaximumMaxAlarmsPerFetch
|
||||||
? cap
|
? cap
|
||||||
: DefaultMaxAlarmsPerFetch;
|
: DefaultMaxAlarmsPerFetch;
|
||||||
}
|
}
|
||||||
@@ -498,7 +512,7 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
|
|||||||
{
|
{
|
||||||
total = ++truncatedFetchCount;
|
total = ++truncatedFetchCount;
|
||||||
elapsed = truncationWarningClock.ElapsedMilliseconds;
|
elapsed = truncationWarningClock.ElapsedMilliseconds;
|
||||||
if (elapsed - lastTruncationWarningMilliseconds < TruncationWarningIntervalMilliseconds)
|
if (!ShouldWarnTruncation(elapsed, lastTruncationWarningMilliseconds))
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
@@ -506,13 +520,24 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
|
|||||||
lastTruncationWarningMilliseconds = elapsed;
|
lastTruncationWarningMilliseconds = elapsed;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Format matches WorkerConsoleLogger's "level=<L> event=<E> k=v k=v"
|
||||||
|
// line shape so the worker's stderr parses uniformly. Deviation, on
|
||||||
|
// purpose: IWorkerLogger exposes only Information and Error, so
|
||||||
|
// "Warning" is a third level string no other worker line emits. A
|
||||||
|
// truncated snapshot is not an error (the poll succeeded and the
|
||||||
|
// snapshot is safe) but it is not routine either, so downgrading it to
|
||||||
|
// Information would bury it. Revisit if the truncation signal becomes
|
||||||
|
// structural — see docs/DesignDecisions.md.
|
||||||
|
//
|
||||||
// Identifiers and counts only — no tag names, values, limits, or
|
// Identifiers and counts only — no tag names, values, limits, or
|
||||||
// comments, per the gateway's "don't log tag values by default" rule.
|
// comments, per the gateway's "don't log tag values by default" rule.
|
||||||
|
// The trailing note stays a single bare k=v token (no spaces, no
|
||||||
|
// semicolons); remediation prose lives in docs/GatewayConfiguration.md.
|
||||||
string message = string.Format(
|
string message = string.Format(
|
||||||
CultureInfo.InvariantCulture,
|
CultureInfo.InvariantCulture,
|
||||||
"level=Warning event=AlarmSnapshotTruncated maxAlarmsPerFetch={0} fetchedRecords={1} "
|
"level=Warning event=AlarmSnapshotTruncated maxAlarmsPerFetch={0} fetchedRecords={1} "
|
||||||
+ "parsedRecords={2} retainedSnapshotSize={3} truncatedFetchesSinceStart={4} "
|
+ "parsedRecords={2} retainedSnapshotSize={3} truncatedFetchesSinceStart={4} "
|
||||||
+ "note=absence-implies-clear suppressed for this poll; raise MxGateway:Alarms:MaxAlarmsPerFetch",
|
+ "note=truncated-snapshot-retained",
|
||||||
maxAlarmsPerFetch,
|
maxAlarmsPerFetch,
|
||||||
fetchedRecordCount,
|
fetchedRecordCount,
|
||||||
parsedRecordCount,
|
parsedRecordCount,
|
||||||
@@ -523,6 +548,25 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
|
|||||||
sink(message);
|
sink(message);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Rate-limit decision for the truncated-fetch warning: emit only when
|
||||||
|
/// a full <see cref="TruncationWarningIntervalMilliseconds"/> has
|
||||||
|
/// elapsed since the last one. A galaxy parked above the cap truncates
|
||||||
|
/// on every poll, so consecutive truncated polls inside one interval
|
||||||
|
/// must produce exactly one line. Exposed as <c>internal static</c>
|
||||||
|
/// so the throttle is unit-testable without the wnwrapConsumer COM
|
||||||
|
/// object.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="elapsedMilliseconds">Consumer-lifetime clock reading for this poll.</param>
|
||||||
|
/// <param name="lastWarningMilliseconds">Clock reading when the last warning was emitted.</param>
|
||||||
|
/// <returns><see langword="true"/> when this poll may emit a warning.</returns>
|
||||||
|
internal static bool ShouldWarnTruncation(
|
||||||
|
long elapsedMilliseconds,
|
||||||
|
long lastWarningMilliseconds)
|
||||||
|
{
|
||||||
|
return elapsedMilliseconds - lastWarningMilliseconds >= TruncationWarningIntervalMilliseconds;
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Pure snapshot-to-transitions diff. Compares the previous polled
|
/// Pure snapshot-to-transitions diff. Compares the previous polled
|
||||||
/// snapshot to the next snapshot and produces one
|
/// snapshot to the next snapshot and produces one
|
||||||
@@ -641,6 +685,13 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
|
|||||||
// first-match-wins (null means "not seen yet"), and a single
|
// first-match-wins (null means "not seen yet"), and a single
|
||||||
// coalesce to string.Empty at the end — which keeps the absent and
|
// coalesce to string.Empty at the end — which keeps the absent and
|
||||||
// empty cases indistinguishable exactly as before.
|
// empty cases indistinguishable exactly as before.
|
||||||
|
//
|
||||||
|
// One theoretical divergence: XmlNode.Name is the QName, so a child
|
||||||
|
// carrying its own default-namespace declaration (<GUID xmlns="...">)
|
||||||
|
// matches here where the unprefixed XPath name test would not. It is
|
||||||
|
// theoretical — wnwrap emits no namespaces, and a namespaced ALARM
|
||||||
|
// wrapper makes the outer SelectNodes return nothing either way — and
|
||||||
|
// it errs permissive (field populated rather than silently dropped).
|
||||||
string? guidHex = null;
|
string? guidHex = null;
|
||||||
string? xmlDate = null;
|
string? xmlDate = null;
|
||||||
string? xmlTime = null;
|
string? xmlTime = null;
|
||||||
|
|||||||
Reference in New Issue
Block a user