perf(misc): cached hot-path lookups, bounded observer queue, alarm-priority stream path

WP2.6 (arch-review remediation, cross-cutting misc):
- SiteExternalSystemRepository: name/ID-indexed ExternalSystemDefinitionCache replaces
  the fetch-all + reverse-map scan on every by-ID/method lookup; loaded once per
  redeploy, invalidated by DeploymentManagerActor after HandleDeployArtifacts applies
  external-system changes. Static JsonSerializerOptions for method-list parsing.
- Inbound API: short-TTL ApiMethodCache fronts the per-request ApiMethod repository
  fetch; invalidated by name via the existing ScriptArtifactChangeSubscriber/
  IScriptArtifactChangeBus pipeline, self-healing via TTL for changes the bus
  doesn't cover (e.g. Management API edits).
- StoreAndForward: the cached-call audit-observer queue — the one unbounded channel
  left in the system — is now bounded (ObserverQueueCapacity, default 10,000) with
  DropOldest overflow and a dropped-notification counter.
- SiteStreamManager: alarm state changes now travel a dedicated publish
  source/broadcast hub, isolated from the (far higher-volume) attribute path, so an
  attribute storm can no longer evict a pending alarm transition; the alarm hand-off
  queue is bounded with a drop counter surfaced on the site health report
  (SiteStreamAlarmDropCount via the new SiteStreamAlarmDropReporter), and publishing
  is skipped entirely at zero subscribers on either path.
- CLI ManagementHttpClient: explicit 30s HttpClient.Timeout on the shared
  construction (was the 100s framework default), overridable via
  SCADABRIDGE_HTTP_TIMEOUT_SECONDS.

Deviation: the failback-probe heartbeat item is NOT included — its only viable
surface (CentralChannelProvider.cs / heartbeat consumers) lives entirely in the
Communication project, explicitly off-limits to this work package this phase.

Tests: SiteRuntime.Tests (550), InboundAPI.Tests (278), StoreAndForward.Tests (133),
CLI.Tests (390), HealthMonitoring.Tests (97) — all green after full solution build.
This commit is contained in:
Joseph Doherty
2026-08-14 20:59:43 -04:00
parent ee193cd2bb
commit a212283104
32 changed files with 1361 additions and 111 deletions
@@ -178,6 +178,22 @@ public interface ISiteHealthCollector
// Default no-op so test fakes do not need to be updated.
}
/// <summary>
/// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer):
/// replace the latest cumulative count of alarm state changes dropped from
/// <c>SiteStreamManager</c>'s dedicated alarm publish queue, used by the next
/// <see cref="CollectReport"/> call. Refreshed periodically by the site-side
/// <c>SiteStreamAlarmDropReporter</c> hosted service. Point-in-time: NOT reset on
/// <see cref="CollectReport"/>. Default interface implementation is a no-op so
/// existing test fakes continue to compile without per-fake updates.
/// </summary>
/// <param name="count">The cumulative dropped-alarm count from <c>SiteStreamManager.AlarmPublishDroppedCount</c>.</param>
void SetSiteStreamAlarmDropCount(long count)
{
// Default no-op so test fakes do not need to be updated. The real
// SiteHealthCollector overrides this with the Interlocked.Exchange store.
}
/// <summary>
/// Sets the hostname of this node.
/// </summary>
@@ -39,6 +39,10 @@ public class SiteHealthCollector : ISiteHealthCollector
private int _scriptQueueDepth;
private int _scriptBusyThreads;
private long _scriptOldestBusyAgeBits = BitConverter.DoubleToInt64Bits(double.NaN);
// WP2.6d: cumulative alarm-publish-queue drop count, refreshed by
// SiteStreamAlarmDropReporter. Point-in-time (not reset on collect), like the
// scheduler gauges above.
private long _siteStreamAlarmDropCount;
private volatile string _nodeHostname = "";
private volatile IReadOnlyList<Commons.Messages.Health.NodeStatus>? _clusterNodes;
private volatile bool _isActiveNode;
@@ -173,6 +177,12 @@ public class SiteHealthCollector : ISiteHealthCollector
return double.IsNaN(value) ? null : value;
}
/// <inheritdoc />
public void SetSiteStreamAlarmDropCount(long count)
{
Interlocked.Exchange(ref _siteStreamAlarmDropCount, count);
}
/// <inheritdoc />
public void SetNodeHostname(string hostname) => _nodeHostname = hostname;
@@ -277,7 +287,8 @@ public class SiteHealthCollector : ISiteHealthCollector
// not run) leaves both report fields null — "no data", not "disconnected
// with an empty backlog".
LocalDbReplicationConnected = localDbReplication?.Item1,
LocalDbOplogBacklog = localDbReplication?.Item2
LocalDbOplogBacklog = localDbReplication?.Item2,
SiteStreamAlarmDropCount = Interlocked.Read(ref _siteStreamAlarmDropCount)
};
}
}