perf(misc): cached hot-path lookups, bounded observer queue, alarm-priority stream path

WP2.6 (arch-review remediation, cross-cutting misc):
- SiteExternalSystemRepository: name/ID-indexed ExternalSystemDefinitionCache replaces
  the fetch-all + reverse-map scan on every by-ID/method lookup; loaded once per
  redeploy, invalidated by DeploymentManagerActor after HandleDeployArtifacts applies
  external-system changes. Static JsonSerializerOptions for method-list parsing.
- Inbound API: short-TTL ApiMethodCache fronts the per-request ApiMethod repository
  fetch; invalidated by name via the existing ScriptArtifactChangeSubscriber/
  IScriptArtifactChangeBus pipeline, self-healing via TTL for changes the bus
  doesn't cover (e.g. Management API edits).
- StoreAndForward: the cached-call audit-observer queue — the one unbounded channel
  left in the system — is now bounded (ObserverQueueCapacity, default 10,000) with
  DropOldest overflow and a dropped-notification counter.
- SiteStreamManager: alarm state changes now travel a dedicated publish
  source/broadcast hub, isolated from the (far higher-volume) attribute path, so an
  attribute storm can no longer evict a pending alarm transition; the alarm hand-off
  queue is bounded with a drop counter surfaced on the site health report
  (SiteStreamAlarmDropCount via the new SiteStreamAlarmDropReporter), and publishing
  is skipped entirely at zero subscribers on either path.
- CLI ManagementHttpClient: explicit 30s HttpClient.Timeout on the shared
  construction (was the 100s framework default), overridable via
  SCADABRIDGE_HTTP_TIMEOUT_SECONDS.

Deviation: the failback-probe heartbeat item is NOT included — its only viable
surface (CentralChannelProvider.cs / heartbeat consumers) lives entirely in the
Communication project, explicitly off-limits to this work package this phase.

Tests: SiteRuntime.Tests (550), InboundAPI.Tests (278), StoreAndForward.Tests (133),
CLI.Tests (390), HealthMonitoring.Tests (97) — all green after full solution build.
This commit is contained in:
Joseph Doherty
2026-08-14 20:59:43 -04:00
parent ee193cd2bb
commit a212283104
32 changed files with 1361 additions and 111 deletions
@@ -0,0 +1,85 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
/// <summary>
/// WP2.6b (arch-review misc — Inbound API per-request SQL): a short-TTL cache of
/// resolved <c>ApiMethod</c> rows, sitting in front of <c>IInboundApiRepository.GetMethodByNameAsync</c>
/// so a repeated <c>POST /api/{methodName}</c> burst for the same method does not each pay a
/// database round-trip.
/// <para>
/// Two invalidation paths, by design:
/// </para>
/// <list type="bullet">
/// <item><description><b>Fast path.</b> <see cref="ScriptArtifactChangeSubscriber"/> calls
/// <see cref="Invalidate"/> for every name in an <c>ApiMethod</c>
/// <c>ScriptArtifactsChanged</c> notification (bundle-import changes today).</description></item>
/// <item><description><b>Fallback.</b> Every entry — including a negative (not-found) result —
/// expires after <see cref="InboundApiOptions.ApiMethodCacheTtl"/> regardless of whether an
/// invalidation notification ever arrives. This is what keeps an operator edit via the
/// Management API/UI (which does not publish to the change bus) correct within a bounded
/// window, and is also the correctness backstop for the bus's at-least-once/in-process-only
/// contract — a missed or duplicate notification is harmless.</description></item>
/// </list>
/// <para>
/// A negative (method not found) result is cached too — bounded by the same short TTL — so a
/// caller with a provisioned-but-nonexistent method scope (a real misconfiguration case, not an
/// enumeration probe: <c>EndpointExtensions</c> only reaches this cache after the in-memory
/// scope check already passed) does not repeatedly pay a miss query.
/// </para>
/// </summary>
public sealed class ApiMethodCache
{
private sealed record Entry(ApiMethod? Method, DateTimeOffset ExpiresAtUtc);
private readonly ConcurrentDictionary<string, Entry> _entries = new(StringComparer.Ordinal);
private readonly TimeSpan _ttl;
private readonly TimeProvider _timeProvider;
/// <summary>Initializes the cache.</summary>
/// <param name="ttl">How long a resolved (or not-found) entry stays valid.</param>
/// <param name="timeProvider">Optional time source; defaults to <see cref="TimeProvider.System"/>.</param>
public ApiMethodCache(TimeSpan ttl, TimeProvider? timeProvider = null)
{
if (ttl <= TimeSpan.Zero)
throw new ArgumentOutOfRangeException(nameof(ttl), ttl, "ApiMethodCache TTL must be positive.");
_ttl = ttl;
_timeProvider = timeProvider ?? TimeProvider.System;
}
/// <summary>
/// Returns the cached entry for <paramref name="methodName"/> if it has not expired;
/// otherwise invokes <paramref name="fetch"/>, caches the result (found or not-found), and
/// returns it. Concurrent callers on a cache miss for the SAME name may each invoke
/// <paramref name="fetch"/> once (last write wins) — an acceptable, rare race in exchange
/// for not serializing every miss behind a lock; the point of the cache is to absorb
/// repeated hits, not to fully dedupe a first-request thundering herd.
/// </summary>
/// <param name="methodName">The method name to resolve.</param>
/// <param name="fetch">Loads the current row from the repository on a cache miss.</param>
/// <param name="cancellationToken">Cancellation token for the fetch.</param>
public async Task<ApiMethod?> GetOrFetchAsync(
string methodName,
Func<CancellationToken, Task<ApiMethod?>> fetch,
CancellationToken cancellationToken)
{
var now = _timeProvider.GetUtcNow();
if (_entries.TryGetValue(methodName, out var cached) && cached.ExpiresAtUtc > now)
{
return cached.Method;
}
var method = await fetch(cancellationToken).ConfigureAwait(false);
_entries[methodName] = new Entry(method, now + _ttl);
return method;
}
/// <summary>Drops the cached entry for one method name. Safe to call for an unknown name.</summary>
/// <param name="methodName">The method name to invalidate.</param>
public void Invalidate(string methodName) => _entries.TryRemove(methodName, out _);
/// <summary>Diagnostic helper — the number of currently-cached entries (including expired-but-not-evicted ones).</summary>
internal int Count => _entries.Count;
}
@@ -91,6 +91,7 @@ public static class EndpointExtensions
var logger = httpContext.RequestServices.GetRequiredService<ILogger<InboundApiEndpoint>>();
var verifier = httpContext.RequestServices.GetRequiredService<IApiKeyVerifier>();
var repository = httpContext.RequestServices.GetRequiredService<IInboundApiRepository>();
var methodCache = httpContext.RequestServices.GetRequiredService<ApiMethodCache>();
var executor = httpContext.RequestServices.GetRequiredService<InboundScriptExecutor>();
var routeHelper = httpContext.RequestServices.GetRequiredService<RouteHelper>();
var options = httpContext.RequestServices.GetRequiredService<IOptions<InboundApiOptions>>().Value;
@@ -150,9 +151,17 @@ public static class EndpointExtensions
// "Echo" does not grant "echo". This is the intended invariant: method
// names are case-sensitive identifiers, and a key's granted scopes must be
// provisioned with the exact casing of the methods they authorize.
// WP2.6b: the per-request repository round-trip is fronted by a short-TTL
// ApiMethodCache (invalidated by name on an ApiMethod ScriptArtifactsChanged
// notification, and self-healing via TTL expiry otherwise — see ApiMethodCache
// and ScriptArtifactChangeSubscriber). Only reached after the in-scope check
// passes, preserving the existing enumeration-safety ordering below.
var inScope = identity.Scopes.Contains(methodName);
var method = inScope
? await repository.GetMethodByNameAsync(methodName, httpContext.RequestAborted)
? await methodCache.GetOrFetchAsync(
methodName,
ct => repository.GetMethodByNameAsync(methodName, ct),
httpContext.RequestAborted)
: null;
if (method == null || !inScope)
@@ -34,4 +34,18 @@ public class InboundApiOptions
/// </para>
/// </summary>
public string ApiKeyPepper { get; set; } = string.Empty;
/// <summary>
/// WP2.6b (arch-review misc — Inbound API per-request SQL): how long a resolved
/// <c>ApiMethod</c> row (found OR not-found) is served from <see cref="ApiMethodCache"/>
/// before the next request re-fetches it from the repository. The
/// <c>ScriptArtifactChangeSubscriber</c>/<c>IScriptArtifactChangeBus</c> pipeline
/// invalidates a specific method's entry immediately on a bundle-import change (the fast
/// path); this TTL is the self-healing fallback for changes the bus does not cover (e.g.
/// an operator edit via the Management API/UI, which does not publish a bus notification),
/// bounding staleness without requiring every mutation path to remember to invalidate.
/// Short by design — the whole point is to absorb repeated-request-storm SQL, not to
/// serve minutes-old method definitions.
/// </summary>
public TimeSpan ApiMethodCacheTtl { get; set; } = TimeSpan.FromSeconds(5);
}
@@ -27,5 +27,9 @@ public sealed class InboundApiOptionsValidator : OptionsValidatorBase<InboundApi
builder.RequireThat(options.MaxRequestBodyBytes > 0,
$"ScadaBridge:InboundApi:MaxRequestBodyBytes must be greater than zero " +
$"(was {options.MaxRequestBodyBytes}); it caps the accepted request body before buffering.");
builder.RequireThat(options.ApiMethodCacheTtl > TimeSpan.Zero,
$"ScadaBridge:InboundApi:ApiMethodCacheTtl must be a positive duration " +
$"(was {options.ApiMethodCacheTtl}); it bounds how long a resolved ApiMethod row is cached.");
}
}
@@ -27,20 +27,27 @@ namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
public sealed class ScriptArtifactChangeSubscriber : IHostedService
{
private readonly InboundScriptExecutor _executor;
private readonly ApiMethodCache _methodCache;
private readonly ILogger<ScriptArtifactChangeSubscriber> _logger;
private readonly IScriptArtifactChangeBus? _bus;
private IDisposable? _subscription;
/// <summary>Initializes the subscriber.</summary>
/// <param name="executor">The compiled-handler cache to invalidate.</param>
/// <param name="methodCache">
/// WP2.6b: the short-TTL resolved-<c>ApiMethod</c>-row cache to invalidate alongside
/// the compiled handler.
/// </param>
/// <param name="logger">Logger instance.</param>
/// <param name="bus">The change bus, or null when the host registers none (site roles, tests).</param>
public ScriptArtifactChangeSubscriber(
InboundScriptExecutor executor,
ApiMethodCache methodCache,
ILogger<ScriptArtifactChangeSubscriber> logger,
IScriptArtifactChangeBus? bus = null)
{
_executor = executor;
_methodCache = methodCache;
_logger = logger;
_bus = bus;
}
@@ -78,6 +85,7 @@ public sealed class ScriptArtifactChangeSubscriber : IHostedService
foreach (var name in notification.Names)
{
_executor.InvalidateMethod(name);
_methodCache.Invalidate(name);
}
_logger.LogInformation(
@@ -1,4 +1,5 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
@@ -19,6 +20,13 @@ public static class ServiceCollectionExtensions
services.AddSingleton<InboundScriptExecutor>();
services.AddScoped<RouteHelper>();
// WP2.6b: short-TTL cache of resolved ApiMethod rows, sitting in front of the
// per-request repository fetch in EndpointExtensions. Invalidated by name via
// ScriptArtifactChangeSubscriber (below) and self-heals via TTL expiry for
// changes the bus does not cover.
services.AddSingleton(sp => new ApiMethodCache(
sp.GetRequiredService<IOptions<InboundApiOptions>>().Value.ApiMethodCacheTtl));
// Routed calls go through the IInstanceRouter seam; the
// production implementation delegates to CommunicationService.
services.AddScoped<IInstanceRouter, CommunicationServiceInstanceRouter>();