e606b24a55
The site's deployed_configurations rows are the durable truth about what should be
running; the set of live Instance Actors is what IS running. Nothing kept the two in
agreement after boot, and nothing surfaced a disagreement.
Row 78 (availability, proven live 2026-08-15) -- deployed_configurations is one of
LocalDb's ten replicated tables, so on a replicated pair a config can appear in a
node's file with no local actor involved at all: the CDC sync applier writes the row
directly. Live sequence: site-a-b's reconcile fetched 4 configs at its solo boot; the
pair was then restarted with a fresh site-a-a file and snapshot replication seeded
site-a-a's table within ~5s -- AFTER its boot-time load had read an empty table, and
invisibly to its own reconcile (central rightly reported "0 fetched", because central's
view of the site was correct). Enabled instances sat dead with their configs present
until yet another restart. Row 67's EnsureInstanceStartedCommand only covers configs
the reconcile itself FETCHES; this is the same gap through a side door.
Row 72 (silent divergence) -- for a STALE ReconcileGapItem the pass writes the newer
config via StoreDeployedConfigIfNewerAsync, a pure file write, while the running
Instance Actor keeps executing the previously compiled config until the next restart
or redeploy. Nothing said "stored revision != running revision".
Both are closed by one periodic sweep owned by the Deployment Manager singleton
(ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0;
0 disables). Each tick re-reads the durable rows off-thread and, on the actor thread,
for every ENABLED stored config either starts a missing Instance Actor (row 78) or --
when the running one's revision no longer matches the stored one -- re-applies the
stored config through the ordinary redeploy path (row 72).
A sweep was chosen over a LocalDb change hook deliberately: a hook covers only the
replication arrival path, and only on the half of the fleet running with replication
enabled, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot
snapshot resync, a hand-restored file, any future path). It is also not available --
ILocalDb 0.3.0 exposes no change-notification or applied-rows callback, so a hook would
mean a new upstream capability in scadaproj plus an in-process bus into this singleton,
a cross-repo change to close a site-local gap.
Row 72's decision is to CONVERGE, not merely warn. The stored row is what central
successfully deployed; an actor serving the config central believes was replaced is the
actual fault, and a surface-only warning leaves the operator to issue by hand exactly
the redeploy the sweep would have performed. Reusing HandleDeploy means the compile
gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm
reset and all-or-nothing-per-instance are the production ones, not a second
implementation -- and a stop still does NOT cancel an in-flight script run, exactly as
for a central redeploy.
Guards, shared with row 67 via the extracted InstanceIsOwnedByADeployPipeline predicate
and the shared StartStoredInstance helper:
- a DISABLED instance is never started (config row, no actor is its correct state);
- the sweep defers to any deploy pipeline owning the instance (compile warm in flight,
predecessor still terminating, deployment awaiting its persistence/init join), so it
can never interleave with a real redeploy -- that window has the config stored and no
actor, which an unguarded sweep would answer by creating a child whose actor name is
still registered;
- inert until boot-time Instance Actor creation has finished, so a tick between
staggered batches does not "heal" instances the boot load is about to create;
- ONE repair attempt per (instance, revision) pair -- this is what stops a periodic
self-heal becoming a restart loop for an actor that dies during init, or for a stored
config the compile gate rejects (a rejection leaves the running instance untouched:
the gate returns before anything is stopped);
- reads never overlap, so a slow read cannot stack ticks into duplicate repairs.
A sweep that finds nothing is completely silent -- no log above Trace, no site event.
A repair emits the same Warning `deployment` site-event pattern as row 67 with the
trigger named ("deployment-manager sweep self-heal") so it is distinguishable from a
startup-reconcile heal; a stale-revision restart names BOTH revisions and a rejected
one adds an Error event. The synthetic redeploy replies to the singleton itself rather
than NoSender, because dead letters are a monitored site health metric.
CreateInstanceActor gains a revisionHash parameter and is the single creation funnel
that records it, so the stale check covers every path that can start an Instance Actor;
a missing entry reads as "unknown", never "stale", so a future path bypassing the funnel
degrades to no-op rather than to spurious restarts.
Fail-before by behavioral revert (sweep timer left unarmed): 5 of 10 new tests fail,
including both live symptoms -- the row-78 repro ("A replication-seeded config for an
enabled instance left no running Instance Actor") and the row-72 repro, whose snapshot
still shows the rev-1 value 111 while the store holds rev-2's 222. The 5 negative
guards (disabled / in-flight redeploy / matching revisions / silent-when-healthy /
interval-zero) are green in both states, as they must be.
SiteRuntime 626/626, Host 505/505, AuditLog 383/383.
Design note: docs/plans/2026-08-15-instance-sweep-self-heal-design.md
206 lines
11 KiB
C#
206 lines
11 KiB
C#
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime;
|
|
|
|
/// <summary>
|
|
/// Configuration options for the Site Runtime component.
|
|
/// Bound from ScadaBridge:SiteRuntime configuration section.
|
|
/// </summary>
|
|
public class SiteRuntimeOptions
|
|
{
|
|
/// <summary>
|
|
/// Number of Instance Actors to create per batch during staggered startup.
|
|
/// Default: 20.
|
|
/// </summary>
|
|
public int StartupBatchSize { get; set; } = 20;
|
|
|
|
/// <summary>
|
|
/// Delay in milliseconds between startup batches to prevent reconnection storms.
|
|
/// Default: 100ms.
|
|
/// </summary>
|
|
public int StartupBatchDelayMs { get; set; } = 100;
|
|
|
|
/// <summary>
|
|
/// Maximum call depth for recursive script calls (CallScript/CallShared).
|
|
/// Default: 10.
|
|
/// </summary>
|
|
public int MaxScriptCallDepth { get; set; } = 10;
|
|
|
|
/// <summary>
|
|
/// Default script execution timeout in seconds.
|
|
/// Default: 30 seconds.
|
|
/// </summary>
|
|
public int ScriptExecutionTimeoutSeconds { get; set; } = 30;
|
|
|
|
/// <summary>
|
|
/// Per-subscriber buffer size for the site-wide Akka stream.
|
|
/// Slow subscribers drop oldest messages when buffer is full.
|
|
/// Default: 1000.
|
|
/// </summary>
|
|
public int StreamBufferSize { get; set; } = 1000;
|
|
|
|
/// <summary>
|
|
/// FLOOR for the number of dedicated threads in the script-execution scheduler.
|
|
/// Script and alarm on-trigger bodies run on these threads instead of the shared
|
|
/// .NET thread pool, so blocking script I/O cannot starve the global pool.
|
|
///
|
|
/// <para>WP3.1: this was a fixed size and is now the lower bound of a grow-only,
|
|
/// instance-scaled pool — see
|
|
/// <see cref="Scripts.ScriptExecutionScheduler.ComputeTargetThreads"/>. Existing
|
|
/// configurations keep exactly the previous behaviour at or below
|
|
/// <c>ScriptExecutionThreadCount * InstancesPerScriptThread</c> deployed instances.</para>
|
|
///
|
|
/// Default: 8.
|
|
/// </summary>
|
|
public int ScriptExecutionThreadCount { get; set; } = 8;
|
|
|
|
/// <summary>
|
|
/// WP3.1: CEILING for the instance-scaled script-execution pool. The pool grows
|
|
/// towards <c>ceil(enabledInstances / 8)</c> threads but never past this value; beyond
|
|
/// it, <see cref="MaxConcurrentRunsPerScript"/> is the real regulator. A 1 MB-stack
|
|
/// dedicated thread is cheap, so 32 (≈ 256 instances at the /8 ratio) is a generous
|
|
/// default. Must be greater than or equal to <see cref="ScriptExecutionThreadCount"/>.
|
|
/// Default: 32.
|
|
/// </summary>
|
|
public int ScriptExecutionMaxThreadCount { get; set; } = 32;
|
|
|
|
/// <summary>
|
|
/// WP3.1 (finding #4): maximum number of trigger-expression evaluations allowed to run
|
|
/// concurrently across the whole process. Trigger expressions are non-blocking by
|
|
/// construction (<see cref="Scripts.TriggerExpressionGlobals"/> exposes only reads over an
|
|
/// in-memory snapshot, and the script trust gate has already denied I/O, network, and
|
|
/// threading), so they run as plain async work on the shared .NET thread pool behind this
|
|
/// gate rather than on the blocking script-execution pool. That is what keeps an alarm's
|
|
/// Expression trigger from queueing behind blocking script bodies.
|
|
/// Default: <c>max(2, Environment.ProcessorCount)</c>.
|
|
/// </summary>
|
|
public int TriggerEvalMaxConcurrency { get; set; } = Math.Max(2, Environment.ProcessorCount);
|
|
|
|
/// <summary>
|
|
/// WP3.1: timeout (seconds) for a single trigger-expression evaluation. Previously
|
|
/// hardcoded at 2 s in both <c>ScriptActor</c> and <c>AlarmActor</c>. The deadline is now
|
|
/// armed when the evaluation is ENQUEUED, not when it is dequeued, so time spent waiting
|
|
/// on <see cref="TriggerEvalMaxConcurrency"/> burns the same budget — a saturated gate
|
|
/// produces a timely "false" instead of an unbounded stall.
|
|
/// Default: 2.
|
|
/// </summary>
|
|
public int TriggerEvalTimeoutSeconds { get; set; } = 2;
|
|
|
|
/// <summary>
|
|
/// WP3.1: maximum number of concurrent runs (queued or executing) for any single script
|
|
/// or alarm on-trigger script. A trigger that arrives while the cap is reached is SHED —
|
|
/// the newest run is refused, the four already in flight (which are closest to their own
|
|
/// deadlines) are kept, and no reordering occurs. A shed increments
|
|
/// <c>ISiteHealthCollector.IncrementScriptRunShed</c>, emits a rate-limited Warning site
|
|
/// event, and — for an Ask-based <c>CallScript</c> — replies with an explicit error rather
|
|
/// than letting the caller hang to its Ask timeout.
|
|
/// Default: 4.
|
|
/// </summary>
|
|
public int MaxConcurrentRunsPerScript { get; set; } = 4;
|
|
|
|
/// <summary>
|
|
/// Max mirrored native alarms retained per source binding before older entries are dropped (logged).
|
|
/// Default: 1000.
|
|
/// </summary>
|
|
public int MirroredAlarmCapPerSource { get; set; } = 1000;
|
|
|
|
/// <summary>
|
|
/// Interval in milliseconds to retry a failed native alarm subscription.
|
|
/// Default: 5000ms.
|
|
/// </summary>
|
|
public int NativeAlarmRetryIntervalMs { get; set; } = 5000;
|
|
|
|
/// <summary>HTTP timeout (seconds) for fetching a deployment config from central (notify-and-fetch).</summary>
|
|
public int ConfigFetchTimeoutSeconds { get; set; } = 30;
|
|
|
|
/// <summary>
|
|
/// Fixed interval (ms) at which an Instance Actor re-sends a tag-subscribe
|
|
/// request that either failed or whose response was lost (S4/UA6). The retry is
|
|
/// armed on every send and cancelled on the first Success reply, so it closes
|
|
/// both the failed-response and the lost-response gaps. Default: 5000ms.
|
|
/// </summary>
|
|
public int TagSubscribeRetryIntervalMs { get; set; } = 5000;
|
|
|
|
/// <summary>
|
|
/// Grace period (ms) after a script's execution timeout elapses before the
|
|
/// stuck-script watchdog declares the script's dedicated thread blocked and
|
|
/// emits a loud site event naming it (S2). The CTS timeout only requests
|
|
/// cooperative cancellation; a script blocked in synchronous I/O never
|
|
/// observes it, so this watchdog is the only operator signal that one of the
|
|
/// bounded script-execution threads is gone. Default: 30000ms.
|
|
/// </summary>
|
|
public int StuckScriptGraceMs { get; set; } = 30000;
|
|
|
|
/// <summary>
|
|
/// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer):
|
|
/// capacity of the bounded hand-off queue feeding <see cref="Streaming.SiteStreamManager"/>'s
|
|
/// dedicated alarm publish path. Alarm state changes are staged through this
|
|
/// queue into their OWN broadcast source — separate from the (far higher-volume)
|
|
/// attribute publish source — so an attribute storm cannot evict a pending alarm
|
|
/// transition. DropOldest overflow; drops are counted (surfaced on the site health
|
|
/// report) and logged. Default: 2000 (alarm volume is inherently far lower than
|
|
/// attribute volume, so this rarely needs to be anywhere near
|
|
/// <see cref="StreamBufferSize"/>).
|
|
/// </summary>
|
|
public int AlarmPublishQueueCapacity { get; set; } = 2000;
|
|
|
|
/// <summary>
|
|
/// WP3.2 (site_events volume policy — docs/plans/2026-08-15-site-events-policy-design.md):
|
|
/// global switch restoring the legacy per-run Started/Completed Info site events for
|
|
/// EVERY instance script run. Off by default: those two rows per run were the dominant
|
|
/// <c>site_events</c> writer (~12 physical row-writes across a replicated pair per run,
|
|
/// counting CDC oplog + row-version rows and their eventual purge tombstones), for
|
|
/// content nobody looks at unless something went wrong. With this off, an interval
|
|
/// aggregate (<see cref="Scripts.ScriptRunSummaryRecorder"/>) replaces the per-run rows
|
|
/// and script Error events (timeout/failure/stuck-watchdog) are unaffected — they are
|
|
/// unconditional regardless of this switch. Read via
|
|
/// <see cref="Microsoft.Extensions.Options.IOptionsMonitor{TOptions}"/> at the top of
|
|
/// every run, so it is hot-togglable (no restart, no redeploy). Default: false.
|
|
/// </summary>
|
|
public bool PerRunScriptEvents { get; set; }
|
|
|
|
/// <summary>
|
|
/// WP3.2: per-script opt-in restoring the legacy per-run Started/Completed Info site
|
|
/// events for NAMED scripts only, for live debugging without flipping the global
|
|
/// <see cref="PerRunScriptEvents"/> switch (and therefore without paying its volume
|
|
/// cost for every other script). Effective only while the global switch is off.
|
|
/// Entries are <c>"{InstanceName}/{ScriptName}"</c> for an exact match, or
|
|
/// <c>"{InstanceName}/*"</c> to opt in every script on that instance. Matched by
|
|
/// <see cref="Scripts.ScriptRunEventPolicy.ShouldEmitPerRun"/>, read from
|
|
/// <see cref="Microsoft.Extensions.Options.IOptionsMonitor{TOptions}"/> per run — hot
|
|
/// -togglable like <see cref="PerRunScriptEvents"/>. Default: empty (no opt-ins).
|
|
/// </summary>
|
|
public List<string> PerRunScriptEventScripts { get; set; } = new();
|
|
|
|
/// <summary>
|
|
/// WP3.2: flush interval (seconds) for <see cref="Scripts.ScriptRunSummaryRecorder"/>'s
|
|
/// interval aggregate — one "script" Info site event per interval summarizing every
|
|
/// instance script run since the last flush (headline totals + a top-50-scripts
|
|
/// breakdown, capped so a pathological deployment cannot mint an oversized row). An
|
|
/// interval with zero activity emits no row. <c>0</c> disables the summary flush
|
|
/// entirely (the recorder still accumulates counters; they are just never emitted or
|
|
/// reset). Default: 300 (5 minutes) — at most 288 summary rows/day, versus 2 rows per
|
|
/// script run under the legacy per-run behaviour.
|
|
/// </summary>
|
|
public int ScriptRunSummaryIntervalSeconds { get; set; } = 300;
|
|
|
|
/// <summary>
|
|
/// Interval (seconds) of the Deployment Manager singleton's <b>instance reconcile sweep</b>
|
|
/// (deferred register rows 78 + 72). Every tick the singleton re-reads the durable
|
|
/// <c>deployed_configurations</c> rows and repairs two silent divergences between the
|
|
/// stored configs and the running Instance Actors:
|
|
/// <list type="bullet">
|
|
/// <item>an ENABLED instance with a stored config and NO running actor is started
|
|
/// (row 78 — configs that arrive by LocalDb CDC replication are written straight into the
|
|
/// file by the sync applier, so they are invisible to both the boot-time load, which has
|
|
/// already run, and the startup reconcile, which reports no gap because central's view is
|
|
/// correct);</item>
|
|
/// <item>a running instance whose stored revision hash no longer matches the revision it
|
|
/// was started from is re-applied through the ordinary redeploy path (row 72 — the startup
|
|
/// reconcile refreshes a stale config row but leaves the running actor serving the config
|
|
/// central believes it replaced).</item>
|
|
/// </list>
|
|
/// The sweep is silent when there is nothing to repair. <c>0</c> disables it entirely
|
|
/// (the row-67 <c>EnsureInstanceStartedCommand</c> path is unaffected). Default: 60.
|
|
/// </summary>
|
|
public int InstanceSweepIntervalSeconds { get; set; } = 60;
|
|
}
|