fix(scripts): build Roslyn ScriptOptions once per process, not per compile

ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.

Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.

Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
  compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run

ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.

Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.

This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
This commit is contained in:
Joseph Doherty
2026-08-12 09:51:08 -04:00
parent 0974de1df5
commit 5a781c706c
5 changed files with 130 additions and 21 deletions
@@ -57,6 +57,26 @@ public class ScriptAnalysisService
"System.Text",
"System.Threading.Tasks");
/// <summary>
/// Options for a sandbox run — <see cref="DefaultOptions"/> plus the sandbox host assembly
/// resolved by file path.
///
/// <para>
/// <b>Built ONCE, deliberately.</b> This was previously rebuilt on every sandbox run, and
/// <c>MetadataReference.CreateFromFile</c> does not cache: each call mints an
/// <c>AssemblyMetadata</c> → <c>PEReader</c> → <c>NativeHeapMemoryBlock</c> holding an
/// unmanaged copy of the assembly metadata that nothing disposes, so every run leaked it
/// for the life of the process. Same defect as the Site-side one confirmed from a live dump
/// on 2026-08-12; smaller blast radius here only because sandbox runs are operator-driven
/// rather than continuous.
/// </para>
/// </summary>
private static readonly ScriptOptions SandboxOptions =
DefaultOptions.WithReferences(DefaultOptions.MetadataReferences.Concat(new[]
{
Microsoft.CodeAnalysis.MetadataReference.CreateFromFile(typeof(SandboxScriptHost).Assembly.Location)
}));
private readonly ISharedScriptCatalog _sharedScripts;
private readonly IMemoryCache _cache;
private readonly IServiceProvider _services;
@@ -191,10 +211,7 @@ public class ScriptAnalysisService
request.TimeoutSeconds ?? SandboxDefaultTimeoutSeconds,
1, SandboxMaxTimeoutSeconds);
var options = DefaultOptions.WithReferences(DefaultOptions.MetadataReferences.Concat(new[]
{
Microsoft.CodeAnalysis.MetadataReference.CreateFromFile(typeof(SandboxScriptHost).Assembly.Location)
}));
var options = SandboxOptions;
var globalsType = request.Kind == ScriptKind.InboundApi
? typeof(SandboxInboundScriptHost)
@@ -202,6 +202,39 @@ public class InboundScriptExecutor
/// <c>null</c> when the script is missing, fails to compile, or violates the
/// script trust model. Does not mutate the handler cache.
/// </summary>
/// <summary>
/// Roslyn scripting options for every inbound-API method compile.
///
/// <para>
/// <b>Built ONCE, deliberately. Do not inline this back into <see cref="Compile"/>.</b>
/// <c>WithReferences(Assembly[])</c> resolves each assembly through
/// <c>MetadataReference.CreateFromFile</c>, which does not cache — every call mints a
/// fresh <c>AssemblyMetadata</c> → <c>PEReader</c> → <c>NativeHeapMemoryBlock</c> holding
/// an unmanaged copy of the assembly metadata that nothing disposes. Per-compile options
/// therefore leak native memory for the life of the process, invisibly to the GC and to
/// gcdump. Method compiles are not one-shot: every method re-registration and every
/// revision change recompiles, so the growth is unbounded on a long-lived central node.
/// </para>
///
/// <para>
/// Same defect, same shape as the Site-side one confirmed from a live dump on 2026-08-12
/// (<c>SiteRuntime.Scripts.ScriptCompilationService.SharedScriptOptions</c>).
/// </para>
/// </summary>
private static readonly ScriptOptions SharedScriptOptions = ScriptOptions.Default
.WithReferences(
typeof(object).Assembly,
typeof(Enumerable).Assembly,
typeof(Dictionary<,>).Assembly,
typeof(RouteHelper).Assembly,
typeof(ScriptParameters).Assembly,
typeof(Microsoft.CSharp.RuntimeBinder.CSharpArgumentInfo).Assembly)
.WithImports(
"System",
"System.Collections.Generic",
"System.Linq",
"System.Threading.Tasks");
private (Func<InboundScriptContext, Task<object?>>? Handler, IReadOnlyList<string> Errors) Compile(ApiMethod method)
{
if (string.IsNullOrWhiteSpace(method.Script))
@@ -224,23 +257,9 @@ public class InboundScriptExecutor
try
{
var scriptOptions = ScriptOptions.Default
.WithReferences(
typeof(object).Assembly,
typeof(Enumerable).Assembly,
typeof(Dictionary<,>).Assembly,
typeof(RouteHelper).Assembly,
typeof(ScriptParameters).Assembly,
typeof(Microsoft.CSharp.RuntimeBinder.CSharpArgumentInfo).Assembly)
.WithImports(
"System",
"System.Collections.Generic",
"System.Linq",
"System.Threading.Tasks");
var compiled = CSharpScript.Create<object?>(
method.Script,
scriptOptions,
SharedScriptOptions,
globalsType: typeof(InboundScriptContext));
var diagnostics = compiled.Compile();
@@ -66,8 +66,31 @@ public class ScriptCompilationService
/// <summary>
/// Shared Roslyn scripting options (references + imports) used by both full
/// script compilation and trigger-expression compilation.
///
/// <para>
/// <b>Built ONCE, deliberately. Do not turn this back into a method.</b>
/// <c>WithReferences(Assembly[])</c> resolves every assembly through
/// <c>MetadataReference.CreateFromFile</c>, which does not cache: each call
/// mints a fresh <c>MetadataReference</c> owning an <c>AssemblyMetadata</c> →
/// <c>PEReader</c> → <c>NativeHeapMemoryBlock</c>, an unmanaged copy of the
/// assembly metadata that nothing disposes. Building the options per compile
/// therefore leaks native memory permanently — no GC reclaims it, and neither
/// gcdump nor <c>GC.GetTotalAllocatedBytes</c> can see it, because it is not
/// on the managed heap.
/// </para>
///
/// <para>
/// This was a live defect: a Site node reached a 2,885 MB working set with only
/// 150 MB of live GC heap, ~2,469 MB of it on the default process heap across
/// ~6,700 undisposed <c>AssemblyMetadata</c> instances (against 473 DLLs on
/// disk). The method form reads as harmless, which is exactly why it survived —
/// see <c>ScriptCompilationServiceTests.Compile_DistinctScripts_ShareOneMetadataReferenceSet_SoNativeMemoryDoesNotGrow</c>,
/// which pins reference-equality of the reference set across compiles.
/// <c>ScriptAnalysisService.DefaultOptions</c> and
/// <c>ScriptTrustPolicy.DefaultReferences</c> already follow this pattern.
/// </para>
/// </summary>
private static ScriptOptions BuildScriptOptions() => ScriptOptions.Default
private static readonly ScriptOptions SharedScriptOptions = ScriptOptions.Default
.WithReferences(ScriptAssemblies)
.WithImports(
"System",
@@ -141,7 +164,7 @@ public class ScriptCompilationService
{
var script = CSharpScript.Create<object?>(
code,
BuildScriptOptions(),
SharedScriptOptions,
globalsType: globalsType);
var diagnostics = script.Compile();