fix(scripts): build Roslyn ScriptOptions once per process, not per compile

ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.

Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.

Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
  compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run

ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.

Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.

This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
This commit is contained in:
Joseph Doherty
2026-08-12 09:51:08 -04:00
parent 0974de1df5
commit 5a781c706c
5 changed files with 130 additions and 21 deletions
@@ -203,4 +203,52 @@ public class ScriptCompilationServiceTests
"var sw = System.Diagnostics.Stopwatch.StartNew(); return sw.ElapsedMilliseconds;");
Assert.Empty(violations);
}
/// <summary>
/// Native-memory leak guard. <c>ScriptOptions.WithReferences(Assembly[])</c> resolves each
/// assembly through <c>MetadataReference.CreateFromFile</c>, and every such reference owns an
/// <c>AssemblyMetadata</c> → <c>PEReader</c> → <c>NativeHeapMemoryBlock</c> — an unmanaged copy
/// of the assembly metadata that nothing here ever disposes. Building the options per compile
/// therefore grows native memory permanently: no GC reclaims it, and it is invisible to
/// <c>GC.GetTotalAllocatedBytes</c> and to gcdump.
///
/// <para>
/// Diagnosed from a live dump of the wonder-app-vd03 Site node (2026-08-12): 2,885 MB working
/// set 78 min after a cold start, of which only 150 MB was live GC heap; VMMap attributed
/// 2,469 MB to the default process heap and <c>dumpheap -stat</c> found 6,740 each of
/// <c>AssemblyMetadata</c> / <c>PEReader</c> / <c>MetadataImageReference</c> against just 473
/// DLLs on disk — i.e. ~1,348 undisposed copies of this service's 5-assembly reference set.
/// </para>
///
/// <para>
/// Asserted on the artifact rather than on memory: a watch-the-bytes test would be flaky, and
/// the leak is native so the managed allocation counters cannot see it at all. Two DISTINCT
/// bodies are required — identical ones would be served from
/// <see cref="SiteScriptCompileCache"/> without a second <c>CompileUncached</c>, and the test
/// would pass without proving anything.
/// </para>
/// </summary>
[Fact]
public void Compile_DistinctScripts_ShareOneMetadataReferenceSet_SoNativeMemoryDoesNotGrow()
{
SiteScriptCompileCache.Clear();
var first = _service.Compile("first", "return 1 + 1;");
var second = _service.Compile("second", "return 2 + 2;");
Assert.True(first.IsSuccess);
Assert.True(second.IsSuccess);
Assert.NotSame(first.CompiledScript, second.CompiledScript); // two real compiles, not a cache hit
var firstRefs = first.CompiledScript!.Options.MetadataReferences;
var secondRefs = second.CompiledScript!.Options.MetadataReferences;
Assert.NotEmpty(firstRefs); // else the reference-equality checks below are vacuous
Assert.Equal(firstRefs.Length, secondRefs.Length);
for (var i = 0; i < firstRefs.Length; i++)
Assert.Same(firstRefs[i], secondRefs[i]);
// The options object itself is cached, so it must not be rebuilt either.
Assert.Same(first.CompiledScript.Options, second.CompiledScript.Options);
}
}