dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 15 Pull Requests Actions Packages Projects Releases Wiki Activity

perf(transport): make large-bundle apply linear in the template rewire passes (ScadaBridge#47)

Browse Source 
Measured first: a scratch scaling probe (seed -> export -> load/preview/apply
into a fresh target and over itself, in-memory provider, every repository
call counted, DetectChanges counted via ChangeTracker.DetectingAllChanges)
showed load, preview and the instance pass scale linearly (500 / 1000 / 2000
instances apply in 0.13 / 0.27 / 0.35 s), but the post-flush template
rewire passes were quadratic: 100 / 400 / 800 composing, derived templates
applied in 0.5 / 8.1 / 26.5 s, with DetectChanges running 5N+10 times and
GetTemplateWithChildrenAsync 2N+2 times.

Cause: ResolveAlarmScriptLinksAsync, ResolveCompositionEdgesAsync and
ResolveInheritanceEdgesAsync found each template with
_dbContext.Templates.Local.FirstOrDefault(...) - per template, and per
composition edge / base lookup. Every DbSet.Local access runs a full
DetectChanges over everything the import has staged. The first two passes
also re-read every template's children one query at a time (a split query
per call on SQL Server).

Fix (no bundle-format or semantic change):
- each pass indexes the tracked templates by name once
  (SnapshotLocalTemplatesByName; ordinal, first wins like FirstOrDefault).
  These passes never stage a template or rename one, so the index stays
  exact for the pass;
- the alarm and composition passes read their templates' children in one
  batched GetTemplatesWithChildrenAsync, matched back by the stub's id;
- the alarm pass only loads templates whose bundled alarms name an
  on-trigger script (the only ones it can rewire).

Result on the same probe: 100 / 400 / 800 templates apply in
0.13 / 0.62 / 0.64 s; DetectChanges is a constant 9; per-template children
reads 0, batched reads 1 (add) / 2 (overwrite).

Evidence: ImportApplyScalingTests (Transport.IntegrationTests) exports 5 and
40 composing, derived templates with alarm -> script links, imports each
into an empty target (Add) and again (Overwrite), and asserts the imported
inheritance / composition / alarm-script links, zero per-template children
reads, at most 3 batched reads, and an identical DetectChanges count for 5
and 40 templates. Fails on the old importer (12 per-template reads), and
with a per-DTO Local snapshot reintroduced in one pass (DetectChanges 51
vs 16). Transport unit (198) and integration (138) suites pass.

Not changed: preview's per-name lookups for database connections,
notification lists and API methods stay per-name - moving them to an
in-memory ordinal match could change name matching under a case-insensitive
SQL collation.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:29:51 -05:00
parent 51d2f6c906
commit 7153a1974e
3 changed files with 297 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs
+83 -21
View File
@@ -3055,10 +3055,18 @@ public sealed class BundleImporter : IBundleImporter
{
if (dtos.Count == 0) return;
// Gitea #47: only a template whose bundled alarms name an on-trigger script
// has a link to rewire, so only those are loaded — and in one batched read,
// not one children query per template. The tracked-template name index is
// built once (each DbSet.Local access runs DetectChanges over the whole
// tracker, which made this pass quadratic in the bundle's size).
var localByName = SnapshotLocalTemplatesByName();
var targets = new List<(TemplateDto Dto, Template Stub)>();
foreach (var dto in dtos)
{
var resolution = ResolveOrDefault(resolutionMap, "Template", dto.Name);
if (resolution.Action == ResolutionAction.Skip) continue;
if (!dto.Alarms.Any(a => !string.IsNullOrEmpty(a.OnTriggerScriptName))) continue;
// Resolve the imported template by its post-rename name. Skip-
// resolved DTOs were already filtered out above; everything else
@@ -3066,16 +3074,19 @@ public sealed class BundleImporter : IBundleImporter
var importedName = resolution.Action == ResolutionAction.Rename
? (resolution.RenameTo ?? dto.Name)
: dto.Name;
if (localByName.TryGetValue(importedName, out var stub))
targets.Add((dto, stub));
}
if (targets.Count == 0) return;
// Use the children-loaded variant so the alarm + script collections
// are populated — the unfiltered GetAllTemplatesAsync returns the
// tracked entities EF assigned ids to during the flush, but the
// navigation collections aren't necessarily included.
var stub = _dbContext.Templates.Local.FirstOrDefault(t =>
string.Equals(t.Name, importedName, StringComparison.Ordinal));
if (stub is null) continue;
var template = await _templateRepo.GetTemplateWithChildrenAsync(stub.Id, ct).ConfigureAwait(false);
if (template is null) continue;
// Use the children-loaded read so the alarm + script collections are
// populated — the tracked entities EF assigned ids to during the flush
// don't necessarily have their navigation collections included.
var loadedById = await LoadTemplatesWithChildrenAsync(targets.Select(t => t.Stub), ct).ConfigureAwait(false);
foreach (var (dto, stub) in targets)
{
if (!loadedById.TryGetValue(stub.Id, out var template)) continue;
foreach (var alarmDto in dto.Alarms)
{
@@ -3142,6 +3153,12 @@ public sealed class BundleImporter : IBundleImporter
{
if (dtos.Count == 0) return;
// Gitea #47: one tracked-template name index for the pass (this pass stages
// composition rows only, never a template, so the index stays exact) and one
// batched children read for every parent, instead of a DbSet.Local scan
// (= a full DetectChanges) and a children query per template + per edge.
var localByName = SnapshotLocalTemplatesByName();
var targets = new List<(TemplateDto Dto, Template Stub)>();
foreach (var dto in dtos)
{
var resolution = ResolveOrDefault(resolutionMap, "Template", dto.Name);
@@ -3152,11 +3169,16 @@ public sealed class BundleImporter : IBundleImporter
? (resolution.RenameTo ?? dto.Name)
: dto.Name;
var parentStub = _dbContext.Templates.Local.FirstOrDefault(t =>
string.Equals(t.Name, importedName, StringComparison.Ordinal));
if (parentStub is null) continue;
var parent = await _templateRepo.GetTemplateWithChildrenAsync(parentStub.Id, ct).ConfigureAwait(false);
if (parent is null) continue;
if (localByName.TryGetValue(importedName, out var parentStub))
targets.Add((dto, parentStub));
}
if (targets.Count == 0) return;
var loadedById = await LoadTemplatesWithChildrenAsync(targets.Select(t => t.Stub), ct).ConfigureAwait(false);
foreach (var (dto, parentStub) in targets)
{
if (!loadedById.TryGetValue(parentStub.Id, out var parent)) continue;
// Clear existing rows (Overwrite-truly-overwrites). EF Core tracks
// the deletes via the navigation; explicit DeleteAsync ensures the
@@ -3200,8 +3222,7 @@ public sealed class BundleImporter : IBundleImporter
// The fallback is a dictionary lookup into the per-apply library
// snapshot (04/P4) — a name not in Local is a pre-existing target
// template this apply has not touched, so its pre-write id is current.
var composedStub = _dbContext.Templates.Local.FirstOrDefault(t =>
string.Equals(t.Name, compDto.ComposedTemplateName, StringComparison.Ordinal));
localByName.TryGetValue(compDto.ComposedTemplateName, out var composedStub);
int? composedId = composedStub?.Id;
if (composedId is null
&& (await library.GetIdByNameAsync(ct).ConfigureAwait(false))
@@ -3258,6 +3279,10 @@ public sealed class BundleImporter : IBundleImporter
{
if (dtos.Count == 0) return;
// Gitea #47: one tracked-template name index for the pass (it only rewrites
// ParentTemplateId, never a name) instead of two DbSet.Local scans per DTO.
var localByName = SnapshotLocalTemplatesByName();
foreach (var dto in dtos)
{
var resolution = ResolveOrDefault(resolutionMap, "Template", dto.Name);
@@ -3268,9 +3293,7 @@ public sealed class BundleImporter : IBundleImporter
: dto.Name;
// The just-imported/updated template is tracked in the Local set.
var template = _dbContext.Templates.Local.FirstOrDefault(t =>
string.Equals(t.Name, importedName, StringComparison.Ordinal));
if (template is null) continue;
if (!localByName.TryGetValue(importedName, out var template)) continue;
// A root-template bundle — or an Overwrite that dropped the base —
// must clear any stale parent edge on the target row.
@@ -3289,8 +3312,7 @@ public sealed class BundleImporter : IBundleImporter
? (baseResolution.RenameTo ?? dto.BaseTemplateName)
: dto.BaseTemplateName;
var baseStub = _dbContext.Templates.Local.FirstOrDefault(t =>
string.Equals(t.Name, baseName, StringComparison.Ordinal));
localByName.TryGetValue(baseName, out var baseStub);
int? baseId = baseStub?.Id;
if (baseId is null
&& (await library.GetIdByNameAsync(ct).ConfigureAwait(false)).TryGetValue(baseName, out var libraryId))
@@ -3324,6 +3346,46 @@ public sealed class BundleImporter : IBundleImporter
}
}
/// <summary>
/// Indexes the change tracker's templates by name (ordinal; the first tracked
/// template wins a duplicate, matching the <c>FirstOrDefault</c> scans this
/// replaces). Gitea #47: every <c>DbSet.Local</c> access runs a full
/// <c>DetectChanges</c> over the tracker — every template, member and instance
/// the import has staged — so the rewire passes take one snapshot per pass
/// instead of one scan per bundled template (and per composition edge), which
/// made a large import's apply quadratic in its size.
/// </summary>
private Dictionary<string, Template> SnapshotLocalTemplatesByName()
{
var byName = new Dictionary<string, Template>(StringComparer.Ordinal);
foreach (var template in _dbContext.Templates.Local)
{
byName.TryAdd(template.Name, template);
}
return byName;
}
/// <summary>
/// Loads the given tracked templates with their child collections in one batched
/// read (Gitea #47 — previously one <c>GetTemplateWithChildrenAsync</c> per
/// template), keyed by id. The read is by name, so the result is keyed by id and
/// looked up by each stub's id: a database collation that matches names
/// case-insensitively can only add rows, never swap the exact one out.
/// </summary>
private async Task<Dictionary<int, Template>> LoadTemplatesWithChildrenAsync(
IEnumerable<Template> stubs, CancellationToken ct)
{
var loaded = await _templateRepo
.GetTemplatesWithChildrenAsync(stubs.Select(t => t.Name), ct)
.ConfigureAwait(false);
var byId = new Dictionary<int, Template>();
foreach (var template in loaded)
{
byId.TryAdd(template.Id, template);
}
return byId;
}
/// <summary>
/// Post-rewire acyclicity guard: reads the fully-merged template graph off the
/// change tracker (staged, not-yet-saved edges included) and rejects the import