dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

refactor(sf,site): both stores take ILocalDb instead of a connection string

Browse Source 
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.

StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.

DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.

Two things the plan did not anticipate, both worth reading:

1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
   directory creation simply moved to LocalDb along with file ownership. It did
   not: the LocalDb library never creates the parent directory, and
   SqliteLocalDb opens the file eagerly in its constructor — so a missing
   directory is a hard boot failure ("SQLite Error 14: unable to open database
   file"), not a degraded start. The default site config points at the RELATIVE
   path ./data/site-localdb.db, so any site node without a pre-existing data/
   directory fails to boot. The docker rig escapes only because its volume mount
   happens to create /app/data — a coincidence that would have hidden this until
   a bare-metal or fresh deployment. This has been latent since Phase 1 made
   LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
   would have widened it. Re-established the guarantee at the layer that now
   owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
   pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
   tests written against the wrong assumption failed with exactly this
   SQLite Error 14 before the fix existed.

2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
   project; the constructor change actually reaches 40 files across 7 test
   projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
   equivalent for, so every one had to move to a real temp file. Rather than
   copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
   tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
   WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.

Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.

Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 03:05:45 -04:00
parent 3dfb288b74
commit f2efeb37b7
57 changed files with 1368 additions and 848 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ParentExecutionIdCorrelationTests.cs
+173 -162
View File
@@ -35,6 +35,7 @@ using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Delivery;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Integration;
@@ -197,174 +198,184 @@ public class ParentExecutionIdCorrelationTests : TestKit, IClassFixture<MsSqlMig
stubClient);
// Site Store-and-Forward — Notify.Send buffers a NotificationSubmit here.
using var safKeepAlive = new Microsoft.Data.Sqlite.SqliteConnection(
$"Data Source=parentexec-saf-{Guid.NewGuid():N};Mode=Memory;Cache=Shared");
safKeepAlive.Open();
var safStorage = new StoreAndForwardStorage(
safKeepAlive.ConnectionString, NullLogger<StoreAndForwardStorage>.Instance);
await safStorage.InitializeAsync();
var storeAndForward = new StoreAndForwardService(
safStorage,
new StoreAndForwardOptions
// The storage takes an ILocalDb and LocalDb has no in-memory mode, so this is a
// real temp file, disposed (the master connection anchors the WAL) and deleted in
// the finally below.
var safLocalDb = TestLocalDb.CreateTemp("parentexec-saf");
var safDbPath = safLocalDb.Path;
try
{
var safStorage = new StoreAndForwardStorage(
safLocalDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await safStorage.InitializeAsync();
var storeAndForward = new StoreAndForwardService(
safStorage,
new StoreAndForwardOptions
{
DefaultRetryInterval = TimeSpan.Zero,
DefaultMaxRetries = 3,
RetryTimerInterval = TimeSpan.FromMinutes(10),
},
NullLogger<StoreAndForwardService>.Instance);
// ── Outbound external-system client (routed run): sync Call succeeds,
// CachedCall completes immediately (WasBuffered=false) so the script
// helper emits the Submit + Attempted + CachedResolve lifecycle. ──
var externalClient = Substitute.For<IExternalSystemClient>();
externalClient
.CallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(), Arg.Any<CancellationToken>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null));
externalClient
.CachedCallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(),
Arg.Any<string?>(), Arg.Any<CancellationToken>(),
Arg.Any<ZB.MOM.WW.ScadaBridge.Commons.Types.TrackedOperationId?>(),
Arg.Any<Guid?>(), Arg.Any<string?>(), Arg.Any<Guid?>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false));
// ── The routing transport stand-in: builds the routed ScriptRuntimeContext
// carrying RouteToCallRequest.ParentExecutionId — exactly what the
// production site handler (DeploymentManagerActor) does. ──
var router = new BridgingInstanceRouter(
siteId,
externalClient,
siteAuditWriter,
cachedForwarder,
storeAndForward);
// ── The inbound API method script: it calls Route.Call into the site
// instance. The real InboundScriptExecutor binds the inbound request's
// ExecutionId onto the RouteHelper, so the routed call carries it as
// ParentExecutionId. ──
var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod(
"submitOrder",
$"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});");
var locator = Substitute.For<IInstanceLocator>();
locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any<CancellationToken>())
.Returns(siteId);
var scriptExecutor = new InboundScriptExecutor(
NullLogger<InboundScriptExecutor>.Instance,
new ServiceCollection().BuildServiceProvider());
Assert.True(scriptExecutor.CompileAndRegister(inboundMethod));
// ── Act — issue the inbound HTTP request through a TestHost pipeline
// fronted by the real AuditWriteMiddleware. The endpoint handler reads
// the middleware-stashed inbound ExecutionId and runs the inbound
// method script with it as parentExecutionId. ──
using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx =>
{
DefaultRetryInterval = TimeSpan.Zero,
DefaultMaxRetries = 3,
RetryTimerInterval = TimeSpan.FromMinutes(10),
},
NullLogger<StoreAndForwardService>.Instance);
var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!;
var route = new RouteHelper(locator, router);
var result = await scriptExecutor.ExecuteAsync(
inboundMethod,
new Dictionary<string, object?>(),
route,
TimeSpan.FromSeconds(30),
ctx.RequestAborted,
parentExecutionId: inboundExecutionId);
// ── Outbound external-system client (routed run): sync Call succeeds,
// CachedCall completes immediately (WasBuffered=false) so the script
// helper emits the Submit + Attempted + CachedResolve lifecycle. ──
var externalClient = Substitute.For<IExternalSystemClient>();
externalClient
.CallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(), Arg.Any<CancellationToken>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null));
externalClient
.CachedCallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(),
Arg.Any<string?>(), Arg.Any<CancellationToken>(),
Arg.Any<ZB.MOM.WW.ScadaBridge.Commons.Types.TrackedOperationId?>(),
Arg.Any<Guid?>(), Arg.Any<string?>(), Arg.Any<Guid?>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false));
ctx.Response.StatusCode = result.Success ? 200 : 500;
await ctx.Response.WriteAsync(result.Success ? "ok" : "fail");
});
// ── The routing transport stand-in: builds the routed ScriptRuntimeContext
// carrying RouteToCallRequest.ParentExecutionId — exactly what the
// production site handler (DeploymentManagerActor) does. ──
var router = new BridgingInstanceRouter(
siteId,
externalClient,
siteAuditWriter,
cachedForwarder,
storeAndForward);
var client = host.GetTestClient();
var response = await client.PostAsync(
"/api/submitOrder",
new StringContent("{}", Encoding.UTF8, "application/json"));
Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode);
// ── The inbound API method script: it calls Route.Call into the site
// instance. The real InboundScriptExecutor binds the inbound request's
// ExecutionId onto the RouteHelper, so the routed call carries it as
// ParentExecutionId. ──
var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod(
"submitOrder",
$"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});");
var locator = Substitute.For<IInstanceLocator>();
locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any<CancellationToken>())
.Returns(siteId);
var scriptExecutor = new InboundScriptExecutor(
NullLogger<InboundScriptExecutor>.Instance,
new ServiceCollection().BuildServiceProvider());
Assert.True(scriptExecutor.CompileAndRegister(inboundMethod));
// The routed run emits its sync-ApiCall and NotifySend audit rows on a
// deliberately fire-and-forget path (alog.md §7 — an audit write must
// never block the user-facing script call). `Notify.Send` therefore
// returns — and the routed `RouteToCallAsync` completes — BEFORE the
// SqliteAuditWriter background loop has flushed the NotifySend row into
// the site hot-path. Wait for all five site rows to be durably present
// in SQLite before the central assertion: this is the production
// durability point (the row IS in SQLite before it is considered
// audited), and pinning it removes the emit-vs-drain race that
// otherwise let the SiteAuditTelemetryADrain forward only four rows on
// its first tick and leave NotifySend stranded for a full drain
// interval under heavy parallel load.
await WaitForSiteRowsPersistedAsync(sqliteWriter);
// ── Act — issue the inbound HTTP request through a TestHost pipeline
// fronted by the real AuditWriteMiddleware. The endpoint handler reads
// the middleware-stashed inbound ExecutionId and runs the inbound
// method script with it as parentExecutionId. ──
using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx =>
// The routed run produced a NotifySend that buffered a NotificationSubmit
// into S&F. Drain that genuine site-produced submission to the central
// NotificationOutboxActor so the NotifyDeliver dispatch rows materialise.
await ForwardBufferedNotificationToCentralAsync(
storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter);
// ── Assert ──────────────────────────────────────────────────────────
await AwaitAssertAsync(async () =>
{
await using var readContext = CreateContext();
var repo = new AuditLogRepository(readContext);
// Every audit row this site produced (sync ApiCall + cached lifecycle
// + NotifySend) plus the central NotifyDeliver rows.
var siteRows = await repo.QueryAsync(
new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }),
new AuditLogPaging(PageSize: 100));
// sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1)
// + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run.
Assert.True(siteRows.Count == 7,
"Expected 7 routed-run audit rows; saw: "
+ string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}")));
Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend);
Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver));
// CORE PROMISE: every routed-run row carries the SAME non-null
// ParentExecutionId — the inbound request's ExecutionId.
var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList();
Assert.Single(parentIds);
Assert.NotNull(parentIds[0]);
var inboundExecutionId = parentIds[0]!.Value;
// The routed run has its OWN distinct ExecutionId — not the parent's.
var routedExecutionIds = siteRows
.Select(r => r.AsRow().ExecutionId)
.Distinct()
.ToList();
Assert.Single(routedExecutionIds);
Assert.NotNull(routedExecutionIds[0]);
var routedExecutionId = routedExecutionIds[0]!.Value;
Assert.NotEqual(inboundExecutionId, routedExecutionId);
// The inbound request's own InboundRequest row is TOP-LEVEL —
// ExecutionId = the propagated id, ParentExecutionId = NULL.
var inboundRows = await repo.QueryAsync(
new AuditLogQueryFilter(ExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 10));
var inboundRow = Assert.Single(inboundRows,
r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest);
Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status);
Assert.Null(inboundRow.AsRow().ParentExecutionId);
// The parentExecutionId filter pulls the routed run's complete
// trust-boundary footprint (all 7 routed rows, none of the inbound).
var byParent = await repo.QueryAsync(
new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 100));
Assert.Equal(7, byParent.Count);
Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId));
// GetExecutionTreeAsync returns BOTH executions in one chain —
// inbound (root) and routed (child), regardless of entry point.
var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId);
AssertChain(treeFromChild, inboundExecutionId, routedExecutionId);
var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId);
AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId);
}, TimeSpan.FromSeconds(90));
}
finally
{
var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!;
var route = new RouteHelper(locator, router);
var result = await scriptExecutor.ExecuteAsync(
inboundMethod,
new Dictionary<string, object?>(),
route,
TimeSpan.FromSeconds(30),
ctx.RequestAborted,
parentExecutionId: inboundExecutionId);
ctx.Response.StatusCode = result.Success ? 200 : 500;
await ctx.Response.WriteAsync(result.Success ? "ok" : "fail");
});
var client = host.GetTestClient();
var response = await client.PostAsync(
"/api/submitOrder",
new StringContent("{}", Encoding.UTF8, "application/json"));
Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode);
// The routed run emits its sync-ApiCall and NotifySend audit rows on a
// deliberately fire-and-forget path (alog.md §7 — an audit write must
// never block the user-facing script call). `Notify.Send` therefore
// returns — and the routed `RouteToCallAsync` completes — BEFORE the
// SqliteAuditWriter background loop has flushed the NotifySend row into
// the site hot-path. Wait for all five site rows to be durably present
// in SQLite before the central assertion: this is the production
// durability point (the row IS in SQLite before it is considered
// audited), and pinning it removes the emit-vs-drain race that
// otherwise let the SiteAuditTelemetryADrain forward only four rows on
// its first tick and leave NotifySend stranded for a full drain
// interval under heavy parallel load.
await WaitForSiteRowsPersistedAsync(sqliteWriter);
// The routed run produced a NotifySend that buffered a NotificationSubmit
// into S&F. Drain that genuine site-produced submission to the central
// NotificationOutboxActor so the NotifyDeliver dispatch rows materialise.
await ForwardBufferedNotificationToCentralAsync(
storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter);
// ── Assert ──────────────────────────────────────────────────────────
await AwaitAssertAsync(async () =>
{
await using var readContext = CreateContext();
var repo = new AuditLogRepository(readContext);
// Every audit row this site produced (sync ApiCall + cached lifecycle
// + NotifySend) plus the central NotifyDeliver rows.
var siteRows = await repo.QueryAsync(
new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }),
new AuditLogPaging(PageSize: 100));
// sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1)
// + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run.
Assert.True(siteRows.Count == 7,
"Expected 7 routed-run audit rows; saw: "
+ string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}")));
Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend);
Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver));
// CORE PROMISE: every routed-run row carries the SAME non-null
// ParentExecutionId — the inbound request's ExecutionId.
var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList();
Assert.Single(parentIds);
Assert.NotNull(parentIds[0]);
var inboundExecutionId = parentIds[0]!.Value;
// The routed run has its OWN distinct ExecutionId — not the parent's.
var routedExecutionIds = siteRows
.Select(r => r.AsRow().ExecutionId)
.Distinct()
.ToList();
Assert.Single(routedExecutionIds);
Assert.NotNull(routedExecutionIds[0]);
var routedExecutionId = routedExecutionIds[0]!.Value;
Assert.NotEqual(inboundExecutionId, routedExecutionId);
// The inbound request's own InboundRequest row is TOP-LEVEL —
// ExecutionId = the propagated id, ParentExecutionId = NULL.
var inboundRows = await repo.QueryAsync(
new AuditLogQueryFilter(ExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 10));
var inboundRow = Assert.Single(inboundRows,
r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest);
Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status);
Assert.Null(inboundRow.AsRow().ParentExecutionId);
// The parentExecutionId filter pulls the routed run's complete
// trust-boundary footprint (all 7 routed rows, none of the inbound).
var byParent = await repo.QueryAsync(
new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 100));
Assert.Equal(7, byParent.Count);
Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId));
// GetExecutionTreeAsync returns BOTH executions in one chain —
// inbound (root) and routed (child), regardless of entry point.
var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId);
AssertChain(treeFromChild, inboundExecutionId, routedExecutionId);
var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId);
AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId);
}, TimeSpan.FromSeconds(90));
safLocalDb.Dispose();
TestLocalDb.DeleteFiles(safDbPath);
}
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj
+2 -1
View File
@@ -76,7 +76,8 @@
central MSSQL AuditLog.
-->
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.InboundAPI/ZB.MOM.WW.ScadaBridge.InboundAPI.csproj" />
</ItemGroup>
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
<ItemGroup>
<!-- M4 Bundle E (Task E3): need ASP.NET Core for the TestHost-based middleware E2E. -->