Files
ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteHealthEndpointTests.cs
T
Joseph Doherty 2e4e41a8f7 fix(auditlog): site audit DB onto the data volume; required path + soft flush
Closes WP1.2 of the arch-review remediation plan (finding #2, High):
SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db",
which on the docker rig resolves onto the container's ephemeral overlayfs
(not the mounted /app/data volume), silently discarding the pending audit
forward-state backlog on every recreate; nothing in docker/ or docker-env2/
overrode it; FlushIntervalMs was validated but never read by the writer loop
(one commit per event even at trickle rate); and no PRAGMA synchronous was
set (SQLite's FULL default fsyncs every commit).

- DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path)
  and is required pre-host for Site nodes only, via a new StartupValidator raw-config
  check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge:
  AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator
  deliberately does NOT check DatabasePath itself, because AddAuditLog runs its
  ValidateOnStart on both Central and Site composition roots but only Site nodes
  ever resolve the writer — checking it there would fail Central's boot too.
- All 8 site-node appsettings under docker/ and docker-env2/ now set
  AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume,
  survives container recreate, same convention as LocalDb:Path); the local-dev
  base appsettings.Site.json sets ./data/auditlog.db to match.
- The writer loop now honors FlushIntervalMs: after draining the immediately
  available burst, it keeps the transaction open (bounded by FlushIntervalMs
  from the first event) waiting for more trickle-rate events before committing,
  instead of flushing (and fsyncing) per event.
- PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by
  design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"),
  so NORMAL's narrower power-loss window is an acceptable trade for far fewer
  fsyncs; WAL mode still guarantees no corruption.
- Tests: StartupValidator site-required/blank/central-exempt cases; writer
  trickle-load single-transaction coalescing + beyond-interval separate-transaction
  regression (new FlushCountForTests seam); options-validator doc updates reflecting
  the moved responsibility. Full suite runs green: AuditLog.Tests 368/368,
  Host.Tests 480/480.

One-time migration note: the existing container-local auditlog.db (wherever it
landed under CWD) is abandoned by this change, not migrated — already-forwarded
rows are safe centrally (AuditLog is the durable copy), and any still-Pending
rows on the abandoned path are lost once. This is the exact bug being fixed, not
a new loss: those rows were already living outside the mounted volume and would
not have survived the next container recreate regardless. Cross-reference
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this
placement bug caused.
2026-08-14 20:13:31 -04:00

165 lines
7.9 KiB
C#

using System.Net;
using System.Text.Json;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Proves the site pipeline actually MAPS the health endpoints — the half
/// <see cref="SiteHealthCheckTests"/> cannot cover, since that fixture builds the site container
/// from <see cref="SiteServiceRegistration"/> and never runs <c>Program</c>'s endpoint wiring.
/// Without this, deleting <c>app.MapZbHealth()</c> from the site branch would leave every
/// registration test green while site nodes served 404 to the overview dashboard.
/// <para>
/// Boots the real <c>Program</c> in the Site role. <c>Program</c> builds its own
/// <c>ConfigurationBuilder</c> before the web host exists, so its role and settings come from
/// PROCESS-WIDE environment variables (<c>SCADABRIDGE_CONFIG</c> selects appsettings.Site.json)
/// rather than from <c>WithWebHostBuilder</c> — which is why this fixture joins
/// <see cref="HostBootCollection"/> and restores every var it sets.
/// </para>
/// </summary>
[Collection(HostBootCollection.Name)]
public class SiteHealthEndpointTests : IDisposable
{
private readonly List<IDisposable> _disposables = new();
private readonly Dictionary<string, string?> _previousEnv = new(StringComparer.Ordinal);
private readonly string _tempDbPath;
private readonly string _tempAuditDbPath;
public SiteHealthEndpointTests()
{
_tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_ep_{Guid.NewGuid()}.db");
_tempAuditDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_ep_audit_{Guid.NewGuid()}.db");
// Whole-key env overrides, the sanctioned path: supplying GrpcPsk concretely makes the
// pre-host secrets expander skip appsettings.Site.json's ${secret:SB-GRPC-PSK-site-1}
// token, so this test needs no seeded secrets store.
//
// The remoting port is moved off the site default (8082) so this fixture cannot collide
// with a locally running node, but it must be a REAL port and seed-nodes[0] must be this
// node itself — StartupValidator enforces both before the host is built, and it runs
// whether or not the Akka hosted service is later removed. Nothing binds it: the hosted
// service is removed below.
SetEnv("SCADABRIDGE_CONFIG", "Site");
SetEnv("ScadaBridge__Node__Role", "Site");
SetEnv("ScadaBridge__Node__SiteId", "TestSite");
SetEnv("ScadaBridge__Node__NodeHostname", "localhost");
SetEnv("ScadaBridge__Node__RemotingPort", "18082");
SetEnv("ScadaBridge__Cluster__SeedNodes__0", "akka.tcp://scadabridge@localhost:18082");
SetEnv("ScadaBridge__Cluster__SeedNodes__1", "akka.tcp://scadabridge@localhost:18085");
SetEnv("ScadaBridge__Communication__GrpcPsk", "test-psk-0123456789");
SetEnv("LocalDb__Path", _tempDbPath);
// arch-review remediation WP1.2: SqliteAuditWriterOptions.DatabasePath has no default
// and StartupValidator now requires it for Site nodes — same reason LocalDb__Path is
// overridden above (appsettings.Site.json's own default is CWD-relative and would
// otherwise litter the test working directory when SiteAuditBacklogReporter probes it).
SetEnv("AuditLog__SiteWriter__DatabasePath", _tempAuditDbPath);
}
private void SetEnv(string key, string? value)
{
_previousEnv[key] = Environment.GetEnvironmentVariable(key);
Environment.SetEnvironmentVariable(key, value);
}
public void Dispose()
{
foreach (var d in _disposables)
{
try { d.Dispose(); } catch { /* best effort */ }
}
foreach (var (key, value) in _previousEnv)
{
Environment.SetEnvironmentVariable(key, value);
}
try { File.Delete(_tempDbPath); } catch { /* best effort */ }
try { File.Delete(_tempAuditDbPath); } catch { /* best effort */ }
GC.SuppressFinalize(this);
}
private HttpClient CreateSiteClient()
{
var factory = new WebApplicationFactory<Program>()
.WithWebHostBuilder(builder =>
{
// WebApplicationFactory defaults the environment to Development, which turns on
// ValidateOnBuild/ValidateScopes. The site graph carries scoped registrations whose
// dependencies are central-only (ReconcileService → IDeploymentManagerRepository,
// AuditLogKpiSampleSource → IAuditLogRepository) and are never resolved on a site
// node, so eager validation fails on a composition that runs fine in production.
// That is a pre-existing property of the site container, not of the health wiring
// under test — run this fixture the way a site node actually runs.
builder.UseEnvironment("Production");
// ConfigureTestServices runs after the app's own registrations, so this drops the
// hosted service that would form a real cluster while leaving the AkkaHostedService
// singleton resolvable (the site pipeline resolves it for shutdown ordering).
builder.ConfigureTestServices(AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly);
});
_disposables.Add(factory);
var client = factory.CreateClient();
_disposables.Add(client);
return client;
}
[Fact]
public async Task Site_MapsHealthReady_WithTheCanonicalJsonBody()
{
var response = await CreateSiteClient().GetAsync("/health/ready");
// Never 404. 200 vs 503 depends on cluster state this fixture does not form, so both are
// accepted — what is asserted is that the tier is mapped and answers in canonical shape.
Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
Assert.True(
response.StatusCode is HttpStatusCode.OK or HttpStatusCode.ServiceUnavailable,
$"Expected 200 or 503, got {(int)response.StatusCode}");
using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
var entries = doc.RootElement.GetProperty("entries");
Assert.True(entries.TryGetProperty("akka-cluster", out _), "ready tier must carry akka-cluster");
Assert.True(entries.TryGetProperty("localdb", out _), "ready tier must carry localdb");
// Readiness must NOT depend on being the primary, or a healthy standby would read unready.
Assert.False(entries.TryGetProperty("active-node", out _), "active-node belongs to the Active tier");
}
[Fact]
public async Task Site_MapsHealthActive_CarryingOnlyTheActiveNodeCheck()
{
var response = await CreateSiteClient().GetAsync("/health/active");
Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
var entries = doc.RootElement.GetProperty("entries");
Assert.Equal(
new[] { "active-node" },
entries.EnumerateObject().Select(p => p.Name).ToArray());
}
[Fact]
public async Task Site_HealthEndpointsAreAnonymous()
{
// The dashboard authenticates nowhere. The site pipeline runs no authentication middleware
// today, so this is a regression pin: adding one later must not silently close these.
var client = CreateSiteClient();
foreach (var path in new[] { "/health/ready", "/health/active", "/healthz" })
{
var response = await client.GetAsync(path);
Assert.NotEqual(HttpStatusCode.Unauthorized, response.StatusCode);
Assert.NotEqual(HttpStatusCode.Forbidden, response.StatusCode);
Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
}
}
}