Files
lmxopcua/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/SecretsReplicationRegistrationTests.cs
T
Joseph Doherty 2254ae3dea
v2-ci / build (push) Successful in 3m55s
v2-ci / unit-tests (push) Failing after 11m7s
fix(secrets): consume Secrets 0.2.2 - clustered-secrets DI deadlock fixed upstream
Bumps the four ZB.MOM.WW.Secrets pins 0.2.1 -> 0.2.2 (closes the OtOpcUa side
of scadaproj#1, tracked here as #482). 0.2.1's Akka replicator deadlocked any
hosted process at startup when Secrets:Replication:Enabled was true: the
package's DI graph closed a circular singleton dependency through factory
lambdas (store decorator -> replicator -> actor provider -> cache invalidator
-> resolver -> store), which MS.DI's StackGuard turns into a silent
cross-thread call-site-lock deadlock. 0.2.2 defers the invalidator edge to
first eviction. The flag stays default-false; enabling remains a
per-environment decision.

The_startup_hook_actually_creates_the_replication_actor is now a real test:
SecretReplicationStarter's docs had promised it since the adoption, and the
upstream fix finally makes a provider-based resolve runnable - container built
exactly as the host does, hook started under a watchdog, replication actor
proven to exist by ActorSelection on a self-joined single-node cluster (no
TestKit needed, which matters because Akka.TestKit.Xunit2 is xunit-v2-only and
this project is on xunit.v3). Also corrects the stale rationale that blamed
the old hang on DistributedPubSub needing a joined cluster - the actor
constructor was never reached; it was the DI cycle.

Verified: SecretsReplicationRegistrationTests 8/8 on the 0.2.2 feed packages;
full slnx build 0 errors; the 2-node Akka live convergence gate re-run against
the published 0.2.2 packages passes 6/6 (write->peer, tombstone propagation
without resurrection, delete visibility through the resolver cache, reverse
direction, wrong-KEK fail-closed).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-18 15:03:35 -04:00

234 lines
11 KiB
C#

using Akka.Actor;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Replication;
using ZB.MOM.WW.Secrets.Sqlite;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// Guards the production DI wiring for cluster secret replication (<c>AddOtOpcUaSecrets</c>).
/// This is the registration half of a defect class that has already shipped once in this library
/// family: every unit test passed because nothing ever built a container, so a decorator that
/// could not actually be constructed went unnoticed.
///
/// Two properties matter here, and both are load-bearing on <b>driver-role</b> nodes that have no
/// auth/AdminUI surface and whose only symptom of a broken registration is drivers failing to
/// open sessions:
/// <list type="number">
/// <item>
/// With replication off (the default), <see cref="ISecretStore"/> must resolve to the
/// plain <see cref="SqliteSecretStore"/> — byte-identical to the pre-replication host.
/// </item>
/// <item>
/// With replication on, <see cref="ISecretStore"/> must resolve to
/// <see cref="ReplicatingSecretStore"/> <b>and</b> the undecorated concrete
/// <see cref="SqliteSecretStore"/> must still resolve, because the decorator is
/// constructed from it. A missing concrete registration is the exact gap that shipped.
/// </item>
/// </list>
/// </summary>
public sealed class SecretsReplicationRegistrationTests
{
/// <summary>
/// Builds a container mirroring the host's registration order: Akka first (the host calls
/// <c>AddAkka</c> before <c>AddOtOpcUaSecrets</c>), then secrets.
/// </summary>
private static ServiceProvider BuildProvider(bool replicationEnabled, ActorSystem? actorSystem = null) =>
BuildServices(replicationEnabled, actorSystem).BuildServiceProvider();
/// <summary>
/// The registrations without building the provider, for assertions about which descriptor
/// won a <c>TryAdd</c> race. Resolving some of these services has side effects (constructing
/// the Akka replicator spawns an actor that needs a joined cluster), so the descriptor is
/// the only place certain defects can be observed without a real cluster.
/// </summary>
private static IServiceCollection BuildServices(bool replicationEnabled, ActorSystem? actorSystem = null)
{
var configuration = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
// A per-test SQLite path keeps these hermetic; nothing here opens the file, because
// resolving the store does not touch the database.
["Secrets:SqlitePath"] = Path.Combine(Path.GetTempPath(), $"otopcua-secrets-{Guid.NewGuid():N}.db"),
["Secrets:MasterKey:Source"] = "Environment",
["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_MASTER_KEY",
["Secrets:RunMigrationsOnStartup"] = "false",
["Secrets:Replication:Enabled"] = replicationEnabled ? "true" : "false",
["Secrets:Replication:AnnounceInterval"] = "00:00:30",
["Secrets:Replication:ActorName"] = "zb-secret-replication",
})
.Build();
var services = new ServiceCollection();
services.AddLogging();
// The replicating store reaches the ActorSystem through SecretReplicationActorProvider.
// The host supplies one via Akka.Hosting; a plain system is equivalent for registration.
if (actorSystem is not null)
services.AddSingleton(actorSystem);
services.AddOtOpcUaSecrets(configuration);
return services;
}
[Fact]
public void Replication_disabled_resolves_the_plain_sqlite_store()
{
using var sp = BuildProvider(replicationEnabled: false);
var store = sp.GetRequiredService<ISecretStore>();
store.ShouldBeOfType<SqliteSecretStore>(
"replication is opt-in — with Secrets:Replication:Enabled false the host must resolve the "
+ "plain local store, so existing nodes are behaviourally unchanged");
}
[Fact]
public void Replication_disabled_does_not_require_an_actor_system()
{
// A driver-role node builds its container before the ActorSystem is reachable for secret
// resolution; the default path must not have taken a dependency on it.
using var sp = BuildProvider(replicationEnabled: false);
Should.NotThrow(() => sp.GetRequiredService<ISecretStore>());
}
[Fact]
public void Replication_enabled_decorates_the_store_so_writes_publish()
{
// Asserted on the ServiceCollection because the registration order IS the defect class this
// file guards (which descriptor won a TryAdd race), and the descriptor is where that is
// visible. An earlier revision of this comment ALSO claimed a provider-based resolve was
// impossible ("hangs against a plain ActorSystem — DistributedPubSub needs a joined
// cluster"). That hang was real but misattributed: it was a circular singleton dependency
// in the 0.2.1 package's own DI wiring (scadaproj#1), fixed in 0.2.2. The provider-based
// resolve is covered by The_startup_hook_actually_creates_the_replication_actor below.
//
// AddSingleton (not TryAdd) appends the decorator, and the LAST registration for a service
// type is what the container resolves.
var last = BuildServices(replicationEnabled: true)
.Last(d => d.ServiceType == typeof(ISecretStore));
last.ImplementationFactory.ShouldNotBeNull(
"with replication enabled the local store must be decorated so writes publish to peers");
}
[Fact]
public async Task The_startup_hook_actually_creates_the_replication_actor()
{
// The test SecretReplicationStarter's docs have promised since the 0.2.x adoption, runnable
// now that the upstream deadlock is fixed: build the container the way the host does, start
// the hook, and prove the replication actor exists. On 0.2.0 this fails because the actor
// is never spawned (inert replicator); on 0.2.1 it deadlocks in the resolve (scadaproj#1).
//
// Akka.TestKit.Xunit2 is xunit-v2-only and this project is on xunit.v3 (CS0433), so this
// uses a plain self-joined single-node cluster — which is also all the actor needs: its
// constructor gets the DistributedPubSub mediator, no peers required.
ActorSystem system = ActorSystem.Create(
"otopcua-secrets-gate",
Akka.Configuration.ConfigurationFactory.ParseString("""
akka {
loglevel = WARNING
actor.provider = cluster
remote.dot-netty.tcp {
hostname = "127.0.0.1"
public-hostname = "127.0.0.1"
port = 0
}
}
"""));
try
{
var cluster = Akka.Cluster.Cluster.Get(system);
cluster.Join(cluster.SelfAddress);
using ServiceProvider sp = BuildProvider(replicationEnabled: true, system);
// Startup order mirrors the host: hosted services run, nothing else has resolved the
// store yet. The hook's resolve is the moment 0.2.1 hung forever, so it runs under a
// watchdog — a regression should fail the test, not the whole run.
var starter = sp.GetServices<IHostedService>().OfType<SecretReplicationStarter>().Single();
Task start = starter.StartAsync(TestContext.Current.CancellationToken);
Task first = await Task.WhenAny(
start, Task.Delay(TimeSpan.FromSeconds(20), TestContext.Current.CancellationToken));
first.ShouldBe(start,
"resolving ISecretStore from the startup hook did not complete — the scadaproj#1 "
+ "DI deadlock has regressed");
await start;
// The store must be the replicating decorator, and the node's replication actor must
// genuinely exist under the configured name — not merely be registered.
sp.GetRequiredService<ISecretStore>().ShouldBeOfType<ReplicatingSecretStore>();
IActorRef actor = await system
.ActorSelection("/user/zb-secret-replication")
.ResolveOne(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken);
actor.Path.Name.ShouldBe("zb-secret-replication");
}
finally
{
await system.Terminate();
}
}
[Fact]
public void Replication_enabled_still_registers_the_undecorated_concrete_store()
{
// The decorator is constructed from the concrete SqliteSecretStore, not from ISecretStore
// (which would recurse). Core registering it only behind ISecretStore is a gap that has
// already shipped in this library once, so pin it.
BuildServices(replicationEnabled: true)
.ShouldContain(d => d.ServiceType == typeof(SqliteSecretStore));
}
[Fact]
public void Replication_enabled_registers_a_startup_hook_that_forces_actor_creation()
{
// The replication actor is created LAZILY on first ISecretStore resolution. A node that
// never reads or writes a secret would therefore never join anti-entropy and would silently
// never converge — so the host must register a startup hook that forces the resolution.
using var sp = BuildProvider(replicationEnabled: true);
var starter = sp.GetServices<IHostedService>().OfType<SecretReplicationStarter>().SingleOrDefault();
starter.ShouldNotBeNull(
"replication must not depend on something happening to resolve ISecretStore later");
}
[Fact]
public void Replication_binds_a_real_replicator_not_the_no_op_sink()
{
// THE REGRESSION GUARD for the upstream defect found during this adoption.
// ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 never bound its own ISecretReplicator:
// AddZbSecretsAkkaReplication called AddZbSecrets FIRST, which does
// TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(), so the package's own TryAdd
// was silently discarded. Writes published into a sink and no actor was ever spawned, with
// no exception and no log line. Fixed in 0.2.1 by registering before that call.
var descriptor = BuildServices(replicationEnabled: true)
.Single(d => d.ServiceType == typeof(ISecretReplicator));
// The no-op is registered by concrete type; the real replicator via a factory, because it
// needs the lazily-created actor ref. A factory here proves the package's descriptor won.
descriptor.ImplementationType.ShouldBeNull(
"ISecretReplicator resolved to the no-op sink — replication would publish into nothing");
descriptor.ImplementationFactory.ShouldNotBeNull();
}
[Fact]
public void Replication_disabled_registers_no_startup_hook()
{
using var sp = BuildProvider(replicationEnabled: false);
sp.GetServices<IHostedService>().OfType<SecretReplicationStarter>().ShouldBeEmpty();
}
}