dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(mesh): resolve dialer + PSK provider lazily — central boots again

Browse Source 
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:

  Akka.Hosting.MissingActorRegistryEntryException: No actor registered
  for key ...ControlPlane.AuditWriterActorKey
     at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
     at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>

WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.

Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.

Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).

Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.

Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.

ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
This commit is contained in:
Joseph Doherty
2026-08-06 07:55:31 -04:00
parent 410c2e2925
commit c0475f2df4
4 changed files with 333 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs
+4 -4
View File
@@ -742,8 +742,8 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
new MeshTransportOptions { Mode = mode, ContactRefreshSeconds = 60 },
clients,
() => null,
new RecordingDialer(),
psk));
() => new RecordingDialer(),
() => psk));
AwaitCondition(
() => psk.KnownClusterPublications.Count >= 1, TimeSpan.FromSeconds(5));
@@ -855,8 +855,8 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
GrpcMode(),
clientFactory ?? new RecordingClientFactory(() => CreateTestProbe()),
() => coordinator,
dialer,
psk));
() => dialer,
() => psk));
return new GrpcRig(actor, dialer, psk);
}
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/ControlPlaneStarterOrderTests.cs
+233
View File
@@ -0,0 +1,233 @@
using System.Diagnostics.CodeAnalysis;
using Akka.Actor;
using Akka.Cluster.Hosting;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary>
/// Guards the one thing every other test in this project structurally cannot see: whether an
/// admin node <b>boots</b> with the real registration graph behind the mesh command boundary.
/// </summary>
/// <remarks>
/// <para>
/// The Phase 1A live gate failed here. <c>WithOtOpcUaControlPlaneSingletons</c>'s first
/// starter resolved <c>IMeshCommandDialer</c> and <c>IMeshPskProvider</c> eagerly, and their
/// graph reaches back into the actor registry — dialer → <c>IMeshPskProvider</c> →
/// <c>ISecretResolver</c> → <c>AddZbSecrets</c> → <c>IAuditWriter</c> →
/// <c>ActorAuditWriter</c>, whose constructor calls
/// <c>registry.Get&lt;AuditWriterActorKey&gt;()</c> for a singleton registered further down
/// the SAME builder. Every central node died on
/// <c>MissingActorRegistryEntryException</c>, in every transport mode, before serving
/// anything.
/// </para>
/// <para>
/// <b>Why the existing tests were blind.</b> <c>MeshGrpcCommandDialerTests</c> activates the
/// dialer from a <c>ServiceCollection</c> that has no secrets registration at all, so its
/// graph stops one hop in; the <c>CentralCommunicationActorTests</c> facts inject fakes
/// directly and never touch DI; and the existing whole-host boots register no mesh provider,
/// so the eager lookups returned <see langword="null"/> and resolved nothing. Every one of
/// them passed while no central node could start.
/// </para>
/// <para>
/// <b>So the assertion is <c>StartAsync</c> itself.</b> The failure happens while the Akka
/// builder's starters execute, which is inside <c>StartAsync</c> — building the
/// <c>ServiceProvider</c> and resolving from it by hand would not reproduce it. The graph
/// below is the production one from <c>IMeshCommandDialer</c> down to the registry-bound
/// audit writer; only the secret store itself is a stand-in, because a real one would add a
/// KEK and a SQLite file to a test whose subject is starter ORDER.
/// </para>
/// </remarks>
public sealed class ControlPlaneStarterOrderTests
{
[Theory]
[InlineData(MeshTransportOptions.ModeDps)]
[InlineData(MeshTransportOptions.ModeClusterClient)]
[InlineData(MeshTransportOptions.ModeGrpc)]
public async Task An_admin_node_boots_with_the_real_mesh_dependency_graph(string mode)
{
// Every mode, because the eager resolve was mode-independent: the starter resolved both
// dependencies before anything looked at MeshTransport:Mode, so Dps — the DEFAULT, and the
// rollback target — died exactly like Grpc did.
using var host = BuildAdminHost(mode);
await Should.NotThrowAsync(() => host.StartAsync());
try
{
var registry = host.Services.GetRequiredService<ActorRegistry>();
registry.TryGet<CentralCommunicationActorKey>(out _).ShouldBeTrue(
"the command boundary must be registered on a booted admin node");
registry.TryGet<AuditWriterActorKey>(out _).ShouldBeTrue(
"the audit singleton the eager resolve reached for too early must exist by now");
}
finally
{
await host.StopAsync();
}
}
[Fact]
public async Task The_deferred_dependencies_are_resolved_and_used_once_the_host_is_up()
{
// Deferring must not become "never resolved" — that would trade a loud boot failure for a
// silent one, with the known-cluster gate permanently unpublished and the dialer permanently
// absent. So this asserts the far end of the deferral: the actor resolved the PSK provider
// FROM THE CONTAINER and drove it, through the ordinary contact-refresh path.
//
// The signal is a TaskCompletionSource rather than a counter, because PreStart runs on the
// actor's own thread and the contact load is a piped DB read: there is no instant after
// StartAsync at which a synchronous assertion is correct.
var signal = new SignallingPskProvider();
using var host = BuildAdminHost(MeshTransportOptions.ModeGrpc, signal);
await host.StartAsync();
try
{
await signal.Published.WaitAsync(TimeSpan.FromSeconds(20));
}
finally
{
await host.StopAsync();
}
}
/// <summary>
/// Builds an admin host through the production cluster bootstrap and the production control-plane
/// starter registration, with the mesh dependency graph wired as it is on a real central node.
/// </summary>
/// <param name="mode">The mesh transport mode to boot under.</param>
/// <param name="pskProvider">
/// Substitute for the real <see cref="MeshPskProvider"/>. Only the resolution test supplies
/// one; the boot test deliberately keeps the production type so the graph it walks is the
/// production graph.
/// </param>
/// <returns>The unstarted host.</returns>
private static IHost BuildAdminHost(string mode, IMeshPskProvider? pskProvider = null)
{
var clusterOptions = new AkkaClusterOptions
{
Port = 0,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
SeedNodes = [],
Roles = ["admin"],
};
return Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(clusterOptions));
services.AddSingleton<IOptions<MeshTransportOptions>>(Options.Create(
new MeshTransportOptions { Mode = mode, DispatchTimeoutSeconds = 10 }));
services.AddDbContextFactory<OtOpcUaConfigDbContext>(
o => o.UseInMemoryDatabase($"starter-order-{Guid.NewGuid():N}"));
services.AddSingleton<IClusterRoleInfo>(new StubClusterRoleInfo());
// THE GRAPH. Production types from the dialer down to the registry-bound writer; the
// order of these registrations is deliberately irrelevant, because the defect was
// never about registration order — it was about WHEN the graph is first walked.
services.AddSingleton<IAuditWriter, RegistryBoundAuditWriter>();
services.AddSingleton<ISecretResolver, AuditingSecretResolver>();
if (pskProvider is null)
services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
else
services.AddSingleton(pskProvider);
services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
services.AddAkka("otopcua-starter-order-test", (ab, sp) =>
{
ab.WithOtOpcUaClusterBootstrap(sp);
ab.WithOtOpcUaControlPlaneSingletons();
});
})
.Build();
}
/// <summary>
/// Stands in for <c>ActorAuditWriter</c>: an <see cref="IAuditWriter"/> that resolves a
/// cluster singleton from the <see cref="ActorRegistry"/> <b>in its constructor</b>. That
/// constructor is the tripwire — it throws
/// <c>MissingActorRegistryEntryException</c> for exactly as long as the audit singleton has
/// not been registered yet.
/// </summary>
private sealed class RegistryBoundAuditWriter : IAuditWriter
{
private readonly IActorRef _proxy;
public RegistryBoundAuditWriter(ActorRegistry registry) => _proxy = registry.Get<AuditWriterActorKey>();
public Task WriteAsync(AuditEvent evt, CancellationToken ct = default)
{
_proxy.Tell(evt);
return Task.CompletedTask;
}
}
/// <summary>
/// An <see cref="IMeshPskProvider"/> that completes a task the first time the actor publishes
/// the known-cluster set. Takes <see cref="ISecretResolver"/> it never uses, purely so that
/// resolving it still walks the same registry-bound chain the real provider does.
/// </summary>
private sealed class SignallingPskProvider : IMeshPskProvider
{
private readonly TaskCompletionSource _published =
new(TaskCreationOptions.RunContinuationsAsynchronously);
/// <summary>Completes once <see cref="SetKnownClusters"/> has been called.</summary>
public Task Published => _published.Task;
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
{
psk = null;
return false;
}
public bool Matches(string clusterId, string presented) => false;
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) => _published.TrySetResult();
public void Invalidate(string clusterId) { }
public void InvalidateAll() { }
}
/// <summary>
/// Stands in for the shared secret store's resolver. Its only load-bearing property is the
/// one that broke boot: it depends on <see cref="IAuditWriter"/>, exactly as
/// <c>AddZbSecrets</c>' resolver registration does.
/// </summary>
private sealed class AuditingSecretResolver : ISecretResolver
{
public AuditingSecretResolver(IAuditWriter auditWriter) => ArgumentNullException.ThrowIfNull(auditWriter);
public Task<string?> GetAsync(SecretName name, CancellationToken ct) => Task.FromResult<string?>(null);
}
/// <summary>Minimal <see cref="IClusterRoleInfo"/>; the admin singleton factories read it.</summary>
private sealed class StubClusterRoleInfo : IClusterRoleInfo
{
public string? ClusterRole => null;
public string? ClusterId => null;
public NodeId LocalNode => NodeId.Parse("127.0.0.1:0");
public IReadOnlySet<string> LocalRoles => new HashSet<string>();
public bool HasRole(string role) => false;
public IReadOnlyList<NodeId> MembersWithRole(string role) => [];
public NodeId? RoleLeader(string role) => null;
public event EventHandler<RoleLeaderChangedEventArgs>? RoleLeaderChanged { add { } remove { } }
}
}