fix(mesh): resolve dialer + PSK provider lazily — central boots again

Phase 1A live gate defect 4. Neither central node booted on this branch, in ANY transport mode: Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ...ControlPlane.AuditWriterActorKey at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons> WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their graph reaches back into the registry — dialer -> IMeshPskProvider -> ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter, whose ctor calls registry.Get<AuditWriterActorKey>() — and WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the same builder. Boot died before the entry existed. Mode-independent: the resolve happened before anything consulted MeshTransport:Mode, so Dps (the default, and the rollback target) died exactly like Grpc. Both dependencies now arrive as Func<> and are invoked in PreStart, by which point the host is fully built — the same shape, for the same class of reason, as the coordinator proxy Func one argument earlier. PreStart re-resolves on restart rather than caching across one; both are container singletons, so a restart gets the same instances. A resolution throw is caught and logged rather than propagated: a throw out of PreStart is an actor failure, the supervisor answers it with a restart, and a broken DI graph is not transient — that is an unbounded restart loop that would take the transport-independent ack path down with it. Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons in production starter order, with the production graph from IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts StartAsync itself — the failure occurs while the builder's starters execute, so building a ServiceProvider and resolving by hand cannot reproduce it. Verified RED against the eager resolve: 3/3 modes fail with MissingActorRegistryEntryException. A second fact pins that the deferral did not become 'never resolved' — the actor must resolve the provider from the container and drive it through the contact refresh, signalled by a TaskCompletionSource (PreStart runs on the actor thread and the contact load is a piped DB read, so no synchronous assertion after StartAsync is correct). Why nothing caught this: the dialer's container-activation test resolves from a collection with no secrets registration, so its graph stops one hop in; the actor facts inject fakes and never touch DI; and the existing whole-host boots register no mesh provider, so the eager lookups returned null and walked nothing. Two aggravators recorded by the gate, no code here: the failed process did not exit (docker Up while /health/ready 503s — Phase 2 readiness concern, tracked), and this is the third instance of the same eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's IClusterRoleInfo boot crash. ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184, 0 warnings in every touched project.
2026-08-06 07:55:31 -04:00
parent 410c2e2925
commit c0475f2df4
4 changed files with 333 additions and 25 deletions
@@ -64,8 +64,15 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    private readonly MeshTransportOptions _options;
    private readonly IMeshClusterClientFactory _clientFactory;
    private readonly Func<IActorRef?> _coordinator;
-    private readonly IMeshCommandDialer? _dialer;
-    private readonly IMeshPskProvider? _pskProvider;
+
+    // Resolved in PreStart, never at construction. See the Props doc: the DI graph behind these two
+    // reaches actor-registry entries that do not exist yet while the Akka builder's starters are
+    // still running, and touching it there is what stopped every central node from booting.
+    private readonly Func<IMeshCommandDialer?>? _dialerFactory;
+    private readonly Func<IMeshPskProvider?>? _pskProviderFactory;
+    private IMeshCommandDialer? _dialer;
+    private IMeshPskProvider? _pskProvider;
+
    private readonly ILoggingAdapter _log = Context.GetLogger();
    private readonly bool _useClusterClient;
    private readonly bool _useGrpc;
@@ -100,24 +107,45 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    ///     time — it resolves on first use and caches.
    /// </param>
    /// <param name="dialer">
-    ///     Places one command call on one node, under <see cref="MeshTransportOptions.ModeGrpc"/>.
-    ///     Optional so the two older transports need nothing new wired; a <see langword="null"/>
-    ///     dialer under <c>Grpc</c> is a mis-wiring and is reported loudly rather than silently
-    ///     dropping every command.
+    ///     Resolves the thing that places one command call on one node, under
+    ///     <see cref="MeshTransportOptions.ModeGrpc"/>. Optional so the two older transports need
+    ///     nothing new wired; a <see langword="null"/> dialer under <c>Grpc</c> is a mis-wiring and
+    ///     is reported loudly rather than silently dropping every command.
    /// </param>
    /// <param name="pskProvider">
-    ///     Central's per-cluster key lookup, told on every contact refresh which clusters exist (the
-    ///     known-cluster gate) and to drop its cache (so a rotated key takes effect within one
-    ///     refresh). Optional for the same reason as <paramref name="dialer"/>.
+    ///     Resolves central's per-cluster key lookup, told on every contact refresh which clusters
+    ///     exist (the known-cluster gate) and to drop its cache (so a rotated key takes effect within
+    ///     one refresh). Optional for the same reason as <paramref name="dialer"/>.
    /// </param>
    /// <returns>The props.</returns>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Both dependencies arrive as <see cref="Func{TResult}"/>, and that is not a style
+    ///         choice.</b> Passing instances meant the caller resolved them while the Akka builder's
+    ///         starters were still running, and the DI graph behind them reaches back into the actor
+    ///         registry: dialer → <c>IMeshPskProvider</c> → <c>ISecretResolver</c> → <c>AddZbSecrets</c>
+    ///         → <c>IAuditWriter</c> → <c>ActorAuditWriter</c>, whose constructor calls
+    ///         <c>registry.Get&lt;AuditWriterActorKey&gt;()</c> — a singleton registered dozens of
+    ///         lines LATER in the very same builder. Every central node failed to boot, in every
+    ///         transport mode, on a <c>MissingActorRegistryEntryException</c>. Deferring to
+    ///         <see cref="PreStart"/> moves the first resolve to a point where the host is fully
+    ///         built, and matches the <see cref="Func{TResult}"/> the coordinator proxy already uses
+    ///         one argument earlier for the same class of reason.
+    ///     </para>
+    ///     <para>
+    ///         This is the third instance of the same failure mode in this family — eager resolution
+    ///         inside an <c>AddAkka</c> configurator, previously <c>scadaproj#1</c> (a DI cycle that
+    ///         deadlocked) and mesh Phase 6's <c>IClusterRoleInfo</c> boot <c>StackOverflow</c>. The
+    ///         rule it keeps re-teaching: inside an Akka builder, resolve nothing you can defer.
+    ///     </para>
+    /// </remarks>
    public static Props Props(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        MeshTransportOptions options,
        IMeshClusterClientFactory clientFactory,
        Func<IActorRef?> coordinator,
-        IMeshCommandDialer? dialer = null,
-        IMeshPskProvider? pskProvider = null) =>
+        Func<IMeshCommandDialer?>? dialer = null,
+        Func<IMeshPskProvider?>? pskProvider = null) =>
        Akka.Actor.Props.Create(() =>
            new CentralCommunicationActor(
                dbFactory, options, clientFactory, coordinator, dialer, pskProvider));
@@ -127,22 +155,22 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    /// <param name="options">The bound mesh-transport options.</param>
    /// <param name="clientFactory">Creates the ClusterClient.</param>
    /// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
-    /// <param name="dialer">Places one gRPC command call on one node; <c>Grpc</c> mode only.</param>
-    /// <param name="pskProvider">Central's per-cluster key lookup; <c>Grpc</c> mode only.</param>
+    /// <param name="dialer">Resolves the gRPC command dialer; invoked in <see cref="PreStart"/>.</param>
+    /// <param name="pskProvider">Resolves the PSK provider; invoked in <see cref="PreStart"/>.</param>
    public CentralCommunicationActor(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        MeshTransportOptions options,
        IMeshClusterClientFactory clientFactory,
        Func<IActorRef?> coordinator,
-        IMeshCommandDialer? dialer = null,
-        IMeshPskProvider? pskProvider = null)
+        Func<IMeshCommandDialer?>? dialer = null,
+        Func<IMeshPskProvider?>? pskProvider = null)
    {
        _dbFactory = dbFactory;
        _options = options;
        _clientFactory = clientFactory;
        _coordinator = coordinator;
-        _dialer = dialer;
-        _pskProvider = pskProvider;
+        _dialerFactory = dialer;
+        _pskProviderFactory = pskProvider;
        _useClusterClient = string.Equals(
            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
        _useGrpc = string.Equals(
@@ -183,6 +211,12 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    /// <inheritdoc />
    protected override void PreStart()
    {
+        // First point at which the host is fully built — see the Props remark. Re-runs on every
+        // actor restart, which re-reads the container rather than caching a stale reference; both
+        // dependencies are container singletons, so a restart gets the same instances back.
+        _dialer = Resolve(_dialerFactory, nameof(IMeshCommandDialer));
+        _pskProvider = Resolve(_pskProviderFactory, nameof(IMeshPskProvider));
+
        if (!_useClusterClient && !_useGrpc)
        {
            _log.Info(
@@ -229,6 +263,41 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
            TimeSpan.FromSeconds(_options.ContactRefreshSeconds));
    }

+    /// <summary>
+    ///     Invokes a dependency factory, converting a resolution failure into a logged absence.
+    /// </summary>
+    /// <typeparam name="T">The dependency's type.</typeparam>
+    /// <param name="factory">The factory, or <see langword="null"/> when nothing was supplied.</param>
+    /// <param name="name">The dependency's name, for the log.</param>
+    /// <returns>The resolved dependency, or <see langword="null"/>.</returns>
+    /// <remarks>
+    ///     Catching is the point. A throw out of <see cref="PreStart"/> is an actor failure, and the
+    ///     supervisor's answer to that is to restart — which calls <see cref="PreStart"/> again, and
+    ///     a broken DI graph is not a transient condition. That is an unbounded restart loop on the
+    ///     node's command boundary, taking the transport-independent ack path down with it. Logging
+    ///     and continuing leaves the ack path serving and hands the two null guards below a genuine
+    ///     absence to report, which is the same outcome as never having been wired.
+    /// </remarks>
+    private T? Resolve<T>(Func<T?>? factory, string name)
+        where T : class
+    {
+        if (factory is null)
+            return null;
+
+        try
+        {
+            return factory();
+        }
+        catch (Exception ex)
+        {
+            _log.Error(
+                ex,
+                "Failed to resolve {Dependency} for the mesh command boundary; continuing without "
+                + "it. The guards below report what that costs", name);
+            return null;
+        }
+    }
+
    /// <inheritdoc />
    protected override void PostStop()
    {
@@ -66,14 +66,20 @@ public static class ServiceCollectionExtensions
            var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
            var options = resolver.GetService<IOptions<MeshTransportOptions>>().Value;

+            // Captured, NOT resolved. Resolving the mesh dialer or PSK provider here made every
+            // central node fail to boot in every transport mode: their graph is dialer →
+            // IMeshPskProvider → ISecretResolver → AddZbSecrets → IAuditWriter → ActorAuditWriter,
+            // whose constructor calls registry.Get<AuditWriterActorKey>() — and WithSingleton
+            // <AuditWriterActorKey> is registered further down THIS builder, so the entry does not
+            // exist while this starter runs. The actor invokes these in PreStart instead, by which
+            // point the host is built. Same shape, same reason, as the coordinator Func below.
+            //
            // Through IServiceProvider rather than the Akka resolver, deliberately: these two are
            // OPTIONAL (Grpc mode only, and admin-role only within that), and GetService's
            // null-for-absent contract is the thing being relied on. The Akka resolver's
            // GetService<T> makes no such promise, and this same block runs in test harnesses whose
            // service collections register neither.
            var serviceProvider = resolver.GetService<IServiceProvider>();
-            var dialer = serviceProvider.GetService<IMeshCommandDialer>();
-            var pskProvider = serviceProvider.GetService<IMeshPskProvider>();

            var actor = system.ActorOf(
                CentralCommunicationActor.Props(
@@ -85,8 +91,8 @@ public static class ServiceCollectionExtensions
                    () => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
                        ? coordinator
                        : null,
-                    dialer,
-                    pskProvider),
+                    () => serviceProvider.GetService<IMeshCommandDialer>(),
+                    () => serviceProvider.GetService<IMeshPskProvider>()),
                MeshPaths.CentralCommunicationName);

            // Registered unconditionally, in both transport modes. Under Dps nothing dials it, and