fix(secrets): Akka replication actor starts with the host (scadaproj#33)

The replication actor was created only when something resolved ISecretStore
(store -> replicator -> SecretReplicationActorProvider.ActorRef). A node that
expanded its ${secret:} references pre-host and never touched the store at
runtime had no actor: no topic subscription, no announces, no pull replies.

AddZbSecretsAkkaReplication now registers SecretReplicationActorStarter, which
creates the actor on IHostApplicationLifetime.ApplicationStarted (after every
hosted service, incl. an Akka.Hosting AkkaHostedService registered later), or
in StartAsync when no lifetime is registered.

- tests: EagerActorStartTests (generic host in both registration orders with a
  no-actor-before-start control; bare container) - all fail without the fix
- docs: README, CHANGELOG
- code-reviews: Secrets-010 Resolved; README regenerated

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 14:54:53 -05:00
parent f60f641069
commit 6112a9f447
8 changed files with 214 additions and 15 deletions
@@ -30,8 +30,9 @@ public static class AkkaSecretsServiceCollectionExtensions
/// distributed failure modes at all; prefer it unless partition tolerance is a stated requirement.
/// </para>
/// <para>
/// <b>Requirements.</b> The application must (1) register its <see cref="ActorSystem"/> in DI
/// before the first secret resolve, and (2) give every node the <b>same KEK</b> — rows carry
/// <b>Requirements.</b> The application must (1) register its <see cref="ActorSystem"/> in DI —
/// the replication actor is created once the host has started (after every hosted service,
/// when <c>IHostApplicationLifetime</c> is available), so the system must be resolvable by then — and (2) give every node the <b>same KEK</b> — rows carry
/// ciphertext only, so a node with a different master key fails closed on a <c>kek_id</c>
/// mismatch. Merging <see cref="AkkaSecretsReplication.SerializationConfig"/> into the actor
/// system's configuration is recommended.
@@ -73,9 +74,10 @@ public static class AkkaSecretsServiceCollectionExtensions
section.Bind(eager);
eager.Validate();
// The actor is created lazily on first use rather than at registration: the ActorSystem is
// typically registered by the application AFTER this call, and a cluster node may also not be
// ready to join at the moment the container is built.
// The actor is created lazily rather than at registration: the ActorSystem is typically
// registered by the application AFTER this call, and a cluster node may also not be ready to
// join at the moment the container is built. SecretReplicationActorStarter (registered below)
// forces it into existence once the host has started (scadaproj#33).
//
// The invalidator MUST stay deferred (scadaproj#1): it is the resolver, the resolver reads
// ISecretStore, and ISecretStore is the ReplicatingSecretStore decorator registered below —
@@ -111,6 +113,11 @@ public static class AkkaSecretsServiceCollectionExtensions
sp.GetRequiredService<ISecretReplicator>(),
sp.GetRequiredService<ILogger<ReplicatingSecretStore>>()));
// Create the actor at host start rather than on the first ISecretStore resolve
// (scadaproj#33), so a node that never touches the store at runtime still replicates.
// AddHostedService de-duplicates, so a repeated call registers one starter.
services.AddHostedService<SecretReplicationActorStarter>();
return services;
}
}
@@ -0,0 +1,73 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
/// <summary>
/// Creates this node's replication actor at host start, so participation in replication does not
/// depend on something happening to resolve <c>ISecretStore</c> (scadaproj#33).
/// </summary>
/// <remarks>
/// <para>
/// The actor is created by <see cref="SecretReplicationActorProvider.ActorRef"/>, which until 0.6.6
/// was read only through the <c>ISecretStore</c> → <c>ISecretReplicator</c> chain. A node that
/// expanded its <c>${secret:}</c> references pre-host and never touched the store at runtime had no
/// actor: no topic subscription, no announces, no pull replies — it silently missed every rotation
/// and served the stale credential on its next restart.
/// </para>
/// <para>
/// When the host exposes <see cref="IHostApplicationLifetime"/>, creation waits for
/// <see cref="IHostApplicationLifetime.ApplicationStarted"/> — after EVERY hosted service has started,
/// including an Akka.Hosting <c>AkkaHostedService</c> that may be registered after this package — so
/// the starter never races the application's own actor-system startup, whatever the registration
/// order. A failure there is logged, because nothing can propagate it; the actor provider caches the
/// failure, so the first store resolve rethrows it as before. Without a lifetime (a bare container
/// whose hosted services are started by hand) the actor is created inside
/// <see cref="StartAsync"/> and a failure propagates.
/// </para>
/// <para>
/// Resolving <see cref="SecretReplicationActorProvider"/> does not re-enter the scadaproj#1 cycle:
/// its cache invalidator is still the deferred one, resolved only at the first eviction.
/// </para>
/// </remarks>
internal sealed class SecretReplicationActorStarter(
IServiceProvider services,
ILogger<SecretReplicationActorStarter> logger) : IHostedService
{
/// <inheritdoc />
public Task StartAsync(CancellationToken cancellationToken)
{
IHostApplicationLifetime? lifetime = services.GetService<IHostApplicationLifetime>();
if (lifetime is null)
{
CreateActor();
return Task.CompletedTask;
}
lifetime.ApplicationStarted.Register(() =>
{
try
{
CreateActor();
}
catch (Exception ex)
{
logger.LogError(
ex,
"The secret replication actor could not be created at startup; this node will not "
+ "receive or announce secret changes until the cause is fixed.");
}
});
return Task.CompletedTask;
}
/// <inheritdoc />
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
// Reading ActorRef is the side effect: the provider's Lazy spawns the actor exactly once, shared
// with every later ISecretReplicator resolve.
private void CreateActor() =>
_ = services.GetRequiredService<SecretReplicationActorProvider>().ActorRef;
}