fix(alarms): refuse a keyed native ack with no Galaxy identity synchronously, from a cached key set (lmxopcua#765)
v2-ci / unit-tests (push) Waiting to run

Item 3 of #765. A keyed (API-key) session whose key had no Galaxy ack identity
got Good and then a revert, because whether the key has an identity lives in
the Secrets store, which must not be read under the SDK lock.

GalaxyAckIdentityDirectory (Driver.Galaxy) keeps a node-wide set of the key ids
that have a galaxy-ack-user secret. It lists this node's Secrets store
(metadata only, nothing decrypted) and is read lock-free.
GalaxyDriver.GetAcknowledgeRefusal (IAlarmAcknowledgePreflight, published
through NativeAlarmAckPreflightView as in #760) now refuses a keyed session
whose key the directory reports Absent, when the driver has no fallback ack
account. A refusal also asks for a refresh.

Fails open: before the first refresh, once the snapshot is older than
StaleAfter, and from a signalled identity-secret change until the next
refresh, the lookup answers Unknown and the ack takes the old Good-then-revert
path.

Refresh: GalaxyAckIdentityRefreshService runs on driver nodes, every
Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds (default 30;
StaleAfterSeconds 0 = three intervals; validated at start). It also refreshes
early on secret changes. The Secrets library has no change event, so
SecretsRegistration decorates ISecretCacheInvalidator, which is called by
AdminUI /api-keys, /admin/secrets and the replication appliers. API-key
changes need no event: a revoked key is already denied before the pre-check.

Tests: Galaxy.Tests GalaxyAckIdentityDirectoryTests (16) and Host.Tests
GalaxyAckIdentityCacheWiringTests (9). Seven separate mutations each turn at
least one test red: pre-check ignores the directory, factory drops it, no
invalidator decoration, no staleness check, no change-pending check, no
refresh on refusal, loop never wakes early.
Docs: Galaxy.md, AlarmTracking.md, Configuration.md, .claude/rules/alarms.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 16:38:35 -05:00
parent c9288a56c0
commit 0702b31029
15 changed files with 1034 additions and 28 deletions
+7 -3
View File
@@ -32,10 +32,14 @@ SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that fie
`ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`).
**A native ack the driver refuses or fails is not left Acknowledged (#760).** Two layers: (1) a refusal the
driver can decide without I/O (`IAlarmAcknowledgePreflight` — Galaxy: anonymous session and no fallback ack
account) fails the Part 9 call `BadUserAccessDenied` via `NativeAlarmAckPreflight`, read lock-free from the
driver can decide without I/O (`IAlarmAcknowledgePreflight` — Galaxy, no fallback ack account: an anonymous session, or a keyed
session whose key `GalaxyAckIdentityDirectory` reports Absent, #765) fails the Part 9 call `BadUserAccessDenied` via `NativeAlarmAckPreflight`, read lock-free from the
`NativeAlarmAckPreflightView` snapshot `DriverHostActor` publishes; never do I/O (e.g. a Secrets lookup) in a
pre-check — it runs under the SDK `Lock`. (2) anything else (keyed session without an identity, MXAccess
pre-check — it runs under the SDK `Lock`; that is why the keyed case reads a cached, periodically refreshed key set
(`Drivers:Galaxy:AckIdentityCache`), which must answer Unknown (= let it through) whenever it is not certain — no
refresh yet, stale, or an identity-secret change signalled through the decorated `ISecretCacheInvalidator` and not
yet re-listed. Only an identity written straight into the store by the `secret` CLI (no in-process signal) can be
refused until the next refresh, which the refusal itself requests. (2) anything else (a keyed session the cache cannot vouch for, MXAccess
non-zero `hresult`, driver still connecting/reconnecting) comes back as `DriverInstanceActor.AlarmAckFailed` — or, for a
disabled/unspawned/stubbed driver, is decided in the host itself (#765) — and `DriverHostActor` re-projects the condition unacknowledged — only while the ack is still pending in
`NativeAlarmProjector` (any driver transition since then wins). A host that came up **Stale** cannot route any
+4 -2
View File
@@ -468,8 +468,10 @@ from the scripted `AlarmCommandRouter`:
not yet spawned) or is a stub on this node is reverted the same way by
the host itself (#765). Only an ack still pending in `NativeAlarmProjector` is
reverted: a driver transition projected since then is authoritative. A
refusal the driver can decide without I/O (Galaxy: an anonymous session
and no fallback ack account) is instead returned synchronously as
refusal the driver can decide without I/O (Galaxy, with no fallback ack
account: an anonymous session, or a keyed session whose key the cached
`GalaxyAckIdentityDirectory` knows has no ack identity — #765) is instead
returned synchronously as
`BadUserAccessDenied` by the node manager's `NativeAlarmAckPreflight`
before anything is applied — as is every native ack while the driver host
is Stale (configuration database unreachable at boot), when no ack can be
+15
View File
@@ -536,6 +536,21 @@ deploy that recreates the driver), not on `ReinitializeAsync`. The init log line
`transport=Libplctag (node default)` when the pin is what selected the stack. There is **no AbLegacy
counterpart**: an AbLegacy driver with no key always runs libplctag (Native is opt-in per driver).
### `Drivers:Galaxy:AckIdentityCache` (cached API-key → Galaxy ack identity set, #765)
- **Purpose:** the node-wide set of API key ids that have a Galaxy ack identity (a `galaxy-ack-user` secret),
which the Galaxy driver's synchronous ack pre-check reads to refuse a keyed session without one
(`BadUserAccessDenied`) instead of returning Good and reverting. See
[`drivers/Galaxy.md` § Alarm acknowledge credential](drivers/Galaxy.md#alarm-acknowledge-credential).
- **Options class:** `GalaxyAckIdentityCacheOptions` (`SectionName = "Drivers:Galaxy:AckIdentityCache"`) —
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/GalaxyAckIdentityCacheOptions.cs`; bound and validated
in `Program.cs`. The directory and its refresh loop exist on driver-role nodes only.
| Key | Type | Default | Meaning |
|---|---|---|---|
| `RefreshIntervalSeconds` | int | `30` | How often the Secrets store is re-listed (1..3600). An in-process identity change or a refusal also triggers an early refresh. |
| `StaleAfterSeconds` | int | `0` | Snapshot age after which keyed acks fall back to Good-then-revert (fail open). `0` = three refresh intervals; otherwise must exceed `RefreshIntervalSeconds`. A bad value fails host start. |
---
## Environment variables
+17 -6
View File
@@ -108,12 +108,23 @@ Galaxy password, so `GalaxyAckCredentialResolver` picks the account per acknowle
3. **Neither** → the driver throws `DriverOperationRefusedException`; **nothing is sent to the
gateway**. The refusal is excluded from the resilience breaker (it says nothing about gateway
health). How the client learns of it (#760):
- **Synchronously** when the driver can tell without I/O — an *anonymous* session on a driver with
no fallback account: the Part 9 `Acknowledge` call itself fails `BadUserAccessDenied` with the
reason, and nothing is applied (`IAlarmAcknowledgePreflight`, published per condition by
`DriverHostActor` into `NativeAlarmAckPreflightView`).
- **By revert** otherwise — a keyed session whose key has no identity (that lives in the Secrets
store, which is not read under the SDK lock), or MXAccess rejecting the account (a non-zero
- **Synchronously** when the driver can tell without I/O, on a driver with no fallback account —
an *anonymous* session (#760), or a *keyed* session whose key has no identity (#765): the Part 9
`Acknowledge` call itself fails `BadUserAccessDenied` with the reason, and nothing is applied
(`IAlarmAcknowledgePreflight`, published per condition by `DriverHostActor` into
`NativeAlarmAckPreflightView`). The keyed case reads `GalaxyAckIdentityDirectory`, a node-wide
cached set of the key ids that have a `galaxy-ack-user` secret, listed from this node's Secrets
store (metadata only — nothing decrypted) every `Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds`
(default 30) by `GalaxyAckIdentityRefreshService`, and early whenever an identity secret changes
in-process (the Secrets cache invalidator is decorated, so AdminUI `/api-keys`, `/admin/secrets` and
replication all signal it) or a refusal asks for one. ⚠️ It **fails open**: before the first refresh,
once the snapshot is older than `StaleAfterSeconds` (default three intervals — e.g. the store cannot
be listed), and from a signalled identity change until the refresh that follows it, every keyed ack
takes the revert path below instead of being refused. An identity written straight into the store
by the `secret` CLI is seen at the next refresh; the refusal itself requests one, so a retry a moment
later goes through.
- **By revert** otherwise — a keyed session the directory cannot vouch for (see above), a key whose
identity is broken (a user with no password secret), or MXAccess rejecting the account (a non-zero
native `hresult`, e.g. a wrong password, now thrown as a refusal): the call returned `Good`, then
`DriverHostActor` re-projects the condition **unacknowledged** (a fresh condition event,
`Retain=true`) and logs the reason. A driver transition projected after the ack wins over a late
@@ -119,6 +119,9 @@ public sealed class GalaxyDriver
// lmxopcua#749: resolves the Galaxy account each acknowledge authenticates as (API key identity →
// fallback account → refused). Built in the ctor — it needs only options + the secret resolver.
private readonly IGalaxyAckCredentialResolver _ackCredentialResolver;
// lmxopcua#765: the cached key → has-Galaxy-identity set the synchronous ack pre-check reads (no I/O).
// Null = no directory wired, so a keyed session is never pre-refused (the pre-#765 behaviour).
private readonly IGalaxyAckIdentityDirectory? _ackIdentityDirectory;
private IGalaxyAlarmFeed? _alarmFeed;
private readonly Lock _alarmHandlersLock = new();
private readonly Lock _alarmFeedLock = new();
@@ -262,15 +265,21 @@ public sealed class GalaxyDriver
/// <param name="options">The Galaxy driver configuration options.</param>
/// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c>.</param>
/// <param name="logger">Optional logger instance for diagnostics.</param>
/// <param name="ackIdentityDirectory">
/// Optional cached set of API keys with a Galaxy ack identity (lmxopcua#765), consulted by the synchronous
/// ack pre-check. Null = keyed sessions are never pre-refused.
/// </param>
public GalaxyDriver(
string driverInstanceId,
GalaxyDriverOptions options,
ISecretResolver secretResolver,
ILogger<GalaxyDriver>? logger = null)
ILogger<GalaxyDriver>? logger = null,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
: this(driverInstanceId, options,
hierarchySource: null, dataReader: null, dataWriter: null, subscriber: null,
alarmAcknowledger: null, alarmFeed: null, logger,
secretResolver: secretResolver ?? throw new ArgumentNullException(nameof(secretResolver)))
secretResolver: secretResolver ?? throw new ArgumentNullException(nameof(secretResolver)),
ackIdentityDirectory: ackIdentityDirectory)
{
}
@@ -307,6 +316,7 @@ public sealed class GalaxyDriver
/// Optional ack-credential seam (lmxopcua#749). Defaults to <see cref="GalaxyAckCredentialResolver"/>
/// over <see cref="GalaxyDriverOptions.AlarmAck"/> and <paramref name="secretResolver"/>.
/// </param>
/// <param name="ackIdentityDirectory">Optional cached key → Galaxy-identity set (lmxopcua#765).</param>
internal GalaxyDriver(
string driverInstanceId,
GalaxyDriverOptions options,
@@ -320,8 +330,10 @@ public sealed class GalaxyDriver
ISecretResolver? secretResolver = null,
IGalaxyDeployWatchSource? deployWatchSource = null,
IGalaxyPlatformSource? platformSource = null,
IGalaxyAckCredentialResolver? ackCredentialResolver = null)
IGalaxyAckCredentialResolver? ackCredentialResolver = null,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
{
_ackIdentityDirectory = ackIdentityDirectory;
_deployWatchSource = deployWatchSource;
_platformSource = platformSource;
_driverInstanceId = !string.IsNullOrWhiteSpace(driverInstanceId)
@@ -1968,17 +1980,32 @@ public sealed class GalaxyDriver
/// <inheritdoc />
/// <remarks>
/// Only the case decidable without I/O (lmxopcua#760): an anonymous session (no API key, so no key-mapped
/// Galaxy identity can exist) on a driver with no <c>AlarmAck.FallbackUser</c>. A keyed session is let
/// through — whether the key has an identity lives in the Secrets store, which must not be read under the
/// SDK lock; <see cref="AcknowledgeAsync"/> still refuses it, and the host reverts the condition.
/// Only cases decidable without I/O, on a driver with no <c>AlarmAck.FallbackUser</c>: an anonymous session
/// (lmxopcua#760 — no API key, so no key-mapped Galaxy identity can exist), and a keyed session whose key the
/// <see cref="IGalaxyAckIdentityDirectory"/> reports <see cref="GalaxyAckIdentityPresence.Absent"/>
/// (lmxopcua#765 — a cached, periodically refreshed view of the Secrets store, because the store itself must
/// not be read under the SDK lock). Anything the directory does not know for certain (no directory, no
/// refresh yet, stale, a change pending) is let through: <see cref="AcknowledgeAsync"/> still refuses it,
/// and the host reverts the condition. A refusal also asks the directory to refresh, so a retry after an
/// identity was set out of band (the <c>secret</c> CLI) is not refused for a whole refresh interval.
/// </remarks>
public string? GetAcknowledgeRefusal(string? apiKeyId)
{
if (!string.IsNullOrWhiteSpace(apiKeyId) || (_options.AlarmAck?.HasFallback ?? false)) return null;
return $"the session is anonymous (no API key) and Galaxy driver '{_driverInstanceId}' has no " +
"AlarmAck.FallbackUser configured, so the acknowledge cannot be sent to Galaxy. Connect with an API key " +
"that has a Galaxy ack identity (AdminUI /api-keys) or configure the driver's fallback ack account.";
if (_options.AlarmAck?.HasFallback ?? false) return null;
if (string.IsNullOrWhiteSpace(apiKeyId))
{
return $"the session is anonymous (no API key) and Galaxy driver '{_driverInstanceId}' has no " +
"AlarmAck.FallbackUser configured, so the acknowledge cannot be sent to Galaxy. Connect with an API key " +
"that has a Galaxy ack identity (AdminUI /api-keys) or configure the driver's fallback ack account.";
}
if (_ackIdentityDirectory is not { } directory
|| directory.GetPresence(apiKeyId) != GalaxyAckIdentityPresence.Absent) return null;
directory.RequestRefresh();
return $"API key '{apiKeyId.Trim()}' has no Galaxy ack identity and Galaxy driver '{_driverInstanceId}' has no " +
"AlarmAck.FallbackUser configured, so the acknowledge cannot be sent to Galaxy. Map a Galaxy ack identity " +
"to the key (AdminUI /api-keys) or configure the driver's fallback ack account.";
}
private static string PrincipalOf(AlarmAcknowledgeRequest ack) =>
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
@@ -31,14 +32,20 @@ public static class GalaxyDriverFactoryExtensions
/// each instance's <c>Gateway.ApiKeySecretRef</c>.
/// </param>
/// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
/// <param name="ackIdentityDirectory">
/// Optional node-wide cached set of API keys with a Galaxy ack identity (lmxopcua#765), shared by every
/// Galaxy driver instance for the synchronous ack pre-check. Null = keyed sessions are never pre-refused.
/// </param>
public static void Register(
DriverFactoryRegistry registry,
ISecretResolver secretResolver,
ILoggerFactory? loggerFactory = null)
ILoggerFactory? loggerFactory = null,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
{
ArgumentNullException.ThrowIfNull(registry);
ArgumentNullException.ThrowIfNull(secretResolver);
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, secretResolver));
registry.Register(DriverTypeName,
(id, json) => CreateInstance(id, json, loggerFactory, secretResolver, ackIdentityDirectory));
}
/// <summary>
@@ -58,9 +65,11 @@ public static class GalaxyDriverFactoryExtensions
/// <param name="driverConfigJson">The driver configuration in JSON format.</param>
/// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
/// <param name="secretResolver">The secret resolver for the driver's <c>secret:</c> API-key arm.</param>
/// <param name="ackIdentityDirectory">Optional cached key → Galaxy-identity set (lmxopcua#765).</param>
/// <returns>The constructed Galaxy driver instance.</returns>
public static GalaxyDriver CreateInstance(
string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory, ISecretResolver secretResolver)
string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory, ISecretResolver secretResolver,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
{
ArgumentNullException.ThrowIfNull(secretResolver);
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
@@ -108,7 +117,8 @@ public static class GalaxyDriverFactoryExtensions
FallbackPasswordSecretRef: NullIfBlank(dto.AlarmAck?.FallbackPasswordSecretRef)),
};
return new GalaxyDriver(driverInstanceId, options, secretResolver, loggerFactory?.CreateLogger<GalaxyDriver>());
return new GalaxyDriver(
driverInstanceId, options, secretResolver, loggerFactory?.CreateLogger<GalaxyDriver>(), ackIdentityDirectory);
}
private static string? NullIfBlank(string? value) => string.IsNullOrWhiteSpace(value) ? null : value.Trim();
@@ -0,0 +1,216 @@
using System.Collections.Frozen;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
/// <summary>What the <see cref="IGalaxyAckIdentityDirectory"/> knows about one API key's Galaxy ack identity.</summary>
public enum GalaxyAckIdentityPresence
{
/// <summary>
/// Not known right now — no successful refresh yet, the last one is stale, a change is pending, or the key
/// id is not one the directory can look up. Callers must treat it as "may have one" (fail open).
/// </summary>
Unknown = 0,
/// <summary>The key has a Galaxy ack user secret (it may still be refused at ack time, e.g. no password).</summary>
Present,
/// <summary>The key has no Galaxy ack user secret as of a fresh refresh.</summary>
Absent,
}
/// <summary>
/// Lock-free answer to "does this API key have a Galaxy ack identity?" (lmxopcua#765), for the synchronous
/// native-ack pre-check (<see cref="ZB.MOM.WW.OtOpcUa.Core.Abstractions.IAlarmAcknowledgePreflight"/>), which
/// runs under the OPC UA SDK lock and must not read the Secrets store.
/// </summary>
public interface IGalaxyAckIdentityDirectory
{
/// <summary>Looks the key up in the cached snapshot. No I/O, never throws.</summary>
/// <param name="apiKeyId">The session's API key id.</param>
/// <returns>The key's presence, or <see cref="GalaxyAckIdentityPresence.Unknown"/> when not known.</returns>
GalaxyAckIdentityPresence GetPresence(string? apiKeyId);
/// <summary>Asks for a refresh soon (coalesced; never blocks). Used after a refusal so a retry sees a new identity.</summary>
void RequestRefresh();
}
/// <summary>
/// Production <see cref="IGalaxyAckIdentityDirectory"/>: a periodically refreshed set of the API key ids that
/// have a Galaxy ack user secret (<see cref="GalaxyAckIdentitySecrets.UserSecretName"/>) in this node's
/// Secrets store — the same store <see cref="GalaxyAckCredentialResolver"/> reads at ack time.
/// </summary>
/// <remarks>
/// <para>
/// Built from <see cref="ISecretStore.ListAsync"/> metadata only — nothing is decrypted, no password is
/// read. A refresh runs every <c>refreshInterval</c> (<see cref="RunAsync"/>), and early on
/// <see cref="NotifySecretChanged"/> / <see cref="RequestRefresh"/>.
/// </para>
/// <para>
/// ⚠️ Fails OPEN: anything not certain answers <see cref="GalaxyAckIdentityPresence.Unknown"/>, which
/// lets the ack through to today's path (the driver resolves at ack time and the host reverts a refusal).
/// That covers no refresh yet, a store with nothing to list, a snapshot older than <c>staleAfter</c>
/// (e.g. the store has been failing), and a change to an identity secret signalled since the snapshot
/// was taken — so an identity just set through AdminUI or arriving by replication is never refused in
/// the window before the next refresh. A change the process is not told about (the <c>secret</c> CLI
/// writing the store directly) is picked up by the next timed refresh, or the refresh a refusal kicks.
/// </para>
/// </remarks>
public sealed class GalaxyAckIdentityDirectory : IGalaxyAckIdentityDirectory, IDisposable
{
private const string UserSuffix = "/galaxy-ack-user";
private readonly ISecretStore? _store;
private readonly TimeProvider _time;
private readonly ILogger _logger;
private readonly SemaphoreSlim _wake = new(0, 1);
// Bumped on every signalled change to an identity secret; a snapshot taken under an older value is not trusted.
private long _changeGeneration;
private volatile Snapshot? _snapshot;
/// <summary>Initializes a new instance of the <see cref="GalaxyAckIdentityDirectory"/> class.</summary>
/// <param name="store">This node's Secrets store; null = nothing to list, so every lookup is Unknown.</param>
/// <param name="refreshInterval">How often <see cref="RunAsync"/> refreshes.</param>
/// <param name="staleAfter">How old a snapshot may get before lookups answer Unknown.</param>
/// <param name="timeProvider">Clock for snapshot age; defaults to the system clock.</param>
/// <param name="logger">Logger; never receives a secret value.</param>
public GalaxyAckIdentityDirectory(
ISecretStore? store, TimeSpan refreshInterval, TimeSpan staleAfter,
TimeProvider? timeProvider = null, ILogger? logger = null)
{
if (refreshInterval <= TimeSpan.Zero)
throw new ArgumentOutOfRangeException(nameof(refreshInterval), refreshInterval, "Must be positive.");
if (staleAfter <= TimeSpan.Zero)
throw new ArgumentOutOfRangeException(nameof(staleAfter), staleAfter, "Must be positive.");
_store = store;
RefreshInterval = refreshInterval;
StaleAfter = staleAfter;
_time = timeProvider ?? TimeProvider.System;
_logger = logger ?? NullLogger.Instance;
}
/// <summary>Gets the refresh period of <see cref="RunAsync"/>.</summary>
public TimeSpan RefreshInterval { get; }
/// <summary>Gets the snapshot age after which every lookup answers Unknown.</summary>
public TimeSpan StaleAfter { get; }
/// <inheritdoc />
public GalaxyAckIdentityPresence GetPresence(string? apiKeyId)
{
var snapshot = _snapshot;
if (snapshot is null) return GalaxyAckIdentityPresence.Unknown;
if (snapshot.Generation != Interlocked.Read(ref _changeGeneration)) return GalaxyAckIdentityPresence.Unknown;
if (_time.GetUtcNow() - snapshot.RefreshedUtc > StaleAfter) return GalaxyAckIdentityPresence.Unknown;
if (TryNormalize(apiKeyId) is not { } keyId) return GalaxyAckIdentityPresence.Unknown;
return snapshot.KeyIds.Contains(keyId) ? GalaxyAckIdentityPresence.Present : GalaxyAckIdentityPresence.Absent;
}
/// <inheritdoc />
public void RequestRefresh()
{
try
{
_wake.Release();
}
catch (SemaphoreFullException)
{
// A refresh is already requested — coalesced.
}
catch (ObjectDisposedException)
{
// Shutting down.
}
}
/// <summary>
/// Tells the directory a secret changed (wired from the Secrets cache-invalidation seam, which AdminUI
/// writes, the Secrets UI and replication all call). A Galaxy ack identity secret marks the current
/// snapshot untrusted until the next refresh, and requests one; any other name is ignored.
/// </summary>
/// <param name="name">The changed secret.</param>
public void NotifySecretChanged(SecretName name)
{
if (!name.Value.StartsWith(GalaxyAckIdentitySecrets.Prefix, StringComparison.Ordinal)) return;
Interlocked.Increment(ref _changeGeneration);
RequestRefresh();
}
/// <summary>Lists the store once and replaces the snapshot. Never throws except on cancellation.</summary>
/// <param name="cancellationToken">Cancels the listing.</param>
/// <returns>True when the snapshot was replaced; false on a failure (the old snapshot keeps ageing).</returns>
public async Task<bool> RefreshAsync(CancellationToken cancellationToken)
{
if (_store is null) return false;
// Read BEFORE listing: a change signalled while the listing runs leaves this snapshot untrusted.
var generation = Interlocked.Read(ref _changeGeneration);
try
{
var rows = await _store.ListAsync(includeDeleted: false, cancellationToken).ConfigureAwait(false);
var keyIds = new HashSet<string>(StringComparer.Ordinal);
foreach (var row in rows)
{
if (row.IsDeleted) continue;
var name = row.Name.Value;
if (!name.StartsWith(GalaxyAckIdentitySecrets.Prefix, StringComparison.Ordinal)
|| !name.EndsWith(UserSuffix, StringComparison.Ordinal)
|| name.Length <= GalaxyAckIdentitySecrets.Prefix.Length + UserSuffix.Length) continue;
if (TryNormalize(name[GalaxyAckIdentitySecrets.Prefix.Length..^UserSuffix.Length]) is { } keyId)
keyIds.Add(keyId);
}
_snapshot = new Snapshot(keyIds.ToFrozenSet(StringComparer.Ordinal), _time.GetUtcNow(), generation);
return true;
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogWarning(
"Galaxy ack identity directory: listing the Secrets store failed ({ExceptionType}); native acks from " +
"keyed sessions fall back to the ack-time check once the snapshot is older than {StaleAfter}",
ex.GetType().Name, StaleAfter);
return false;
}
}
/// <summary>Refreshes now, then every <see cref="RefreshInterval"/> or sooner when asked, until cancelled.</summary>
/// <param name="cancellationToken">Stops the loop.</param>
/// <returns>A task that completes when cancelled.</returns>
public async Task RunAsync(CancellationToken cancellationToken)
{
try
{
while (!cancellationToken.IsCancellationRequested)
{
await RefreshAsync(cancellationToken).ConfigureAwait(false);
await _wake.WaitAsync(RefreshInterval, cancellationToken).ConfigureAwait(false);
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// Normal shutdown.
}
}
/// <inheritdoc />
public void Dispose() => _wake.Dispose();
// Same shape GalaxyAckIdentitySecrets accepts (trimmed, lower-case ASCII letters/digits); anything else is
// not a key id the resolver could look up, so the directory does not claim to know about it.
private static string? TryNormalize(string? apiKeyId)
{
if (string.IsNullOrWhiteSpace(apiKeyId)) return null;
var id = apiKeyId.Trim().ToLowerInvariant();
foreach (var c in id)
{
if (!char.IsAsciiLetterOrDigit(c)) return null;
}
return id;
}
private sealed record Snapshot(FrozenSet<string> KeyIds, DateTimeOffset RefreshedUtc, long Generation);
}
@@ -0,0 +1,72 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Node-level settings for the cached set of API keys that have a Galaxy ack identity (lmxopcua#765), bound
/// from <c>Drivers:Galaxy:AckIdentityCache</c>. The set lets the synchronous native-ack pre-check refuse a
/// keyed session without an identity (<c>BadUserAccessDenied</c>) instead of returning Good and reverting.
/// Overridable by environment variables <c>Drivers__Galaxy__AckIdentityCache__RefreshIntervalSeconds</c> /
/// <c>…__StaleAfterSeconds</c>.
/// </summary>
public sealed class GalaxyAckIdentityCacheOptions
{
/// <summary>The configuration section path.</summary>
public const string SectionName = "Drivers:Galaxy:AckIdentityCache";
/// <summary>How often the set is re-read from the Secrets store, in seconds (1..3600). Default 30.</summary>
public int RefreshIntervalSeconds { get; set; } = 30;
/// <summary>
/// How old the set may get, in seconds, before keyed acks fall back to the pre-#765 Good-then-revert path
/// (fail open, e.g. while the store cannot be listed). 0 (the default) = three refresh intervals. When set,
/// must be greater than <see cref="RefreshIntervalSeconds"/>.
/// </summary>
public int StaleAfterSeconds { get; set; }
/// <summary>Gets the refresh period.</summary>
public TimeSpan RefreshInterval => TimeSpan.FromSeconds(RefreshIntervalSeconds);
/// <summary>Gets the staleness bound (explicit, or three refresh intervals).</summary>
public TimeSpan StaleAfter => StaleAfterSeconds > 0
? TimeSpan.FromSeconds(StaleAfterSeconds)
: TimeSpan.FromSeconds(RefreshIntervalSeconds * 3L);
}
/// <summary>Fail-fast startup validator for <see cref="GalaxyAckIdentityCacheOptions"/>.</summary>
public sealed class GalaxyAckIdentityCacheOptionsValidator : OptionsValidatorBase<GalaxyAckIdentityCacheOptions>
{
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, GalaxyAckIdentityCacheOptions options)
{
builder.RequireThat(
options.RefreshIntervalSeconds is >= 1 and <= 3600,
$"{GalaxyAckIdentityCacheOptions.SectionName}:RefreshIntervalSeconds is {options.RefreshIntervalSeconds} — " +
"expected 1..3600.");
builder.RequireThat(
options.StaleAfterSeconds == 0 || options.StaleAfterSeconds > options.RefreshIntervalSeconds,
$"{GalaxyAckIdentityCacheOptions.SectionName}:StaleAfterSeconds is {options.StaleAfterSeconds} — expected 0 " +
$"(three refresh intervals) or more than RefreshIntervalSeconds ({options.RefreshIntervalSeconds}); otherwise " +
"the set would go stale between refreshes and never be used.");
}
}
/// <summary>Composition-root registration of <see cref="GalaxyAckIdentityCacheOptions"/>.</summary>
public static class GalaxyAckIdentityCacheRegistration
{
/// <summary>
/// Binds + validates <c>Drivers:Galaxy:AckIdentityCache</c>. A bad value fails host start. Without this call
/// the driver factories use the compiled defaults.
/// </summary>
/// <param name="services">The service collection.</param>
/// <param name="configuration">The host configuration.</param>
/// <returns>The same collection.</returns>
public static IServiceCollection AddGalaxyAckIdentityCache(this IServiceCollection services, IConfiguration configuration)
{
services.AddValidatedOptions<GalaxyAckIdentityCacheOptions, GalaxyAckIdentityCacheOptionsValidator>(
configuration, GalaxyAckIdentityCacheOptions.SectionName);
return services;
}
}
@@ -0,0 +1,26 @@
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Decorates the Secrets library's <see cref="ISecretCacheInvalidator"/> so a secret change also reaches the
/// <see cref="GalaxyAckIdentityDirectory"/> (lmxopcua#765). The library has no change event; this seam is the
/// one every in-process writer already calls — the AdminUI <c>/api-keys</c> ack-identity service, the
/// <c>/admin/secrets</c> editor and delete modal, and the replication appliers (Akka / SQL Server).
/// </summary>
/// <param name="inner">The library's invalidator (the resolver cache).</param>
/// <param name="directory">The node's directory, or null on a node that runs no drivers.</param>
internal sealed class SecretChangeSignallingInvalidator(
ISecretCacheInvalidator inner, GalaxyAckIdentityDirectory? directory) : ISecretCacheInvalidator
{
/// <summary>Gets the decorated invalidator (test seam).</summary>
internal ISecretCacheInvalidator Inner => inner;
/// <inheritdoc />
public void Invalidate(SecretName name)
{
inner.Invalidate(name);
directory?.NotifySecretChanged(name);
}
}
@@ -1,6 +1,8 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
@@ -67,6 +69,7 @@ public static class SecretsRegistration
if (!IsReplicationEnabled(configuration))
{
services.AddZbSecrets(configuration, SecretsSectionPath);
DecorateCacheInvalidator(services);
return services;
}
@@ -77,6 +80,27 @@ public static class SecretsRegistration
services.AddSingleton<SecretReplicationStarter>();
services.AddHostedService(sp => sp.GetRequiredService<SecretReplicationStarter>());
DecorateCacheInvalidator(services);
return services;
}
/// <summary>
/// Wraps the library's <see cref="ISecretCacheInvalidator"/> registration in
/// <see cref="SecretChangeSignallingInvalidator"/>, so every in-process secret write and replicated apply
/// also tells the Galaxy ack identity directory (lmxopcua#765). Every consumer resolves the invalidator
/// through the interface (the replication appliers lazily), so replacing the descriptor reaches them all.
/// The directory is resolved optionally — admin-only nodes register none and get a pass-through.
/// </summary>
/// <param name="services">The service collection, after <c>AddZbSecrets</c>.</param>
internal static void DecorateCacheInvalidator(IServiceCollection services)
{
var existing = services.LastOrDefault(d => d.ServiceType == typeof(ISecretCacheInvalidator) && !d.IsKeyedService);
if (existing is null) return;
services.Remove(existing);
services.AddSingleton<ISecretCacheInvalidator>(sp => new SecretChangeSignallingInvalidator(
(ISecretCacheInvalidator)(existing.ImplementationInstance
?? existing.ImplementationFactory?.Invoke(sp)
?? ActivatorUtilities.CreateInstance(sp, existing.ImplementationType!)),
sp.GetService<GalaxyAckIdentityDirectory>()));
}
}
@@ -61,6 +61,22 @@ public static class DriverFactoryBootstrap
services.TryAddSingleton<RedundancyRoleView>();
services.TryAddSingleton<IDriverPrimaryRoleSource>(sp => sp.GetRequiredService<RedundancyRoleView>());
// lmxopcua#765: the node-wide cached set of API keys with a Galaxy ack identity, read without I/O by every
// Galaxy driver's synchronous ack pre-check. Listed from this node's Secrets store (optional here, so a
// test graph without one gets a directory that always answers Unknown = fail open), refreshed by the
// hosted loop below and early on identity-secret changes (SecretsRegistration decorates the invalidator).
services.TryAddSingleton(sp =>
{
var options = sp.GetService<IOptions<GalaxyAckIdentityCacheOptions>>()?.Value ?? new GalaxyAckIdentityCacheOptions();
return new Driver.Galaxy.Runtime.GalaxyAckIdentityDirectory(
sp.GetService<ISecretStore>(), options.RefreshInterval, options.StaleAfter,
sp.GetService<TimeProvider>(),
sp.GetService<ILoggerFactory>()?.CreateLogger<Driver.Galaxy.Runtime.GalaxyAckIdentityDirectory>());
});
services.TryAddSingleton<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(
sp => sp.GetRequiredService<Driver.Galaxy.Runtime.GalaxyAckIdentityDirectory>());
services.AddHostedService<GalaxyAckIdentityRefreshService>();
services.AddSingleton<DriverFactoryRegistry>(sp =>
{
var registry = new DriverFactoryRegistry();
@@ -86,7 +102,7 @@ public static class DriverFactoryBootstrap
// failures; without this it writes to NullLogger in production.
ConfigureAbTransportLogging(sp);
Register(registry, loggerFactory, secretResolver, primaryRoleSource, ResolveAbCipDefaultTransport(sp),
scriptRoot, scripting);
sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting);
return registry;
});
services.AddSingleton<IDriverFactory>(sp =>
@@ -215,6 +231,7 @@ public static class DriverFactoryBootstrap
ISecretResolver secretResolver,
IDriverPrimaryRoleSource primaryRoleSource,
Driver.AbCip.AbCipTransport abCipDefaultTransport,
Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory galaxyAckIdentities,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null)
{
@@ -222,7 +239,7 @@ public static class DriverFactoryBootstrap
Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting);
Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory, galaxyAckIdentities);
Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Mqtt.MqttDriverFactoryExtensions.Register(registry, loggerFactory, primaryRoleSource);
// Tier A (the Register default): fully managed — HttpClient + System.Xml.Linq, no native SDK,
@@ -0,0 +1,16 @@
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Host.Drivers;
/// <summary>
/// Runs the <see cref="GalaxyAckIdentityDirectory"/> refresh loop on driver nodes (lmxopcua#765): an initial
/// refresh at start, then every <c>Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds</c>, and early when
/// an identity secret changes or a refusal asks for one.
/// </summary>
/// <param name="directory">The node's directory.</param>
public sealed class GalaxyAckIdentityRefreshService(GalaxyAckIdentityDirectory directory) : BackgroundService
{
/// <inheritdoc />
protected override Task ExecuteAsync(CancellationToken stoppingToken) => directory.RunAsync(stoppingToken);
}
@@ -194,6 +194,8 @@ builder.Services.AddValidatedOptions<LdapOptions, LdapOptionsValidator>(builder.
// Test Connect probe (admin role, which must probe the stack the driver would run). A bad value fails host
// start here rather than silently falling back.
builder.Services.AddAbCipDriverDefaults(builder.Configuration);
// lmxopcua#765: refresh/staleness of the cached API-key → Galaxy ack identity set (driver nodes consume it).
builder.Services.AddGalaxyAckIdentityCache(builder.Configuration);
if (hasDriver)
{
@@ -0,0 +1,356 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime;
/// <summary>
/// lmxopcua#765 item 3 — a keyed session whose API key has no Galaxy ack identity is refused synchronously
/// (<c>BadUserAccessDenied</c> via <see cref="ZB.MOM.WW.OtOpcUa.Core.Abstractions.IAlarmAcknowledgePreflight"/>)
/// from a cached, periodically refreshed set read without I/O, and everything the set is not certain about
/// fails open to the Good-then-revert path.
/// </summary>
public sealed class GalaxyAckIdentityDirectoryTests
{
private const string WithIdentity = "0a1b2c3d4e5f6071";
private const string WithoutIdentity = "ffeeddccbbaa0099";
private static readonly TimeSpan Interval = TimeSpan.FromSeconds(30);
private static readonly TimeSpan StaleAfter = TimeSpan.FromSeconds(90);
// ------------------------------------------------------------------ driver pre-check
[Fact]
public async Task Keyed_session_without_an_identity_is_refused_synchronously()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
var refusal = driver.GetAcknowledgeRefusal(WithoutIdentity);
refusal.ShouldNotBeNull();
refusal.ShouldContain(WithoutIdentity);
refusal.ShouldContain("no Galaxy ack identity");
refusal.ShouldContain("drv");
}
[Fact]
public async Task Keyed_session_with_an_identity_proceeds()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
driver.GetAcknowledgeRefusal(WithIdentity).ShouldBeNull();
driver.GetAcknowledgeRefusal(WithIdentity.ToUpperInvariant()).ShouldBeNull("key ids are case-insensitive");
}
[Fact]
public async Task A_fallback_account_lets_a_keyed_session_without_an_identity_through()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory, Opts() with
{
AlarmAck = new GalaxyAlarmAckOptions("ack-svc", "PLANT", "secret:otopcua/galaxy/ack-fallback"),
});
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
}
[Fact]
public void Without_a_directory_a_keyed_session_is_never_pre_refused()
{
using var driver = NewDriver(directory: null);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
driver.GetAcknowledgeRefusal(apiKeyId: null).ShouldNotBeNull("the #760 anonymous refusal is unchanged");
}
[Fact]
public async Task A_refusal_asks_for_a_refresh_so_a_retry_sees_an_identity_set_out_of_band()
{
var (directory, store, _) = await RefreshedDirectoryAsync(TimeSpan.FromHours(1));
using var driver = NewDriver(directory);
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var loop = directory.RunAsync(cts.Token);
await WaitForAsync(() => store.ListCalls >= 2, cts.Token); // RunAsync's own initial refresh
// Set by the `secret` CLI: nothing in this process is told.
store.AddIdentity(WithoutIdentity);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull();
// The refusal kicked a refresh well inside the one-hour interval.
await WaitForAsync(() => driver.GetAcknowledgeRefusal(WithoutIdentity) is null, cts.Token);
await cts.CancelAsync();
await loop;
}
// ------------------------------------------------------------------ fail open
[Fact]
public void Before_the_first_refresh_every_key_is_Unknown_and_proceeds()
{
using var directory = new GalaxyAckIdentityDirectory(new FakeSecretStore(), Interval, StaleAfter);
using var driver = NewDriver(directory);
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
}
[Fact]
public async Task A_stale_snapshot_is_Unknown_and_proceeds()
{
var (directory, store, clock) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull("positive control: fresh snapshot refuses");
// The store starts failing: the old snapshot is kept, and ages out.
store.Fail = true;
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeFalse();
clock.Advance(StaleAfter - TimeSpan.FromSeconds(1));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
clock.Advance(TimeSpan.FromSeconds(2));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
directory.GetPresence(WithIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
}
[Fact]
public async Task A_signalled_identity_change_makes_the_snapshot_Unknown_until_the_next_refresh()
{
var (directory, store, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull("positive control");
// AdminUI sets the identity: the write invalidates the secret, the directory is told.
store.AddIdentity(WithoutIdentity);
directory.NotifySecretChanged(new SecretName(GalaxyAckIdentitySecrets.UserSecretName(WithoutIdentity)));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull("never refuse in the window before the refresh");
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Present);
}
[Fact]
public async Task A_change_signalled_during_a_refresh_leaves_that_snapshot_untrusted()
{
var (directory, store, _) = await RefreshedDirectoryAsync();
store.OnList = () => directory.NotifySecretChanged(
new SecretName(GalaxyAckIdentitySecrets.PasswordSecretName(WithoutIdentity)));
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
[Fact]
public async Task An_unrelated_secret_change_does_not_untrust_the_snapshot()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
directory.NotifySecretChanged(new SecretName("otopcua/galaxy/ack-fallback"));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
}
[Fact]
public async Task A_directory_without_a_store_is_always_Unknown()
{
using var directory = new GalaxyAckIdentityDirectory(store: null, Interval, StaleAfter);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeFalse();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
[Fact]
public async Task A_key_id_the_resolver_could_not_look_up_is_Unknown()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
directory.GetPresence("../etc").ShouldBe(GalaxyAckIdentityPresence.Unknown);
directory.GetPresence(" ").ShouldBe(GalaxyAckIdentityPresence.Unknown);
directory.GetPresence(null).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
// ------------------------------------------------------------------ refresh
[Fact]
public async Task Refresh_picks_up_added_and_removed_identities()
{
var (directory, store, _) = await RefreshedDirectoryAsync();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
store.AddIdentity(WithoutIdentity);
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent, "a cache, until refreshed");
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Present);
store.RemoveIdentity(WithIdentity, softDelete: true);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent, "a deleted user secret is no identity");
}
[Fact]
public async Task Only_user_secrets_count_as_an_identity()
{
var store = new FakeSecretStore();
store.Add(GalaxyAckIdentitySecrets.PasswordSecretName(WithoutIdentity)); // password without user
store.Add("otopcua/galaxy/ack-fallback");
using var directory = new GalaxyAckIdentityDirectory(store, Interval, StaleAfter);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
}
[Fact]
public async Task The_loop_refreshes_on_its_interval()
{
var store = new FakeSecretStore();
using var directory = new GalaxyAckIdentityDirectory(store, TimeSpan.FromMilliseconds(50), TimeSpan.FromMinutes(1));
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var loop = directory.RunAsync(cts.Token);
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Absent, cts.Token);
store.AddIdentity(WithIdentity);
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Present, cts.Token);
await cts.CancelAsync();
await loop; // completes cleanly on cancellation
}
[Fact]
public async Task The_loop_refreshes_early_on_a_signalled_change()
{
var store = new FakeSecretStore();
using var directory = new GalaxyAckIdentityDirectory(store, TimeSpan.FromHours(1), TimeSpan.FromHours(3));
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var loop = directory.RunAsync(cts.Token);
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Absent, cts.Token);
store.AddIdentity(WithIdentity);
directory.NotifySecretChanged(new SecretName(GalaxyAckIdentitySecrets.UserSecretName(WithIdentity)));
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Present, cts.Token);
await cts.CancelAsync();
await loop;
}
// ------------------------------------------------------------------ helpers
private static async Task<(GalaxyAckIdentityDirectory Directory, FakeSecretStore Store, ManualClock Clock)>
RefreshedDirectoryAsync(TimeSpan? interval = null)
{
var store = new FakeSecretStore();
store.AddIdentity(WithIdentity);
var clock = new ManualClock();
var directory = new GalaxyAckIdentityDirectory(
store, interval ?? Interval, interval is { } i ? i * 3 : StaleAfter, clock);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
return (directory, store, clock);
}
private static async Task WaitForAsync(Func<bool> condition, CancellationToken ct)
{
while (!condition())
{
ct.ThrowIfCancellationRequested();
await Task.Delay(10, ct);
}
}
private static GalaxyDriverOptions Opts() => new(
new GalaxyGatewayOptions("https://mxgw.test:5001", "dev:key"),
new GalaxyMxAccessOptions("OtOpcUa-A"),
new GalaxyRepositoryOptions(),
new GalaxyReconnectOptions());
private static GalaxyDriver NewDriver(IGalaxyAckIdentityDirectory? directory, GalaxyDriverOptions? options = null) =>
new("drv", options ?? Opts(), hierarchySource: null, ackIdentityDirectory: directory);
private sealed class ManualClock : TimeProvider
{
private DateTimeOffset _now = new(2026, 11, 9, 12, 0, 0, TimeSpan.Zero);
public void Advance(TimeSpan by) => _now += by;
public override DateTimeOffset GetUtcNow() => _now;
}
/// <summary>Metadata-only store: <see cref="ListAsync"/> is all the directory may call.</summary>
private sealed class FakeSecretStore : ISecretStore
{
private readonly Dictionary<string, bool> _rows = new(StringComparer.Ordinal); // name → deleted
private readonly Lock _lock = new();
private int _listCalls;
public bool Fail { get; set; }
public Action? OnList { get; set; }
public int ListCalls => Volatile.Read(ref _listCalls);
public void Add(string name)
{
lock (_lock) _rows[name] = false;
}
public void AddIdentity(string keyId)
{
Add(GalaxyAckIdentitySecrets.PasswordSecretName(keyId));
Add(GalaxyAckIdentitySecrets.UserSecretName(keyId));
}
public void RemoveIdentity(string keyId, bool softDelete)
{
lock (_lock)
{
foreach (var name in new[]
{
GalaxyAckIdentitySecrets.UserSecretName(keyId), GalaxyAckIdentitySecrets.PasswordSecretName(keyId),
})
{
if (softDelete) _rows[name] = true;
else _rows.Remove(name);
}
}
}
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct)
{
Interlocked.Increment(ref _listCalls);
OnList?.Invoke();
if (Fail) throw new InvalidOperationException("store unavailable");
lock (_lock)
{
// Deleted rows are returned regardless, to prove the directory filters them itself.
IReadOnlyList<SecretMetadata> rows = _rows.Select(r => new SecretMetadata
{
Name = new SecretName(r.Key),
ContentType = SecretContentType.Text,
KekId = "k",
Revision = 1,
IsDeleted = r.Value,
CreatedUtc = DateTimeOffset.UnixEpoch,
UpdatedUtc = DateTimeOffset.UnixEpoch,
}).ToList();
return Task.FromResult(rows);
}
}
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) => throw Unexpected();
public Task UpsertAsync(StoredSecret row, CancellationToken ct) => throw Unexpected();
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct) => throw Unexpected();
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) => throw Unexpected();
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) => throw Unexpected();
public Task<bool> ApplyRewrapAsync(StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
throw Unexpected();
// The directory must read metadata only — never a row (that would decrypt-adjacent I/O per key).
private static InvalidOperationException Unexpected() => new("the directory may only list metadata");
}
}
@@ -0,0 +1,208 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.Drivers;
using ZB.MOM.WW.Secrets;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
/// <summary>
/// lmxopcua#765 item 3 — the host wiring of the cached API-key → Galaxy ack identity set: the driver registry
/// hands every Galaxy driver the node's directory (so the synchronous pre-check can refuse a keyed session
/// without an identity), a refresh loop runs on driver nodes, secret writes signal it through the decorated
/// invalidator, and the refresh interval is configurable.
/// </summary>
[Trait("Category", "Unit")]
public sealed class GalaxyAckIdentityCacheWiringTests
{
private const string WithIdentity = "0a1b2c3d4e5f6071";
private const string WithoutIdentity = "ffeeddccbbaa0099";
private const string GalaxyConfig = """
{
"Gateway": { "Endpoint": "https://mxgw.test:5001", "ApiKeySecretRef": "dev:key" },
"MxAccess": { "ClientName": "OtOpcUa-A" }
}
""";
[Fact]
public async Task A_registry_built_Galaxy_driver_refuses_a_keyed_session_without_an_identity()
{
var store = new ListOnlySecretStore(GalaxyAckIdentitySecrets.UserSecretName(WithIdentity));
using var sp = DriverServices(store).BuildServiceProvider();
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
var driver = sp.GetRequiredService<DriverFactoryRegistry>()
.TryGet(GalaxyDriverFactoryExtensions.DriverTypeName)!.Invoke("galaxy-1", GalaxyConfig);
var preflight = driver.ShouldBeAssignableTo<IAlarmAcknowledgePreflight>()!;
preflight.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull();
preflight.GetAcknowledgeRefusal(WithIdentity).ShouldBeNull();
(driver as IDisposable)?.Dispose();
}
[Fact]
public void Driver_nodes_run_the_refresh_loop_and_share_one_directory()
{
var services = DriverServices(new ListOnlySecretStore());
services.AddOtOpcUaDriverFactories(); // a fused admin,driver node may reach it twice
// By descriptor: resolving every IHostedService would need the resilience publisher's whole graph.
services.Count(d => d.ServiceType == typeof(IHostedService)
&& d.ImplementationType == typeof(GalaxyAckIdentityRefreshService)).ShouldBe(1);
using var sp = services.BuildServiceProvider();
sp.GetRequiredService<IGalaxyAckIdentityDirectory>()
.ShouldBeSameAs(sp.GetRequiredService<GalaxyAckIdentityDirectory>());
}
[Fact]
public void The_refresh_interval_and_staleness_are_configurable()
{
var services = DriverServices(new ListOnlySecretStore());
services.AddGalaxyAckIdentityCache(Config(new()
{
["Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds"] = "7",
["Drivers:Galaxy:AckIdentityCache:StaleAfterSeconds"] = "50",
}));
using var sp = services.BuildServiceProvider();
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
directory.RefreshInterval.ShouldBe(TimeSpan.FromSeconds(7));
directory.StaleAfter.ShouldBe(TimeSpan.FromSeconds(50));
}
[Fact]
public void Defaults_are_30s_and_three_intervals()
{
using var sp = DriverServices(new ListOnlySecretStore()).BuildServiceProvider();
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
directory.RefreshInterval.ShouldBe(TimeSpan.FromSeconds(30));
directory.StaleAfter.ShouldBe(TimeSpan.FromSeconds(90));
}
[Theory]
[InlineData("0", "0")]
[InlineData("3601", "0")]
[InlineData("30", "30")]
public void Bad_cache_settings_fail_validation(string interval, string staleAfter)
{
var services = new ServiceCollection();
services.AddGalaxyAckIdentityCache(Config(new()
{
["Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds"] = interval,
["Drivers:Galaxy:AckIdentityCache:StaleAfterSeconds"] = staleAfter,
}));
using var sp = services.BuildServiceProvider();
Should.Throw<OptionsValidationException>(() => sp.GetRequiredService<IOptions<GalaxyAckIdentityCacheOptions>>().Value);
}
[Fact]
public async Task AddOtOpcUaSecrets_decorates_the_invalidator_so_an_identity_write_untrusts_the_directory()
{
// Registered before AddOtOpcUaSecrets, so the library's TryAdd keeps this store for the directory's listing.
var services = DriverServices(new ListOnlySecretStore());
services.AddLogging();
services.AddOtOpcUaSecrets(Config(new()
{
// Resolving the invalidator never opens the store; the path only has to pass validation.
["Secrets:SqlitePath"] = Path.Combine(Path.GetTempPath(), $"otopcua-secrets-{Guid.NewGuid():N}.db"),
["Secrets:MasterKey:Source"] = "Environment",
["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_MASTER_KEY",
["Secrets:RunMigrationsOnStartup"] = "false",
}));
using var sp = services.BuildServiceProvider();
var invalidator = sp.GetRequiredService<ISecretCacheInvalidator>();
var decorated = invalidator.ShouldBeOfType<SecretChangeSignallingInvalidator>();
decorated.Inner.ShouldBeOfType<DefaultSecretResolver>();
decorated.Inner.ShouldBeSameAs(sp.GetRequiredService<DefaultSecretResolver>(), "the resolver cache is still invalidated");
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
// What ApiKeyGalaxyAckIdentityService.SetAsync does after writing the user secret.
invalidator.Invalidate(new SecretName(GalaxyAckIdentitySecrets.UserSecretName(WithoutIdentity)));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
[Fact]
public void Without_a_directory_the_decorated_invalidator_is_a_pass_through()
{
var inner = new RecordingInvalidator();
var services = new ServiceCollection();
services.AddSingleton<ISecretCacheInvalidator>(_ => inner);
SecretsRegistration.DecorateCacheInvalidator(services);
using var sp = services.BuildServiceProvider();
var name = new SecretName("otopcua/apikeys/abc/galaxy-ack-user");
sp.GetRequiredService<ISecretCacheInvalidator>().Invalidate(name);
inner.Names.ShouldBe([name]);
}
// ------------------------------------------------------------------ helpers
private static IServiceCollection DriverServices(ISecretStore store)
{
var services = new ServiceCollection();
services.AddSingleton<ISecretResolver>(new NullResolver());
services.AddSingleton(store);
services.AddOtOpcUaDriverFactories();
return services;
}
private static IConfiguration Config(Dictionary<string, string?> values) =>
new ConfigurationBuilder().AddInMemoryCollection(values).Build();
private sealed class NullResolver : ISecretResolver
{
public Task<string?> GetAsync(SecretName name, CancellationToken ct) => Task.FromResult<string?>(null);
}
private sealed class RecordingInvalidator : ISecretCacheInvalidator
{
public List<SecretName> Names { get; } = [];
public void Invalidate(SecretName name) => Names.Add(name);
}
private sealed class ListOnlySecretStore(params string[] names) : ISecretStore
{
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct) =>
Task.FromResult<IReadOnlyList<SecretMetadata>>(names.Select(n => new SecretMetadata
{
Name = new SecretName(n),
ContentType = SecretContentType.Text,
KekId = "k",
Revision = 1,
CreatedUtc = DateTimeOffset.UnixEpoch,
UpdatedUtc = DateTimeOffset.UnixEpoch,
}).ToList());
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) => throw new NotSupportedException();
public Task UpsertAsync(StoredSecret row, CancellationToken ct) => throw new NotSupportedException();
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct) => throw new NotSupportedException();
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) => throw new NotSupportedException();
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) => throw new NotSupportedException();
public Task<bool> ApplyRewrapAsync(StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
throw new NotSupportedException();
}
}