fix(alarms): refuse a keyed native ack with no Galaxy identity synchronously, from a cached key set (lmxopcua#765)
v2-ci / unit-tests (push) Waiting to run

Item 3 of #765. A keyed (API-key) session whose key had no Galaxy ack identity
got Good and then a revert, because whether the key has an identity lives in
the Secrets store, which must not be read under the SDK lock.

GalaxyAckIdentityDirectory (Driver.Galaxy) keeps a node-wide set of the key ids
that have a galaxy-ack-user secret. It lists this node's Secrets store
(metadata only, nothing decrypted) and is read lock-free.
GalaxyDriver.GetAcknowledgeRefusal (IAlarmAcknowledgePreflight, published
through NativeAlarmAckPreflightView as in #760) now refuses a keyed session
whose key the directory reports Absent, when the driver has no fallback ack
account. A refusal also asks for a refresh.

Fails open: before the first refresh, once the snapshot is older than
StaleAfter, and from a signalled identity-secret change until the next
refresh, the lookup answers Unknown and the ack takes the old Good-then-revert
path.

Refresh: GalaxyAckIdentityRefreshService runs on driver nodes, every
Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds (default 30;
StaleAfterSeconds 0 = three intervals; validated at start). It also refreshes
early on secret changes. The Secrets library has no change event, so
SecretsRegistration decorates ISecretCacheInvalidator, which is called by
AdminUI /api-keys, /admin/secrets and the replication appliers. API-key
changes need no event: a revoked key is already denied before the pre-check.

Tests: Galaxy.Tests GalaxyAckIdentityDirectoryTests (16) and Host.Tests
GalaxyAckIdentityCacheWiringTests (9). Seven separate mutations each turn at
least one test red: pre-check ignores the directory, factory drops it, no
invalidator decoration, no staleness check, no change-pending check, no
refresh on refusal, loop never wakes early.
Docs: Galaxy.md, AlarmTracking.md, Configuration.md, .claude/rules/alarms.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 16:38:35 -05:00
parent c9288a56c0
commit 0702b31029
15 changed files with 1034 additions and 28 deletions
+7 -3
View File
@@ -32,10 +32,14 @@ SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that fie
`ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`). `ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`).
**A native ack the driver refuses or fails is not left Acknowledged (#760).** Two layers: (1) a refusal the **A native ack the driver refuses or fails is not left Acknowledged (#760).** Two layers: (1) a refusal the
driver can decide without I/O (`IAlarmAcknowledgePreflight` — Galaxy: anonymous session and no fallback ack driver can decide without I/O (`IAlarmAcknowledgePreflight` — Galaxy, no fallback ack account: an anonymous session, or a keyed
account) fails the Part 9 call `BadUserAccessDenied` via `NativeAlarmAckPreflight`, read lock-free from the session whose key `GalaxyAckIdentityDirectory` reports Absent, #765) fails the Part 9 call `BadUserAccessDenied` via `NativeAlarmAckPreflight`, read lock-free from the
`NativeAlarmAckPreflightView` snapshot `DriverHostActor` publishes; never do I/O (e.g. a Secrets lookup) in a `NativeAlarmAckPreflightView` snapshot `DriverHostActor` publishes; never do I/O (e.g. a Secrets lookup) in a
pre-check — it runs under the SDK `Lock`. (2) anything else (keyed session without an identity, MXAccess pre-check — it runs under the SDK `Lock`; that is why the keyed case reads a cached, periodically refreshed key set
(`Drivers:Galaxy:AckIdentityCache`), which must answer Unknown (= let it through) whenever it is not certain — no
refresh yet, stale, or an identity-secret change signalled through the decorated `ISecretCacheInvalidator` and not
yet re-listed. Only an identity written straight into the store by the `secret` CLI (no in-process signal) can be
refused until the next refresh, which the refusal itself requests. (2) anything else (a keyed session the cache cannot vouch for, MXAccess
non-zero `hresult`, driver still connecting/reconnecting) comes back as `DriverInstanceActor.AlarmAckFailed` — or, for a non-zero `hresult`, driver still connecting/reconnecting) comes back as `DriverInstanceActor.AlarmAckFailed` — or, for a
disabled/unspawned/stubbed driver, is decided in the host itself (#765) — and `DriverHostActor` re-projects the condition unacknowledged — only while the ack is still pending in disabled/unspawned/stubbed driver, is decided in the host itself (#765) — and `DriverHostActor` re-projects the condition unacknowledged — only while the ack is still pending in
`NativeAlarmProjector` (any driver transition since then wins). A host that came up **Stale** cannot route any `NativeAlarmProjector` (any driver transition since then wins). A host that came up **Stale** cannot route any
+4 -2
View File
@@ -468,8 +468,10 @@ from the scripted `AlarmCommandRouter`:
not yet spawned) or is a stub on this node is reverted the same way by not yet spawned) or is a stub on this node is reverted the same way by
the host itself (#765). Only an ack still pending in `NativeAlarmProjector` is the host itself (#765). Only an ack still pending in `NativeAlarmProjector` is
reverted: a driver transition projected since then is authoritative. A reverted: a driver transition projected since then is authoritative. A
refusal the driver can decide without I/O (Galaxy: an anonymous session refusal the driver can decide without I/O (Galaxy, with no fallback ack
and no fallback ack account) is instead returned synchronously as account: an anonymous session, or a keyed session whose key the cached
`GalaxyAckIdentityDirectory` knows has no ack identity — #765) is instead
returned synchronously as
`BadUserAccessDenied` by the node manager's `NativeAlarmAckPreflight` `BadUserAccessDenied` by the node manager's `NativeAlarmAckPreflight`
before anything is applied — as is every native ack while the driver host before anything is applied — as is every native ack while the driver host
is Stale (configuration database unreachable at boot), when no ack can be is Stale (configuration database unreachable at boot), when no ack can be
+15
View File
@@ -536,6 +536,21 @@ deploy that recreates the driver), not on `ReinitializeAsync`. The init log line
`transport=Libplctag (node default)` when the pin is what selected the stack. There is **no AbLegacy `transport=Libplctag (node default)` when the pin is what selected the stack. There is **no AbLegacy
counterpart**: an AbLegacy driver with no key always runs libplctag (Native is opt-in per driver). counterpart**: an AbLegacy driver with no key always runs libplctag (Native is opt-in per driver).
### `Drivers:Galaxy:AckIdentityCache` (cached API-key → Galaxy ack identity set, #765)
- **Purpose:** the node-wide set of API key ids that have a Galaxy ack identity (a `galaxy-ack-user` secret),
which the Galaxy driver's synchronous ack pre-check reads to refuse a keyed session without one
(`BadUserAccessDenied`) instead of returning Good and reverting. See
[`drivers/Galaxy.md` § Alarm acknowledge credential](drivers/Galaxy.md#alarm-acknowledge-credential).
- **Options class:** `GalaxyAckIdentityCacheOptions` (`SectionName = "Drivers:Galaxy:AckIdentityCache"`) —
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/GalaxyAckIdentityCacheOptions.cs`; bound and validated
in `Program.cs`. The directory and its refresh loop exist on driver-role nodes only.
| Key | Type | Default | Meaning |
|---|---|---|---|
| `RefreshIntervalSeconds` | int | `30` | How often the Secrets store is re-listed (1..3600). An in-process identity change or a refusal also triggers an early refresh. |
| `StaleAfterSeconds` | int | `0` | Snapshot age after which keyed acks fall back to Good-then-revert (fail open). `0` = three refresh intervals; otherwise must exceed `RefreshIntervalSeconds`. A bad value fails host start. |
--- ---
## Environment variables ## Environment variables
+17 -6
View File
@@ -108,12 +108,23 @@ Galaxy password, so `GalaxyAckCredentialResolver` picks the account per acknowle
3. **Neither** → the driver throws `DriverOperationRefusedException`; **nothing is sent to the 3. **Neither** → the driver throws `DriverOperationRefusedException`; **nothing is sent to the
gateway**. The refusal is excluded from the resilience breaker (it says nothing about gateway gateway**. The refusal is excluded from the resilience breaker (it says nothing about gateway
health). How the client learns of it (#760): health). How the client learns of it (#760):
- **Synchronously** when the driver can tell without I/O — an *anonymous* session on a driver with - **Synchronously** when the driver can tell without I/O, on a driver with no fallback account —
no fallback account: the Part 9 `Acknowledge` call itself fails `BadUserAccessDenied` with the an *anonymous* session (#760), or a *keyed* session whose key has no identity (#765): the Part 9
reason, and nothing is applied (`IAlarmAcknowledgePreflight`, published per condition by `Acknowledge` call itself fails `BadUserAccessDenied` with the reason, and nothing is applied
`DriverHostActor` into `NativeAlarmAckPreflightView`). (`IAlarmAcknowledgePreflight`, published per condition by `DriverHostActor` into
- **By revert** otherwise — a keyed session whose key has no identity (that lives in the Secrets `NativeAlarmAckPreflightView`). The keyed case reads `GalaxyAckIdentityDirectory`, a node-wide
store, which is not read under the SDK lock), or MXAccess rejecting the account (a non-zero cached set of the key ids that have a `galaxy-ack-user` secret, listed from this node's Secrets
store (metadata only — nothing decrypted) every `Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds`
(default 30) by `GalaxyAckIdentityRefreshService`, and early whenever an identity secret changes
in-process (the Secrets cache invalidator is decorated, so AdminUI `/api-keys`, `/admin/secrets` and
replication all signal it) or a refusal asks for one. ⚠️ It **fails open**: before the first refresh,
once the snapshot is older than `StaleAfterSeconds` (default three intervals — e.g. the store cannot
be listed), and from a signalled identity change until the refresh that follows it, every keyed ack
takes the revert path below instead of being refused. An identity written straight into the store
by the `secret` CLI is seen at the next refresh; the refusal itself requests one, so a retry a moment
later goes through.
- **By revert** otherwise — a keyed session the directory cannot vouch for (see above), a key whose
identity is broken (a user with no password secret), or MXAccess rejecting the account (a non-zero
native `hresult`, e.g. a wrong password, now thrown as a refusal): the call returned `Good`, then native `hresult`, e.g. a wrong password, now thrown as a refusal): the call returned `Good`, then
`DriverHostActor` re-projects the condition **unacknowledged** (a fresh condition event, `DriverHostActor` re-projects the condition **unacknowledged** (a fresh condition event,
`Retain=true`) and logs the reason. A driver transition projected after the ack wins over a late `Retain=true`) and logs the reason. A driver transition projected after the ack wins over a late
@@ -119,6 +119,9 @@ public sealed class GalaxyDriver
// lmxopcua#749: resolves the Galaxy account each acknowledge authenticates as (API key identity → // lmxopcua#749: resolves the Galaxy account each acknowledge authenticates as (API key identity →
// fallback account → refused). Built in the ctor — it needs only options + the secret resolver. // fallback account → refused). Built in the ctor — it needs only options + the secret resolver.
private readonly IGalaxyAckCredentialResolver _ackCredentialResolver; private readonly IGalaxyAckCredentialResolver _ackCredentialResolver;
// lmxopcua#765: the cached key → has-Galaxy-identity set the synchronous ack pre-check reads (no I/O).
// Null = no directory wired, so a keyed session is never pre-refused (the pre-#765 behaviour).
private readonly IGalaxyAckIdentityDirectory? _ackIdentityDirectory;
private IGalaxyAlarmFeed? _alarmFeed; private IGalaxyAlarmFeed? _alarmFeed;
private readonly Lock _alarmHandlersLock = new(); private readonly Lock _alarmHandlersLock = new();
private readonly Lock _alarmFeedLock = new(); private readonly Lock _alarmFeedLock = new();
@@ -262,15 +265,21 @@ public sealed class GalaxyDriver
/// <param name="options">The Galaxy driver configuration options.</param> /// <param name="options">The Galaxy driver configuration options.</param>
/// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c>.</param> /// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c>.</param>
/// <param name="logger">Optional logger instance for diagnostics.</param> /// <param name="logger">Optional logger instance for diagnostics.</param>
/// <param name="ackIdentityDirectory">
/// Optional cached set of API keys with a Galaxy ack identity (lmxopcua#765), consulted by the synchronous
/// ack pre-check. Null = keyed sessions are never pre-refused.
/// </param>
public GalaxyDriver( public GalaxyDriver(
string driverInstanceId, string driverInstanceId,
GalaxyDriverOptions options, GalaxyDriverOptions options,
ISecretResolver secretResolver, ISecretResolver secretResolver,
ILogger<GalaxyDriver>? logger = null) ILogger<GalaxyDriver>? logger = null,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
: this(driverInstanceId, options, : this(driverInstanceId, options,
hierarchySource: null, dataReader: null, dataWriter: null, subscriber: null, hierarchySource: null, dataReader: null, dataWriter: null, subscriber: null,
alarmAcknowledger: null, alarmFeed: null, logger, alarmAcknowledger: null, alarmFeed: null, logger,
secretResolver: secretResolver ?? throw new ArgumentNullException(nameof(secretResolver))) secretResolver: secretResolver ?? throw new ArgumentNullException(nameof(secretResolver)),
ackIdentityDirectory: ackIdentityDirectory)
{ {
} }
@@ -307,6 +316,7 @@ public sealed class GalaxyDriver
/// Optional ack-credential seam (lmxopcua#749). Defaults to <see cref="GalaxyAckCredentialResolver"/> /// Optional ack-credential seam (lmxopcua#749). Defaults to <see cref="GalaxyAckCredentialResolver"/>
/// over <see cref="GalaxyDriverOptions.AlarmAck"/> and <paramref name="secretResolver"/>. /// over <see cref="GalaxyDriverOptions.AlarmAck"/> and <paramref name="secretResolver"/>.
/// </param> /// </param>
/// <param name="ackIdentityDirectory">Optional cached key → Galaxy-identity set (lmxopcua#765).</param>
internal GalaxyDriver( internal GalaxyDriver(
string driverInstanceId, string driverInstanceId,
GalaxyDriverOptions options, GalaxyDriverOptions options,
@@ -320,8 +330,10 @@ public sealed class GalaxyDriver
ISecretResolver? secretResolver = null, ISecretResolver? secretResolver = null,
IGalaxyDeployWatchSource? deployWatchSource = null, IGalaxyDeployWatchSource? deployWatchSource = null,
IGalaxyPlatformSource? platformSource = null, IGalaxyPlatformSource? platformSource = null,
IGalaxyAckCredentialResolver? ackCredentialResolver = null) IGalaxyAckCredentialResolver? ackCredentialResolver = null,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
{ {
_ackIdentityDirectory = ackIdentityDirectory;
_deployWatchSource = deployWatchSource; _deployWatchSource = deployWatchSource;
_platformSource = platformSource; _platformSource = platformSource;
_driverInstanceId = !string.IsNullOrWhiteSpace(driverInstanceId) _driverInstanceId = !string.IsNullOrWhiteSpace(driverInstanceId)
@@ -1968,17 +1980,32 @@ public sealed class GalaxyDriver
/// <inheritdoc /> /// <inheritdoc />
/// <remarks> /// <remarks>
/// Only the case decidable without I/O (lmxopcua#760): an anonymous session (no API key, so no key-mapped /// Only cases decidable without I/O, on a driver with no <c>AlarmAck.FallbackUser</c>: an anonymous session
/// Galaxy identity can exist) on a driver with no <c>AlarmAck.FallbackUser</c>. A keyed session is let /// (lmxopcua#760 — no API key, so no key-mapped Galaxy identity can exist), and a keyed session whose key the
/// through — whether the key has an identity lives in the Secrets store, which must not be read under the /// <see cref="IGalaxyAckIdentityDirectory"/> reports <see cref="GalaxyAckIdentityPresence.Absent"/>
/// SDK lock; <see cref="AcknowledgeAsync"/> still refuses it, and the host reverts the condition. /// (lmxopcua#765 — a cached, periodically refreshed view of the Secrets store, because the store itself must
/// not be read under the SDK lock). Anything the directory does not know for certain (no directory, no
/// refresh yet, stale, a change pending) is let through: <see cref="AcknowledgeAsync"/> still refuses it,
/// and the host reverts the condition. A refusal also asks the directory to refresh, so a retry after an
/// identity was set out of band (the <c>secret</c> CLI) is not refused for a whole refresh interval.
/// </remarks> /// </remarks>
public string? GetAcknowledgeRefusal(string? apiKeyId) public string? GetAcknowledgeRefusal(string? apiKeyId)
{ {
if (!string.IsNullOrWhiteSpace(apiKeyId) || (_options.AlarmAck?.HasFallback ?? false)) return null; if (_options.AlarmAck?.HasFallback ?? false) return null;
return $"the session is anonymous (no API key) and Galaxy driver '{_driverInstanceId}' has no " + if (string.IsNullOrWhiteSpace(apiKeyId))
"AlarmAck.FallbackUser configured, so the acknowledge cannot be sent to Galaxy. Connect with an API key " + {
"that has a Galaxy ack identity (AdminUI /api-keys) or configure the driver's fallback ack account."; return $"the session is anonymous (no API key) and Galaxy driver '{_driverInstanceId}' has no " +
"AlarmAck.FallbackUser configured, so the acknowledge cannot be sent to Galaxy. Connect with an API key " +
"that has a Galaxy ack identity (AdminUI /api-keys) or configure the driver's fallback ack account.";
}
if (_ackIdentityDirectory is not { } directory
|| directory.GetPresence(apiKeyId) != GalaxyAckIdentityPresence.Absent) return null;
directory.RequestRefresh();
return $"API key '{apiKeyId.Trim()}' has no Galaxy ack identity and Galaxy driver '{_driverInstanceId}' has no " +
"AlarmAck.FallbackUser configured, so the acknowledge cannot be sent to Galaxy. Map a Galaxy ack identity " +
"to the key (AdminUI /api-keys) or configure the driver's fallback ack account.";
} }
private static string PrincipalOf(AlarmAcknowledgeRequest ack) => private static string PrincipalOf(AlarmAcknowledgeRequest ack) =>
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions; using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting; using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config; using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions; using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy; namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
@@ -31,14 +32,20 @@ public static class GalaxyDriverFactoryExtensions
/// each instance's <c>Gateway.ApiKeySecretRef</c>. /// each instance's <c>Gateway.ApiKeySecretRef</c>.
/// </param> /// </param>
/// <param name="loggerFactory">The optional logger factory for creating drivers.</param> /// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
/// <param name="ackIdentityDirectory">
/// Optional node-wide cached set of API keys with a Galaxy ack identity (lmxopcua#765), shared by every
/// Galaxy driver instance for the synchronous ack pre-check. Null = keyed sessions are never pre-refused.
/// </param>
public static void Register( public static void Register(
DriverFactoryRegistry registry, DriverFactoryRegistry registry,
ISecretResolver secretResolver, ISecretResolver secretResolver,
ILoggerFactory? loggerFactory = null) ILoggerFactory? loggerFactory = null,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
{ {
ArgumentNullException.ThrowIfNull(registry); ArgumentNullException.ThrowIfNull(registry);
ArgumentNullException.ThrowIfNull(secretResolver); ArgumentNullException.ThrowIfNull(secretResolver);
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, secretResolver)); registry.Register(DriverTypeName,
(id, json) => CreateInstance(id, json, loggerFactory, secretResolver, ackIdentityDirectory));
} }
/// <summary> /// <summary>
@@ -58,9 +65,11 @@ public static class GalaxyDriverFactoryExtensions
/// <param name="driverConfigJson">The driver configuration in JSON format.</param> /// <param name="driverConfigJson">The driver configuration in JSON format.</param>
/// <param name="loggerFactory">The optional logger factory for creating drivers.</param> /// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
/// <param name="secretResolver">The secret resolver for the driver's <c>secret:</c> API-key arm.</param> /// <param name="secretResolver">The secret resolver for the driver's <c>secret:</c> API-key arm.</param>
/// <param name="ackIdentityDirectory">Optional cached key → Galaxy-identity set (lmxopcua#765).</param>
/// <returns>The constructed Galaxy driver instance.</returns> /// <returns>The constructed Galaxy driver instance.</returns>
public static GalaxyDriver CreateInstance( public static GalaxyDriver CreateInstance(
string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory, ISecretResolver secretResolver) string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory, ISecretResolver secretResolver,
IGalaxyAckIdentityDirectory? ackIdentityDirectory = null)
{ {
ArgumentNullException.ThrowIfNull(secretResolver); ArgumentNullException.ThrowIfNull(secretResolver);
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId); ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
@@ -108,7 +117,8 @@ public static class GalaxyDriverFactoryExtensions
FallbackPasswordSecretRef: NullIfBlank(dto.AlarmAck?.FallbackPasswordSecretRef)), FallbackPasswordSecretRef: NullIfBlank(dto.AlarmAck?.FallbackPasswordSecretRef)),
}; };
return new GalaxyDriver(driverInstanceId, options, secretResolver, loggerFactory?.CreateLogger<GalaxyDriver>()); return new GalaxyDriver(
driverInstanceId, options, secretResolver, loggerFactory?.CreateLogger<GalaxyDriver>(), ackIdentityDirectory);
} }
private static string? NullIfBlank(string? value) => string.IsNullOrWhiteSpace(value) ? null : value.Trim(); private static string? NullIfBlank(string? value) => string.IsNullOrWhiteSpace(value) ? null : value.Trim();
@@ -0,0 +1,216 @@
using System.Collections.Frozen;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
/// <summary>What the <see cref="IGalaxyAckIdentityDirectory"/> knows about one API key's Galaxy ack identity.</summary>
public enum GalaxyAckIdentityPresence
{
/// <summary>
/// Not known right now — no successful refresh yet, the last one is stale, a change is pending, or the key
/// id is not one the directory can look up. Callers must treat it as "may have one" (fail open).
/// </summary>
Unknown = 0,
/// <summary>The key has a Galaxy ack user secret (it may still be refused at ack time, e.g. no password).</summary>
Present,
/// <summary>The key has no Galaxy ack user secret as of a fresh refresh.</summary>
Absent,
}
/// <summary>
/// Lock-free answer to "does this API key have a Galaxy ack identity?" (lmxopcua#765), for the synchronous
/// native-ack pre-check (<see cref="ZB.MOM.WW.OtOpcUa.Core.Abstractions.IAlarmAcknowledgePreflight"/>), which
/// runs under the OPC UA SDK lock and must not read the Secrets store.
/// </summary>
public interface IGalaxyAckIdentityDirectory
{
/// <summary>Looks the key up in the cached snapshot. No I/O, never throws.</summary>
/// <param name="apiKeyId">The session's API key id.</param>
/// <returns>The key's presence, or <see cref="GalaxyAckIdentityPresence.Unknown"/> when not known.</returns>
GalaxyAckIdentityPresence GetPresence(string? apiKeyId);
/// <summary>Asks for a refresh soon (coalesced; never blocks). Used after a refusal so a retry sees a new identity.</summary>
void RequestRefresh();
}
/// <summary>
/// Production <see cref="IGalaxyAckIdentityDirectory"/>: a periodically refreshed set of the API key ids that
/// have a Galaxy ack user secret (<see cref="GalaxyAckIdentitySecrets.UserSecretName"/>) in this node's
/// Secrets store — the same store <see cref="GalaxyAckCredentialResolver"/> reads at ack time.
/// </summary>
/// <remarks>
/// <para>
/// Built from <see cref="ISecretStore.ListAsync"/> metadata only — nothing is decrypted, no password is
/// read. A refresh runs every <c>refreshInterval</c> (<see cref="RunAsync"/>), and early on
/// <see cref="NotifySecretChanged"/> / <see cref="RequestRefresh"/>.
/// </para>
/// <para>
/// ⚠️ Fails OPEN: anything not certain answers <see cref="GalaxyAckIdentityPresence.Unknown"/>, which
/// lets the ack through to today's path (the driver resolves at ack time and the host reverts a refusal).
/// That covers no refresh yet, a store with nothing to list, a snapshot older than <c>staleAfter</c>
/// (e.g. the store has been failing), and a change to an identity secret signalled since the snapshot
/// was taken — so an identity just set through AdminUI or arriving by replication is never refused in
/// the window before the next refresh. A change the process is not told about (the <c>secret</c> CLI
/// writing the store directly) is picked up by the next timed refresh, or the refresh a refusal kicks.
/// </para>
/// </remarks>
public sealed class GalaxyAckIdentityDirectory : IGalaxyAckIdentityDirectory, IDisposable
{
private const string UserSuffix = "/galaxy-ack-user";
private readonly ISecretStore? _store;
private readonly TimeProvider _time;
private readonly ILogger _logger;
private readonly SemaphoreSlim _wake = new(0, 1);
// Bumped on every signalled change to an identity secret; a snapshot taken under an older value is not trusted.
private long _changeGeneration;
private volatile Snapshot? _snapshot;
/// <summary>Initializes a new instance of the <see cref="GalaxyAckIdentityDirectory"/> class.</summary>
/// <param name="store">This node's Secrets store; null = nothing to list, so every lookup is Unknown.</param>
/// <param name="refreshInterval">How often <see cref="RunAsync"/> refreshes.</param>
/// <param name="staleAfter">How old a snapshot may get before lookups answer Unknown.</param>
/// <param name="timeProvider">Clock for snapshot age; defaults to the system clock.</param>
/// <param name="logger">Logger; never receives a secret value.</param>
public GalaxyAckIdentityDirectory(
ISecretStore? store, TimeSpan refreshInterval, TimeSpan staleAfter,
TimeProvider? timeProvider = null, ILogger? logger = null)
{
if (refreshInterval <= TimeSpan.Zero)
throw new ArgumentOutOfRangeException(nameof(refreshInterval), refreshInterval, "Must be positive.");
if (staleAfter <= TimeSpan.Zero)
throw new ArgumentOutOfRangeException(nameof(staleAfter), staleAfter, "Must be positive.");
_store = store;
RefreshInterval = refreshInterval;
StaleAfter = staleAfter;
_time = timeProvider ?? TimeProvider.System;
_logger = logger ?? NullLogger.Instance;
}
/// <summary>Gets the refresh period of <see cref="RunAsync"/>.</summary>
public TimeSpan RefreshInterval { get; }
/// <summary>Gets the snapshot age after which every lookup answers Unknown.</summary>
public TimeSpan StaleAfter { get; }
/// <inheritdoc />
public GalaxyAckIdentityPresence GetPresence(string? apiKeyId)
{
var snapshot = _snapshot;
if (snapshot is null) return GalaxyAckIdentityPresence.Unknown;
if (snapshot.Generation != Interlocked.Read(ref _changeGeneration)) return GalaxyAckIdentityPresence.Unknown;
if (_time.GetUtcNow() - snapshot.RefreshedUtc > StaleAfter) return GalaxyAckIdentityPresence.Unknown;
if (TryNormalize(apiKeyId) is not { } keyId) return GalaxyAckIdentityPresence.Unknown;
return snapshot.KeyIds.Contains(keyId) ? GalaxyAckIdentityPresence.Present : GalaxyAckIdentityPresence.Absent;
}
/// <inheritdoc />
public void RequestRefresh()
{
try
{
_wake.Release();
}
catch (SemaphoreFullException)
{
// A refresh is already requested — coalesced.
}
catch (ObjectDisposedException)
{
// Shutting down.
}
}
/// <summary>
/// Tells the directory a secret changed (wired from the Secrets cache-invalidation seam, which AdminUI
/// writes, the Secrets UI and replication all call). A Galaxy ack identity secret marks the current
/// snapshot untrusted until the next refresh, and requests one; any other name is ignored.
/// </summary>
/// <param name="name">The changed secret.</param>
public void NotifySecretChanged(SecretName name)
{
if (!name.Value.StartsWith(GalaxyAckIdentitySecrets.Prefix, StringComparison.Ordinal)) return;
Interlocked.Increment(ref _changeGeneration);
RequestRefresh();
}
/// <summary>Lists the store once and replaces the snapshot. Never throws except on cancellation.</summary>
/// <param name="cancellationToken">Cancels the listing.</param>
/// <returns>True when the snapshot was replaced; false on a failure (the old snapshot keeps ageing).</returns>
public async Task<bool> RefreshAsync(CancellationToken cancellationToken)
{
if (_store is null) return false;
// Read BEFORE listing: a change signalled while the listing runs leaves this snapshot untrusted.
var generation = Interlocked.Read(ref _changeGeneration);
try
{
var rows = await _store.ListAsync(includeDeleted: false, cancellationToken).ConfigureAwait(false);
var keyIds = new HashSet<string>(StringComparer.Ordinal);
foreach (var row in rows)
{
if (row.IsDeleted) continue;
var name = row.Name.Value;
if (!name.StartsWith(GalaxyAckIdentitySecrets.Prefix, StringComparison.Ordinal)
|| !name.EndsWith(UserSuffix, StringComparison.Ordinal)
|| name.Length <= GalaxyAckIdentitySecrets.Prefix.Length + UserSuffix.Length) continue;
if (TryNormalize(name[GalaxyAckIdentitySecrets.Prefix.Length..^UserSuffix.Length]) is { } keyId)
keyIds.Add(keyId);
}
_snapshot = new Snapshot(keyIds.ToFrozenSet(StringComparer.Ordinal), _time.GetUtcNow(), generation);
return true;
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogWarning(
"Galaxy ack identity directory: listing the Secrets store failed ({ExceptionType}); native acks from " +
"keyed sessions fall back to the ack-time check once the snapshot is older than {StaleAfter}",
ex.GetType().Name, StaleAfter);
return false;
}
}
/// <summary>Refreshes now, then every <see cref="RefreshInterval"/> or sooner when asked, until cancelled.</summary>
/// <param name="cancellationToken">Stops the loop.</param>
/// <returns>A task that completes when cancelled.</returns>
public async Task RunAsync(CancellationToken cancellationToken)
{
try
{
while (!cancellationToken.IsCancellationRequested)
{
await RefreshAsync(cancellationToken).ConfigureAwait(false);
await _wake.WaitAsync(RefreshInterval, cancellationToken).ConfigureAwait(false);
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// Normal shutdown.
}
}
/// <inheritdoc />
public void Dispose() => _wake.Dispose();
// Same shape GalaxyAckIdentitySecrets accepts (trimmed, lower-case ASCII letters/digits); anything else is
// not a key id the resolver could look up, so the directory does not claim to know about it.
private static string? TryNormalize(string? apiKeyId)
{
if (string.IsNullOrWhiteSpace(apiKeyId)) return null;
var id = apiKeyId.Trim().ToLowerInvariant();
foreach (var c in id)
{
if (!char.IsAsciiLetterOrDigit(c)) return null;
}
return id;
}
private sealed record Snapshot(FrozenSet<string> KeyIds, DateTimeOffset RefreshedUtc, long Generation);
}
@@ -0,0 +1,72 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Node-level settings for the cached set of API keys that have a Galaxy ack identity (lmxopcua#765), bound
/// from <c>Drivers:Galaxy:AckIdentityCache</c>. The set lets the synchronous native-ack pre-check refuse a
/// keyed session without an identity (<c>BadUserAccessDenied</c>) instead of returning Good and reverting.
/// Overridable by environment variables <c>Drivers__Galaxy__AckIdentityCache__RefreshIntervalSeconds</c> /
/// <c>…__StaleAfterSeconds</c>.
/// </summary>
public sealed class GalaxyAckIdentityCacheOptions
{
/// <summary>The configuration section path.</summary>
public const string SectionName = "Drivers:Galaxy:AckIdentityCache";
/// <summary>How often the set is re-read from the Secrets store, in seconds (1..3600). Default 30.</summary>
public int RefreshIntervalSeconds { get; set; } = 30;
/// <summary>
/// How old the set may get, in seconds, before keyed acks fall back to the pre-#765 Good-then-revert path
/// (fail open, e.g. while the store cannot be listed). 0 (the default) = three refresh intervals. When set,
/// must be greater than <see cref="RefreshIntervalSeconds"/>.
/// </summary>
public int StaleAfterSeconds { get; set; }
/// <summary>Gets the refresh period.</summary>
public TimeSpan RefreshInterval => TimeSpan.FromSeconds(RefreshIntervalSeconds);
/// <summary>Gets the staleness bound (explicit, or three refresh intervals).</summary>
public TimeSpan StaleAfter => StaleAfterSeconds > 0
? TimeSpan.FromSeconds(StaleAfterSeconds)
: TimeSpan.FromSeconds(RefreshIntervalSeconds * 3L);
}
/// <summary>Fail-fast startup validator for <see cref="GalaxyAckIdentityCacheOptions"/>.</summary>
public sealed class GalaxyAckIdentityCacheOptionsValidator : OptionsValidatorBase<GalaxyAckIdentityCacheOptions>
{
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, GalaxyAckIdentityCacheOptions options)
{
builder.RequireThat(
options.RefreshIntervalSeconds is >= 1 and <= 3600,
$"{GalaxyAckIdentityCacheOptions.SectionName}:RefreshIntervalSeconds is {options.RefreshIntervalSeconds} — " +
"expected 1..3600.");
builder.RequireThat(
options.StaleAfterSeconds == 0 || options.StaleAfterSeconds > options.RefreshIntervalSeconds,
$"{GalaxyAckIdentityCacheOptions.SectionName}:StaleAfterSeconds is {options.StaleAfterSeconds} — expected 0 " +
$"(three refresh intervals) or more than RefreshIntervalSeconds ({options.RefreshIntervalSeconds}); otherwise " +
"the set would go stale between refreshes and never be used.");
}
}
/// <summary>Composition-root registration of <see cref="GalaxyAckIdentityCacheOptions"/>.</summary>
public static class GalaxyAckIdentityCacheRegistration
{
/// <summary>
/// Binds + validates <c>Drivers:Galaxy:AckIdentityCache</c>. A bad value fails host start. Without this call
/// the driver factories use the compiled defaults.
/// </summary>
/// <param name="services">The service collection.</param>
/// <param name="configuration">The host configuration.</param>
/// <returns>The same collection.</returns>
public static IServiceCollection AddGalaxyAckIdentityCache(this IServiceCollection services, IConfiguration configuration)
{
services.AddValidatedOptions<GalaxyAckIdentityCacheOptions, GalaxyAckIdentityCacheOptionsValidator>(
configuration, GalaxyAckIdentityCacheOptions.SectionName);
return services;
}
}
@@ -0,0 +1,26 @@
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Decorates the Secrets library's <see cref="ISecretCacheInvalidator"/> so a secret change also reaches the
/// <see cref="GalaxyAckIdentityDirectory"/> (lmxopcua#765). The library has no change event; this seam is the
/// one every in-process writer already calls — the AdminUI <c>/api-keys</c> ack-identity service, the
/// <c>/admin/secrets</c> editor and delete modal, and the replication appliers (Akka / SQL Server).
/// </summary>
/// <param name="inner">The library's invalidator (the resolver cache).</param>
/// <param name="directory">The node's directory, or null on a node that runs no drivers.</param>
internal sealed class SecretChangeSignallingInvalidator(
ISecretCacheInvalidator inner, GalaxyAckIdentityDirectory? directory) : ISecretCacheInvalidator
{
/// <summary>Gets the decorated invalidator (test seam).</summary>
internal ISecretCacheInvalidator Inner => inner;
/// <inheritdoc />
public void Invalidate(SecretName name)
{
inner.Invalidate(name);
directory?.NotifySecretChanged(name);
}
}
@@ -1,6 +1,8 @@
using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.DependencyInjection; using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection; using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
@@ -67,6 +69,7 @@ public static class SecretsRegistration
if (!IsReplicationEnabled(configuration)) if (!IsReplicationEnabled(configuration))
{ {
services.AddZbSecrets(configuration, SecretsSectionPath); services.AddZbSecrets(configuration, SecretsSectionPath);
DecorateCacheInvalidator(services);
return services; return services;
} }
@@ -77,6 +80,27 @@ public static class SecretsRegistration
services.AddSingleton<SecretReplicationStarter>(); services.AddSingleton<SecretReplicationStarter>();
services.AddHostedService(sp => sp.GetRequiredService<SecretReplicationStarter>()); services.AddHostedService(sp => sp.GetRequiredService<SecretReplicationStarter>());
DecorateCacheInvalidator(services);
return services; return services;
} }
/// <summary>
/// Wraps the library's <see cref="ISecretCacheInvalidator"/> registration in
/// <see cref="SecretChangeSignallingInvalidator"/>, so every in-process secret write and replicated apply
/// also tells the Galaxy ack identity directory (lmxopcua#765). Every consumer resolves the invalidator
/// through the interface (the replication appliers lazily), so replacing the descriptor reaches them all.
/// The directory is resolved optionally — admin-only nodes register none and get a pass-through.
/// </summary>
/// <param name="services">The service collection, after <c>AddZbSecrets</c>.</param>
internal static void DecorateCacheInvalidator(IServiceCollection services)
{
var existing = services.LastOrDefault(d => d.ServiceType == typeof(ISecretCacheInvalidator) && !d.IsKeyedService);
if (existing is null) return;
services.Remove(existing);
services.AddSingleton<ISecretCacheInvalidator>(sp => new SecretChangeSignallingInvalidator(
(ISecretCacheInvalidator)(existing.ImplementationInstance
?? existing.ImplementationFactory?.Invoke(sp)
?? ActivatorUtilities.CreateInstance(sp, existing.ImplementationType!)),
sp.GetService<GalaxyAckIdentityDirectory>()));
}
} }
@@ -61,6 +61,22 @@ public static class DriverFactoryBootstrap
services.TryAddSingleton<RedundancyRoleView>(); services.TryAddSingleton<RedundancyRoleView>();
services.TryAddSingleton<IDriverPrimaryRoleSource>(sp => sp.GetRequiredService<RedundancyRoleView>()); services.TryAddSingleton<IDriverPrimaryRoleSource>(sp => sp.GetRequiredService<RedundancyRoleView>());
// lmxopcua#765: the node-wide cached set of API keys with a Galaxy ack identity, read without I/O by every
// Galaxy driver's synchronous ack pre-check. Listed from this node's Secrets store (optional here, so a
// test graph without one gets a directory that always answers Unknown = fail open), refreshed by the
// hosted loop below and early on identity-secret changes (SecretsRegistration decorates the invalidator).
services.TryAddSingleton(sp =>
{
var options = sp.GetService<IOptions<GalaxyAckIdentityCacheOptions>>()?.Value ?? new GalaxyAckIdentityCacheOptions();
return new Driver.Galaxy.Runtime.GalaxyAckIdentityDirectory(
sp.GetService<ISecretStore>(), options.RefreshInterval, options.StaleAfter,
sp.GetService<TimeProvider>(),
sp.GetService<ILoggerFactory>()?.CreateLogger<Driver.Galaxy.Runtime.GalaxyAckIdentityDirectory>());
});
services.TryAddSingleton<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(
sp => sp.GetRequiredService<Driver.Galaxy.Runtime.GalaxyAckIdentityDirectory>());
services.AddHostedService<GalaxyAckIdentityRefreshService>();
services.AddSingleton<DriverFactoryRegistry>(sp => services.AddSingleton<DriverFactoryRegistry>(sp =>
{ {
var registry = new DriverFactoryRegistry(); var registry = new DriverFactoryRegistry();
@@ -86,7 +102,7 @@ public static class DriverFactoryBootstrap
// failures; without this it writes to NullLogger in production. // failures; without this it writes to NullLogger in production.
ConfigureAbTransportLogging(sp); ConfigureAbTransportLogging(sp);
Register(registry, loggerFactory, secretResolver, primaryRoleSource, ResolveAbCipDefaultTransport(sp), Register(registry, loggerFactory, secretResolver, primaryRoleSource, ResolveAbCipDefaultTransport(sp),
scriptRoot, scripting); sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting);
return registry; return registry;
}); });
services.AddSingleton<IDriverFactory>(sp => services.AddSingleton<IDriverFactory>(sp =>
@@ -215,6 +231,7 @@ public static class DriverFactoryBootstrap
ISecretResolver secretResolver, ISecretResolver secretResolver,
IDriverPrimaryRoleSource primaryRoleSource, IDriverPrimaryRoleSource primaryRoleSource,
Driver.AbCip.AbCipTransport abCipDefaultTransport, Driver.AbCip.AbCipTransport abCipDefaultTransport,
Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory galaxyAckIdentities,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null, ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null) ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null)
{ {
@@ -222,7 +239,7 @@ public static class DriverFactoryBootstrap
Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory); Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting); Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting);
Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry, loggerFactory); Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory); Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory, galaxyAckIdentities);
Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory); Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Mqtt.MqttDriverFactoryExtensions.Register(registry, loggerFactory, primaryRoleSource); Driver.Mqtt.MqttDriverFactoryExtensions.Register(registry, loggerFactory, primaryRoleSource);
// Tier A (the Register default): fully managed — HttpClient + System.Xml.Linq, no native SDK, // Tier A (the Register default): fully managed — HttpClient + System.Xml.Linq, no native SDK,
@@ -0,0 +1,16 @@
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Host.Drivers;
/// <summary>
/// Runs the <see cref="GalaxyAckIdentityDirectory"/> refresh loop on driver nodes (lmxopcua#765): an initial
/// refresh at start, then every <c>Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds</c>, and early when
/// an identity secret changes or a refusal asks for one.
/// </summary>
/// <param name="directory">The node's directory.</param>
public sealed class GalaxyAckIdentityRefreshService(GalaxyAckIdentityDirectory directory) : BackgroundService
{
/// <inheritdoc />
protected override Task ExecuteAsync(CancellationToken stoppingToken) => directory.RunAsync(stoppingToken);
}
@@ -194,6 +194,8 @@ builder.Services.AddValidatedOptions<LdapOptions, LdapOptionsValidator>(builder.
// Test Connect probe (admin role, which must probe the stack the driver would run). A bad value fails host // Test Connect probe (admin role, which must probe the stack the driver would run). A bad value fails host
// start here rather than silently falling back. // start here rather than silently falling back.
builder.Services.AddAbCipDriverDefaults(builder.Configuration); builder.Services.AddAbCipDriverDefaults(builder.Configuration);
// lmxopcua#765: refresh/staleness of the cached API-key → Galaxy ack identity set (driver nodes consume it).
builder.Services.AddGalaxyAckIdentityCache(builder.Configuration);
if (hasDriver) if (hasDriver)
{ {
@@ -0,0 +1,356 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime;
/// <summary>
/// lmxopcua#765 item 3 — a keyed session whose API key has no Galaxy ack identity is refused synchronously
/// (<c>BadUserAccessDenied</c> via <see cref="ZB.MOM.WW.OtOpcUa.Core.Abstractions.IAlarmAcknowledgePreflight"/>)
/// from a cached, periodically refreshed set read without I/O, and everything the set is not certain about
/// fails open to the Good-then-revert path.
/// </summary>
public sealed class GalaxyAckIdentityDirectoryTests
{
private const string WithIdentity = "0a1b2c3d4e5f6071";
private const string WithoutIdentity = "ffeeddccbbaa0099";
private static readonly TimeSpan Interval = TimeSpan.FromSeconds(30);
private static readonly TimeSpan StaleAfter = TimeSpan.FromSeconds(90);
// ------------------------------------------------------------------ driver pre-check
[Fact]
public async Task Keyed_session_without_an_identity_is_refused_synchronously()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
var refusal = driver.GetAcknowledgeRefusal(WithoutIdentity);
refusal.ShouldNotBeNull();
refusal.ShouldContain(WithoutIdentity);
refusal.ShouldContain("no Galaxy ack identity");
refusal.ShouldContain("drv");
}
[Fact]
public async Task Keyed_session_with_an_identity_proceeds()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
driver.GetAcknowledgeRefusal(WithIdentity).ShouldBeNull();
driver.GetAcknowledgeRefusal(WithIdentity.ToUpperInvariant()).ShouldBeNull("key ids are case-insensitive");
}
[Fact]
public async Task A_fallback_account_lets_a_keyed_session_without_an_identity_through()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory, Opts() with
{
AlarmAck = new GalaxyAlarmAckOptions("ack-svc", "PLANT", "secret:otopcua/galaxy/ack-fallback"),
});
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
}
[Fact]
public void Without_a_directory_a_keyed_session_is_never_pre_refused()
{
using var driver = NewDriver(directory: null);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
driver.GetAcknowledgeRefusal(apiKeyId: null).ShouldNotBeNull("the #760 anonymous refusal is unchanged");
}
[Fact]
public async Task A_refusal_asks_for_a_refresh_so_a_retry_sees_an_identity_set_out_of_band()
{
var (directory, store, _) = await RefreshedDirectoryAsync(TimeSpan.FromHours(1));
using var driver = NewDriver(directory);
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var loop = directory.RunAsync(cts.Token);
await WaitForAsync(() => store.ListCalls >= 2, cts.Token); // RunAsync's own initial refresh
// Set by the `secret` CLI: nothing in this process is told.
store.AddIdentity(WithoutIdentity);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull();
// The refusal kicked a refresh well inside the one-hour interval.
await WaitForAsync(() => driver.GetAcknowledgeRefusal(WithoutIdentity) is null, cts.Token);
await cts.CancelAsync();
await loop;
}
// ------------------------------------------------------------------ fail open
[Fact]
public void Before_the_first_refresh_every_key_is_Unknown_and_proceeds()
{
using var directory = new GalaxyAckIdentityDirectory(new FakeSecretStore(), Interval, StaleAfter);
using var driver = NewDriver(directory);
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
}
[Fact]
public async Task A_stale_snapshot_is_Unknown_and_proceeds()
{
var (directory, store, clock) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull("positive control: fresh snapshot refuses");
// The store starts failing: the old snapshot is kept, and ages out.
store.Fail = true;
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeFalse();
clock.Advance(StaleAfter - TimeSpan.FromSeconds(1));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
clock.Advance(TimeSpan.FromSeconds(2));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
directory.GetPresence(WithIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull();
}
[Fact]
public async Task A_signalled_identity_change_makes_the_snapshot_Unknown_until_the_next_refresh()
{
var (directory, store, _) = await RefreshedDirectoryAsync();
using var driver = NewDriver(directory);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull("positive control");
// AdminUI sets the identity: the write invalidates the secret, the directory is told.
store.AddIdentity(WithoutIdentity);
directory.NotifySecretChanged(new SecretName(GalaxyAckIdentitySecrets.UserSecretName(WithoutIdentity)));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
driver.GetAcknowledgeRefusal(WithoutIdentity).ShouldBeNull("never refuse in the window before the refresh");
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Present);
}
[Fact]
public async Task A_change_signalled_during_a_refresh_leaves_that_snapshot_untrusted()
{
var (directory, store, _) = await RefreshedDirectoryAsync();
store.OnList = () => directory.NotifySecretChanged(
new SecretName(GalaxyAckIdentitySecrets.PasswordSecretName(WithoutIdentity)));
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
[Fact]
public async Task An_unrelated_secret_change_does_not_untrust_the_snapshot()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
directory.NotifySecretChanged(new SecretName("otopcua/galaxy/ack-fallback"));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
}
[Fact]
public async Task A_directory_without_a_store_is_always_Unknown()
{
using var directory = new GalaxyAckIdentityDirectory(store: null, Interval, StaleAfter);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeFalse();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
[Fact]
public async Task A_key_id_the_resolver_could_not_look_up_is_Unknown()
{
var (directory, _, _) = await RefreshedDirectoryAsync();
directory.GetPresence("../etc").ShouldBe(GalaxyAckIdentityPresence.Unknown);
directory.GetPresence(" ").ShouldBe(GalaxyAckIdentityPresence.Unknown);
directory.GetPresence(null).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
// ------------------------------------------------------------------ refresh
[Fact]
public async Task Refresh_picks_up_added_and_removed_identities()
{
var (directory, store, _) = await RefreshedDirectoryAsync();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
store.AddIdentity(WithoutIdentity);
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent, "a cache, until refreshed");
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Present);
store.RemoveIdentity(WithIdentity, softDelete: true);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent, "a deleted user secret is no identity");
}
[Fact]
public async Task Only_user_secrets_count_as_an_identity()
{
var store = new FakeSecretStore();
store.Add(GalaxyAckIdentitySecrets.PasswordSecretName(WithoutIdentity)); // password without user
store.Add("otopcua/galaxy/ack-fallback");
using var directory = new GalaxyAckIdentityDirectory(store, Interval, StaleAfter);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
}
[Fact]
public async Task The_loop_refreshes_on_its_interval()
{
var store = new FakeSecretStore();
using var directory = new GalaxyAckIdentityDirectory(store, TimeSpan.FromMilliseconds(50), TimeSpan.FromMinutes(1));
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var loop = directory.RunAsync(cts.Token);
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Absent, cts.Token);
store.AddIdentity(WithIdentity);
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Present, cts.Token);
await cts.CancelAsync();
await loop; // completes cleanly on cancellation
}
[Fact]
public async Task The_loop_refreshes_early_on_a_signalled_change()
{
var store = new FakeSecretStore();
using var directory = new GalaxyAckIdentityDirectory(store, TimeSpan.FromHours(1), TimeSpan.FromHours(3));
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var loop = directory.RunAsync(cts.Token);
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Absent, cts.Token);
store.AddIdentity(WithIdentity);
directory.NotifySecretChanged(new SecretName(GalaxyAckIdentitySecrets.UserSecretName(WithIdentity)));
await WaitForAsync(() => directory.GetPresence(WithIdentity) == GalaxyAckIdentityPresence.Present, cts.Token);
await cts.CancelAsync();
await loop;
}
// ------------------------------------------------------------------ helpers
private static async Task<(GalaxyAckIdentityDirectory Directory, FakeSecretStore Store, ManualClock Clock)>
RefreshedDirectoryAsync(TimeSpan? interval = null)
{
var store = new FakeSecretStore();
store.AddIdentity(WithIdentity);
var clock = new ManualClock();
var directory = new GalaxyAckIdentityDirectory(
store, interval ?? Interval, interval is { } i ? i * 3 : StaleAfter, clock);
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
return (directory, store, clock);
}
private static async Task WaitForAsync(Func<bool> condition, CancellationToken ct)
{
while (!condition())
{
ct.ThrowIfCancellationRequested();
await Task.Delay(10, ct);
}
}
private static GalaxyDriverOptions Opts() => new(
new GalaxyGatewayOptions("https://mxgw.test:5001", "dev:key"),
new GalaxyMxAccessOptions("OtOpcUa-A"),
new GalaxyRepositoryOptions(),
new GalaxyReconnectOptions());
private static GalaxyDriver NewDriver(IGalaxyAckIdentityDirectory? directory, GalaxyDriverOptions? options = null) =>
new("drv", options ?? Opts(), hierarchySource: null, ackIdentityDirectory: directory);
private sealed class ManualClock : TimeProvider
{
private DateTimeOffset _now = new(2026, 11, 9, 12, 0, 0, TimeSpan.Zero);
public void Advance(TimeSpan by) => _now += by;
public override DateTimeOffset GetUtcNow() => _now;
}
/// <summary>Metadata-only store: <see cref="ListAsync"/> is all the directory may call.</summary>
private sealed class FakeSecretStore : ISecretStore
{
private readonly Dictionary<string, bool> _rows = new(StringComparer.Ordinal); // name → deleted
private readonly Lock _lock = new();
private int _listCalls;
public bool Fail { get; set; }
public Action? OnList { get; set; }
public int ListCalls => Volatile.Read(ref _listCalls);
public void Add(string name)
{
lock (_lock) _rows[name] = false;
}
public void AddIdentity(string keyId)
{
Add(GalaxyAckIdentitySecrets.PasswordSecretName(keyId));
Add(GalaxyAckIdentitySecrets.UserSecretName(keyId));
}
public void RemoveIdentity(string keyId, bool softDelete)
{
lock (_lock)
{
foreach (var name in new[]
{
GalaxyAckIdentitySecrets.UserSecretName(keyId), GalaxyAckIdentitySecrets.PasswordSecretName(keyId),
})
{
if (softDelete) _rows[name] = true;
else _rows.Remove(name);
}
}
}
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct)
{
Interlocked.Increment(ref _listCalls);
OnList?.Invoke();
if (Fail) throw new InvalidOperationException("store unavailable");
lock (_lock)
{
// Deleted rows are returned regardless, to prove the directory filters them itself.
IReadOnlyList<SecretMetadata> rows = _rows.Select(r => new SecretMetadata
{
Name = new SecretName(r.Key),
ContentType = SecretContentType.Text,
KekId = "k",
Revision = 1,
IsDeleted = r.Value,
CreatedUtc = DateTimeOffset.UnixEpoch,
UpdatedUtc = DateTimeOffset.UnixEpoch,
}).ToList();
return Task.FromResult(rows);
}
}
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) => throw Unexpected();
public Task UpsertAsync(StoredSecret row, CancellationToken ct) => throw Unexpected();
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct) => throw Unexpected();
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) => throw Unexpected();
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) => throw Unexpected();
public Task<bool> ApplyRewrapAsync(StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
throw Unexpected();
// The directory must read metadata only — never a row (that would decrypt-adjacent I/O per key).
private static InvalidOperationException Unexpected() => new("the directory may only list metadata");
}
}
@@ -0,0 +1,208 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.Drivers;
using ZB.MOM.WW.Secrets;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
/// <summary>
/// lmxopcua#765 item 3 — the host wiring of the cached API-key → Galaxy ack identity set: the driver registry
/// hands every Galaxy driver the node's directory (so the synchronous pre-check can refuse a keyed session
/// without an identity), a refresh loop runs on driver nodes, secret writes signal it through the decorated
/// invalidator, and the refresh interval is configurable.
/// </summary>
[Trait("Category", "Unit")]
public sealed class GalaxyAckIdentityCacheWiringTests
{
private const string WithIdentity = "0a1b2c3d4e5f6071";
private const string WithoutIdentity = "ffeeddccbbaa0099";
private const string GalaxyConfig = """
{
"Gateway": { "Endpoint": "https://mxgw.test:5001", "ApiKeySecretRef": "dev:key" },
"MxAccess": { "ClientName": "OtOpcUa-A" }
}
""";
[Fact]
public async Task A_registry_built_Galaxy_driver_refuses_a_keyed_session_without_an_identity()
{
var store = new ListOnlySecretStore(GalaxyAckIdentitySecrets.UserSecretName(WithIdentity));
using var sp = DriverServices(store).BuildServiceProvider();
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
var driver = sp.GetRequiredService<DriverFactoryRegistry>()
.TryGet(GalaxyDriverFactoryExtensions.DriverTypeName)!.Invoke("galaxy-1", GalaxyConfig);
var preflight = driver.ShouldBeAssignableTo<IAlarmAcknowledgePreflight>()!;
preflight.GetAcknowledgeRefusal(WithoutIdentity).ShouldNotBeNull();
preflight.GetAcknowledgeRefusal(WithIdentity).ShouldBeNull();
(driver as IDisposable)?.Dispose();
}
[Fact]
public void Driver_nodes_run_the_refresh_loop_and_share_one_directory()
{
var services = DriverServices(new ListOnlySecretStore());
services.AddOtOpcUaDriverFactories(); // a fused admin,driver node may reach it twice
// By descriptor: resolving every IHostedService would need the resilience publisher's whole graph.
services.Count(d => d.ServiceType == typeof(IHostedService)
&& d.ImplementationType == typeof(GalaxyAckIdentityRefreshService)).ShouldBe(1);
using var sp = services.BuildServiceProvider();
sp.GetRequiredService<IGalaxyAckIdentityDirectory>()
.ShouldBeSameAs(sp.GetRequiredService<GalaxyAckIdentityDirectory>());
}
[Fact]
public void The_refresh_interval_and_staleness_are_configurable()
{
var services = DriverServices(new ListOnlySecretStore());
services.AddGalaxyAckIdentityCache(Config(new()
{
["Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds"] = "7",
["Drivers:Galaxy:AckIdentityCache:StaleAfterSeconds"] = "50",
}));
using var sp = services.BuildServiceProvider();
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
directory.RefreshInterval.ShouldBe(TimeSpan.FromSeconds(7));
directory.StaleAfter.ShouldBe(TimeSpan.FromSeconds(50));
}
[Fact]
public void Defaults_are_30s_and_three_intervals()
{
using var sp = DriverServices(new ListOnlySecretStore()).BuildServiceProvider();
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
directory.RefreshInterval.ShouldBe(TimeSpan.FromSeconds(30));
directory.StaleAfter.ShouldBe(TimeSpan.FromSeconds(90));
}
[Theory]
[InlineData("0", "0")]
[InlineData("3601", "0")]
[InlineData("30", "30")]
public void Bad_cache_settings_fail_validation(string interval, string staleAfter)
{
var services = new ServiceCollection();
services.AddGalaxyAckIdentityCache(Config(new()
{
["Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds"] = interval,
["Drivers:Galaxy:AckIdentityCache:StaleAfterSeconds"] = staleAfter,
}));
using var sp = services.BuildServiceProvider();
Should.Throw<OptionsValidationException>(() => sp.GetRequiredService<IOptions<GalaxyAckIdentityCacheOptions>>().Value);
}
[Fact]
public async Task AddOtOpcUaSecrets_decorates_the_invalidator_so_an_identity_write_untrusts_the_directory()
{
// Registered before AddOtOpcUaSecrets, so the library's TryAdd keeps this store for the directory's listing.
var services = DriverServices(new ListOnlySecretStore());
services.AddLogging();
services.AddOtOpcUaSecrets(Config(new()
{
// Resolving the invalidator never opens the store; the path only has to pass validation.
["Secrets:SqlitePath"] = Path.Combine(Path.GetTempPath(), $"otopcua-secrets-{Guid.NewGuid():N}.db"),
["Secrets:MasterKey:Source"] = "Environment",
["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_MASTER_KEY",
["Secrets:RunMigrationsOnStartup"] = "false",
}));
using var sp = services.BuildServiceProvider();
var invalidator = sp.GetRequiredService<ISecretCacheInvalidator>();
var decorated = invalidator.ShouldBeOfType<SecretChangeSignallingInvalidator>();
decorated.Inner.ShouldBeOfType<DefaultSecretResolver>();
decorated.Inner.ShouldBeSameAs(sp.GetRequiredService<DefaultSecretResolver>(), "the resolver cache is still invalidated");
var directory = sp.GetRequiredService<GalaxyAckIdentityDirectory>();
(await directory.RefreshAsync(CancellationToken.None)).ShouldBeTrue();
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Absent);
// What ApiKeyGalaxyAckIdentityService.SetAsync does after writing the user secret.
invalidator.Invalidate(new SecretName(GalaxyAckIdentitySecrets.UserSecretName(WithoutIdentity)));
directory.GetPresence(WithoutIdentity).ShouldBe(GalaxyAckIdentityPresence.Unknown);
}
[Fact]
public void Without_a_directory_the_decorated_invalidator_is_a_pass_through()
{
var inner = new RecordingInvalidator();
var services = new ServiceCollection();
services.AddSingleton<ISecretCacheInvalidator>(_ => inner);
SecretsRegistration.DecorateCacheInvalidator(services);
using var sp = services.BuildServiceProvider();
var name = new SecretName("otopcua/apikeys/abc/galaxy-ack-user");
sp.GetRequiredService<ISecretCacheInvalidator>().Invalidate(name);
inner.Names.ShouldBe([name]);
}
// ------------------------------------------------------------------ helpers
private static IServiceCollection DriverServices(ISecretStore store)
{
var services = new ServiceCollection();
services.AddSingleton<ISecretResolver>(new NullResolver());
services.AddSingleton(store);
services.AddOtOpcUaDriverFactories();
return services;
}
private static IConfiguration Config(Dictionary<string, string?> values) =>
new ConfigurationBuilder().AddInMemoryCollection(values).Build();
private sealed class NullResolver : ISecretResolver
{
public Task<string?> GetAsync(SecretName name, CancellationToken ct) => Task.FromResult<string?>(null);
}
private sealed class RecordingInvalidator : ISecretCacheInvalidator
{
public List<SecretName> Names { get; } = [];
public void Invalidate(SecretName name) => Names.Add(name);
}
private sealed class ListOnlySecretStore(params string[] names) : ISecretStore
{
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct) =>
Task.FromResult<IReadOnlyList<SecretMetadata>>(names.Select(n => new SecretMetadata
{
Name = new SecretName(n),
ContentType = SecretContentType.Text,
KekId = "k",
Revision = 1,
CreatedUtc = DateTimeOffset.UnixEpoch,
UpdatedUtc = DateTimeOffset.UnixEpoch,
}).ToList());
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) => throw new NotSupportedException();
public Task UpsertAsync(StoredSecret row, CancellationToken ct) => throw new NotSupportedException();
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct) => throw new NotSupportedException();
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) => throw new NotSupportedException();
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) => throw new NotSupportedException();
public Task<bool> ApplyRewrapAsync(StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
throw new NotSupportedException();
}
}