using System.Collections.Concurrent;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
///
/// Memoizes the grant-walk half of an ACL decision per (session, node, operation) for one
/// authorization generation (#520 stage 6).
///
///
///
/// Why this exists. Stage 5 gated write, alarm commands and HistoryRead — all of which are
/// request-rate operations where rebuilding the decision each time costs nothing. Stage 6 gates
/// Read, Browse and subscription creation, which are not: one Browse of a device folder asks the
/// same question once per child, and a client that reads a thousand nodes asks it a thousand times.
/// The trie walk is cheap individually and the repetition is what makes it worth avoiding.
///
///
/// ⚠️ Only the grant walk is cached — never the key check. A cached verdict is valid because
/// grants, principals and the trie are all fixed for the life of a generation, so nothing the walk
/// reads can change without advancing. That reasoning does not
/// extend to the calling key's usability: a key's ExpiresUtc passes with the clock, not with
/// a deployment, so a cached allow would keep an expired key working until the next deploy —
/// re-introducing, in a subtler form, exactly the open-session hole stage 5 closed.
/// therefore runs revocation and expiry on every call and consults
/// this only afterwards.
///
///
/// Invalidation is wholesale, by generation. Per-entry invalidation would need to know which
/// nodes a changed grant covers, which is the trie walk run backwards; dropping everything on a
/// deploy costs one repopulation and cannot be wrong. Deployments are rare relative to reads.
///
///
/// ⚠️ The invariant that makes that safe is per-shard, not global: a shard can only ever answer for
/// the generation it was built under. The wholesale drop is necessarily two steps — publish the new
/// generation, then purge the shards — because the publish is the compare-exchange that elects
/// the single thread allowed to purge. Between them the stamp already matches while the previous
/// generation's shards are still in the table, so comparing the stamp alone would let a caller arriving
/// in that window read a verdict the new trie never made. Each shard therefore carries its own
/// generation: a mismatched shard is a miss on the read path and is displaced on the store path, which
/// makes the window inert rather than merely narrow.
///
///
/// Storage is sharded per calling KEY, under a two-level bound. The shard key is
/// ApiKeyId ?? Principal, so sessions sharing one API key share one shard and one
/// cap — the isolation is across distinct keys, not across
/// sessions. Overflowing a shard clears that shard only, so a crawl by one key cannot evict
/// another key's entries. A second, global bound clears every shard, and
/// exists purely so many keys each staying under their own cap cannot add up to unbounded growth.
/// The flat cache this replaced had only the global bound, which made one machine consumer
/// crawling a large tree an eviction event for every other caller on the server — the callers
/// doing ordinary work paid for the crawl. Both caps are enforced against
/// counters, never
/// , which locks the whole table.
///
///
public sealed class SessionAclDecisionCache
{
///
/// Process-wide entry ceiling, past which every shard clears. The outer of the two bounds,
/// and a rare fallback: it exists so a large fleet of distinct keys each staying well under
/// still cannot grow the cache without limit.
///
///
/// Enforced against an counter rather than
/// , which takes every lock in the table — a
/// price the store path would pay on every miss to answer a question that only matters at the
/// ceiling. The counter lags — an insert is in its shard's dictionary a moment before it is booked
/// — but it does not drift: every booked entry is unbooked exactly once, when the shard
/// holding it is sealed. (Not every entry is booked: one whose loses
/// the compare-exchange to a seal is never booked at all, because it is already unreachable in a
/// displaced shard.) See for how that survives an eviction racing an
/// insert.
///
public const int MaxEntries = 100_000;
///
/// Per-key entry ceiling (the shard key is ApiKeyId ?? Principal, so every session
/// using the same API key shares this one budget), past which that shard alone clears. The inner
/// and normally the only bound reached: it is what keeps one machine consumer crawling a 50k-node
/// tree from evicting the decisions taken for every OTHER key on the server. The name predates the
/// shard key and is kept for source compatibility.
///
///
/// Clearing beats evicting within a shard: an LRU needs per-entry bookkeeping on the hot path
/// to save a trie walk that is itself cheap, so the eviction policy would cost more than the miss it
/// prevents. The failure mode of clearing is a burst of recomputation, which is exactly the steady
/// state of not having a cache at all — and sharding confines that burst to the key that caused it,
/// which is the part the old flat cache got wrong.
///
public const int MaxEntriesPerSession = 20_000;
///
/// Sentinel written over a shard's the instant it stops being the
/// live shard for its key. It is the whole of the eviction race fix: it is set by the same
/// that captures the count being subtracted from
/// , so an insert either books its increment before that capture
/// (and is therefore included in the subtraction) or sees the sentinel and books nothing at all.
/// There is no third outcome, which is what makes the counter exact rather than approximate.
/// Negative so it can never be mistaken for a real count, and far enough from zero that the
/// increment path cannot walk it back to a positive value.
///
private const int SealedShard = int.MinValue;
private readonly ConcurrentDictionary _shards = new(StringComparer.Ordinal);
private long _generation = -1;
private int _globalCount;
/// The authorization generation the cached decisions were taken under.
public long Generation => Interlocked.Read(ref _generation);
///
/// Exact number of decisions currently memoized, across every shard. Diagnostics and tests only —
/// it enumerates the shards and takes each one's locks, so it must stay off the store path (which
/// uses the approximate counters above instead).
///
public int Count
{
get
{
var total = 0;
foreach (var shard in _shards.Values) total += shard.Entries.Count;
return total;
}
}
///
/// The running counter both ceilings are enforced against. Exposed for the concurrency audit only:
/// the property a test wants is almost always , and the single interesting fact
/// about this one is that the two agree.
///
internal int TrackedCount => Volatile.Read(ref _globalCount);
///
/// Sum of the live shards' own counters — the middle term of the invariant
/// TrackedCount == LiveShardCount == Count. Reading the three separately is what tells a
/// failing audit whether a shard's counter drifted from its dictionary or the global counter
/// drifted from the shards.
///
internal int LiveShardCount
{
get
{
var total = 0;
foreach (var shard in _shards.Values) total += Volatile.Read(ref shard.ApproxCount);
return total;
}
}
///
/// Looks a decision up without allocating anything — the hot half, and the reason the store half is
/// a separate method rather than a parameter.
///
///
///
/// Why split from . A closure-taking API forces its caller to
/// build the closure — and everything the closure captures — before it knows whether the
/// answer is already memoized, so the allocation the cache exists to avoid is paid on the hit
/// path too. Read and Browse ask this question once per node, which is precisely where paying
/// for a walk that never runs is worth removing. remains, delegating
/// here, but has no production caller — it is exercised by the cache's own tests.
///
///
/// The generation is checked (and a stale cache dropped) here rather than in
/// alone, so a lookup landing first after a deploy cannot read the previous
/// generation's verdicts.
///
///
/// The authorization generation the decision is being taken under.
/// Stable identity of the calling session — its key id, else its principal.
/// The operation being attempted.
/// The realm the node was resolved in.
/// The node's s= identifier.
/// The memoized verdict, when one exists.
/// when a decision was memoized for this key.
public bool TryGet(
long generation,
string sessionKey,
OpcUaOperation operation,
AddressSpaceRealm realm,
string nodeIdentifier,
out bool decision)
{
InvalidateIfStale(generation);
decision = false;
// TryGetValue rather than GetOrAdd: a session that only ever hits must not mint a shard, and a
// session whose shard was just cleared must not resurrect one until it actually stores something.
//
// The shard's OWN generation decides, not the stamp InvalidateIfStale just compared: inside the
// rollover window the stamp already matches and the purge has not run, so the stamp cannot
// distinguish this generation's shard from the last one's. A mismatch is treated as a plain miss and
// is NOT displaced from here — this path stays allocation-free and non-mutating, and the shard is
// reclaimed a moment later by the purging thread or by the next Set on that key.
return _shards.TryGetValue(sessionKey, out var shard)
&& shard.Generation == generation
&& shard.Entries.TryGetValue(new ShardKey(operation, realm, nodeIdentifier), out decision);
}
///
/// Memoizes a decision, subject to the two entry bounds and the overtaken-generation drop rule.
///
/// The generation the decision was taken under — not re-read from the
/// caller, so a deploy landing mid-walk is detected here.
/// Stable identity of the calling session — its key id, else its principal.
/// The operation that was attempted.
/// The realm the node was resolved in.
/// The node's s= identifier.
/// The verdict the grant walk produced.
public void Set(
long generation,
string sessionKey,
OpcUaOperation operation,
AddressSpaceRealm realm,
string nodeIdentifier,
bool decision)
{
// Re-read the generation AFTER the walk. A deploy landing mid-walk means the verdict may have been
// taken against either generation, and storing it under the new one would pin a decision the new
// trie never made. Dropping it costs one recomputation on the next call.
if (Generation != generation) return;
var shard = _shards.GetOrAdd(sessionKey, static (_, gen) => new Shard(gen), generation);
if (Volatile.Read(ref _globalCount) >= MaxEntries)
{
// The outer bound: no single session is over its own ceiling, so there is nothing narrower to
// drop. Re-acquire the shard — ClearAll drops the shard objects themselves.
//
// Deliberately NOT elected: concurrent callers crossing the ceiling together all run ClearAll,
// and that is safe rather than merely tolerated. ClearAll displaces each shard with a per-shard
// TryRemove, so exactly one thread wins each shard and seals it exactly once; the losers just
// re-enumerate an emptying dictionary. The redundancy costs an enumeration, not correctness,
// and a CAS-elected clear would add hot-path state for a branch that is a rare fallback.
ClearAll();
shard = _shards.GetOrAdd(sessionKey, static (_, gen) => new Shard(gen), generation);
}
else if (Volatile.Read(ref shard.ApproxCount) >= MaxEntriesPerSession)
{
// The inner bound. The replacement is pre-sized to what the shard it displaces was holding: this
// key has just demonstrated it fills its whole budget, and it is about to do so again, so the
// refill would otherwise walk the dictionary's whole growth cascade — a rehash under every one of
// the table's locks at each doubling — to arrive back at a size already known. Clamped to the
// ceiling because the count read here is the pre-seal one — a thread that lost the swap reads
// the sentinel and clamps to no hint at all, which is the right answer for it. Only this site
// hints; a shard minted by GetOrAdd has no such evidence and stays at the default capacity.
//
// The trade: the replacement is built before the swap is attempted, so a thread that loses
// discards a pre-sized dictionary rather than an empty one. That is a bigger throwaway than
// before, and it is accepted — the loss window is a single compare-and-swap wide, and it is
// paid only by threads that overflowed the same shard simultaneously, against a refill cost
// paid by every eviction.
//
// A shard from a PREVIOUS generation contributes no hint: the fill level it reached belongs to a
// set of grants that no longer exists, and this very Displace replaces it with a fresh shard
// stamped to the current generation regardless. Sizing this generation's storage from the last
// one's crawl would be a guess dressed as evidence.
shard = Displace(
sessionKey, shard, generation,
capacityHint: shard.Generation == generation
? Math.Clamp(Volatile.Read(ref shard.ApproxCount), 0, MaxEntriesPerSession)
: 0);
}
// The shard must belong to THIS generation before anything is filed in it. Reached in two ways: a
// deploy landed since the shard was minted (the common case — one displacement per key per deploy,
// and the read path has been missing on it since the stamp moved), or the rollover window closed
// between the check at the top of this method and here.
if (shard.Generation != generation)
{
shard = Displace(sessionKey, shard, generation, capacityHint: 0);
// Still mismatched means a racer re-minted the shard under a generation that has since moved
// again. Drop the decision rather than loop: filing this generation's verdict in another
// generation's shard is the same error in the other direction, and the line above already
// dropped a verdict for the identical reason.
if (shard.Generation != generation) return;
}
// TryAdd, not an indexer assignment: only a genuine insert may move the counters, or a racing pair
// of walks over the same key would inflate them and pull the ceilings in for good.
if (shard.Entries.TryAdd(new ShardKey(operation, realm, nodeIdentifier), decision))
TryBookInsert(shard);
}
///
/// Replaces a session's live shard with a fresh one for and returns
/// whatever shard is live afterwards.
///
///
/// This REPLACES the shard object rather than emptying the dictionary in place — the move
/// makes, and for the same reason. Emptying in place left the clear and the
/// counter reset as two steps over storage a racing was still inserting into: the
/// racer could land in the just-cleared dictionary and increment both counters between them, and the
/// then subtracted an amount that included the
/// racer's fresh entry, under-counting for good. Swapping the object makes
/// the displaced shard unreachable in one atomic step, after which settles its
/// count exactly — and because only the thread that wins the compare-and-swap seals, no shard is
/// sealed twice however many threads arrive here at once. The returned shard is re-read from the
/// table rather than assumed to be the one just built: whoever won the swap, the live shard is now a
/// different object than the caller held.
///
/// The key whose shard is being displaced.
/// The shard the caller found live.
/// The generation the replacement is built for.
/// Initial entry capacity for the replacement; zero leaves it at the default.
/// The live shard after the swap.
private Shard Displace(string sessionKey, Shard shard, long generation, int capacityHint)
{
if (_shards.TryUpdate(sessionKey, new Shard(generation, capacityHint), shard))
Interlocked.Add(ref _globalCount, -Seal(shard));
return _shards.GetOrAdd(sessionKey, static (_, gen) => new Shard(gen), generation);
}
///
/// Books a just-completed insert against the shard's own counter and the global one, unless the
/// shard was displaced first — in which case the entry lives in storage nothing can reach any more
/// and must not be counted.
///
///
/// A plain here would race the seal in the other
/// direction: an increment landing after the displacing thread captured the count would leave
/// carrying an entry no live shard holds. The compare-exchange is what
/// linearises the two — either it wins while the shard is still live (and the seal that follows
/// captures and subtracts this increment along with the rest), or it observes
/// and books nothing. The loop retries only against other inserts
/// on the same shard; the sealed case exits.
///
/// The shard's counter is moved before the global one, and that order matters: a seal
/// landing between the two subtracts an increment the global counter has not received yet, so
/// can dip transiently below the live total. That is self-healing —
/// the thread in the gap always completes its increment — and it errs towards not
/// clearing, which is the harmless direction. The reverse order cannot self-heal: it would book
/// globally before the seal could see the entry, leaving a permanent over-count.
///
///
/// The shard the entry was added to.
private void TryBookInsert(Shard shard)
{
while (true)
{
var seen = Volatile.Read(ref shard.ApproxCount);
// Displaced between the TryAdd and here: the entry is lost with the shard, and counting it
// would be exactly the over-count this method exists to prevent.
if (seen < 0) return;
if (Interlocked.CompareExchange(ref shard.ApproxCount, seen + 1, seen) != seen) continue;
Interlocked.Increment(ref _globalCount);
return;
}
}
///
/// Retires a shard that has just been removed from (or replaced in) and
/// returns the number of entries it was still accounted for, for the caller to subtract from
/// .
///
///
/// Only the thread that actually displaced the shard may call this — '
/// compare-and-swap removal/replacement elects exactly one, so no shard is ever sealed twice. The
/// already-sealed branch is defence in depth: it returns zero rather than subtracting the
/// sentinel, which would drive the global counter deeply negative and disable both ceilings.
///
/// The displaced shard.
/// The count to subtract from .
private static int Seal(Shard shard)
{
var captured = Interlocked.Exchange(ref shard.ApproxCount, SealedShard);
return captured < 0 ? 0 : captured;
}
///
/// Returns the memoized decision for this key, computing and storing it when absent or when the
/// generation has moved on. A convenience over + .
///
/// Retained for tests; production has no caller. uses
/// the split / pair precisely so a hit pays for no closure,
/// and no other caller has appeared since. Kept because it is the readable single-expression
/// form of the same protocol and is how the cache's tests drive it — not because a production
/// caller is expected back.
///
///
/// The authorization generation the decision is being taken under.
/// Stable identity of the calling session — its key id, else its principal.
/// The operation being attempted.
/// The realm the node was resolved in.
/// The node's s= identifier.
/// The grant walk to run on a miss.
/// when the operation is permitted.
public bool GetOrAdd(
long generation,
string sessionKey,
OpcUaOperation operation,
AddressSpaceRealm realm,
string nodeIdentifier,
Func compute)
{
ArgumentNullException.ThrowIfNull(compute);
if (TryGet(generation, sessionKey, operation, realm, nodeIdentifier, out var cached)) return cached;
var decided = compute();
Set(generation, sessionKey, operation, realm, nodeIdentifier, decided);
return decided;
}
/// Drops every memoized decision. Called when the authorization generation advances.
public void Clear() => ClearAll();
///
/// Moves the generation stamp without clearing — reproducing, deterministically, the exact
/// view a racer gets inside 's rollover window, where the new
/// generation is already published but the winner's has not run yet.
///
///
/// Tests only. That window is a few instructions wide, so a concurrency test can only make it
/// probable; this makes it certain. The invariant it exercises — a shard can only serve the
/// generation it was built under — is what makes the window harmless, so the pin has to be able
/// to stand inside it.
///
/// The generation to publish.
internal void AdvanceGenerationForTest(long generation) =>
Interlocked.Exchange(ref _generation, generation);
///
/// Advances the cache-wide stamp when this call presents a newer generation, and purges on the way.
/// The stamp ends up holding the highest generation any caller has presented — never a lower one,
/// and never one call late.
///
///
///
/// ⚠️ The comparison is <=, not !=, and that is a straggler guard. The
/// generation only ever increases (DataPlaneAuthorityState counts it up rather than
/// deriving it from the revision hash, precisely so it cannot go backwards), but a request
/// can still present an older one: it captured its authority snapshot before a deploy and then
/// lost the scheduling race to a request that captured after. An inequality test treats that
/// straggler as a deploy, rolls the stamp backward, and purges the new generation's
/// freshly-populated shards — so the post-deploy window, when the cache is coldest and doing the
/// most work, is exactly when it thrashes. Verdicts are safe either way now that each shard
/// carries its own generation; what is at stake here is only whether the cache is any use.
///
///
/// ⚠️ That guard lives INSIDE the loop, against a freshly re-read stamp — hoisting it to a
/// single check before the loop reintroduces exactly the backward roll it exists to prevent: a
/// straggler that lost its exchange would spin back round and re-attempt against the newer stamp
/// it just lost to, and eventually succeed in rolling it back. Re-reading means the straggler
/// sees the newer value on its second pass and leaves.
///
///
/// Why a loop rather than one exchange. A single shot defers publication by a call: two
/// callers read the same baseline, the older one's exchange lands first, and the newer one's
/// fails and falls out having published nothing. Its own lookup then misses and the
/// that follows is dropped by the Generation != generation guard, which
/// costs a store in the post-deploy window where the cache is coldest. Retrying against the moved
/// stamp closes that: the higher generation is published on the call that presents it.
///
///
/// The elected clear survives the loop. Only the thread whose exchange actually moved the
/// stamp clears, so a burst of concurrent requests arriving just after a deploy still repopulates
/// once rather than each clearing the others' fresh entries. A retry can now clear too, but only
/// by winning a move to a strictly higher generation — so clears stay bounded by the number of
/// distinct generations presented, not by request volume.
///
///
/// It terminates. Each iteration either returns through the guard or exchanges against a
/// seen strictly larger than the last iteration's (a failed exchange means someone raised
/// the stamp), and _generation only ever rises — so the guard is reached within a bounded
/// number of passes.
///
///
/// The straggler's own lookup is unaffected by returning early: its shard-generation check misses
/// against the newer shards regardless, so it recomputes, which is the correct outcome for a
/// request holding a superseded snapshot. _generation starts at -1, below any real
/// generation, so the first one — generation 0 included — still passes.
///
///
/// The generation the calling request is being decided under.
private void InvalidateIfStale(long generation)
{
while (true)
{
var seen = Interlocked.Read(ref _generation);
if (generation <= seen) return;
if (Interlocked.CompareExchange(ref _generation, generation, seen) == seen)
{
ClearAll();
return;
}
}
}
///
/// Drops every shard and settles the global counter. Dropping the shard objects rather than emptying
/// them also sheds the shards of sessions that have since closed. A walk racing this holds a shard
/// that is no longer in the table; its entry is simply lost, which is the same outcome the clear
/// itself produces.
///
///
/// Each shard is removed and then ed, and only its sealed count is subtracted —
/// rather than the whole counter being slammed to zero. Zeroing is wrong in the other direction
/// from the eviction race: a walk that mints a fresh shard while this is running is a live
/// entry, and resetting the counter after it books its increment would lose it. Subtracting exactly
/// what is being dropped leaves the counter equal to what remains, whatever arrives mid-clear.
///
private void ClearAll()
{
foreach (var key in _shards.Keys)
if (_shards.TryRemove(key, out var shard))
Interlocked.Add(ref _globalCount, -Seal(shard));
}
///
/// One session's decisions, plus the entry count its own ceiling is enforced on. A shard is
/// immutable in identity: it is never emptied, only replaced, so a reference obtained from
/// either still is the live shard or has been sealed — it can never turn
/// into a different generation of the same session's entries under its holder.
///
/// The authorization generation this shard's entries were decided under.
///
/// Initial entry capacity. Zero leaves at the default — the right choice
/// everywhere except the per-shard eviction swap, which alone has evidence of the size the
/// replacement is about to grow back to.
///
private sealed class Shard(long generation, int capacity = 0)
{
public readonly ConcurrentDictionary Entries = capacity > 0
? new ConcurrentDictionary(Environment.ProcessorCount, capacity)
: new ConcurrentDictionary();
///
/// The generation this shard's entries were decided under, fixed for its lifetime. It is what the
/// read path compares — the cache-wide stamp cannot be trusted alone, because it is published one
/// step before the shards it invalidates are purged.
///
public readonly long Generation = generation;
///
/// Read and written with , never derived from
/// on the store path. The name is
/// historical: while the shard is live this tracks exactly (entries are
/// only ever added, and every add books here), and once displaced it holds
/// . It is lagging — an insert is visible in
/// a moment before it is booked here — but never wrong.
///
public int ApproxCount;
}
private readonly record struct ShardKey(
OpcUaOperation Operation, AddressSpaceRealm Realm, string NodeIdentifier);
}