feat(acl): keys + ACLs reach driver-only nodes via the artifact (#520 stage 4)
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys now ride the deployment artifact instead. - ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried. Only the peppered HMAC ships, never a secret, and the pepper is node config that is deliberately not in the artifact. - DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are scoped to the node's own cluster; keys are not, because ApiKey carries no ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it. - OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply, BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until an unrelated structural change redeployed. - ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation immediate there. - OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4 named. NodeScope.ClusterId is required and the evaluator fails closed on a mismatch, so until now no scope it built could have evaluated. Nothing reads it until stages 5/6. The generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs, silently. Sessions are not stamped with it yet — that belongs with the first seam that reads a UserAuthorizationState, in stage 5, rather than being built now and consulted by nothing. Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key authenticates, wrong secret denied, both halves of the revocation asymmetry observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an independent Python HMAC so the verifier was checked against something other than itself. Rig left clean. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
@@ -37,6 +37,24 @@ public enum ClusterFilterMode
|
||||
Suppress,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The data-plane authorization inputs decoded out of a deployment artifact for one node (#520).
|
||||
/// </summary>
|
||||
/// <param name="ClusterId">The node's cluster per its <c>ClusterNode</c> row, or null on a
|
||||
/// single-cluster / legacy artifact where no scoping applies.</param>
|
||||
/// <param name="Keys">Every <c>ApiKey</c> the artifact carries, revoked and expired included — the
|
||||
/// lookup filters those, so that a revocation is visible rather than indistinguishable from a
|
||||
/// missing key.</param>
|
||||
/// <param name="Grants">The <c>NodeAcl</c> rows scoped to <paramref name="ClusterId"/>.</param>
|
||||
public sealed record DataPlaneAuthorityProjection(
|
||||
string? ClusterId,
|
||||
IReadOnlyList<ApiKey> Keys,
|
||||
IReadOnlyList<NodeAcl> Grants)
|
||||
{
|
||||
/// <summary>The projection of an absent or unparseable artifact: no cluster, no keys, no grants.</summary>
|
||||
public static readonly DataPlaneAuthorityProjection Empty = new(null, [], []);
|
||||
}
|
||||
|
||||
/// <summary>Resolved scoping decision for a node against an artifact.</summary>
|
||||
/// <param name="Mode">
|
||||
/// None = apply everything (single-cluster / legacy); ScopeTo = filter to <paramref name="ClusterId"/>;
|
||||
@@ -534,6 +552,170 @@ public static class DeploymentArtifact
|
||||
: all;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Parse the data-plane authorization inputs a node needs to authenticate OPC UA sessions:
|
||||
/// the <c>ApiKeys</c> the artifact carries and the <c>NodeAcls</c> granting their principals
|
||||
/// permissions (#520, stage 4).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is the ONLY key source a <b>driver-only</b> node has — per-cluster mesh Phase 4 left
|
||||
/// it with no ConfigDb, so without this it denies every UserName token.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Grants are scoped to the node's own cluster (<see cref="ResolveClusterScope"/>); keys are
|
||||
/// not, because <c>ApiKey</c> carries no ClusterId — a key is a fleet-wide credential and it
|
||||
/// is the ACL rows that bind it to a cluster's nodes. A <see cref="ClusterFilterMode.Suppress"/>
|
||||
/// scope — the node's row is absent from a multi-cluster artifact — yields NO grants, matching
|
||||
/// the driver-spawn path's reading that an unlisted node was not deployed to.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Lenient like every other parser here: an empty or malformed blob yields empty lists. The
|
||||
/// caller must NOT treat that as "no keys are valid" and install it — see issue #485; the
|
||||
/// apply paths only reach this with bytes that already passed their own emptiness guard.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="blob">The deployment artifact blob to parse.</param>
|
||||
/// <param name="nodeId">The node's identity, used to resolve cluster scope for the grants.</param>
|
||||
/// <returns>The node's cluster (null when single-cluster), its keys, and its cluster's grants.</returns>
|
||||
public static DataPlaneAuthorityProjection ParseDataPlaneAuthority(ReadOnlySpan<byte> blob, string nodeId)
|
||||
{
|
||||
if (blob.IsEmpty) return DataPlaneAuthorityProjection.Empty;
|
||||
|
||||
var scope = ResolveClusterScope(blob, nodeId);
|
||||
|
||||
try
|
||||
{
|
||||
using var doc = JsonDocument.Parse(blob.ToArray());
|
||||
var root = doc.RootElement;
|
||||
|
||||
var keys = new List<ApiKey>();
|
||||
foreach (var el in EnumerateArray(root, "ApiKeys"))
|
||||
{
|
||||
var key = TryReadApiKey(el);
|
||||
if (key is not null) keys.Add(key);
|
||||
}
|
||||
|
||||
var grants = new List<NodeAcl>();
|
||||
foreach (var el in EnumerateArray(root, "NodeAcls"))
|
||||
{
|
||||
var acl = TryReadNodeAcl(el);
|
||||
if (acl is null) continue;
|
||||
if (scope.Mode == ClusterFilterMode.Suppress) continue;
|
||||
if (scope.Mode == ClusterFilterMode.ScopeTo
|
||||
&& !string.Equals(acl.ClusterId, scope.ClusterId, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
grants.Add(acl);
|
||||
}
|
||||
|
||||
return new DataPlaneAuthorityProjection(scope.ClusterId, keys, grants);
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return DataPlaneAuthorityProjection.Empty;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Reads one <c>ApiKeys</c> element; null when a field the credential check depends on is
|
||||
/// missing, so a half-written row can never authenticate.</summary>
|
||||
private static ApiKey? TryReadApiKey(JsonElement el)
|
||||
{
|
||||
var apiKeyId = ReadString(el, "ApiKeyId");
|
||||
var secretHash = ReadString(el, "SecretHash");
|
||||
var principal = ReadString(el, "Principal");
|
||||
if (string.IsNullOrWhiteSpace(apiKeyId)
|
||||
|| string.IsNullOrWhiteSpace(secretHash)
|
||||
|| string.IsNullOrWhiteSpace(principal))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return new ApiKey
|
||||
{
|
||||
ApiKeyId = apiKeyId,
|
||||
Name = ReadString(el, "Name") ?? apiKeyId,
|
||||
SecretHash = secretHash,
|
||||
Principal = principal,
|
||||
// A row whose Enabled flag did not survive serialisation is treated as REVOKED. The other
|
||||
// direction would resurrect a revoked key out of a malformed artifact.
|
||||
Enabled = ReadBool(el, "Enabled") ?? false,
|
||||
CreatedUtc = ReadDateTime(el, "CreatedUtc") ?? default,
|
||||
ExpiresUtc = ReadDateTime(el, "ExpiresUtc"),
|
||||
Notes = ReadNullableString(el, "Notes"),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Reads one <c>NodeAcls</c> element; null when identity, cluster or scope kind is unreadable.</summary>
|
||||
private static NodeAcl? TryReadNodeAcl(JsonElement el)
|
||||
{
|
||||
var nodeAclId = ReadString(el, "NodeAclId");
|
||||
var clusterId = ReadString(el, "ClusterId");
|
||||
var principal = ReadString(el, "Principal");
|
||||
if (string.IsNullOrWhiteSpace(nodeAclId)
|
||||
|| string.IsNullOrWhiteSpace(clusterId)
|
||||
|| string.IsNullOrWhiteSpace(principal))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
// ConfigComposer serialises enums as NUMBERS (its JsonSerializerOptions carries no
|
||||
// JsonStringEnumConverter), so these read as int. Keep the two sides in step — a string-enum
|
||||
// converter added to the composer would silently drop every grant here.
|
||||
var scopeKind = ReadInt32(el, "ScopeKind");
|
||||
if (scopeKind is null || !Enum.IsDefined(typeof(NodeAclScopeKind), scopeKind.Value)) return null;
|
||||
|
||||
return new NodeAcl
|
||||
{
|
||||
NodeAclId = nodeAclId,
|
||||
ClusterId = clusterId,
|
||||
Principal = principal,
|
||||
ScopeKind = (NodeAclScopeKind)scopeKind.Value,
|
||||
ScopeId = ReadNullableString(el, "ScopeId"),
|
||||
// An unreadable bitmask grants nothing rather than defaulting to something permissive.
|
||||
PermissionFlags = (NodePermissions)(ReadInt32(el, "PermissionFlags") ?? 0),
|
||||
Notes = ReadNullableString(el, "Notes"),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Reads a boolean property; null when absent or not a JSON boolean.</summary>
|
||||
private static bool? ReadBool(JsonElement el, string property) =>
|
||||
el.TryGetProperty(property, out var p)
|
||||
? p.ValueKind switch
|
||||
{
|
||||
JsonValueKind.True => true,
|
||||
JsonValueKind.False => false,
|
||||
_ => null,
|
||||
}
|
||||
: null;
|
||||
|
||||
/// <summary>Reads an int property; null when absent, not a number, or out of int range.</summary>
|
||||
private static int? ReadInt32(JsonElement el, string property) =>
|
||||
el.TryGetProperty(property, out var p) && p.ValueKind == JsonValueKind.Number
|
||||
&& p.TryGetInt32(out var v)
|
||||
? v
|
||||
: null;
|
||||
|
||||
/// <summary>Reads a UTC DateTime property; null when absent, JSON null, or unparseable.</summary>
|
||||
private static DateTime? ReadDateTime(JsonElement el, string property)
|
||||
{
|
||||
if (!el.TryGetProperty(property, out var p) || p.ValueKind != JsonValueKind.String) return null;
|
||||
if (!p.TryGetDateTime(out var v)) return null;
|
||||
|
||||
// Normalise to UTC, because every consumer compares against DateTime.UtcNow and an expiry that
|
||||
// silently shifted by the node's timezone is a key that outlives (or predeceases) its expiry.
|
||||
// Unspecified must be SpecifyKind, NOT ToUniversalTime: EF materialises datetime2 as Unspecified,
|
||||
// so the composer writes these UTC instants without an offset and ToUniversalTime would subtract
|
||||
// the node's local offset from a value that was already UTC.
|
||||
return v.Kind switch
|
||||
{
|
||||
DateTimeKind.Utc => v,
|
||||
DateTimeKind.Local => v.ToUniversalTime(),
|
||||
_ => DateTime.SpecifyKind(v, DateTimeKind.Utc),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Reads a string property, tolerating a wrong JSON type: a property that is present but
|
||||
/// NOT a JSON string (number / bool / object / array) yields null rather than throwing
|
||||
/// <see cref="InvalidOperationException"/> from <c>GetString()</c>. Keeps the artifact-decode lenient
|
||||
|
||||
@@ -9,6 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Observability;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Authorization;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Health;
|
||||
@@ -95,6 +96,10 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
private readonly NodeId? _localNode;
|
||||
private readonly IDbContextFactory<OtOpcUaConfigDbContext>? _dbFactory;
|
||||
private readonly AddressSpaceApplier? _applier;
|
||||
/// <summary>#520 stage 4 — receives the API keys + ACL grants carried by each applied artifact, so
|
||||
/// OPC UA session activation can authenticate. Null when nothing consumes it (tests, and any node
|
||||
/// that is not serving an OPC UA data plane).</summary>
|
||||
private readonly IDataPlaneAuthoritySink? _dataPlaneAuthority;
|
||||
private readonly IActorRef? _dbHealthProbe;
|
||||
/// <summary>Per-cluster mesh Phase 4: this is a driver-only node with NO ConfigDb (LocalDb is its
|
||||
/// config store). There is no DB to probe, so the ServiceLevel DB-reachability input is retired —
|
||||
@@ -166,7 +171,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
TimeSpan? staleWindow = null,
|
||||
TimeSpan? probeFreshnessWindow = null,
|
||||
TimeSpan? healthTickInterval = null,
|
||||
bool dbLess = false) =>
|
||||
bool dbLess = false,
|
||||
IDataPlaneAuthoritySink? dataPlaneAuthority = null) =>
|
||||
Akka.Actor.Props.Create(() => new OpcUaPublishActor(
|
||||
sink ?? NullOpcUaAddressSpaceSink.Instance,
|
||||
serviceLevel ?? NullServiceLevelPublisher.Instance,
|
||||
@@ -178,7 +184,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
staleWindow,
|
||||
probeFreshnessWindow,
|
||||
healthTickInterval,
|
||||
dbLess)).WithDispatcher(DispatcherId);
|
||||
dbLess,
|
||||
dataPlaneAuthority)).WithDispatcher(DispatcherId);
|
||||
|
||||
/// <summary>Test-only Props that omits the pinned-dispatcher requirement and skips the
|
||||
/// DPS subscribe so unit tests can spin up the actor on a vanilla TestKit cluster.</summary>
|
||||
@@ -214,7 +221,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
TimeSpan? staleWindow = null,
|
||||
TimeSpan? probeFreshnessWindow = null,
|
||||
TimeSpan? healthTickInterval = null,
|
||||
bool dbLess = false) =>
|
||||
bool dbLess = false,
|
||||
IDataPlaneAuthoritySink? dataPlaneAuthority = null) =>
|
||||
Akka.Actor.Props.Create(() => new OpcUaPublishActor(
|
||||
sink ?? NullOpcUaAddressSpaceSink.Instance,
|
||||
serviceLevel ?? NullServiceLevelPublisher.Instance,
|
||||
@@ -226,7 +234,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
staleWindow,
|
||||
probeFreshnessWindow,
|
||||
healthTickInterval,
|
||||
dbLess));
|
||||
dbLess,
|
||||
dataPlaneAuthority));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="OpcUaPublishActor"/> class.</summary>
|
||||
/// <param name="sink">The OPC UA address space sink.</param>
|
||||
@@ -259,9 +268,11 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
TimeSpan? staleWindow = null,
|
||||
TimeSpan? probeFreshnessWindow = null,
|
||||
TimeSpan? healthTickInterval = null,
|
||||
bool dbLess = false)
|
||||
bool dbLess = false,
|
||||
IDataPlaneAuthoritySink? dataPlaneAuthority = null)
|
||||
{
|
||||
_sink = sink;
|
||||
_dataPlaneAuthority = dataPlaneAuthority;
|
||||
_serviceLevel = serviceLevel;
|
||||
_subscribeRedundancyTopic = subscribeRedundancyTopic;
|
||||
_localNode = localNode;
|
||||
@@ -412,6 +423,14 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
return;
|
||||
}
|
||||
|
||||
// #520 stage 4 — install this artifact's API keys + ACL grants BEFORE the address-space work
|
||||
// below, and deliberately ahead of the `plan.IsEmpty` early-return: an artifact that revokes a
|
||||
// key or edits a grant without touching a single node produces an EMPTY address-space plan, so
|
||||
// feeding this after that return would leave a revoked key authenticating until some unrelated
|
||||
// structural change happened to redeploy. Authorization and topology change on their own
|
||||
// schedules.
|
||||
FeedDataPlaneAuthority(artifact);
|
||||
|
||||
var composition = _localNode is { } ln
|
||||
? DeploymentArtifact.ParseComposition(artifact, ln.Value,
|
||||
inconsistency => _log.Warning("OpcUaPublish {Node}: cross-cluster binding — {Message}", ln, inconsistency))
|
||||
@@ -491,6 +510,43 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Decode the artifact's data-plane authorization inputs (API keys + this cluster's ACL grants)
|
||||
/// and install them for OPC UA session activation (#520 stage 4). No-op when nothing consumes
|
||||
/// them.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Wrapped so a decode failure can never break a deploy — the same discipline as the historised-ref
|
||||
/// feed. The cost of that choice is explicit: the node keeps serving the PREVIOUS key set, which is
|
||||
/// the safe direction for availability but means a revocation carried by an artifact this node
|
||||
/// could not decode has not taken effect. It is logged at Error for exactly that reason.
|
||||
/// </remarks>
|
||||
private void FeedDataPlaneAuthority(byte[] artifact)
|
||||
{
|
||||
if (_dataPlaneAuthority is null) return;
|
||||
|
||||
try
|
||||
{
|
||||
var nodeId = _localNode?.Value ?? string.Empty;
|
||||
var projection = DeploymentArtifact.ParseDataPlaneAuthority(artifact, nodeId);
|
||||
// The revision hash is SHA-256 of the artifact bytes — the same derivation ConfigComposer uses,
|
||||
// which is what lets a node recognise a re-applied revision without the hash riding the wire.
|
||||
var revision = Convert.ToHexStringLower(System.Security.Cryptography.SHA256.HashData(artifact));
|
||||
var snapshot = _dataPlaneAuthority.Apply(
|
||||
projection.ClusterId, revision, projection.Keys, projection.Grants);
|
||||
|
||||
_log.Info(
|
||||
"OpcUaPublish: data-plane authority generation {Generation} installed (cluster={Cluster}, revision={Revision}, keys={Keys}, grants={Grants})",
|
||||
snapshot.GenerationId, projection.ClusterId ?? "(single-cluster)", revision[..12],
|
||||
projection.Keys.Count, projection.Grants.Count);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Error(ex,
|
||||
"OpcUaPublish: failed to install data-plane authority from the applied artifact — the node keeps the PREVIOUS keys and grants, so any revocation in this deployment has NOT taken effect");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Read a specific deployment's artifact blob from ConfigDb (the one just applied,
|
||||
/// which may not be Sealed yet). Empty array on any failure — <see cref="HandleRebuild"/> treats that as
|
||||
/// "no answer" and abandons the rebuild, leaving the served address space untouched (issue #485).</summary>
|
||||
|
||||
@@ -17,6 +17,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Authorization;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
@@ -467,6 +468,11 @@ public static class ServiceCollectionExtensions
|
||||
loggerFactory.CreateLogger<AddressSpaceApplier>(),
|
||||
provisioning: provisioning,
|
||||
historizedSubscriptions: historizedSubscriptions);
|
||||
// #520 stage 4 — the sink that receives each applied artifact's API keys + ACL grants, so OPC
|
||||
// UA session activation can authenticate. Optional: GetService (not GetRequiredService), because
|
||||
// a node with no driver role never registers it and test harnesses build this graph without it;
|
||||
// the actor treats null as "nothing consumes authorization inputs" and skips the decode.
|
||||
var dataPlaneAuthority = resolver.GetService<IDataPlaneAuthoritySink>();
|
||||
var publishActor = system.ActorOf(
|
||||
OpcUaPublishActor.Props(
|
||||
sink: addressSpaceSink,
|
||||
@@ -475,7 +481,8 @@ public static class ServiceCollectionExtensions
|
||||
dbFactory: dbFactory,
|
||||
applier: applier,
|
||||
dbHealthProbe: dbHealth,
|
||||
dbLess: dbFactory is null),
|
||||
dbLess: dbFactory is null,
|
||||
dataPlaneAuthority: dataPlaneAuthority),
|
||||
OpcUaPublishActorName);
|
||||
registry.Register<OpcUaPublishActorKey>(publishActor);
|
||||
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Cluster\ZB.MOM.WW.OtOpcUa.Cluster.csproj"/>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Configuration\ZB.MOM.WW.OtOpcUa.Configuration.csproj"/>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.OpcUaServer\ZB.MOM.WW.OtOpcUa.OpcUaServer.csproj"/>
|
||||
<!-- #520 stage 4: the artifact-apply path feeds IDataPlaneAuthoritySink. Core already arrives
|
||||
transitively via OpcUaServer, but a transitive reference is not a stated dependency — make it
|
||||
explicit so removing it from OpcUaServer does not silently break this one. -->
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core\ZB.MOM.WW.OtOpcUa.Core.csproj"/>
|
||||
<!--
|
||||
Concrete OtOpcUa.Driver.* assemblies are loaded reflectively at runtime (Phase 6,
|
||||
Task 41+) and intentionally NOT project-referenced here. The driver contracts live in
|
||||
|
||||
Reference in New Issue
Block a user