mxaccessgw/docs/Authentication.md

Gateway Authentication

The gateway authenticates inbound gRPC callers with API keys: a bearer token is parsed, its secret is hashed with a peppered HMAC and compared in constant time against a stored hash, and administrative and verification events are recorded to an audit trail.

The peppered-HMAC pipeline itself — token parsing, secret generation, hashing, constant-time compare, the SQLite schema, the key store, the verifier, and schema migration — lives in the shared ZB.MOM.WW.Auth.ApiKeys package, of which this gateway is the donor. The gateway does not reimplement or fork those types; it binds the library through AddZbApiKeyAuth and layers gateway-specific concerns on top: constraint enforcement, the gRPC authorization interceptor, hot-path decorators, the admin CLI, the dashboard, and a canonical audit store that supersedes the library's own audit table. This document describes the consumer side — the token format, the options the gateway binds, the pieces it adds, and where the library boundary sits. For the library internals (the concrete ApiKeyVerifier, the SQLite stores, the schema and migrator), read the ZB.MOM.WW.Auth.ApiKeys sources; they are not duplicated in this repository.

Token Format

API keys travel in the HTTP Authorization header as a bearer token shaped mxgw_<keyId>_<secret>. The mxgw_ prefix scopes parsing to gateway tokens, the <keyId> segment is the public identifier used for lookup, and <secret> is the high-entropy portion verified against a stored hash. The prefix and the pepper configuration key the gateway pins are constants on AuthStoreServiceCollectionExtensions (TokenPrefix = "mxgw", PepperSecretName = "MxGateway:ApiKeyPepper"); they are supplied to the library at registration so the library's parser and pepper provider use the gateway's contract. The library parser rejects a malformed token before any database round-trip, and only a well-formed mxgw_<keyId>_<secret> token reaches the store lookup.

Secrets And Peppered Hashing

New secret material is high-entropy: the library generates 32 random bytes and encodes them URL-safe base64 (no padding) so a secret embeds in a header without escaping. The gateway never persists a plaintext secret — only its hash.

Secrets are hashed with HMAC-SHA256 keyed by a server-side pepper. The pepper lives outside the database and is resolved from configuration under the MxGateway:ApiKeyPepper key (the library's pepper provider reads it). Keeping the pepper out of the SQLite file means an attacker who exfiltrates only the database holds the hashes but lacks the keying material to brute-force candidate secrets, even if the hash algorithm is known.

When the pepper is not configured, the library surfaces the failure as an InvalidOperationException whose message reports the pepper is unavailable rather than persisting a key with an unkeyed hash. The dashboard management path (DashboardApiKeyManagementService) catches that condition and returns the friendly "API key pepper is not configured." result instead of faulting the Blazor circuit; it currently matches on the message text, so a library wording change would need to be reflected there (a typed pepper-unavailable exception is a pending library improvement).

Verification

The gateway consumes the library's IApiKeyVerifier from GatewayGrpcAuthorizationInterceptor. The verifier's flow is:

1. Parse the Authorization header into the key id and presented secret.
2. Look up the stored key record by key id.
3. Reject a revoked record, and reject an expired record whose ExpiresUtc is in the past. Expiry is opt-in — keys created without an expiry never expire; an expired key fails opaquely, indistinguishable to the client from any other auth failure.
4. Hash the presented secret with the configured pepper.
5. Compare hashes in constant time to avoid a timing oracle.
6. Stamp a LastUsedUtc timestamp and return a shared ApiKeyIdentity carrying the key id, key prefix, display name, scopes, and the opaque constraints JSON.

A verification failure is opaque to the client: the interceptor returns Unauthenticated/PermissionDenied without disclosing which check failed, while the failure detail is available server-side for audit.

GatewayApiKeyIdentityMapper.ToGatewayIdentity maps the library's shared ApiKeyIdentity onto the gateway's own ApiKeyIdentity (Security/Authentication/ApiKeyIdentity.cs), which exposes the deserialized ApiKeyConstraints — parsed from the opaque constraints JSON via ApiKeyConstraintSerializer — that the downstream ConstraintEnforcer and the request-identity accessor enforce. The gateway identity exposes only non-secret fields (KeyId, KeyPrefix, DisplayName, Scopes, Constraints).

Hot-path caching and last-used coalescing

Left unmediated, every authenticated gRPC call costs a SQLite read plus a last_used_utc write (the library verifier couples MarkUsed into VerifyAsync), which makes the auth store the throughput ceiling on the bulk-read workload. The gateway layers two decorators over the shared library's registrations (in AuthStoreServiceCollectionExtensions) — it does not edit the library:

• CachingApiKeyVerifier wraps the library IApiKeyVerifier with an IMemoryCache entry per successful verification, keyed on a SHA-256 hash of the presented token (never the plaintext secret). A cache hit within MxGateway:Security:ApiKeyVerificationCacheSeconds (default 15 s) returns the cached result without touching the store, so both the read and the coupled write are skipped. Only successes are cached; failures always reach the inner verifier. On a gateway-initiated revoke/rotate/delete the dashboard admin service calls IApiKeyCacheInvalidator.Invalidate(keyId), evicting the cached entry immediately. Invalidate bumps a per-key generation counter before it evicts, and VerifyAsync snapshots that generation before the inner verify and re-checks it after writing the cache entry (set-then-recheck); a revoke that lands while a verification is still in flight in the inner library therefore discards that verification's repopulation instead of re-caching the just-revoked identity for a full TTL (SEC-34). The short TTL remains the backstop for two bounded-staleness windows it cannot close directly: (1) out-of-band mutations (a direct DB edit, or a revoke run by the separate apikey CLI process, whose in-memory cache is not the running gateway's cache); and (2) a key whose ExpiresUtc passes while cached keeps authenticating until the entry's TTL elapses — expiry is enforced by the inner library verifier, which a cache hit never reaches, and the verification identity the library returns carries no expiry timestamp, so the cache cannot cap an entry at the key's expiry (capping it needs the donor library to surface expiry on the verification identity). The default 15 s TTL bounds both windows.
• CoalescingMarkApiKeyStore wraps the library IApiKeyStore and forwards at most one MarkUsed write per key per MxGateway:Security:ApiKeyLastUsedCoalesceSeconds (default 60 s), so even under a cache miss the last_used_utc write is bounded to roughly one per key per minute rather than one per RPC. last_used_utc is a coarse staleness hint, not an audit record (audit rows are written separately), so bounded staleness of up to one window is acceptable.

GatewayApiKeyIdentityMapper additionally memoizes the constraints-JSON deserialization by blob, so the per-call parse on the mapped identity collapses to a dictionary lookup. Both windows are configurable and may be set to 0 to disable the respective mechanism; see GatewayConfiguration.

Failures are never cached — a wrong secret always reaches the store — so the failure path is shielded by ApiKeyFailureLimiter instead, consulted before VerifyAsync runs. It counts failures over one sliding MxGateway:Security:ApiKeyFailureWindowSeconds window in two layers: a composite (transport peer, key id) partition capped at ApiKeyFailureLimit, and a per-key-id aggregate across all peers capped at ApiKeyFailureAggregateLimit. The key id never partitions on its own — it is public, so an attacker-supplied one would otherwise let any peer throttle a key it does not hold — and it joins the partition only when the presented token is validly shaped (mxgw_<keyId>_<secret>, key id at most 64 characters), with at most 32 key-id partitions per address before the overflow collapses onto that address's fallback partition. An over-limit state admits one probe per ApiKeyFailureProbeIntervalSeconds through to the real verifier and refuses everything else with ResourceExhausted before the store read, so a legitimate holder presenting the correct secret always reaches the constant-time compare and resets both layers; the counter's LRU eviction (ApiKeyFailureTrackedPeers) prefers expired windows and will not drop an over-limit partition below a 2x overshoot ceiling, so the memory bound cannot be turned into a way to clear an active block. See Authorization for the enforcement path.

Storage

API-key state lives in a dedicated SQLite database owned by the shared library. SQLite is sufficient because credential volume is small, the gateway runs as a single process, and the file is straightforward to back up and rotate independently of the main application data.

The database path is GatewayOptions.Authentication.SqlitePath. Its code default is derived from Environment.GetFolderPath(SpecialFolder.CommonApplicationData) (C:\ProgramData\MxGateway\gateway-auth.db on Windows, /usr/share/MxGateway/gateway-auth.db or the container equivalent elsewhere) so the credential store is never written relative to the launch working directory on a non-Windows host. appsettings.json no longer ships an explicit path (SEC-33): the removed Windows literal matched the Windows code default and, being non-rooted on a Unix host, would have resolved against the CWD there; deployed hosts override it through the NSSM environment (MxGateway__Authentication__SqlitePath). GatewayOptionsValidator rejects a SqlitePath that is not rooted on the host running the gateway (Path.IsPathRooted, current OS) — a relative filename or a foreign-platform literal fails fast at startup rather than scattering the store by launch CWD (SEC-01, SEC-33).

The library owns the SQLite schema and connection factory. The api_keys table carries the key id, key prefix, secret-hash blob, display name, serialized scopes, optional serialized constraints, and the created_utc, last_used_utc, revoked_utc, and expires_utc timestamps. Because the schema, stores, and migrator belong to ZB.MOM.WW.Auth.ApiKeys, this document does not restate their column readers or SQL; consult the library for that detail.

Audit trail

The library emits its own API-key audit entries (from the admin verbs — create, revoke, rotate, init-db, and constraint denials), but the gateway overrides the library's IApiKeyAuditStore registration with CanonicalForwardingApiKeyAuditStore. That adapter canonicalizes every library-emitted ApiKeyAuditEntry onto the gateway's AuditEvent shape and routes it through IAuditWriter (CanonicalAuditWriter) into SqliteCanonicalAuditStore, which persists to a single audit_event table (columns event_id, occurred_at_utc, actor, action, outcome, category, target, source_node, correlation_id, details_json). Reads for the dashboard "recent audit" view go back through the same adapter, which maps audit_event rows back to ApiKeyAuditEntry values so the existing view keeps working unchanged.

Consequently the library's own api_key_audit table is left in place but unused after adoption — nothing writes to it once the override is registered. The canonical audit_event table is the single durable record of both API-key administrative actions and the dashboard's own audit vocabulary (dashboard-create-key, dashboard-rotate-key, dashboard-revoke-key, dashboard-delete-key, and the session Close/Kill actions). This is why any prose that describes credential audits as landing in api_key_audit is stale: the canonical store is audit_event.

Registration

AuthStoreServiceCollectionExtensions.AddSqliteAuthStore(IConfiguration) wires the whole subsystem. It does not register the library types directly — it delegates to the shared provider and then layers the gateway concerns:

public static IServiceCollection AddSqliteAuthStore(
    this IServiceCollection services,
    IConfiguration configuration)
{
    // Pin the gateway's token prefix ("mxgw") and pepper key ("MxGateway:ApiKeyPepper")
    // as fallback defaults UNDER the supplied configuration, then register the shared
    // provider: it binds ApiKeyOptions from MxGateway:Authentication and wires the SQLite
    // stores, the configuration-backed pepper provider, the verifier, the migrator, and
    // the migration hosted service.
    services.AddZbApiKeyAuth(effectiveConfig, AuthenticationSectionPath);

    // SEC-08 hot-path decorators layered over the library registrations.
    services.AddMemoryCache();
    // CoalescingMarkApiKeyStore decorates IApiKeyStore; CachingApiKeyVerifier decorates
    // IApiKeyVerifier and also serves as IApiKeyCacheInvalidator.

    // Canonical audit: override the library's IApiKeyAuditStore so every API-key audit
    // event is forwarded through IAuditWriter into the audit_event table.
    services.AddSingleton<IApiKeyAuditStore, CanonicalForwardingApiKeyAuditStore>();

    // The shared admin command set (ApiKeyAdminCommands) and the gateway CLI runner.
    services.AddSingleton<ApiKeyAdminCliRunner>();

    return services;
}

The decorators wrap the library's last registration for each interface rather than replacing the library types, preserving singleton semantics; the audit override is registered after AddZbApiKeyAuth so it wins as the resolved IApiKeyAuditStore.

Admin CLI

ApiKeyAdminCommandLineParser.Parse recognises a leading apikey argument and dispatches to one of the subcommands declared by ApiKeyAdminCommandKind. Each parsed invocation produces an ApiKeyAdminCommand (or an ApiKeyAdminParseResult carrying an error). ApiKeyAdminCliRunner then runs the migrator, invokes the shared ApiKeyAdminCommands verb, and writes text or JSON output via ApiKeyAdminOutput. The returned ApiKeyAdminListedKey projection deliberately omits the secret hash so listing a database never surfaces hash material.

The supported subcommands match ApiKeyAdminCommandKind exactly:

Subcommand	Required options	Behaviour
init-db	none	Runs the migrator and records an audit entry.
create-key	--key-id, --display-name	Generates a new secret, stores its peppered hash and optional constraints, and prints the assembled mxgw_<keyId>_<secret> token. Optional --expires sets an expiry (absolute ISO-8601 UTC, or a relative <N>d/<N>h from now); omit it for a non-expiring key.
list-keys	none	Lists every stored key with its scopes, constraints, revocation state, and expiry (active/expired/revoked).
revoke-key	--key-id	Marks the key revoked if it is currently active.
rotate-key	--key-id	Replaces the secret hash and prints the new token.

Examples:

mxgateway apikey init-db
mxgateway apikey create-key --key-id ops.alice --display-name "Alice (ops)" --scopes read,write
mxgateway apikey create-key --key-id area1.reader --display-name "Area 1 reader" --scopes invoke:read,metadata:read --read-subtree "Area1/*" --browse-subtree "Area1/*"
mxgateway apikey create-key --key-id ops.temp --display-name "Temp contractor" --scopes invoke:read --expires 90d
mxgateway apikey create-key --key-id team-a.svc --display-name "Team A service" --scopes session:open,invoke:read --dashboard-tags team-a
mxgateway apikey create-key --key-id ops.audit --display-name "Audit window" --scopes metadata:read --expires 2027-01-01T00:00:00Z
mxgateway apikey list-keys --json
mxgateway apikey revoke-key --key-id ops.alice
mxgateway apikey rotate-key --key-id ops.alice

Constraint flags are optional. --read-subtree, --write-subtree, --read-tag-glob, --write-tag-glob, and --browse-subtree are repeatable. --max-write-classification accepts one integer. --read-alarm-only and --read-historized-only are boolean flags. --dashboard-tags takes a comma-separated list (--dashboard-tags team-a,team-b) and is repeatable; its segments are trimmed and de-duplicated ordinal-ignore-case, and an empty segment is rejected rather than dropped so a stray comma cannot silently persist a grant the operator did not write. It is the only constraint flag that splits its value on commas (--scopes, which is not a constraint, is the other flag that does): the repeatable subtree and glob flags each take exactly one value per occurrence, so --read-subtree "Area1/*,Area2/*" is a single literal pattern containing a comma, not two patterns. Repeat the flag instead. Existing rows with null constraints remain fully unconstrained after migration; rows written before --dashboard-tags existed deserialize as untagged, unchanged in every other respect.

--dashboard-tags is not a data-access constraint — it only labels the key for dashboard event visibility, and sessions the key opens inherit it. See Authorization.

list-keys prints the tags as a trailing tab-separated column (- when untagged); the values are operator-chosen labels, not key material.

Key ids are restricted by the parser to ASCII letters, digits, periods, and hyphens so they remain safe to embed in the token format and in URL paths used by administrative tooling.

The CLI is not the only management surface: the dashboard API Keys page creates, rotates, revokes, and deletes (revoked-only) keys through the same shared admin command set. Every destructive dashboard action is gated by a confirmation dialog and emits its own audit event (dashboard-create-key, dashboard-rotate-key, dashboard-revoke-key, dashboard-delete-key) into the canonical audit_event store. The page also surfaces expiry: each row shows an Expires column (Never when unset) and a status badge that reads Expired, Expiring (within seven days), Revoked, or Active. This staleness surfacing is display-only; expiry is set at creation time via apikey create-key --expires, not from the dashboard. See Gateway Dashboard Design.

Related Documentation

• Gateway Configuration
• Authorization
• Gateway Dashboard Design
• Diagnostics