mxaccessgw/docs/Authentication.md

# Gateway Authentication

The gateway authentication subsystem verifies inbound API key credentials against a SQLite-backed key store, hashes secrets with a configurable pepper, and records administrative and verification events to an audit trail.

## Token Format

API keys travel in the HTTP `Authorization` header as a bearer token shaped `mxgw_<keyId>_<secret>`. The `mxgw_` prefix scopes parsing to gateway tokens, the `<keyId>` segment is the public identifier used for lookup, and `<secret>` is the high-entropy portion that the gateway verifies against a stored hash.

`ApiKeyParser` enforces the format and rejects malformed tokens before any database round-trip:

```csharp
public bool TryParseAuthorizationHeader(string? authorizationHeader, out ParsedApiKey? apiKey)
{
    apiKey = null;

    if (string.IsNullOrWhiteSpace(authorizationHeader)
        || !authorizationHeader.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
    {
        return false;
    }

    string token = authorizationHeader[BearerPrefix.Length..].Trim();

    if (!token.StartsWith(TokenPrefix, StringComparison.OrdinalIgnoreCase))
    {
        return false;
    }
```

A successful parse produces a `ParsedApiKey(KeyId, Secret)` record. The `IApiKeyParser` interface exists so verification consumers can be tested without depending on header-format details.

## Parsing and Secrets

### Secret generation

`ApiKeySecretGenerator.Generate()` is the single source of new secret material. It uses 32 bytes from `RandomNumberGenerator.Fill` and encodes with URL-safe base64 (no padding) so secrets can be embedded in headers without escaping:

```csharp
public static string Generate()
{
    Span<byte> bytes = stackalloc byte[32];
    RandomNumberGenerator.Fill(bytes);

    return Convert.ToBase64String(bytes)
        .TrimEnd('=')
        .Replace('+', '-')
        .Replace('/', '_');
}
```

### Peppered hashing

`ApiKeySecretHasher` (registered behind `IApiKeySecretHasher`) hashes secrets with `HMACSHA256` keyed by a server-side pepper. The pepper lives outside the database and is resolved by `IConfiguration` lookup against the configured `PepperSecretName`:

```csharp
public byte[] HashSecret(string secret)
{
    string pepper = GetPepper();
    byte[] pepperBytes = Encoding.UTF8.GetBytes(pepper);
    byte[] secretBytes = Encoding.UTF8.GetBytes(secret);

    using HMACSHA256 hmac = new(pepperBytes);

    return hmac.ComputeHash(secretBytes);
}
```

The pepper is intentionally not stored alongside the hash: an attacker who exfiltrates only the SQLite file holds the hashes but lacks the keying material to brute-force candidate secrets, even if the stored hash algorithm and salt scheme are known. If the pepper is missing the hasher throws `ApiKeyPepperUnavailableException`, which the verifier converts to a distinct failure code rather than treating it as a credential mismatch.

## Verification

`ApiKeyVerifier` (`IApiKeyVerifier`) implements the verification flow:

1. Parse the `Authorization` header into a `ParsedApiKey`.
2. Look up the `ApiKeyRecord` by `KeyId` through `IApiKeyStore.FindByKeyIdAsync`.
3. Reject revoked records (`RevokedUtc is not null`).
4. Hash the presented secret with the configured pepper.
5. Compare hashes with `CryptographicOperations.FixedTimeEquals` to avoid timing oracles.
6. Record a `LastUsedUtc` timestamp via `MarkKeyUsedAsync` and return an `ApiKeyIdentity`.

```csharp
if (!CryptographicOperations.FixedTimeEquals(presentedHash, storedKey.SecretHash))
{
    return ApiKeyVerificationResult.Fail(ApiKeyVerificationFailure.SecretMismatch);
}

await keyStore.MarkKeyUsedAsync(storedKey.KeyId, DateTimeOffset.UtcNow, cancellationToken)
    .ConfigureAwait(false);

return ApiKeyVerificationResult.Success(new ApiKeyIdentity(
    KeyId: storedKey.KeyId,
    KeyPrefix: storedKey.KeyPrefix,
    DisplayName: storedKey.DisplayName,
    Scopes: storedKey.Scopes,
    Constraints: storedKey.Constraints));
```

`ApiKeyVerificationResult` carries either an `ApiKeyIdentity` or a discriminated `ApiKeyVerificationFailure` value. The failure enum distinguishes parse errors, missing pepper, missing or revoked keys, and secret mismatch so the calling middleware can emit precise audit detail without leaking which check failed to the client.

`ApiKeyIdentity` exposes only non-secret fields (`KeyId`, `KeyPrefix`,
`DisplayName`, `Scopes`, and `Constraints`) and is the type downstream
authorization code consumes.

### Hot-path caching and last-used coalescing

Left unmediated, every authenticated gRPC call costs a SQLite read plus a
`last_used_utc` **write** (the library verifier couples `MarkKeyUsed` into
`VerifyAsync`), which makes the auth store the throughput ceiling on the
bulk-read workload. The gateway layers two decorators over the shared library's
registrations (in `AuthStoreServiceCollectionExtensions`) — it does not edit the
library:

- **`CachingApiKeyVerifier`** wraps `IApiKeyVerifier` with an `IMemoryCache`
  entry per successful verification, keyed on a SHA-256 hash of the presented
  token (never the plaintext secret). A cache hit within
  `MxGateway:Security:ApiKeyVerificationCacheSeconds` (default 15 s) returns the
  cached result without touching the store, so both the read and the coupled
  write are skipped. Only successes are cached; failures always reach the inner
  verifier. On a gateway-initiated revoke/rotate/delete the dashboard admin
  service calls `IApiKeyCacheInvalidator.Invalidate(keyId)`, evicting the cached
  entry immediately. The short TTL is the backstop for out-of-band mutations
  (a direct DB edit, or a revoke run by the separate `apikey` CLI process, whose
  in-memory cache is not the running gateway's cache).
- **`CoalescingMarkApiKeyStore`** wraps `IApiKeyStore` and forwards at most one
  `MarkUsed` write per key per `MxGateway:Security:ApiKeyLastUsedCoalesceSeconds`
  (default 60 s), so even under a cache miss the `last_used_utc` write is bounded
  to roughly one per key per minute rather than one per RPC. `last_used_utc` is a
  coarse staleness hint, not an audit record (audit rows are written separately),
  so bounded staleness of up to one window is acceptable.

`GatewayApiKeyIdentityMapper` additionally memoizes the constraints-JSON
deserialization by blob, so the per-call parse on the mapped identity collapses to
a dictionary lookup. Both windows are configurable and may be set to `0` to
disable the respective mechanism; see [GatewayConfiguration](./GatewayConfiguration.md).

## Storage

The gateway keeps API key state in a dedicated SQLite database. SQLite is sufficient because credential volume is small, the gateway runs as a single process, and the file is straightforward to back up and rotate independently of the main application data.

The database path is `GatewayOptions.Authentication.SqlitePath`. Its code default is derived from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` (`C:\ProgramData\MxGateway\gateway-auth.db` on Windows, `/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) so the credential store is never written relative to the launch working directory on a non-Windows host. The production hosts pin the explicit Windows path in `appsettings.json`. `GatewayOptionsValidator` rejects a non-rooted (relative) `SqlitePath` so a bad override fails fast at startup rather than scattering the store by launch CWD (SEC-01).

### Connection factory

`AuthSqliteConnectionFactory` reads `GatewayOptions.Authentication.SqlitePath`, ensures the parent directory exists, and builds a connection string in `ReadWriteCreate` mode so first-run installations can create the file without manual provisioning. Connection pooling is enabled and the connection string carries a non-zero `DefaultTimeout`:

```csharp
SqliteConnectionStringBuilder builder = new()
{
    DataSource = sqlitePath,
    Mode = SqliteOpenMode.ReadWriteCreate,
    Pooling = true,
    DefaultTimeout = (int)BusyTimeout.TotalSeconds,
};
```

Every store opens its connection through `OpenConnectionAsync`, which opens the connection and then applies `PRAGMA journal_mode=WAL` and `PRAGMA busy_timeout`. WAL is a persistent database-level setting so re-applying it per connection is a cheap no-op; `busy_timeout` is per-connection state. Because `MarkKeyUsedAsync` runs on every authenticated request and `SqliteApiKeyAuditStore` appends on every denial, this lets concurrent readers and writers retry briefly instead of surfacing `SQLITE_BUSY` as a hard failure on the request path.

### Schema

`SqliteAuthSchema` declares table names and the current schema version as constants. Three tables are involved:

- `api_keys` stores `key_id`, `key_prefix`, the `secret_hash` blob,
  `display_name`, serialized `scopes`, optional serialized `constraints`, and
  the `created_utc`, `last_used_utc`, and `revoked_utc` timestamps.
- `api_key_audit` is an append-only log keyed by an autoincrement `audit_id` with `key_id`, `event_type`, `remote_address`, `created_utc`, and `details` columns.
- `schema_version` carries a single row whose `version` column is matched against `SqliteAuthSchema.CurrentVersion`.

### Read paths

`SqliteApiKeyStore` (`IApiKeyStore`) handles the two reads needed at request time: `FindByKeyIdAsync` returns any record (so revoked keys can be reported distinctly) and `FindActiveByKeyIdAsync` filters to non-revoked rows. `MarkKeyUsedAsync` updates `last_used_utc` only for non-revoked rows so a freshly revoked key cannot have its timestamp refreshed by a racing verification.

`ApiKeyRecord` is the in-memory projection. `ApiKeyRecordReader.Read` is shared by every read path so column ordering is defined in one place:

```csharp
public static ApiKeyRecord Read(SqliteDataReader reader)
{
    return new ApiKeyRecord(
        KeyId: reader.GetString(0),
        KeyPrefix: reader.GetString(1),
        SecretHash: (byte[])reader["secret_hash"],
        DisplayName: reader.GetString(3),
        Scopes: ApiKeyScopeSerializer.Deserialize(reader.GetString(4)),
        Constraints: ApiKeyConstraintSerializer.Deserialize(reader.IsDBNull(5) ? null : reader.GetString(5)),
        CreatedUtc: DateTimeOffset.Parse(reader.GetString(6), System.Globalization.CultureInfo.InvariantCulture),
        LastUsedUtc: ReadNullableDateTimeOffset(reader, 7),
        RevokedUtc: ReadNullableDateTimeOffset(reader, 8));
}
```

### Write paths

`SqliteApiKeyAdminStore` (`IApiKeyAdminStore`) implements administrative mutations: `CreateAsync` accepts an `ApiKeyCreateRequest`, `RevokeAsync` sets `revoked_utc` only when not already revoked, `RotateAsync` replaces `secret_hash`, clears `last_used_utc`, and clears `revoked_utc` so a rotated key is immediately usable, and `DeleteAsync` permanently removes a row but only when `revoked_utc IS NOT NULL` — active keys are untouched (returns false) so the revoke event lands in the audit log before the row disappears.

Because `RotateAsync` clears `revoked_utc`, rotating a previously revoked key reactivates it. The dashboard API Keys page therefore offers the Rotate (and Revoke) actions only for keys whose status is `Active`; revoked keys instead show a Delete action that calls `DeleteAsync`, so an operator can permanently remove a revoked row without ever risking un-revocation as a side effect of a rotation.

### Audit trail

`SqliteApiKeyAuditStore` (`IApiKeyAuditStore`) appends `ApiKeyAuditEntry` values to the `api_key_audit` table and stamps each row with a UTC timestamp inside the store rather than trusting the caller. `ListRecentAsync` returns the most recent rows ordered by `audit_id` descending and projects them into `ApiKeyAuditRecord`. Rows are kept even after the referenced key is revoked because the audit history is the durable record of administrative action; the `key_id` column is nullable to accommodate non-key-scoped events such as `init-db`.

## Migration

Schema bring-up is centralised behind `IAuthStoreMigrator`. `SqliteAuthStoreMigrator` executes the migration inside a single transaction so a partial failure leaves the database untouched, refuses to start when the on-disk schema version is newer than the binary supports, and idempotently creates the v1 schema:

```csharp
if (existingVersion > SqliteAuthSchema.CurrentVersion)
{
    throw new AuthStoreMigrationException(
        $"Auth database schema version {existingVersion} is newer than supported version {SqliteAuthSchema.CurrentVersion}.");
}

await ApplyVersionOneAsync(connection, transaction, cancellationToken).ConfigureAwait(false);

await transaction.CommitAsync(cancellationToken).ConfigureAwait(false);
```

`AuthStoreMigrationHostedService` runs the migrator at startup, but only when API-key authentication is enabled and `RunMigrationsOnStartup` is true. Operators who manage schema out-of-band can disable the hosted run and use the admin CLI's `init-db` command instead.

`AuthStoreMigrationException` is a sealed `InvalidOperationException` so it can be caught precisely without swallowing unrelated failures.

## Admin CLI

`ApiKeyAdminCommandLineParser.Parse` recognises a leading `apikey` argument and dispatches to one of the subcommands declared by `ApiKeyAdminCommandKind`. Each parsed invocation produces an `ApiKeyAdminCommand` (or an `ApiKeyAdminParseResult` carrying an error). `ApiKeyAdminCliRunner` then executes the command, runs the migrator first, calls the relevant store method, appends an audit row, and writes either text or JSON output via `ApiKeyAdminOutput`. The returned `ApiKeyAdminListedKey` projection deliberately omits the `secret_hash` so listing a database does not surface hash material.

The supported subcommands match `ApiKeyAdminCommandKind` exactly:

| Subcommand | Required options | Behaviour |
|------------|------------------|-----------|
| `init-db` | none | Runs the migrator and records an audit entry. |
| `create-key` | `--key-id`, `--display-name` | Generates a new secret, stores its peppered hash and optional constraints, and prints the assembled `mxgw_<keyId>_<secret>` token. |
| `list-keys` | none | Lists every stored key with its scopes, constraints, and revocation state. |
| `revoke-key` | `--key-id` | Sets `revoked_utc` if the key is currently active. |
| `rotate-key` | `--key-id` | Replaces the secret hash and prints the new token. |

Examples:

```bash
mxgateway apikey init-db
mxgateway apikey create-key --key-id ops.alice --display-name "Alice (ops)" --scopes read,write
mxgateway apikey create-key --key-id area1.reader --display-name "Area 1 reader" --scopes invoke:read,metadata:read --read-subtree "Area1/*" --browse-subtree "Area1/*"
mxgateway apikey list-keys --json
mxgateway apikey revoke-key --key-id ops.alice
mxgateway apikey rotate-key --key-id ops.alice
```

Constraint flags are optional. `--read-subtree`, `--write-subtree`,
`--read-tag-glob`, `--write-tag-glob`, and `--browse-subtree` are repeatable.
`--max-write-classification` accepts one integer. `--read-alarm-only` and
`--read-historized-only` are boolean flags. Existing rows with null
constraints remain fully unconstrained after migration.

Key ids are restricted by the parser to ASCII letters, digits, periods, and hyphens so they remain safe to embed in the token format and in URL paths used by administrative tooling.

The CLI is not the only management surface: the dashboard API Keys page
creates, rotates, revokes, and deletes (revoked-only) keys through the same
`IApiKeyAdminStore`. Every destructive dashboard action is gated by a
confirmation dialog and emits its own audit event
(`dashboard-create-key`, `dashboard-rotate-key`, `dashboard-revoke-key`,
`dashboard-delete-key`). See
[Gateway Dashboard Design](./GatewayDashboardDesign.md#api-keys-page).

## Scope Serialization

Scopes are persisted as a single TEXT column rather than a join table because the set is small, never queried by membership at the database level, and changes atomically with the owning row. `ApiKeyScopeSerializer.Serialize` writes a JSON array sorted with `StringComparer.Ordinal` so equivalent scope sets produce byte-identical column values, which makes audit diffing and database comparisons deterministic:

```csharp
public static string Serialize(IReadOnlySet<string> scopes)
{
    return JsonSerializer.Serialize(scopes.Order(StringComparer.Ordinal));
}

public static IReadOnlySet<string> Deserialize(string value)
{
    if (string.IsNullOrWhiteSpace(value))
    {
        return new HashSet<string>(StringComparer.Ordinal);
    }

    string[]? scopes = JsonSerializer.Deserialize<string[]>(value);

    return new HashSet<string>(scopes ?? [], StringComparer.Ordinal);
}
```

`Deserialize` tolerates an empty column by returning an empty set so older rows or hand-edited records do not crash the verifier.

## Registration

`AuthStoreServiceCollectionExtensions.AddSqliteAuthStore` wires every service in this subsystem as a singleton and registers the migration hosted service:

```csharp
public static IServiceCollection AddSqliteAuthStore(this IServiceCollection services)
{
    services.AddSingleton<IApiKeyParser, ApiKeyParser>();
    services.AddSingleton<IApiKeySecretHasher, ApiKeySecretHasher>();
    services.AddSingleton<IApiKeyVerifier, ApiKeyVerifier>();
    services.AddSingleton<ApiKeyAdminCliRunner>();
    services.AddSingleton<AuthSqliteConnectionFactory>();
    services.AddSingleton<IAuthStoreMigrator, SqliteAuthStoreMigrator>();
    services.AddSingleton<IApiKeyStore, SqliteApiKeyStore>();
    services.AddSingleton<IApiKeyAdminStore, SqliteApiKeyAdminStore>();
    services.AddSingleton<IApiKeyAuditStore, SqliteApiKeyAuditStore>();
    services.AddHostedService<AuthStoreMigrationHostedService>();

    return services;
}
```

Singletons are safe because each operation opens its own short-lived `SqliteConnection` through the factory; there is no shared mutable state inside the services.

## Related Documentation

- [Gateway Configuration](./GatewayConfiguration.md)
- [Authorization](./Authorization.md)
- [Gateway Dashboard Design](./GatewayDashboardDesign.md)
- [Diagnostics](./Diagnostics.md)