Joseph Doherty
df710e18a9
The gRPC auth failure limiter partitioned on the key id parsed out of the *unauthenticated* token and rejected with ResourceExhausted before VerifyAsync ran. Key ids are not secret — they ride in every token and are listed on the dashboard — so any network peer could send 10 garbage-secret requests per minute and deny that key indefinitely: the legitimate holder's correct secret was refused before it was ever checked, and the success-path Reset that would clear the block sat behind the verification the block prevented (SEC-31). The tracked map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the `mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked entry and reset the window (SEC-32). ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an ApiKeyThrottleDecision result. Two layers share one sliding window — a composite (transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30) that bounds a source-rotating sprayer. An over-limit state is now a valve rather than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default 5) is admitted through to the real verifier, so the correct secret always reaches the constant-time compare and resets both layers. Guarantees preserved: guessing stays bounded per window, and the failure path still spends no store read per attempt. SEC-32 rides the same change set: the interceptor validates token shape (literal `mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a key-id partition, each transport peer may mint at most 32 of them before the overflow collapses onto its fallback partition, and eviction prefers fully expired windows and never drops an over-limit partition below a 2x transient overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key material may appear there. Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys, the Authentication hot-path paragraph, the Authorization SEC-11 section, and the limiter / SecurityOptions XML remarks (the old NAT rationale described the defective keying). Tracking rows flipped to Done with a change-log entry. Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs aggregate trip points, probe cadence, absolute-block mode, reset across both layers, junk-spray eviction resistance, the per-peer cap, and expired-window eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20 total); GatewayOptionsValidatorTests covers both new keys including 0 as a supported disable value (66 total).
284 lines
16 KiB
Markdown
284 lines
16 KiB
Markdown
# Gateway Authentication
|
|
|
|
The gateway authenticates inbound gRPC callers with API keys: a bearer token is
|
|
parsed, its secret is hashed with a peppered HMAC and compared in constant time
|
|
against a stored hash, and administrative and verification events are recorded to
|
|
an audit trail.
|
|
|
|
The peppered-HMAC pipeline itself — token parsing, secret generation, hashing,
|
|
constant-time compare, the SQLite schema, the key store, the verifier, and schema
|
|
migration — lives in the shared **`ZB.MOM.WW.Auth.ApiKeys`** package, of which
|
|
this gateway is the donor. The gateway does not reimplement or fork those types;
|
|
it binds the library through `AddZbApiKeyAuth` and layers gateway-specific
|
|
concerns on top: constraint enforcement, the gRPC authorization interceptor,
|
|
hot-path decorators, the admin CLI, the dashboard, and a canonical audit store
|
|
that supersedes the library's own audit table. This document describes the
|
|
consumer side — the token format, the options the gateway binds, the pieces it
|
|
adds, and where the library boundary sits. For the library internals (the concrete
|
|
`ApiKeyVerifier`, the SQLite stores, the schema and migrator), read the
|
|
`ZB.MOM.WW.Auth.ApiKeys` sources; they are not duplicated in this repository.
|
|
|
|
## Token Format
|
|
|
|
API keys travel in the HTTP `Authorization` header as a bearer token shaped
|
|
`mxgw_<keyId>_<secret>`. The `mxgw_` prefix scopes parsing to gateway tokens, the
|
|
`<keyId>` segment is the public identifier used for lookup, and `<secret>` is the
|
|
high-entropy portion verified against a stored hash. The prefix and the pepper
|
|
configuration key the gateway pins are constants on
|
|
`AuthStoreServiceCollectionExtensions`
|
|
(`TokenPrefix = "mxgw"`, `PepperSecretName = "MxGateway:ApiKeyPepper"`); they are
|
|
supplied to the library at registration so the library's parser and pepper
|
|
provider use the gateway's contract. The library parser rejects a malformed token
|
|
before any database round-trip, and only a well-formed `mxgw_<keyId>_<secret>`
|
|
token reaches the store lookup.
|
|
|
|
## Secrets And Peppered Hashing
|
|
|
|
New secret material is high-entropy: the library generates 32 random bytes and
|
|
encodes them URL-safe base64 (no padding) so a secret embeds in a header without
|
|
escaping. The gateway never persists a plaintext secret — only its hash.
|
|
|
|
Secrets are hashed with `HMAC-SHA256` keyed by a server-side **pepper**. The
|
|
pepper lives outside the database and is resolved from configuration under the
|
|
`MxGateway:ApiKeyPepper` key (the library's pepper provider reads it). Keeping the
|
|
pepper out of the SQLite file means an attacker who exfiltrates only the database
|
|
holds the hashes but lacks the keying material to brute-force candidate secrets,
|
|
even if the hash algorithm is known.
|
|
|
|
When the pepper is not configured, the library surfaces the failure as an
|
|
`InvalidOperationException` whose message reports the pepper is unavailable rather
|
|
than persisting a key with an unkeyed hash. The dashboard management path
|
|
(`DashboardApiKeyManagementService`) catches that condition and returns the
|
|
friendly "API key pepper is not configured." result instead of faulting the Blazor
|
|
circuit; it currently matches on the message text, so a library wording change
|
|
would need to be reflected there (a typed pepper-unavailable exception is a pending
|
|
library improvement).
|
|
|
|
## Verification
|
|
|
|
The gateway consumes the library's `IApiKeyVerifier` from
|
|
`GatewayGrpcAuthorizationInterceptor`. The verifier's flow is:
|
|
|
|
1. Parse the `Authorization` header into the key id and presented secret.
|
|
2. Look up the stored key record by key id.
|
|
3. Reject a revoked record, and reject an expired record whose `ExpiresUtc` is in
|
|
the past. Expiry is opt-in — keys created without an expiry never expire; an
|
|
expired key fails opaquely, indistinguishable to the client from any other auth
|
|
failure.
|
|
4. Hash the presented secret with the configured pepper.
|
|
5. Compare hashes in constant time to avoid a timing oracle.
|
|
6. Stamp a `LastUsedUtc` timestamp and return a shared `ApiKeyIdentity` carrying
|
|
the key id, key prefix, display name, scopes, and the opaque constraints JSON.
|
|
|
|
A verification failure is opaque to the client: the interceptor returns
|
|
`Unauthenticated`/`PermissionDenied` without disclosing which check failed, while
|
|
the failure detail is available server-side for audit.
|
|
|
|
`GatewayApiKeyIdentityMapper.ToGatewayIdentity` maps the library's shared
|
|
`ApiKeyIdentity` onto the gateway's own `ApiKeyIdentity`
|
|
(`Security/Authentication/ApiKeyIdentity.cs`), which exposes the deserialized
|
|
`ApiKeyConstraints` — parsed from the opaque constraints JSON via
|
|
`ApiKeyConstraintSerializer` — that the downstream `ConstraintEnforcer` and the
|
|
request-identity accessor enforce. The gateway identity exposes only non-secret
|
|
fields (`KeyId`, `KeyPrefix`, `DisplayName`, `Scopes`, `Constraints`).
|
|
|
|
### Hot-path caching and last-used coalescing
|
|
|
|
Left unmediated, every authenticated gRPC call costs a SQLite read plus a
|
|
`last_used_utc` **write** (the library verifier couples `MarkUsed` into
|
|
`VerifyAsync`), which makes the auth store the throughput ceiling on the
|
|
bulk-read workload. The gateway layers two decorators over the shared library's
|
|
registrations (in `AuthStoreServiceCollectionExtensions`) — it does not edit the
|
|
library:
|
|
|
|
- **`CachingApiKeyVerifier`** wraps the library `IApiKeyVerifier` with an
|
|
`IMemoryCache` entry per successful verification, keyed on a SHA-256 hash of the
|
|
presented token (never the plaintext secret). A cache hit within
|
|
`MxGateway:Security:ApiKeyVerificationCacheSeconds` (default 15 s) returns the
|
|
cached result without touching the store, so both the read and the coupled write
|
|
are skipped. Only successes are cached; failures always reach the inner verifier.
|
|
On a gateway-initiated revoke/rotate/delete the dashboard admin service calls
|
|
`IApiKeyCacheInvalidator.Invalidate(keyId)`, evicting the cached entry
|
|
immediately. The short TTL is the backstop for out-of-band mutations (a direct DB
|
|
edit, or a revoke run by the separate `apikey` CLI process, whose in-memory cache
|
|
is not the running gateway's cache).
|
|
- **`CoalescingMarkApiKeyStore`** wraps the library `IApiKeyStore` and forwards at
|
|
most one `MarkUsed` write per key per
|
|
`MxGateway:Security:ApiKeyLastUsedCoalesceSeconds` (default 60 s), so even under a
|
|
cache miss the `last_used_utc` write is bounded to roughly one per key per minute
|
|
rather than one per RPC. `last_used_utc` is a coarse staleness hint, not an audit
|
|
record (audit rows are written separately), so bounded staleness of up to one
|
|
window is acceptable.
|
|
|
|
`GatewayApiKeyIdentityMapper` additionally memoizes the constraints-JSON
|
|
deserialization by blob, so the per-call parse on the mapped identity collapses to
|
|
a dictionary lookup. Both windows are configurable and may be set to `0` to disable
|
|
the respective mechanism; see
|
|
[GatewayConfiguration](./GatewayConfiguration.md).
|
|
|
|
Failures are never cached — a wrong secret always reaches the store — so the
|
|
failure path is shielded by `ApiKeyFailureLimiter` instead, consulted before
|
|
`VerifyAsync` runs. It counts failures over one sliding
|
|
`MxGateway:Security:ApiKeyFailureWindowSeconds` window in two layers: a composite
|
|
`(transport peer, key id)` partition capped at `ApiKeyFailureLimit`, and a
|
|
per-key-id aggregate across all peers capped at `ApiKeyFailureAggregateLimit`. The
|
|
key id never partitions on its own — it is public, so an attacker-supplied one
|
|
would otherwise let any peer throttle a key it does not hold — and it joins the
|
|
partition only when the presented token is validly shaped
|
|
(`mxgw_<keyId>_<secret>`, key id at most 64 characters), with at most 32 key-id
|
|
partitions per address before the overflow collapses onto that address's fallback
|
|
partition. An over-limit state admits one probe per
|
|
`ApiKeyFailureProbeIntervalSeconds` through to the real verifier and refuses
|
|
everything else with `ResourceExhausted` before the store read, so a legitimate
|
|
holder presenting the correct secret always reaches the constant-time compare and
|
|
resets both layers; the counter's LRU eviction (`ApiKeyFailureTrackedPeers`)
|
|
prefers expired windows and will not drop an over-limit partition below a 2x
|
|
overshoot ceiling, so the memory bound cannot be turned into a way to clear an
|
|
active block. See [Authorization](./Authorization.md) for the enforcement path.
|
|
|
|
## Storage
|
|
|
|
API-key state lives in a dedicated SQLite database owned by the shared library.
|
|
SQLite is sufficient because credential volume is small, the gateway runs as a
|
|
single process, and the file is straightforward to back up and rotate independently
|
|
of the main application data.
|
|
|
|
The database path is `GatewayOptions.Authentication.SqlitePath`. Its code default
|
|
is derived from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)`
|
|
(`C:\ProgramData\MxGateway\gateway-auth.db` on Windows,
|
|
`/usr/share/MxGateway/gateway-auth.db` or the container equivalent elsewhere) so the
|
|
credential store is never written relative to the launch working directory on a
|
|
non-Windows host. The production hosts pin the explicit Windows path in
|
|
`appsettings.json`. `GatewayOptionsValidator` rejects a non-rooted (relative)
|
|
`SqlitePath` so a bad override fails fast at startup rather than scattering the store
|
|
by launch CWD (SEC-01).
|
|
|
|
The library owns the SQLite schema and connection factory. The `api_keys` table
|
|
carries the key id, key prefix, secret-hash blob, display name, serialized scopes,
|
|
optional serialized constraints, and the `created_utc`, `last_used_utc`,
|
|
`revoked_utc`, and `expires_utc` timestamps. Because the schema, stores, and migrator
|
|
belong to `ZB.MOM.WW.Auth.ApiKeys`, this document does not restate their column
|
|
readers or SQL; consult the library for that detail.
|
|
|
|
### Audit trail
|
|
|
|
The library emits its own API-key audit entries (from the admin verbs — create,
|
|
revoke, rotate, `init-db`, and constraint denials), but the gateway **overrides**
|
|
the library's `IApiKeyAuditStore` registration with
|
|
`CanonicalForwardingApiKeyAuditStore`. That adapter canonicalizes every
|
|
library-emitted `ApiKeyAuditEntry` onto the gateway's `AuditEvent` shape and routes
|
|
it through `IAuditWriter` (`CanonicalAuditWriter`) into `SqliteCanonicalAuditStore`,
|
|
which persists to a single **`audit_event`** table (columns `event_id`,
|
|
`occurred_at_utc`, `actor`, `action`, `outcome`, `category`, `target`,
|
|
`source_node`, `correlation_id`, `details_json`). Reads for the dashboard "recent
|
|
audit" view go back through the same adapter, which maps `audit_event` rows back to
|
|
`ApiKeyAuditEntry` values so the existing view keeps working unchanged.
|
|
|
|
Consequently the library's own `api_key_audit` table is left in place but
|
|
**unused** after adoption — nothing writes to it once the override is registered.
|
|
The canonical `audit_event` table is the single durable record of both API-key
|
|
administrative actions and the dashboard's own audit vocabulary
|
|
(`dashboard-create-key`, `dashboard-rotate-key`, `dashboard-revoke-key`,
|
|
`dashboard-delete-key`, and the session Close/Kill actions). This is why any prose
|
|
that describes credential audits as landing in `api_key_audit` is stale: the
|
|
canonical store is `audit_event`.
|
|
|
|
## Registration
|
|
|
|
`AuthStoreServiceCollectionExtensions.AddSqliteAuthStore(IConfiguration)` wires the
|
|
whole subsystem. It does not register the library types directly — it delegates to
|
|
the shared provider and then layers the gateway concerns:
|
|
|
|
```csharp
|
|
public static IServiceCollection AddSqliteAuthStore(
|
|
this IServiceCollection services,
|
|
IConfiguration configuration)
|
|
{
|
|
// Pin the gateway's token prefix ("mxgw") and pepper key ("MxGateway:ApiKeyPepper")
|
|
// as fallback defaults UNDER the supplied configuration, then register the shared
|
|
// provider: it binds ApiKeyOptions from MxGateway:Authentication and wires the SQLite
|
|
// stores, the configuration-backed pepper provider, the verifier, the migrator, and
|
|
// the migration hosted service.
|
|
services.AddZbApiKeyAuth(effectiveConfig, AuthenticationSectionPath);
|
|
|
|
// SEC-08 hot-path decorators layered over the library registrations.
|
|
services.AddMemoryCache();
|
|
// CoalescingMarkApiKeyStore decorates IApiKeyStore; CachingApiKeyVerifier decorates
|
|
// IApiKeyVerifier and also serves as IApiKeyCacheInvalidator.
|
|
|
|
// Canonical audit: override the library's IApiKeyAuditStore so every API-key audit
|
|
// event is forwarded through IAuditWriter into the audit_event table.
|
|
services.AddSingleton<IApiKeyAuditStore, CanonicalForwardingApiKeyAuditStore>();
|
|
|
|
// The shared admin command set (ApiKeyAdminCommands) and the gateway CLI runner.
|
|
services.AddSingleton<ApiKeyAdminCliRunner>();
|
|
|
|
return services;
|
|
}
|
|
```
|
|
|
|
The decorators wrap the library's last registration for each interface rather than
|
|
replacing the library types, preserving singleton semantics; the audit override is
|
|
registered after `AddZbApiKeyAuth` so it wins as the resolved `IApiKeyAuditStore`.
|
|
|
|
## Admin CLI
|
|
|
|
`ApiKeyAdminCommandLineParser.Parse` recognises a leading `apikey` argument and
|
|
dispatches to one of the subcommands declared by `ApiKeyAdminCommandKind`. Each
|
|
parsed invocation produces an `ApiKeyAdminCommand` (or an `ApiKeyAdminParseResult`
|
|
carrying an error). `ApiKeyAdminCliRunner` then runs the migrator, invokes the shared
|
|
`ApiKeyAdminCommands` verb, and writes text or JSON output via `ApiKeyAdminOutput`.
|
|
The returned `ApiKeyAdminListedKey` projection deliberately omits the secret hash so
|
|
listing a database never surfaces hash material.
|
|
|
|
The supported subcommands match `ApiKeyAdminCommandKind` exactly:
|
|
|
|
| Subcommand | Required options | Behaviour |
|
|
|------------|------------------|-----------|
|
|
| `init-db` | none | Runs the migrator and records an audit entry. |
|
|
| `create-key` | `--key-id`, `--display-name` | Generates a new secret, stores its peppered hash and optional constraints, and prints the assembled `mxgw_<keyId>_<secret>` token. Optional `--expires` sets an expiry (absolute ISO-8601 UTC, or a relative `<N>d`/`<N>h` from now); omit it for a non-expiring key. |
|
|
| `list-keys` | none | Lists every stored key with its scopes, constraints, revocation state, and expiry (`active`/`expired`/`revoked`). |
|
|
| `revoke-key` | `--key-id` | Marks the key revoked if it is currently active. |
|
|
| `rotate-key` | `--key-id` | Replaces the secret hash and prints the new token. |
|
|
|
|
Examples:
|
|
|
|
```bash
|
|
mxgateway apikey init-db
|
|
mxgateway apikey create-key --key-id ops.alice --display-name "Alice (ops)" --scopes read,write
|
|
mxgateway apikey create-key --key-id area1.reader --display-name "Area 1 reader" --scopes invoke:read,metadata:read --read-subtree "Area1/*" --browse-subtree "Area1/*"
|
|
mxgateway apikey create-key --key-id ops.temp --display-name "Temp contractor" --scopes invoke:read --expires 90d
|
|
mxgateway apikey create-key --key-id ops.audit --display-name "Audit window" --scopes metadata:read --expires 2027-01-01T00:00:00Z
|
|
mxgateway apikey list-keys --json
|
|
mxgateway apikey revoke-key --key-id ops.alice
|
|
mxgateway apikey rotate-key --key-id ops.alice
|
|
```
|
|
|
|
Constraint flags are optional. `--read-subtree`, `--write-subtree`,
|
|
`--read-tag-glob`, `--write-tag-glob`, and `--browse-subtree` are repeatable.
|
|
`--max-write-classification` accepts one integer. `--read-alarm-only` and
|
|
`--read-historized-only` are boolean flags. Existing rows with null constraints
|
|
remain fully unconstrained after migration.
|
|
|
|
Key ids are restricted by the parser to ASCII letters, digits, periods, and hyphens
|
|
so they remain safe to embed in the token format and in URL paths used by
|
|
administrative tooling.
|
|
|
|
The CLI is not the only management surface: the dashboard API Keys page creates,
|
|
rotates, revokes, and deletes (revoked-only) keys through the same shared admin
|
|
command set. Every destructive dashboard action is gated by a confirmation dialog
|
|
and emits its own audit event (`dashboard-create-key`, `dashboard-rotate-key`,
|
|
`dashboard-revoke-key`, `dashboard-delete-key`) into the canonical `audit_event`
|
|
store. The page also surfaces expiry: each row shows an `Expires` column (`Never`
|
|
when unset) and a status badge that reads `Expired`, `Expiring` (within seven days),
|
|
`Revoked`, or `Active`. This staleness surfacing is display-only; expiry is set at
|
|
creation time via `apikey create-key --expires`, not from the dashboard. See
|
|
[Gateway Dashboard Design](./GatewayDashboardDesign.md#api-keys-page).
|
|
|
|
## Related Documentation
|
|
|
|
- [Gateway Configuration](./GatewayConfiguration.md)
|
|
- [Authorization](./Authorization.md)
|
|
- [Gateway Dashboard Design](./GatewayDashboardDesign.md)
|
|
- [Diagnostics](./Diagnostics.md)
|