Joseph Doherty
205 lines
9.4 KiB
Markdown
205 lines
9.4 KiB
Markdown
# Inbound API Key Re-issue Runbook
|
|
|
|
**Status:** BREAKING change — action required on every environment that uses the
|
|
inbound API (`POST /api/{methodName}`).
|
|
**Date:** 2026-06-02
|
|
**Migration:** `RetireInboundApiKeyStore`
|
|
|
|
This runbook covers the migration of inbound API authentication from the legacy SQL
|
|
Server `X-API-Key` scheme to the shared `ZB.MOM.WW.Auth.ApiKeys` store. After this
|
|
change **all existing inbound API keys are invalidated** and every API client must be
|
|
re-issued a new credential.
|
|
|
|
---
|
|
|
|
## 1. What changed and why
|
|
|
|
| | Before | After |
|
|
|---|---|---|
|
|
| Header | `X-API-Key: <key>` | `Authorization: Bearer sbk_<keyId>_<secret>` |
|
|
| Verification | Deterministic HMAC hash, looked up in SQL Server | Peppered, constant-time HMAC compare in the shared `ZB.MOM.WW.Auth.ApiKeys` verifier |
|
|
| Storage | SQL Server `ApiKeys` table (config DB) | `ZB.MOM.WW.Auth.ApiKeys` SQLite store |
|
|
| Authorization | `ApiMethod.ApprovedApiKeyIds` CSV linking methods to key IDs | Per-key **scopes**, where each scope string is an allowed method name (ordinal, case-sensitive) |
|
|
|
|
**Why:** the inbound credential path now reuses the shared auth library that the rest
|
|
of the `ZB.MOM.WW.*` family uses, with a single, tested, peppered verifier and a
|
|
proper one-time-token issuance model. The deterministic SQL Server hash table and its
|
|
method-link CSV are retired. The legacy `ApiKeyHasher` / `IApiKeyHasher` and the
|
|
in-repo `ApiKeyValidator` are gone — inbound auth runs through `IApiKeyVerifier`.
|
|
|
|
> The old `X-API-Key` credentials are **not migrated**. There is no automated
|
|
> conversion: the stored hashes are not reversible, and the new tokens have a
|
|
> different shape (`sbk_<keyId>_<secret>`). Every key must be re-issued.
|
|
|
|
---
|
|
|
|
## 2. Required configuration (per environment)
|
|
|
|
Set these under the ScadaBridge configuration for each environment (appsettings,
|
|
environment variables, or your secret store):
|
|
|
|
| Key | Value | Notes |
|
|
|---|---|---|
|
|
| `ScadaBridge:InboundApi:ApiKeyStore:SqlitePath` | Filesystem path to the SQLite key store | Defaults to `<content-root>/data/inbound-api-keys.sqlite` if unset. Choose a durable, backed-up path on a writable volume. Used when `Provider` is `Sqlite` (the default) and, when `Provider=SqlServer`, as the source file `ImportFromSqlitePath` imports once at boot. |
|
|
| `ScadaBridge:InboundApi:ApiKeyStore:Provider` | `Sqlite` (default) or `SqlServer` | `SqlServer` shares one row set across a central pair (design 2026-09-02, register row 108). Case-insensitive; an unrecognized value fails startup. |
|
|
| `ScadaBridge:InboundApi:ApiKeyStore:SqlServer:ConnectionString` | Defaults to `ScadaBridge:Database:ConfigurationDb` | Only used when `Provider=SqlServer`. Must be a literal or env override, **never** a `${secret:}` reference — the secrets store is itself SQL-backed. |
|
|
| `ScadaBridge:InboundApi:ApiKeyStore:ImportFromSqlitePath` | Defaults to `SqlitePath`'s default | Only used when `Provider=SqlServer`. The one-shot import source; renamed `*.imported` after a successful import. |
|
|
| `ScadaBridge:InboundApi:ApiKeyPepper` | A strong, random string, **≥ 16 characters** | **DIFFERENT per environment.** Keep it secret (secret store, not source control). This is the HMAC pepper that binds every stored key to this deployment; it is also the verifier's pepper secret. |
|
|
|
|
Notes:
|
|
- The pepper must be present and at least 16 characters or the host fails fast at
|
|
startup (`AddZbApiKeyAuth`).
|
|
- Changing the pepper after keys are issued invalidates all keys in that environment
|
|
(they would no longer verify). Set it once, per environment, and keep it stable.
|
|
- The token prefix is `sbk` and migrations run on startup by default
|
|
(`ScadaBridge:InboundApi:ApiKeyStore:RunMigrationsOnStartup = true`); these are
|
|
wired by the Host and normally need no operator change.
|
|
|
|
### Central pairs
|
|
|
|
A central pair (QA `vq03`/`vq04`, both docker rigs) should run `Provider=SqlServer` — under
|
|
the `Sqlite` default a key exists on whichever node served the management call, and a failover
|
|
401s every inbound client until keys are re-issued on the survivor (`StartupValidator` warns
|
|
about this at boot). To move an existing pair over:
|
|
|
|
1. Stop **both** central nodes together.
|
|
2. Set `ScadaBridge:InboundApi:ApiKeyStore:Provider = SqlServer` on both nodes.
|
|
3. If the app's SQL login lacks CREATE SCHEMA/TABLE rights, have a DBA run
|
|
`docs/operations/sql/zbauth-schema.sql` against the configuration database first — the
|
|
migrator then finds the version row already present and skips DDL.
|
|
4. Start both nodes together.
|
|
5. Check the import log line on each node (`grep -i "api-key store import\|zbauth"` in the
|
|
node's log) — one node reports the keys it inserted, the other reports the ones it found
|
|
already present (skipped); together they cover the union of both nodes' prior SQLite files.
|
|
6. Run `security api-key list` against each node's management endpoint and confirm both show
|
|
the same keys.
|
|
7. After a quarantine period (once you're confident the import is correct and no rollback is
|
|
needed), delete the `*.imported` SQLite files (and their `-wal`/`-shm` siblings) from both
|
|
nodes' volumes.
|
|
|
|
**Rollback:** revert `Provider` to `Sqlite` (or remove it) and rename the `*.imported` files
|
|
back to their original names on both nodes; no SQL Server rows need to be removed — the SQLite
|
|
code path never reads them.
|
|
|
|
---
|
|
|
|
## 3. Database migration step
|
|
|
|
Apply the EF Core migration `RetireInboundApiKeyStore` to the SQL Server
|
|
configuration database. It:
|
|
|
|
- drops the `ApiKeys` table, and
|
|
- drops the `ApprovedApiKeyIds` column from `ApiMethods`.
|
|
|
|
If migrations are applied automatically on deploy (the default for the central node),
|
|
this happens as part of the rollout. To apply manually:
|
|
|
|
```bash
|
|
dotnet ef database update RetireInboundApiKeyStore \
|
|
--project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase \
|
|
--startup-project src/ZB.MOM.WW.ScadaBridge.Host
|
|
```
|
|
|
|
> Applying this migration **permanently drops** the old key data. Take a database
|
|
> backup first if you need a record of the prior `ApiKeys` rows for audit purposes
|
|
> (the hashes are not usable credentials, but the names/enabled flags may be of
|
|
> record-keeping value).
|
|
|
|
The new inbound keys live in the **SQLite** store (section 2), not in SQL Server.
|
|
|
|
---
|
|
|
|
## 4. Operator re-issue procedure
|
|
|
|
Re-issue one key per client. Each key is created with the exact method names it is
|
|
allowed to call (its scopes).
|
|
|
|
### Option A — Admin UI
|
|
|
|
1. Navigate to **`/admin/api-keys`** in the central UI.
|
|
2. **Create** a new key: enter a display name and select the allowed method(s).
|
|
3. The one-time token `sbk_<keyId>_<secret>` is shown **exactly once** — copy it now.
|
|
It cannot be retrieved later.
|
|
4. Distribute the token securely to the owning client.
|
|
|
|
### Option B — CLI
|
|
|
|
```bash
|
|
scadabridge --url <central-url> security api-key create \
|
|
--name <client-name> \
|
|
--methods <method1,method2>
|
|
```
|
|
|
|
- `--methods` is a comma-separated list of allowed method names — these become the
|
|
key's scopes. A method name must match the registered `ApiMethod.Name` **exactly**
|
|
(case-sensitive).
|
|
- The command prints `API key created. KeyId: <id>` and then the one-time token on
|
|
stdout (the "save this now — it will not be shown again" advisory goes to stderr, so
|
|
piping stdout captures only the token).
|
|
|
|
Capture the `sbk_…` token at issue time; it is the only moment the secret is available.
|
|
|
|
To later change which methods a key may call:
|
|
|
|
```bash
|
|
scadabridge --url <central-url> security api-key set-methods --key-id <id> --methods <m1,m2>
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Client change
|
|
|
|
Each API client must replace its header:
|
|
|
|
- **Remove:** `X-API-Key: <old-key>`
|
|
- **Add:** `Authorization: Bearer sbk_<keyId>_<secret>`
|
|
|
|
Example:
|
|
|
|
```http
|
|
POST /api/CreateOrder HTTP/1.1
|
|
Host: scadabridge.example.com
|
|
Authorization: Bearer sbk_7f3a...._9c1e....
|
|
Content-Type: application/json
|
|
|
|
```
|
|
|
|
The token is the full `sbk_<keyId>_<secret>` string exactly as issued — do not split
|
|
or transform it.
|
|
|
|
---
|
|
|
|
## 6. Verification
|
|
|
|
1. **Authn (valid key):** call an allowed method with the new Bearer token → `200`
|
|
(or the method's normal result).
|
|
2. **Authn (no/old credential):** call with no `Authorization` header, or with the old
|
|
`X-API-Key` header only → `401` with `{"error":"Invalid or missing API key"}`.
|
|
3. **Authz (out of scope):** call a method the key is **not** scoped for → `403` with
|
|
`{"error":"API key not approved for this method"}`. A non-existent method name
|
|
returns the identical `403` body (enumeration-safe — by design).
|
|
4. **Audit:** a successful call records the verified key's display name as the audit
|
|
actor; an auth failure records `Actor=null`. Confirm via the audit log.
|
|
5. Confirm no client is still sending `X-API-Key` (those requests now fail `401`).
|
|
|
|
---
|
|
|
|
## 7. Rollback
|
|
|
|
The migration `Down` recreates the `ApiKeys` table and the `ApprovedApiKeyIds` column,
|
|
**but the dropped key rows are not restored** — `Down` only rebuilds empty structures.
|
|
Rolling the migration back does **not** recover any credential.
|
|
|
|
Therefore "rollback" means **reverting the deployment** to the prior build (which still
|
|
speaks `X-API-Key`), not reverting the keys:
|
|
|
|
1. Redeploy the previous ScadaBridge build.
|
|
2. If you took a SQL Server backup before section 3, restore the `ApiKeys` table from
|
|
it so the old keys verify again.
|
|
3. Without that backup, the old keys are gone and must be re-created under the legacy
|
|
scheme as well.
|
|
|
|
Because rollback is costly and lossy, prefer rolling **forward**: complete the re-issue
|
|
in section 4 and fix any straggler clients rather than reverting.
|