# Inbound API Key Re-issue Runbook

**Status:** BREAKING change — action required on every environment that uses the
inbound API (`POST /api/{methodName}`).
**Date:** 2026-06-02
**Migration:** `RetireInboundApiKeyStore`

This runbook covers the migration of inbound API authentication from the legacy SQL
Server `X-API-Key` scheme to the shared `ZB.MOM.WW.Auth.ApiKeys` store. After this
change **all existing inbound API keys are invalidated** and every API client must be
re-issued a new credential.

---

## 1. What changed and why

| | Before | After |
|---|---|---|
| Header | `X-API-Key: <key>` | `Authorization: Bearer sbk_<keyId>_<secret>` |
| Verification | Deterministic HMAC hash, looked up in SQL Server | Peppered, constant-time HMAC compare in the shared `ZB.MOM.WW.Auth.ApiKeys` verifier |
| Storage | SQL Server `ApiKeys` table (config DB) | `ZB.MOM.WW.Auth.ApiKeys` SQLite store |
| Authorization | `ApiMethod.ApprovedApiKeyIds` CSV linking methods to key IDs | Per-key **scopes**, where each scope string is an allowed method name (ordinal, case-sensitive) |

**Why:** the inbound credential path now reuses the shared auth library that the rest
of the `ZB.MOM.WW.*` family uses, with a single, tested, peppered verifier and a
proper one-time-token issuance model. The deterministic SQL Server hash table and its
method-link CSV are retired. The legacy `ApiKeyHasher` / `IApiKeyHasher` and the
in-repo `ApiKeyValidator` are gone — inbound auth runs through `IApiKeyVerifier`.

> The old `X-API-Key` credentials are **not migrated**. There is no automated
> conversion: the stored hashes are not reversible, and the new tokens have a
> different shape (`sbk_<keyId>_<secret>`). Every key must be re-issued.

---

## 2. Required configuration (per environment)

Set these under the ScadaBridge configuration for each environment (appsettings,
environment variables, or your secret store):

| Key | Value | Notes |
|---|---|---|
| `ScadaBridge:InboundApi:ApiKeyStore:SqlitePath` | Filesystem path to the SQLite key store | Defaults to `<content-root>/data/inbound-api-keys.sqlite` if unset. Choose a durable, backed-up path on a writable volume. Used when `Provider` is `Sqlite` (the default) and, when `Provider=SqlServer`, as the source file `ImportFromSqlitePath` imports once at boot. |
| `ScadaBridge:InboundApi:ApiKeyStore:Provider` | `Sqlite` (default) or `SqlServer` | `SqlServer` shares one row set across a central pair (design 2026-09-02, register row 108). Case-insensitive; an unrecognized value fails startup. |
| `ScadaBridge:InboundApi:ApiKeyStore:SqlServer:ConnectionString` | Defaults to `ScadaBridge:Database:ConfigurationDb` | Only used when `Provider=SqlServer`. Must be a literal or env override, **never** a `${secret:}` reference — the secrets store is itself SQL-backed. |
| `ScadaBridge:InboundApi:ApiKeyStore:ImportFromSqlitePath` | Defaults to `SqlitePath`'s default | Only used when `Provider=SqlServer`. The one-shot import source; renamed `*.imported` after a successful import. |
| `ScadaBridge:InboundApi:ApiKeyPepper` | A strong, random string, **≥ 16 characters** | **DIFFERENT per environment.** Keep it secret (secret store, not source control). This is the HMAC pepper that binds every stored key to this deployment; it is also the verifier's pepper secret. |

Notes:
- The pepper must be present and at least 16 characters or the host fails fast at
  startup (`AddZbApiKeyAuth`).
- Changing the pepper after keys are issued invalidates all keys in that environment
  (they would no longer verify). Set it once, per environment, and keep it stable.
- The token prefix is `sbk` and migrations run on startup by default
  (`ScadaBridge:InboundApi:ApiKeyStore:RunMigrationsOnStartup = true`); these are
  wired by the Host and normally need no operator change.

### Central pairs

A central pair (QA `vq03`/`vq04`, both docker rigs) should run `Provider=SqlServer` — under
the `Sqlite` default a key exists on whichever node served the management call, and a failover
401s every inbound client until keys are re-issued on the survivor (`StartupValidator` warns
about this at boot). To move an existing pair over:

1. Stop **both** central nodes together.
2. Set `ScadaBridge:InboundApi:ApiKeyStore:Provider = SqlServer` on both nodes.
3. If the app's SQL login lacks CREATE SCHEMA/TABLE rights, have a DBA run
   `docs/operations/sql/zbauth-schema.sql` against the configuration database first — the
   migrator then finds the version row already present and skips DDL.
4. Start both nodes together.
5. Check the import log line on each node (`grep -i "api-key store import\|zbauth"` in the
   node's log) — one node reports the keys it inserted, the other reports the ones it found
   already present (skipped); together they cover the union of both nodes' prior SQLite files.
6. Run `security api-key list` against each node's management endpoint and confirm both show
   the same keys.
7. After a quarantine period (once you're confident the import is correct and no rollback is
   needed), delete the `*.imported` SQLite files (and their `-wal`/`-shm` siblings) from both
   nodes' volumes.

**Rollback:** revert `Provider` to `Sqlite` (or remove it) and rename the `*.imported` files
back to their original names on both nodes; no SQL Server rows need to be removed — the SQLite
code path never reads them.

---

## 3. Database migration step

Apply the EF Core migration `RetireInboundApiKeyStore` to the SQL Server
configuration database. It:

- drops the `ApiKeys` table, and
- drops the `ApprovedApiKeyIds` column from `ApiMethods`.

If migrations are applied automatically on deploy (the default for the central node),
this happens as part of the rollout. To apply manually:

```bash
dotnet ef database update RetireInboundApiKeyStore \
  --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase \
  --startup-project src/ZB.MOM.WW.ScadaBridge.Host
```

> Applying this migration **permanently drops** the old key data. Take a database
> backup first if you need a record of the prior `ApiKeys` rows for audit purposes
> (the hashes are not usable credentials, but the names/enabled flags may be of
> record-keeping value).

The new inbound keys live in the **SQLite** store (section 2), not in SQL Server.

---

## 4. Operator re-issue procedure

Re-issue one key per client. Each key is created with the exact method names it is
allowed to call (its scopes).

### Option A — Admin UI

1. Navigate to **`/admin/api-keys`** in the central UI.
2. **Create** a new key: enter a display name and select the allowed method(s).
3. The one-time token `sbk_<keyId>_<secret>` is shown **exactly once** — copy it now.
   It cannot be retrieved later.
4. Distribute the token securely to the owning client.

### Option B — CLI

```bash
scadabridge --url <central-url> security api-key create \
  --name <client-name> \
  --methods <method1,method2>
```

- `--methods` is a comma-separated list of allowed method names — these become the
  key's scopes. A method name must match the registered `ApiMethod.Name` **exactly**
  (case-sensitive).
- The command prints `API key created. KeyId: <id>` and then the one-time token on
  stdout (the "save this now — it will not be shown again" advisory goes to stderr, so
  piping stdout captures only the token).

Capture the `sbk_…` token at issue time; it is the only moment the secret is available.

To later change which methods a key may call:

```bash
scadabridge --url <central-url> security api-key set-methods --key-id <id> --methods <m1,m2>
```

---

## 5. Client change

Each API client must replace its header:

- **Remove:** `X-API-Key: <old-key>`
- **Add:** `Authorization: Bearer sbk_<keyId>_<secret>`

Example:

```http
POST /api/CreateOrder HTTP/1.1
Host: scadabridge.example.com
Authorization: Bearer sbk_7f3a...._9c1e....
Content-Type: application/json

{ "orderId": "..." }
```

The token is the full `sbk_<keyId>_<secret>` string exactly as issued — do not split
or transform it.

---

## 6. Verification

1. **Authn (valid key):** call an allowed method with the new Bearer token → `200`
   (or the method's normal result).
2. **Authn (no/old credential):** call with no `Authorization` header, or with the old
   `X-API-Key` header only → `401` with `{"error":"Invalid or missing API key"}`.
3. **Authz (out of scope):** call a method the key is **not** scoped for → `403` with
   `{"error":"API key not approved for this method"}`. A non-existent method name
   returns the identical `403` body (enumeration-safe — by design).
4. **Audit:** a successful call records the verified key's display name as the audit
   actor; an auth failure records `Actor=null`. Confirm via the audit log.
5. Confirm no client is still sending `X-API-Key` (those requests now fail `401`).

---

## 7. Rollback

The migration `Down` recreates the `ApiKeys` table and the `ApprovedApiKeyIds` column,
**but the dropped key rows are not restored** — `Down` only rebuilds empty structures.
Rolling the migration back does **not** recover any credential.

Therefore "rollback" means **reverting the deployment** to the prior build (which still
speaks `X-API-Key`), not reverting the keys:

1. Redeploy the previous ScadaBridge build.
2. If you took a SQL Server backup before section 3, restore the `ApiKeys` table from
   it so the old keys verify again.
3. Without that backup, the old keys are gone and must be re-created under the legacy
   scheme as well.

Because rollback is costly and lossy, prefer rolling **forward**: complete the re-issue
in section 4 and fix any straggler clients rather than reverting.