ScadaBridge/docs/operations/inbound-api-key-reissue.md

Inbound API Key Re-issue Runbook

Status: BREAKING change — action required on every environment that uses the inbound API (POST /api/{methodName}). Date: 2026-06-02 Migration: RetireInboundApiKeyStore

This runbook covers the migration of inbound API authentication from the legacy SQL Server X-API-Key scheme to the shared ZB.MOM.WW.Auth.ApiKeys store. After this change all existing inbound API keys are invalidated and every API client must be re-issued a new credential.

---

1. What changed and why

	Before	After
Header	X-API-Key: <key>	Authorization: Bearer sbk_<keyId>_<secret>
Verification	Deterministic HMAC hash, looked up in SQL Server	Peppered, constant-time HMAC compare in the shared ZB.MOM.WW.Auth.ApiKeys verifier
Storage	SQL Server ApiKeys table (config DB)	ZB.MOM.WW.Auth.ApiKeys SQLite store
Authorization	ApiMethod.ApprovedApiKeyIds CSV linking methods to key IDs	Per-key scopes, where each scope string is an allowed method name (ordinal, case-sensitive)

Why: the inbound credential path now reuses the shared auth library that the rest of the ZB.MOM.WW.* family uses, with a single, tested, peppered verifier and a proper one-time-token issuance model. The deterministic SQL Server hash table and its method-link CSV are retired. The legacy ApiKeyHasher / IApiKeyHasher and the in-repo ApiKeyValidator are gone — inbound auth runs through IApiKeyVerifier.

The old X-API-Key credentials are not migrated. There is no automated conversion: the stored hashes are not reversible, and the new tokens have a different shape (sbk_<keyId>_<secret>). Every key must be re-issued.

---

2. Required configuration (per environment)

Set these under the ScadaBridge configuration for each environment (appsettings, environment variables, or your secret store):

Key	Value	Notes
ScadaBridge:InboundApi:ApiKeyStore:SqlitePath	Filesystem path to the SQLite key store	Defaults to <content-root>/data/inbound-api-keys.sqlite if unset. Choose a durable, backed-up path on a writable volume. Used when Provider is Sqlite (the default) and, when Provider=SqlServer, as the source file ImportFromSqlitePath imports once at boot.
ScadaBridge:InboundApi:ApiKeyStore:Provider	Sqlite (default) or SqlServer	SqlServer shares one row set across a central pair (design 2026-09-02, register row 108). Case-insensitive; an unrecognized value fails startup.
ScadaBridge:InboundApi:ApiKeyStore:SqlServer:ConnectionString	Defaults to ScadaBridge:Database:ConfigurationDb	Only used when Provider=SqlServer. Must be a literal or env override, never a ${secret:} reference — the secrets store is itself SQL-backed.
ScadaBridge:InboundApi:ApiKeyStore:ImportFromSqlitePath	Defaults to SqlitePath's default	Only used when Provider=SqlServer. The one-shot import source; renamed *.imported after a successful import.
ScadaBridge:InboundApi:ApiKeyPepper	A strong, random string, ≥ 16 characters	DIFFERENT per environment. Keep it secret (secret store, not source control). This is the HMAC pepper that binds every stored key to this deployment; it is also the verifier's pepper secret.

Notes:

• The pepper must be present and at least 16 characters or the host fails fast at startup (AddZbApiKeyAuth).
• Changing the pepper after keys are issued invalidates all keys in that environment (they would no longer verify). Set it once, per environment, and keep it stable.
• The token prefix is sbk and migrations run on startup by default (ScadaBridge:InboundApi:ApiKeyStore:RunMigrationsOnStartup = true); these are wired by the Host and normally need no operator change.

Central pairs

A central pair (QA vq03/vq04, both docker rigs) should run Provider=SqlServer — under the Sqlite default a key exists on whichever node served the management call, and a failover 401s every inbound client until keys are re-issued on the survivor (StartupValidator warns about this at boot). To move an existing pair over:

1. Stop both central nodes together.
2. Set ScadaBridge:InboundApi:ApiKeyStore:Provider = SqlServer on both nodes.
3. If the app's SQL login lacks CREATE SCHEMA/TABLE rights, have a DBA run docs/operations/sql/zbauth-schema.sql against the configuration database first — the migrator then finds the version row already present and skips DDL.
4. Start both nodes together.
5. Check the import log line on each node (grep -i "api-key store import\|zbauth" in the node's log) — one node reports the keys it inserted, the other reports the ones it found already present (skipped); together they cover the union of both nodes' prior SQLite files.
6. Run security api-key list against each node's management endpoint and confirm both show the same keys.
7. After a quarantine period (once you're confident the import is correct and no rollback is needed), delete the *.imported SQLite files (and their -wal/-shm siblings) from both nodes' volumes.

Rollback: revert Provider to Sqlite (or remove it) and rename the *.imported files back to their original names on both nodes; no SQL Server rows need to be removed — the SQLite code path never reads them.

---

3. Database migration step

Apply the EF Core migration RetireInboundApiKeyStore to the SQL Server configuration database. It:

• drops the ApiKeys table, and
• drops the ApprovedApiKeyIds column from ApiMethods.

If migrations are applied automatically on deploy (the default for the central node), this happens as part of the rollout. To apply manually:

dotnet ef database update RetireInboundApiKeyStore \
  --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase \
  --startup-project src/ZB.MOM.WW.ScadaBridge.Host

Applying this migration permanently drops the old key data. Take a database backup first if you need a record of the prior ApiKeys rows for audit purposes (the hashes are not usable credentials, but the names/enabled flags may be of record-keeping value).

The new inbound keys live in the SQLite store (section 2), not in SQL Server.

---

4. Operator re-issue procedure

Re-issue one key per client. Each key is created with the exact method names it is allowed to call (its scopes).

Option A — Admin UI

1. Navigate to /admin/api-keys in the central UI.
2. Create a new key: enter a display name and select the allowed method(s).
3. The one-time token sbk_<keyId>_<secret> is shown exactly once — copy it now. It cannot be retrieved later.
4. Distribute the token securely to the owning client.

Option B — CLI

scadabridge --url <central-url> security api-key create \
  --name <client-name> \
  --methods <method1,method2>

• --methods is a comma-separated list of allowed method names — these become the key's scopes. A method name must match the registered ApiMethod.Name exactly (case-sensitive).
• The command prints API key created. KeyId: <id> and then the one-time token on stdout (the "save this now — it will not be shown again" advisory goes to stderr, so piping stdout captures only the token).

Capture the sbk_… token at issue time; it is the only moment the secret is available.

To later change which methods a key may call:

scadabridge --url <central-url> security api-key set-methods --key-id <id> --methods <m1,m2>

---

5. Client change

Each API client must replace its header:

• Remove: X-API-Key: <old-key>
• Add: Authorization: Bearer sbk_<keyId>_<secret>

Example:

POST /api/CreateOrder HTTP/1.1
Host: scadabridge.example.com
Authorization: Bearer sbk_7f3a...._9c1e....
Content-Type: application/json

{ "orderId": "..." }

The token is the full sbk_<keyId>_<secret> string exactly as issued — do not split or transform it.

---

6. Verification

1. Authn (valid key): call an allowed method with the new Bearer token → 200 (or the method's normal result).
2. Authn (no/old credential): call with no Authorization header, or with the old X-API-Key header only → 401 with {"error":"Invalid or missing API key"}.
3. Authz (out of scope): call a method the key is not scoped for → 403 with {"error":"API key not approved for this method"}. A non-existent method name returns the identical 403 body (enumeration-safe — by design).
4. Audit: a successful call records the verified key's display name as the audit actor; an auth failure records Actor=null. Confirm via the audit log.
5. Confirm no client is still sending X-API-Key (those requests now fail 401).

---

7. Rollback

The migration Down recreates the ApiKeys table and the ApprovedApiKeyIds column, but the dropped key rows are not restored — Down only rebuilds empty structures. Rolling the migration back does not recover any credential.

Therefore "rollback" means reverting the deployment to the prior build (which still speaks X-API-Key), not reverting the keys:

1. Redeploy the previous ScadaBridge build.
2. If you took a SQL Server backup before section 3, restore the ApiKeys table from it so the old keys verify again.
3. Without that backup, the old keys are gone and must be re-created under the legacy scheme as well.

Because rollback is costly and lossy, prefer rolling forward: complete the re-issue in section 4 and fix any straggler clients rather than reverting.