ScadaBridge/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md

# Design — Shared SQL Server store for inbound API keys (central pair)

**Date:** 2026-09-02 · **Status:** Proposed · **Register:** row 108 (`docs/plans/2026-07-08-deferred-work-register.md`) · **Plan:** `2026-09-02-inbound-api-key-shared-store.md`

## Problem

Inbound API keys (`POST /api/{methodName}`, `Bearer sbk_<keyId>_<secret>`) are stored by the shared
`ZB.MOM.WW.Auth.ApiKeys` library in a **per-node SQLite file**
(`ScadaBridge:InboundApi:ApiKeyStore:SqlitePath`, `/app/data/inbound-api-keys.sqlite` on the rig,
`app\data\inbound-api-keys.sqlite` on the Windows installs). Key admin runs on whichever central node
serves the management call — through Traefik that is always the active node — so **a key exists on
exactly one node**. Consequences, all observed or verified on 2026-09-02:

- After an active-central crash the survivor answers **401 `KeyNotFound`** to every inbound client
  (`docker/verify-delmia-notifier.sh`, first run). The rig now needs `docker/sync-inbound-api-keys.sh`
  to copy the file by hand before a failover drill can pass.
- A key **revoked** on one node stays valid on the other. Revocation is not cluster-wide.
- The standby-relay path only works because the standby forwards *before* authenticating.
- Prod `wonder-app-vd03` is single-node and unaffected; the QA pair `vq03`/`vq04` has the gap today.
- `Component-InboundAPI.md` §Storage still says keys live in the configuration database — stale since
  the 2026-06-02 re-arch (`RetireInboundApiKeyStore` dropped the SQL table and moved keys to SQLite).

The library ships **only** a SQLite store (`Sqlite/SqliteApiKeyStore`, `SqliteApiKeyAdminStore`,
`SqliteApiKeyAuditStore`, `SqliteAuthStoreMigrator`). There is no provider to switch to. This is
**not** the secrets store: `ZB.MOM.WW.Secrets` already has a shared SQL Server store for central
(`Secrets 0.5.0`, `ZbSecretsHub`), and `${secret:}` values are unaffected.

## Decision

**Move central's inbound key store to the shared SQL Server (the configuration database, own schema
`zbauth`) by adding a SQL Server provider to the shared library, mirroring what Secrets 0.5.0 did.**
SQLite stays the library default and the right answer for single-process consumers
(MxAccessGateway, HistorianGateway, a single-node ScadaBridge). Every reader and writer of a ScadaBridge
central pair then sees one row per key: create, revoke, rotate, scope edits and `last_used` are
cluster-wide by construction, and there is nothing to replicate or reconcile.

Alternatives rejected:

| Option | Why not |
|---|---|
| Replicate the SQLite file (rig script, robocopy, LocalDb CDC) | Copy-then-restart is operator-driven and still per-node between copies; LocalDb is the *site* replication engine and central does not host it; a mid-copy revoke is lost. |
| Shared SQLite on a network share | SQLite over SMB/UNC is unsafe under concurrent writers; QA nodes are separate boxes. |
| Central-to-central Akka replication of key rows | New replication mechanism for one table when a shared HA database already exists at central. |
| Keys as `${secret:}` entries in the secrets hub | Verification needs `last_used`, revoke/rotate/scope semantics and an audit log — a key/value secret store is the wrong shape. |

## Library change — `ZB.MOM.WW.Auth` 0.2.1 → **0.3.0** (`~/Desktop/scadaproj/ZB.MOM.WW.Auth`)

Additive for every existing consumer; the four existing packages bump in lockstep and a fifth is added.

1. **`Abstractions`** — new `IApiKeyStoreMigrator { Task MigrateAsync(CancellationToken ct); }`
   (same shape as Secrets' `ISecretsStoreMigrator`).
2. **`ApiKeys`** — `SqliteAuthStoreMigrator : IApiKeyStoreMigrator`; `AddZbApiKeyAuth` registers
   `IApiKeyStoreMigrator` with `TryAdd` (SQLite) so a provider registered *before* it displaces every
   store — the exact mechanism `AddZbSecretsSqlServerStore` uses; `ApiKeyMigrationHostedService` and
   `ApiKeyAdminCommands` depend on the interface. `ApiKeyAdminCommands` **keeps** its current ctor
   (delegating), so MxAccessGateway/HistorianGateway/ScadaBridge compile unchanged at 0.3.0.
3. **New package `ZB.MOM.WW.Auth.ApiKeys.SqlServer`** (references `ApiKeys` for `ScopeSerializer`
   and the hasher; adds `Microsoft.Data.SqlClient` 6.0.2, the family-pinned version):
   - `SqlServerApiKeyOptions { ConnectionString (required), Schema = "zbauth" }`, validated at start.
   - `SqlServerAuthSchema` — T-SQL for `[zbauth].[schema_version]`, `[zbauth].[api_keys]`,
     `[zbauth].[api_key_audit]` and the two indexes (see plan Task 3 for the DDL). Column types map
     the SQLite schema one-to-one: `key_id nvarchar(64) PK`, `secret_hash varbinary(64)`
     (HMAC-SHA256 = 32 bytes), `scopes nvarchar(max)` in the existing `ScopeSerializer` format,
     `*_utc datetimeoffset(7)`.
   - `SqlServerApiKeyStoreMigrator : IApiKeyStoreMigrator` — idempotent `IF NOT EXISTS` DDL with the
     **same retry set Secrets learned live on the central pair's first boot** (deadlock 1205, lost
     CREATE 2714/1913, PK 2627): both central nodes run it at every startup, concurrently.
   - `SqlServerApiKeyStore`, `SqlServerApiKeyAdminStore`, `SqlServerApiKeyAuditStore` — parameterised
     `SqlCommand` ports of the SQLite stores, same semantics (`FindActiveByKeyId` = not revoked;
     `SetEnabled(false)` = revoke; `Rotate` preserves expiry; `List` ordered by `created_utc`).
   - `SqliteApiKeyStoreImporter.ImportAsync(sqlitePath, ct)` — one-shot copy of `api_keys` and
     `api_key_audit` into SQL Server, **insert-where-not-exists on `key_id`** (audit rows keyed by
     `(key_id, event_type, created_utc)`), returns counts. Idempotent, so both nodes of a pair may run
     it and the result is the **union** of their files.
   - `AddZbApiKeyAuthSqlServer(config, apiKeySectionPath, sqlServerSectionPath)` — registers the SQL
     implementations, then calls `AddZbApiKeyAuth`.
   - Tests: contract tests shared with SQLite via a small abstract fixture, run against a live SQL
     Server only when `AUTH_SQLSERVER_CONNSTR` is set (`Skip.IfNot`, as Secrets does with
     `SECRETS_SQLSERVER_CONNSTR`); the rig's `infra/` MSSQL serves.

## ScadaBridge change

- **Config** (`ScadaBridge:InboundApi:ApiKeyStore`): new `Provider` = `Sqlite` (default) | `SqlServer`;
  `SqlServer:ConnectionString` **defaults to `ScadaBridge:Database:ConfigurationDb`** when empty (no
  new secret to distribute — the pair already shares that database); `SqlServer:Schema` = `zbauth`;
  `ImportFromSqlitePath` defaults to the current `SqlitePath` default.
- **Host composition** (`Program.cs`, central branch): `Provider == SqlServer` →
  `AddZbApiKeyAuthSqlServer`, else `AddZbApiKeyAuth` as today. `ApiKeyAdminCommands` is composed over
  `IApiKeyStoreMigrator`. A new hosted service `InboundApiKeyStoreImportService` runs **after** the
  library migration: if `Provider == SqlServer` and the SQLite file exists, import it and rename it
  `.imported` (the `SiteLocalDbLegacyMigrator` pattern — run once, keep the original).
- **`StartupValidator`**: a central node whose Akka seed list names more than one node and whose
  `Provider` is `Sqlite` logs a **Warning** naming this design (availability-first, matching the
  auto-down decision — not a hard fail, so an un-reconfigured QA still boots).
- **Docker rig**: both centrals get `ScadaBridge__InboundApi__ApiKeyStore__Provider=SqlServer`; the
  connection string falls back to the shared `ScadaBridgeConfig` database on `scadabridge-mssql`.
  `docker/sync-inbound-api-keys.sh` is **deleted** once `verify-delmia-notifier.sh` passes without it.
- **Windows installs**: appsettings are user-held. QA (`vq03`/`vq04`): stop both centrals **together**,
  add `Provider`, start both — the first to boot creates `zbauth` and imports its file, the second
  imports its own (union). Prod `vd03`: same change for consistency (single node, import once). If the
  app login lacks DDL rights, a DBA runs `docs/operations/sql/zbauth-schema.sql` first; the migrator
  finds the version row and skips DDL. Recorded per environment in a deployment record.

## Behaviour after the change

| Case | Before | After |
|---|---|---|
| Key created on active, active crashes | survivor: 401 for every caller | survivor verifies against the same rows: 200 |
| Key revoked on one node | still valid on the peer | revoked everywhere immediately |
| `last_used_utc` | per node | one value, whichever node verified last |
| SQL Server unreachable | inbound auth unaffected (local file) | inbound auth fails **closed** (401/500) — same blast radius as the config DB today, since method scripts already need it |
| Single-node install with `Provider=Sqlite` | unchanged | unchanged |

## Out of scope

- Site nodes (they host no inbound API). MxAccessGateway/HistorianGateway stay on SQLite.
- Encrypting `secret_hash` at rest (it is a peppered HMAC; the pepper stays in config, never in SQL).
- Retiring the per-node SQLite code path from the library.