ScadaBridge/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md

Design — Shared SQL Server store for inbound API keys (central pair)

Date: 2026-09-02 · Status: Proposed · Register: row 108 (docs/plans/2026-07-08-deferred-work-register.md) · Plan: 2026-09-02-inbound-api-key-shared-store.md

Problem

Inbound API keys (POST /api/{methodName}, Bearer sbk_<keyId>_<secret>) are stored by the shared ZB.MOM.WW.Auth.ApiKeys library in a per-node SQLite file (ScadaBridge:InboundApi:ApiKeyStore:SqlitePath, /app/data/inbound-api-keys.sqlite on the rig, app\data\inbound-api-keys.sqlite on the Windows installs). Key admin runs on whichever central node serves the management call — through Traefik that is always the active node — so a key exists on exactly one node. Consequences, all observed or verified on 2026-09-02:

• After an active-central crash the survivor answers 401 KeyNotFound to every inbound client (docker/verify-delmia-notifier.sh, first run). The rig now needs docker/sync-inbound-api-keys.sh to copy the file by hand before a failover drill can pass.
• A key revoked on one node stays valid on the other. Revocation is not cluster-wide.
• The standby-relay path only works because the standby forwards before authenticating.
• Prod wonder-app-vd03 is single-node and unaffected; the QA pair vq03/vq04 has the gap today.
• Component-InboundAPI.md §Storage still says keys live in the configuration database — stale since the 2026-06-02 re-arch (RetireInboundApiKeyStore dropped the SQL table and moved keys to SQLite).

The library ships only a SQLite store (Sqlite/SqliteApiKeyStore, SqliteApiKeyAdminStore, SqliteApiKeyAuditStore, SqliteAuthStoreMigrator). There is no provider to switch to. This is not the secrets store: ZB.MOM.WW.Secrets already has a shared SQL Server store for central (Secrets 0.5.0, ZbSecretsHub), and ${secret:} values are unaffected.

Decision

Move central's inbound key store to the shared SQL Server (the configuration database, own schema zbauth) by adding a SQL Server provider to the shared library, mirroring what Secrets 0.5.0 did. SQLite stays the library default and the right answer for single-process consumers (MxAccessGateway, HistorianGateway, a single-node ScadaBridge). Every reader and writer of a ScadaBridge central pair then sees one row per key: create, revoke, rotate, scope edits and last_used are cluster-wide by construction, and there is nothing to replicate or reconcile.

Alternatives rejected:

Option	Why not
Replicate the SQLite file (rig script, robocopy, LocalDb CDC)	Copy-then-restart is operator-driven and still per-node between copies; LocalDb is the site replication engine and central does not host it; a mid-copy revoke is lost.
Shared SQLite on a network share	SQLite over SMB/UNC is unsafe under concurrent writers; QA nodes are separate boxes.
Central-to-central Akka replication of key rows	New replication mechanism for one table when a shared HA database already exists at central.
Keys as ${secret:} entries in the secrets hub	Verification needs last_used, revoke/rotate/scope semantics and an audit log — a key/value secret store is the wrong shape.

Library change — ZB.MOM.WW.Auth 0.2.1 → 0.3.0 (~/Desktop/scadaproj/ZB.MOM.WW.Auth)

Additive for every existing consumer; the four existing packages bump in lockstep and a fifth is added.

1. Abstractions — new IApiKeyStoreMigrator { Task MigrateAsync(CancellationToken ct); } (same shape as Secrets' ISecretsStoreMigrator).
2. ApiKeys — SqliteAuthStoreMigrator : IApiKeyStoreMigrator; AddZbApiKeyAuth registers IApiKeyStoreMigrator with TryAdd (SQLite) so a provider registered before it displaces every store — the exact mechanism AddZbSecretsSqlServerStore uses; ApiKeyMigrationHostedService and ApiKeyAdminCommands depend on the interface. ApiKeyAdminCommands keeps its current ctor (delegating), so MxAccessGateway/HistorianGateway/ScadaBridge compile unchanged at 0.3.0.
3. New package ZB.MOM.WW.Auth.ApiKeys.SqlServer (references ApiKeys for ScopeSerializer and the hasher; adds Microsoft.Data.SqlClient 6.0.2, the family-pinned version):
   • SqlServerApiKeyOptions { ConnectionString (required), Schema = "zbauth" }, validated at start.
   • SqlServerAuthSchema — T-SQL for [zbauth].[schema_version], [zbauth].[api_keys], [zbauth].[api_key_audit] and the two indexes (see plan Task 3 for the DDL). Column types map the SQLite schema one-to-one: key_id nvarchar(64) PK, secret_hash varbinary(64) (HMAC-SHA256 = 32 bytes), scopes nvarchar(max) in the existing ScopeSerializer format, *_utc datetimeoffset(7).
   • SqlServerApiKeyStoreMigrator : IApiKeyStoreMigrator — idempotent IF NOT EXISTS DDL with the same retry set Secrets learned live on the central pair's first boot (deadlock 1205, lost CREATE 2714/1913, PK 2627): both central nodes run it at every startup, concurrently.
   • SqlServerApiKeyStore, SqlServerApiKeyAdminStore, SqlServerApiKeyAuditStore — parameterised SqlCommand ports of the SQLite stores, same semantics (FindActiveByKeyId = not revoked; SetEnabled(false) = revoke; Rotate preserves expiry; List ordered by created_utc).
   • SqliteApiKeyStoreImporter.ImportAsync(sqlitePath, ct) — one-shot copy of api_keys and api_key_audit into SQL Server, insert-where-not-exists on key_id (audit rows keyed by (key_id, event_type, created_utc)), returns counts. Idempotent, so both nodes of a pair may run it and the result is the union of their files.
   • AddZbApiKeyAuthSqlServer(config, apiKeySectionPath, sqlServerSectionPath) — registers the SQL implementations, then calls AddZbApiKeyAuth.
   • Tests: contract tests shared with SQLite via a small abstract fixture, run against a live SQL Server only when AUTH_SQLSERVER_CONNSTR is set (Skip.IfNot, as Secrets does with SECRETS_SQLSERVER_CONNSTR); the rig's infra/ MSSQL serves.

ScadaBridge change

• Config (ScadaBridge:InboundApi:ApiKeyStore): new Provider = Sqlite (default) | SqlServer; SqlServer:ConnectionString defaults to ScadaBridge:Database:ConfigurationDb when empty (no new secret to distribute — the pair already shares that database); SqlServer:Schema = zbauth; ImportFromSqlitePath defaults to the current SqlitePath default.
• Host composition (Program.cs, central branch): Provider == SqlServer → AddZbApiKeyAuthSqlServer, else AddZbApiKeyAuth as today. ApiKeyAdminCommands is composed over IApiKeyStoreMigrator. A new hosted service InboundApiKeyStoreImportService runs after the library migration: if Provider == SqlServer and the SQLite file exists, import it and rename it .imported (the SiteLocalDbLegacyMigrator pattern — run once, keep the original).
• StartupValidator: a central node whose Akka seed list names more than one node and whose Provider is Sqlite logs a Warning naming this design (availability-first, matching the auto-down decision — not a hard fail, so an un-reconfigured QA still boots).
• Docker rig: both centrals get ScadaBridge__InboundApi__ApiKeyStore__Provider=SqlServer; the connection string falls back to the shared ScadaBridgeConfig database on scadabridge-mssql. docker/sync-inbound-api-keys.sh is deleted once verify-delmia-notifier.sh passes without it.
• Windows installs: appsettings are user-held. QA (vq03/vq04): stop both centrals together, add Provider, start both — the first to boot creates zbauth and imports its file, the second imports its own (union). Prod vd03: same change for consistency (single node, import once). If the app login lacks DDL rights, a DBA runs docs/operations/sql/zbauth-schema.sql first; the migrator finds the version row and skips DDL. Recorded per environment in a deployment record.

Behaviour after the change

Case	Before	After
Key created on active, active crashes	survivor: 401 for every caller	survivor verifies against the same rows: 200
Key revoked on one node	still valid on the peer	revoked everywhere immediately
last_used_utc	per node	one value, whichever node verified last
SQL Server unreachable	inbound auth unaffected (local file)	inbound auth fails closed (401/500) — same blast radius as the config DB today, since method scripts already need it
Single-node install with Provider=Sqlite	unchanged	unchanged

Out of scope

• Site nodes (they host no inbound API). MxAccessGateway/HistorianGateway stay on SQLite.
• Encrypting secret_hash at rest (it is a peppered HMAC; the pepper stays in config, never in SQL).
• Retiring the per-node SQLite code path from the library.