fix(localdb): sync endpoint enforces the configured ApiKey (scadaproj#36)

LocalDbSyncService never read request headers, so setting
LocalDb:Replication:ApiKey on both nodes left the passive endpoint open
(LocalDb-001). It now requires Authorization: Bearer <key> (constant-time)
before the handshake and before newest-wins takeover, refusing anything else
with Unauthenticated. A blank key keeps the old accept-all behaviour and
logs a startup warning. Wire-compatible: the initiator has always sent the
header when the key is set; ScadaBridge/OtOpcUa interceptors use the same key.

LocalDb 0.3.5 -> 0.3.6 (unpublished).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 14:49:20 -05:00
parent d549a782fd
commit 48813bba6f
8 changed files with 228 additions and 18 deletions
+3
View File
@@ -6,6 +6,9 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA
**`0.3.5` (unpublished, scadaproj#26):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change.
**`0.3.6` (unpublished, scadaproj#36–#39):** code-review fixes, all wire-compatible (no proto, `lib_schema_version` or on-disk schema change).
- **#36 LocalDb-001:** the passive `LocalDbSyncService` now enforces `LocalDb:Replication:ApiKey` itself (Bearer, constant-time, checked before the handshake and before newest-wins takeover; `Unauthenticated` otherwise). Blank key = unchanged behaviour plus a startup warning. ScadaBridge's and OtOpcUa's own `LocalDbSyncAuthInterceptor`s use the same key and header, so they keep working; they are now redundant except for their fail-closed-when-blank rule.
## Index history (migrated from the scadaproj root CLAUDE.md)
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+1 -1
View File
@@ -5,7 +5,7 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>latest</LangVersion>
<Version>0.3.5</Version>
<Version>0.3.6</Version>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<!-- NU1510 (package-pruning advice): the Replication and Tests projects carry a
+15 -5
View File
@@ -232,7 +232,7 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
| Key | Default | Meaning |
|---|---|---|
| `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). |
| `ApiKey` | `null` | Bearer token the initiator presents on its channel; `null` ⇒ no key header. |
| `ApiKey` | `null` | Shared pair key — set the **same** value on both nodes. The initiator sends it as `Authorization: Bearer <key>`; since 0.3.6 the passive endpoint **requires** it and refuses any other stream with `Unauthenticated`. Blank ⇒ no header sent and no check (any caller is accepted; a startup warning is logged). |
| `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. |
| `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
@@ -380,10 +380,20 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
- The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
rejects.
- **Peer auth**: `ApiKey` is sent by the initiator as a static `Authorization: Bearer …`
header on its channel. The library's passive service does **not** itself verify the key
— the passive host enforces whatever gRPC auth the host application configures (e.g.
`ZB.MOM.WW.Auth`). Configure host-side authorization to actually gate inbound streams.
- **Peer auth** (0.3.6+): with `ApiKey` set, the initiator sends `Authorization: Bearer <key>`
and the passive `LocalDbSyncService` verifies it (constant-time) **before** the handshake and
before it displaces the current stream — a caller without the key gets `Unauthenticated`, sees
no table names or schema digests, and cannot evict the real peer. With `ApiKey` blank on a
node, that node's endpoint accepts any caller (exactly the pre-0.3.6 behaviour) and logs a
warning at startup; protect it by network isolation or a host interceptor. Up to 0.3.5 the
library never checked the key, so ScadaBridge and OtOpcUa each added a fail-closed
`LocalDbSyncAuthInterceptor` over the same key — those keep working unchanged and are now
redundant with the library check (they still add fail-closed-when-blank, which the library
deliberately does not, so a key-less pair keeps syncing).
- **Rollout:** the wire is unchanged (the initiator has always sent the header when the key
is set). Set the key on both nodes; a node with the key refuses a peer without it, so a
pair with mismatched keys stops syncing until both agree. A pair where neither node has a
key is unaffected.
## Testing
@@ -1,3 +1,5 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
@@ -57,6 +59,13 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
_logger = loggerFactory.CreateLogger<LocalDbSyncService>();
_metrics = metrics;
_status = status as SyncStatus;
if (string.IsNullOrWhiteSpace(_options.ApiKey))
{
_logger.LogWarning(
"LocalDb sync endpoint has no LocalDb:Replication:ApiKey: it accepts any caller that can reach it " +
"unless the host adds its own authentication. Set the same key on both nodes of the pair.");
}
}
/// <summary>
@@ -66,8 +75,8 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
/// <para>
/// <b>Newest wins.</b> The guard exists to SERIALIZE sessions — a node syncs with exactly one
/// peer, and two sessions against one database would fight over the same watermarks — not to
/// fence out an impostor; authentication is the interceptor's job, and an authenticated caller
/// on this key IS the peer by definition. Refusing the newcomer therefore protected nothing and
/// fence out an impostor; authentication happens before the guard (<see cref="IsAuthorized"/>, plus any
/// host interceptor), and an authenticated caller on this key IS the peer by definition. Refusing the newcomer therefore protected nothing and
/// cost everything in the case that actually happens: the dialling node crashes or is
/// partitioned, this handler stays parked forever inside <c>MoveNext</c> on a half-open stream
/// that will never produce another byte, and every reconnect from the rebooted peer is rejected
@@ -87,6 +96,14 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
IServerStreamWriter<SyncMessage> responseStream,
ServerCallContext context)
{
// Authenticate BEFORE the takeover: an unauthenticated caller must not be able to displace
// the real peer's stream, nor see this node's Handshake (table names + schema digests).
if (!IsAuthorized(_options.ApiKey, context.RequestHeaders, out var reason))
{
_logger.LogWarning("Rejected a LocalDb sync stream from {Peer}: {Reason}.", context.Peer, reason);
throw new RpcException(new Status(StatusCode.Unauthenticated, "LocalDb sync authentication failed."));
}
// The linked source exists so the reader pump is deterministically torn down on EVERY exit
// path — including a session fault for a local (non-transport) reason, where the call token
// never fires and the pump would otherwise keep reading a stream whose handler has exited.
@@ -145,6 +162,50 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
}
}
/// <summary>
/// The library's own sync-endpoint authentication (LocalDb-001). With no
/// <see cref="ReplicationOptions.ApiKey"/> configured (null, empty or whitespace) every caller is
/// admitted — the pre-0.3.6 behaviour, so a pair with no key keeps working unchanged. With a key
/// configured the call must carry <c>Authorization: Bearer &lt;key&gt;</c> — exactly what
/// <see cref="SyncBackgroundService"/> has always sent when the same key is configured — compared
/// in constant time over UTF-8 bytes.
/// </summary>
internal static bool IsAuthorized(string? expectedKey, Metadata headers, out string reason)
{
reason = "";
if (string.IsNullOrWhiteSpace(expectedKey))
return true;
string? presented = null;
foreach (var entry in headers)
{
// gRPC lowercases header names on the wire; compare case-insensitively anyway.
if (entry.IsBinary || !string.Equals(entry.Key, "authorization", StringComparison.OrdinalIgnoreCase))
continue;
if (entry.Value is { } value && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
{
presented = value[BearerPrefix.Length..];
break;
}
}
if (presented is null)
{
reason = "no bearer token presented";
return false;
}
if (!CryptographicOperations.FixedTimeEquals(Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expectedKey)))
{
reason = "bearer token did not match";
return false;
}
return true;
}
private const string BearerPrefix = "Bearer ";
/// <summary>
/// Claims the single-stream guard for <paramref name="mine"/>, cancelling and awaiting whatever
/// held it. Throws <c>AlreadyExists</c> only when the incumbent refuses to end.
@@ -8,7 +8,15 @@ public sealed class ReplicationOptions
/// <summary>gRPC address of the peer to dial. Required for the initiator role (validated in the sync host).</summary>
public string PeerAddress { get; set; } = "";
/// <summary>API key presented to / expected from the peer; null disables key auth.</summary>
/// <summary>
/// Shared key that authenticates the pair partner. Set the SAME value on both nodes. The initiator
/// sends it as <c>Authorization: Bearer &lt;key&gt;</c>; since 0.3.6 the passive
/// <see cref="LocalDbSyncService"/> also requires it (constant-time compare) and refuses any other
/// stream with <c>Unauthenticated</c>, before the handshake and before displacing the current
/// stream. Null, empty or whitespace disables key auth on both sides: the initiator sends no
/// header and the endpoint accepts any caller (a warning is logged at startup), so protect it some
/// other way — network isolation or a host interceptor.
/// </summary>
public string? ApiKey { get; set; }
/// <summary>How often the change pump flushes pending oplog deltas to the peer.</summary>
@@ -156,6 +156,119 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
await bg.StopAsync(CancellationToken.None);
}
// ---- LocalDb-001 (scadaproj#36): the passive endpoint enforces ApiKey ------------------
[Fact]
public async Task ApiKeyConfigured_BothSides_TwoNodesConverge()
{
var serverHost = await BuildServerAsync(NewDbPath(), apiKey: "pair-key", sink: null);
var serverDb = serverHost.Services.GetRequiredService<ILocalDb>();
await serverDb.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2, 'S', 20)");
var handler = serverHost.GetTestServer().CreateHandler();
var (clientProvider, bg) = BuildClient(NewDbPath(), "http://localhost", apiKey: "pair-key", () => ChannelOver(handler));
var clientDb = clientProvider.GetRequiredService<ILocalDb>();
await clientDb.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'C', 10)");
using var cts = new CancellationTokenSource(RunTimeout);
await bg.StartAsync(cts.Token);
await WaitForAsync(
async () => (await ReadOrders(clientDb)).Count == 2 && (await ReadOrders(serverDb)).Count == 2,
RunTimeout);
await bg.StopAsync(CancellationToken.None);
}
[Theory]
[InlineData(null)]
[InlineData("Bearer wrong-key")]
[InlineData("pair-key")] // no Bearer scheme
[InlineData("Bearer pair-key-and-more")]
public async Task ApiKeyConfigured_StreamWithoutTheKey_IsRefused_BeforeTheHandshake(string? authorization)
{
var serverHost = await BuildServerAsync(NewDbPath(), apiKey: "pair-key", sink: null);
var handler = serverHost.GetTestServer().CreateHandler();
var client = new LocalDbSync.LocalDbSyncClient(ChannelOver(handler));
var headers = new Metadata();
if (authorization is not null)
headers.Add("Authorization", authorization);
using var cts = new CancellationTokenSource(RunTimeout);
using var call = client.Sync(headers, cancellationToken: cts.Token);
var ex = await Assert.ThrowsAsync<RpcException>(() => call.ResponseStream.MoveNext(cts.Token));
Assert.Equal(StatusCode.Unauthenticated, ex.StatusCode);
}
[Fact]
public async Task ApiKeyConfigured_StreamWithTheKey_GetsTheHandshake()
{
var serverHost = await BuildServerAsync(NewDbPath(), apiKey: "pair-key", sink: null);
var client = new LocalDbSync.LocalDbSyncClient(ChannelOver(serverHost.GetTestServer().CreateHandler()));
using var cts = new CancellationTokenSource(RunTimeout);
using var call = client.Sync(new Metadata { { "Authorization", "Bearer pair-key" } }, cancellationToken: cts.Token);
Assert.True(await call.ResponseStream.MoveNext(cts.Token));
Assert.Equal(SyncMessage.MsgOneofCase.Handshake, call.ResponseStream.Current.MsgCase);
}
[Fact]
public async Task ApiKeyConfigured_UnauthenticatedStream_DoesNotDisplaceThePeer()
{
var serverHost = await BuildServerAsync(NewDbPath(), apiKey: "pair-key", sink: null);
var serverDb = serverHost.Services.GetRequiredService<ILocalDb>();
var handler = serverHost.GetTestServer().CreateHandler();
var (clientProvider, bg) = BuildClient(NewDbPath(), "http://localhost", apiKey: "pair-key", () => ChannelOver(handler));
var clientDb = clientProvider.GetRequiredService<ILocalDb>();
await clientDb.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'C', 10)");
using var cts = new CancellationTokenSource(RunTimeout);
await bg.StartAsync(cts.Token);
await WaitForAsync(async () => (await ReadOrders(serverDb)).Count == 1, RunTimeout);
var attemptsBefore = bg.ConnectionAttempts;
var intruder = new LocalDbSync.LocalDbSyncClient(ChannelOver(handler));
using var call = intruder.Sync(cancellationToken: cts.Token);
var ex = await Assert.ThrowsAsync<RpcException>(() => call.ResponseStream.MoveNext(cts.Token));
Assert.Equal(StatusCode.Unauthenticated, ex.StatusCode);
// The real peer's stream is untouched: it neither redials nor stops replicating.
await clientDb.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (3, 'C2', 30)");
await WaitForAsync(async () => (await ReadOrders(serverDb)).Count == 2, RunTimeout);
Assert.Equal(attemptsBefore, bg.ConnectionAttempts);
await bg.StopAsync(CancellationToken.None);
}
[Fact]
public async Task NoApiKeyOnServer_ClientKeyIsIgnored_Unchanged()
{
// Mixed config during a rollout: the key is set on the initiator first. The passive node has
// none, so it keeps accepting — exactly the pre-0.3.6 behaviour.
var serverHost = await BuildServerAsync(NewDbPath(), apiKey: null, sink: null);
var serverDb = serverHost.Services.GetRequiredService<ILocalDb>();
var handler = serverHost.GetTestServer().CreateHandler();
var (clientProvider, bg) = BuildClient(NewDbPath(), "http://localhost", apiKey: "pair-key", () => ChannelOver(handler));
var clientDb = clientProvider.GetRequiredService<ILocalDb>();
await clientDb.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'C', 10)");
using var cts = new CancellationTokenSource(RunTimeout);
await bg.StartAsync(cts.Token);
await WaitForAsync(async () => (await ReadOrders(serverDb)).Count == 1, RunTimeout);
await bg.StopAsync(CancellationToken.None);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void IsAuthorized_NoConfiguredKey_AdmitsEveryCaller(string? key)
{
Assert.True(LocalDbSyncService.IsAuthorized(key, new Metadata(), out _));
}
// Shipped variant: in-proc against GrpcSyncDuplex with a flow-control-stalled fake writer
// (WriteAsync never completes until its token fires) — pins requirement 2 (Send honors its
// CancellationToken even when transport-blocked) deterministically; a true stalled-gRPC-peer
+19 -3
View File
@@ -10,7 +10,7 @@
| Last reviewed | 2026-11-09 |
| Reviewer | Claude |
| Commit reviewed | `91fb2b4` |
| Open findings | 7 |
| Open findings | 6 |
## Summary
@@ -73,7 +73,7 @@ and #26 (Microsoft.Data.Sqlite pool bump) are excluded.
| Severity | Medium |
| Priority | P2 |
| Category | Security & secret handling |
| Status | Open |
| Status | Resolved |
| Location | `ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/LocalDbSyncService.cs:85`, `ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/ReplicationOptions.cs:11`, `ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/DependencyInjection/ReplicationServiceCollectionExtensions.cs:77` |
**Description**
@@ -125,7 +125,23 @@ the `ApiKey` XML doc either way. Both consumers could then delete their copies.
**Resolution**
_Unresolved._
Resolved 2026-11-09 (scadaproj#36, LocalDb `0.3.6`, unpublished). Re-verified: at `91fb2b4` nothing in the library reads
request headers. `LocalDbSyncService.Sync` now calls `IsAuthorized` before the takeover and before the handshake. When
`ApiKey` is set, the stream must carry `Authorization: Bearer <key>`, compared with `CryptographicOperations.FixedTimeEquals`
over UTF-8; anything else is refused with `Unauthenticated`. The check sits before the takeover, so an unauthenticated
caller can neither read the Handshake (table names and digests) nor displace the real peer.
As the task requires, a blank `ApiKey` keeps the old behaviour (accept), and the service logs a startup warning. That
rules out the review's fail-closed-by-default suggestion: a key-less pair must keep syncing. Wire compatibility holds,
because the initiator has sent this header whenever the key is set since `0.1.0`. The ScadaBridge and OtOpcUa
interceptors check the same key and the same header, so they are unaffected. The `ApiKey` XML doc, the README config
table and the README Ops notes are corrected.
Tests (`GrpcAdapterTests`): two-node convergence with the key on both sides. Missing, wrong, scheme-less and
over-long tokens are each refused before the handshake. A correct token gets the Handshake. An unauthenticated
stream does not displace a live peer: there is no redial, and replication continues. A key on the initiator only
still syncs, and a blank, empty or whitespace key admits every caller. With the check disabled, the 5 refusal and
non-displacement cases fail.
### LocalDb-002 — Snapshot streams tables by name and rows by `pk_json` text, so FK children land before parents and are dead-lettered for good
+5 -6
View File
@@ -40,7 +40,7 @@ code-reviews/
## Summary
10 of 10 libraries reviewed. 23 pending findings across all libraries.
10 of 10 libraries reviewed. 22 pending findings across all libraries.
Coverage: Secrets, LocalDb, GalaxyRepository and Overview first reviewed, and Auth, Health,
Theme, Audit, Configuration and Telemetry re-reviewed, on 2026-11-09 (#15). Each finding also
@@ -50,9 +50,9 @@ carries a P1/P2/P3 priority (P1 = Critical/High, P2 = Medium, P3 = Low).
|----------|---------------|
| Critical | 0 |
| High | 0 |
| Medium | 4 |
| Medium | 3 |
| Low | 19 |
| **Total** | **23** |
| **Total** | **22** |
## Library Status
@@ -63,7 +63,7 @@ carries a P1/P2/P3 priority (P1 = Critical/High, P2 = Medium, P3 = Low).
| [Configuration](Configuration/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 4 |
| [GalaxyRepository](GalaxyRepository/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 8 |
| [Health](Health/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 7 |
| [LocalDb](LocalDb/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/4/3 | 7 | 7 |
| [LocalDb](LocalDb/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/3/3 | 6 | 7 |
| [Overview](Overview/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 5 |
| [Secrets](Secrets/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/13 | 13 | 19 |
| [Telemetry](Telemetry/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 11 |
@@ -84,11 +84,10 @@ _None open._
_None open._
### Medium (4)
### Medium (3)
| ID | Library | Title |
|----|---------|-------|
| LocalDb-001 | [LocalDb](LocalDb/findings.md) | Sync endpoint is unauthenticated by default, and `ApiKey` is documented as "expected from the peer" |
| LocalDb-002 | [LocalDb](LocalDb/findings.md) | Snapshot streams tables by name and rows by `pk_json` text, so FK children land before parents and are dead-lettered for good |
| LocalDb-003 | [LocalDb](LocalDb/findings.md) | `INSERT OR REPLACE` apply silently deletes a different row on a secondary `UNIQUE` conflict, so the pair diverges |
| LocalDb-004 | [LocalDb](LocalDb/findings.md) | `FailClosedOnDrift` is bypassed on the snapshot path, so a future-stamped snapshot row moves the local HLC forward permanently |