dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/scadaproj:f42aa1403359cd92a348721a5a3c292d26c99264
Branches Tags
dohertj2/scadaproj:main dohertj2/scadaproj:low/T5
..
compare: dohertj2/scadaproj:low/T5
Branches Tags
dohertj2/scadaproj:main dohertj2/scadaproj:low/T5

26 Commits
f42aa14033 .. low/T5

Author SHA1 Message Date
Joseph Doherty 41bdc178c2 fix(localdb): refuse cascade-delete FK parents; make one entry's row+ledger writes atomic (0.3.3, ScadaBridge arch-review P4-7) 
Two halves of the same INSERT OR REPLACE hazard in the LWW apply path. Wire-compatible
with every earlier peer: no proto change, no lib_schema_version bump, no on-disk schema
change.

1. RegisterReplicated now REFUSES a table that another table references with a
   row-mutating ON DELETE action (CASCADE / SET NULL / SET DEFAULT). Applying a remote row
   uses INSERT OR REPLACE, whose implicit DELETE fires that referential action — while the
   apply path holds the capture guard raised, precisely so its own writes are not
   re-captured. The child rows it deletes or nulls therefore produce no oplog entry: the
   peer keeps them and the pair diverges with nothing reporting a fault. Registration is
   the only point at which the problem is still visible, so it is refused there with an
   actionable LocalDbRegistrationException naming the referencing table and its action.
   NO ACTION / RESTRICT stay allowed: they mutate no child row, so the implicit delete just
   fails and the applier dead-letters the entry — loud, not silent. Registration-time only;
   a cascading child table created LATER is not caught (SQLite offers no DDL hook).

   No current consumer has a foreign key on a replicated table (ScadaBridge's ten tables
   have none), so this is a guard against a future foot-gun, not a live fix.

2. One entry's row write and its ledger write are now atomic. They were two statements
   under the poison catch, so a failure landing between them committed a deleted/replaced
   row carrying a STALE LIVE row-version entry — exactly the shape SnapshotStreamer warns
   about and skips, so that key never travels again and the pair is permanently divergent
   for it. A per-entry SAVEPOINT makes the pair all-or-nothing while keeping the catch's
   per-ENTRY granularity: entries applied earlier in the same batch survive the rollback,
   the poison entry is still dead-lettered, and the batch still continues. The RELEASE runs
   outside the catch on purpose, so a failure there rolls the whole batch back rather than
   being swallowed as poison. Measured cost on a 100k-row backlog drain: ~7%
   (178.6k rows/s -> 166.1k rows/s).

Tests: both hazards proven against the unfixed code first (FK parent registered silently;
an injected ledger-write abort left the row deleted with a live ledger entry). Adds FK
guard coverage (parent refused, child allowed, SET NULL / SET DEFAULT refused, NO ACTION /
RESTRICT allowed, unrelated cascades ignored) and two applier atomicity tests. The
statement-count lock-in moves 7 -> 9 for the two new savepoint statements.

Version 0.3.2 -> 0.3.3; README limitations + release notes document the constraint.
 2026-08-19 16:22:44 -04:00
Joseph Doherty e9dc13689a docs(index): LocalDb 0.3.2 published — cap-prune session fault, sync keepalives, newest-wins stream guard (ScadaBridge arch-review P4-1 + P4-3); ScadaBridge pins 0.3.2 2026-08-19 15:06:47 -04:00
Joseph Doherty f62b017cc7 fix(localdb): fault live sync session on cap-prune; keepalive + newest-wins sync guard (0.3.2, ScadaBridge arch-review P4-1 + P4-3) 2026-08-19 15:02:04 -04:00
Joseph Doherty dbcbc5bd41 docs(index): LocalDb 0.3.1 published — handshake-fault control-lane flush (register row 41) 2026-08-18 07:30:59 -04:00
Joseph Doherty 1145df682e docs(index): OtOpcUa now registers LocalDb tables conditionally — baseline on enable, deregister self-heal on disable (row 39 closed) 2026-08-18 07:25:18 -04:00
Joseph Doherty 3e5a5fde81 fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1) 
A session that faulted during the handshake cancelled its writer loop immediately,
which could kill its OWN Handshake while it still sat in the control lane: the
enqueue -> receive -> validate -> fault path has no yields, while the writer's
wakeup waits in the thread-pool queue, so with the peer's handshake already
buffered the cancel won the race almost every time (39/40 in the new pinning
test). Mismatch detection is symmetric — the peer computes its fail-closed
verdict FROM our Handshake — so the healthy peer never learned anything: in
tests it waited out the full 15 s receive cap (the register-row-41 flake on
SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it
would misclassify a permanent schema mismatch as a generic disconnect and
reconnect-loop at the fast backoff instead of taking the classified
FailedPrecondition -> LogError + max-backoff path.

RunAsync's fault teardown now completes both lanes so the writer drains what is
already enqueued and exits on its own, with a 2 s deferred cancel as the backstop
for a wedged transport. Caller-initiated cancellation is unchanged (the linked
token is already cancelled, so shutdown is not delayed), and nothing on the wire
changes — a 0.3.0 peer simply receives the Handshake it was always owed.

New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins
the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green
post-fix).

BatchBudgetTests hardened separately: the register's other flaky test
(Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the
scripted-peer channel without ever observing the session task, so ANY session
fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now
races the session task at every read site in the class, so the next flake reports
the session's real exception instead. (The handshake-race root cause does not
structurally apply to this test — no handshake fault occurs there — so its
intermittent timeout is attributed to the fault-hiding wait, now removed.)

Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
 2026-08-18 07:11:14 -04:00
Joseph Doherty b62981ff80 fix(build): anchor pack/push scripts to their own repo root (0038d46 pattern, all seven) 
Secrets/push.sh, Theme/pack.sh+push.sh, Audit/pack.sh+push.sh and Auth/pack.sh+push.sh
all carried the CWD-pack bug LocalDb's push.sh was cured of in 0038d46: a bare
'dotnet pack -c Release -o ./artifacts' packs whatever solution sits in the CALLER's
working directory, and the push scripts then shove the caller's ./artifacts/*.nupkg
onto the live Gitea feed (the incident that briefly published 25 unrelated packages).

Each script now cds to its own repo root, names its .slnx explicitly, writes to an
absolute artifacts path, and (push.sh) pushes an anchored glob. Verified by bash -n on
all seven plus a positive control: Audit's pack.sh run from a foreign CWD packed only
ZB.MOM.WW.Audit into its own artifacts dir and left the caller's directory untouched.
Push scripts deliberately NOT executed (they publish); pack half verified only.

Closes ScadaBridge deferred-work register row 42 (cross-repo).
 2026-08-18 07:00:42 -04:00
Joseph Doherty 31a7c8a88a docs(index): LocalDb consumer pins — OtOpcUa 0.1.3 -> 0.2.1 (R4 merged), skew note updated for 0.3.0 2026-08-15 04:15:28 -04:00
Joseph Doherty 7757b9c6fc Merge branch 'localdb-per-table-snapshot' — LocalDb 0.3.0: per-table needs_snapshot, capability-negotiated partial snapshots (ScadaBridge R1) 2026-08-15 04:15:02 -04:00
Joseph Doherty 0038d46c9a fix(build): push.sh must pack ITSELF, not the caller's working directory 
dotnet pack with no project argument packs whatever solution sits in the caller's
CWD, so invoking this script by absolute path from another repo's checkout packed
and published 25 unrelated packages to the live Gitea feed under its name. Now it
cds to its own repo root and names the solution and an absolute output path.
 2026-08-15 03:36:48 -04:00
Joseph Doherty cea3930b94 feat(localdb): per-table needs_snapshot (0.3.0, on-disk schema v3) 
Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a
snapshot streams every registered table in full — so opting one table into
replication late re-sent all of them, in both directions at once on a pair
enabling replication late. On a database holding a high-volume table (an event
log at its retention cap) that is the largest transfer this library performs.
0.2.1 documented the cost and deferred the fix as needing an on-disk change.

Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created
by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3
transition is a version stamp only. RegisterReplicated(..., baselineExistingRows:
true) now flags that table alone. The database-wide flag survives unchanged for
the paths that genuinely reach every table — a cap prune or an impossible peer ack
loses delta continuity for the oplog as a whole. DeregisterReplicated drops a
table's pending flag with its ledger rows, since nothing would ever stream (and so
clear) it afterwards.

Wire compatibility is additive and negotiated by CAPABILITY, not version:
lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for
equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake
gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope
(field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits
and what a pre-0.3.0 receiver assumes. A peer that does not advertise the
capability is served a full snapshot: a scoped one would make it advance
last_applied_remote_seq to the snapshot's as-of and silently discard every delta
below it for the tables left out.

For the same reason a partial snapshot moves no watermark on either side. The
receiver keeps last_applied_remote_seq (carrying only the HLC across, which is
MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot
overlaps are still pumped and merged by LWW.

Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather
than on enqueue, so an ungraceful exit part-way through re-streams the same scope
on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it
composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot.
Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is
no ack to wait for — with the handshake's horizon gap check as the net.

Tests 199/199 (was 188): single-table baseline streams only that table, mixed-
version interop (peer without the capability gets a full snapshot), debt standing
until SnapshotAck, partial snapshot not advancing the receiver's watermark, no
SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3
upgrade and per-table flag isolation.

README, ILocalDb remarks (which named this as the follow-up) and the umbrella
CLAUDE.md index updated.
 2026-08-15 03:34:54 -04:00
Joseph Doherty f627cee9c0 Merge branch 'localdb-perf-remediation' 2026-08-15 01:46:38 -04:00
Joseph Doherty 14cb283172 docs(index): LocalDb 0.2.1 — HLC anchor on deregister + bounded sync inbox 
The umbrella LocalDb row and the pack/test quick-reference now name 0.2.1 (188 tests,
3 nupkgs on the feed). Records both findings, the snapshot-out-of-handshake change that
bounding the inbox required, and the deferred per-table needs_snapshot. Consumer pins are
unchanged: ScadaBridge still pins 0.2.0, OtOpcUa still pins 0.1.3.
 2026-08-14 23:35:57 -04:00
Joseph Doherty 522cfbfda6 fix(localdb): flush the HLC anchor on deregister, bound the sync inbox (0.2.1) 
Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers:
no proto change, no lib_schema_version bump, no on-disk schema change.

F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock
recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc),
last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the
table's rows from BOTH ledgers. On a file whose registered tables have all been
deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above
zero to recover from: the reopened clock restarts from wall-clock time and can re-issue
stamps below HLCs peers already hold, which LWW discards as older. The current clock is now
flushed inside the deregistration transaction, before the prune, as a separate command so
it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional
transaction (SQLite rejects a command with no transaction on a connection that has one) and
is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower
value can only be stale.

F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what
the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every
registered table, needs_snapshot is per-database, and both sides of a pair that enables
replication late baseline unconditionally — so a mutual full snapshot of a large dataset
(an event log near its storage cap) was buffered whole, in memory, on both nodes at once.
The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is
~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full
inbox stops draining the gRPC stream, which closes the peer's flow-control window. The
pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the
message, and every inbound message is protocol state.

Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node
owes was streamed from INSIDE the handshake, before the receive loop existed, so neither
side drained its inbox while sending; a mutual snapshot converged only because the inbox
was unbounded. The snapshot now runs as the first phase of the outbound task, started
alongside the receive loop, so the inbox is drained for the whole life of the session.
Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the
snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved
(the pump still starts only after the snapshot, from its as-of seq).

Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in
__localdb_peer_state and narrowing it needs an on-disk schema change, which this fix
deliberately avoids. Documented as a follow-up, and the cost is now written down where it
is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that
baselining one table re-streams every registered table in both directions, and that
high-volume tables should be trimmed before enabling replication late.

188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an
ungraceful reopen (a second instance over the same file with no Dispose — fails without the
flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure
under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous
in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that
last one deadlocks and times out against the old handshake-phase snapshot ordering.

Version 0.2.1.
 2026-08-14 23:34:55 -04:00
Joseph Doherty 1ce0c5955f docs: umbrella index — LocalDb 0.2.0 + ScadaBridge consumption 2026-08-14 22:21:41 -04:00
Joseph Doherty 9377fa1678 perf(localdb): dereg API, O(1) backlog depth, byte-budget batching, prepare-once apply (0.2.0) 
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all
wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The
on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened.

DeregisterReplicated(table). There was no way to stop capturing once a file had been
registered, so a node that does not replicate kept paying three triggers inside every
write transaction, appending to an oplog nothing would ever drain. Drops the triggers and
prunes the table's oplog + row-version rows in one transaction; idempotent, and
deliberately usable on a table this process never registered, since stale triggers left by
an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and
already is enforced: the handshake compares registered-table digests fail-closed, so a node
that drops a table its peer still replicates stops syncing loudly rather than diverging.

RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the
snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node
that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding
alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at
the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot,
and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this
node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly
greater, which makes a baselined row lose every comparison against a real remote write and
win only where the peer has no version of that key at all. Two nodes baselining the same key
both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is
available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is
unchanged unless asked. Pruning the ledger on deregister is safe precisely because
re-registration can re-baseline; the two APIs are a pair.

O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a
minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly
when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at
0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k
backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack
pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with
seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a
stalled watermark — without it the caps would re-flag on every tick forever.

Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a
proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge
pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB
gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed
serialized size, with the row count demoted to a secondary cap. A row larger than the budget
is sent alone rather than stalling the stream on it. Since more, smaller messages would
otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY
queued are coalesced — one transaction and one ack per burst, bounded by the sender's own
caps, and a no-op under a trickle.

LwwApplier prepares each statement once. It went through the generic transaction façade,
which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path
that drains a reconnect backlog. Statements are now held for the batch and rebound.

Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever
used — every oplog access path is by seq or a full hlc-predicate delete — while costing a
b-tree insert inside every captured consumer write.

184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves
identity/watermarks/data, deregistration idempotency and trigger absence and cross-process
cleanup, baseline seeding at the floor with LWW safety proved through the real applier,
depth asserted equal to the counting query across every prune scenario plus a constant-time
assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both
the delta and snapshot paths, prepare-once statement counting, and coalescing that must not
swallow the message that ends its run.

Version 0.2.0.
 2026-08-14 22:08:43 -04:00
Joseph Doherty 92246c9734 docs(inventory): full QA §7.1 refresh for the 2026-08-14 VQ04 singleton clone — 10-instance fleet, psbridge :6150 + glauth :3893 port rows, VQ04→VQ03 Galaxy/alarm dependency; CLAUDE.md Overview note updated 2026-08-14 12:39:14 -04:00
Joseph Doherty 1fee9755d7 docs(deployment): VQ04 now runs its own MxGateway/HistorianGateway/Overview (cloned from VQ03 2026-08-14, shared API-key stores); QA access = psbridge :6150, servecli retired 2026-08-14 12:35:56 -04:00
Joseph Doherty 205bddb3b8 docs(plan): flag ignitionoee LDAP as ON HOLD -- needs new UG_WW_* groups minted first 2026-08-13 15:01:15 -04:00
Joseph Doherty 206c588608 docs(plans): VD03 NAM AD LDAP cutover plan (not executed) 2026-08-13 09:24:17 -04:00
Joseph Doherty bd929d98fc docs(ldap): e2e NAM AD login verified via HG docker -- svc_igntadmin, stage diagnostics 2026-08-13 09:17:42 -04:00
Joseph Doherty 990f4c018d docs: HG e42ff31 is on main, not a branch 2026-08-13 08:58:21 -04:00
Joseph Doherty 5da4ab3ebd docs(index): HistorianGateway UI-sweep branch merged to main (e42ff31) — Theme 0.4.1 + Health 0.3.0 + Secrets 0.6.2 + Auth 0.2.1 now on its default 2026-08-13 08:58:09 -04:00
Joseph Doherty 3d2f3f59b2 docs(index): Auth 0.2.1 adoption commits pushed on all four apps 2026-08-13 08:52:25 -04:00
Joseph Doherty 58c2e14c38 docs(index): Auth 0.2.1 adopted by all four apps; FallbackServers live-gated on HG + ScadaBridge rig 2026-08-13 08:49:25 -04:00
Joseph Doherty 58f605609d feat(infra): redundant GLAuth pair for FallbackServers failover drills 
Two local glauth containers (zb-ldap-primary :3893 / zb-ldap-backup :3894)
simulating the NAM two-DC topology, so the primary can be killed during a
drill without touching the shared 10.100.0.35 instance. Drill recipe +
verified-PASS record (HistorianGateway, 2026-08-13) in the README.
 2026-08-13 08:28:05 -04:00
43 changed files with 4575 additions and 263 deletions
Expand all files Collapse all files
CLAUDE.md
+12 -10
View File
File diff suppressed because one or more lines are too long
ZB.MOM.WW.Audit/build/pack.sh
+7 -1
View File
@@ -1,4 +1,10 @@
#!/usr/bin/env bash
# pack.sh — produce the ZB.MOM.WW.Audit NuGet package into ./artifacts.
set -euo pipefail
dotnet pack -c Release -o ./artifacts
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
# neither a project argument nor a fixed working directory packs whatever solution happens
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"
ZB.MOM.WW.Audit/build/push.sh
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
ZB.MOM.WW.Auth/build/pack.sh
+7 -1
View File
@@ -6,4 +6,10 @@
set -euo pipefail
dotnet pack -c Release -o ./artifacts
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
# neither a project argument nor a fixed working directory packs whatever solution happens
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"
ZB.MOM.WW.Auth/build/push.sh
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
ZB.MOM.WW.LocalDb/Directory.Build.props
+1 -1
View File
@@ -5,7 +5,7 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>latest</LangVersion>
<Version>0.1.3</Version>
<Version>0.3.3</Version>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<!-- NU1510 (package-pruning advice): the Replication and Tests projects carry a
ZB.MOM.WW.LocalDb/README.md
+188 -18
View File
@@ -10,16 +10,127 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
last-writer-wins over a hybrid logical clock. All timestamps are UTC.
## Packages (all `0.1.3`)
## Packages (all `0.3.3`)
| Package | Contents |
|---|---|
| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table)` (PK validation + trigger install + oplog schema), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK validation + trigger install + oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). |
| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. |
> Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
>
> - `0.3.3` (2026-08-19) — one Low finding from the ScadaBridge adversarial architecture review
> (P4-7), in two halves. Wire-compatible with every earlier peer: no proto change, no
> `lib_schema_version` bump, no on-disk schema change.
> - **`RegisterReplicated` now refuses a table that another table cascades a DELETE onto.**
> Applying a remote row uses `INSERT OR REPLACE`, whose implicit DELETE fires `ON DELETE
> CASCADE` / `SET NULL` / `SET DEFAULT` on every child — while the apply path holds the capture
> guard raised, so the child rows it touches produce no oplog entry, the peer keeps them, and the
> pair diverges with nothing reporting a fault. Registration is the only place that is visible,
> so it is refused there with an actionable `LocalDbRegistrationException` naming the referencing
> table and its action. `NO ACTION` / `RESTRICT` are still allowed: they mutate no child row (the
> implicit delete simply fails, which the applier dead-letters — loud, not silent). No current
> consumer has a foreign key on a replicated table, so this is a guard against a future foot-gun
> rather than a live fix.
> - **Applying one entry is atomic across its row write and its ledger write.** They were two
> statements under the poison catch, so a failure landing between them committed a
> deleted/replaced row carrying a STALE LIVE ledger version — the shape `SnapshotStreamer` warns
> about and skips, meaning that key never travels again. A per-entry `SAVEPOINT` makes the pair
> all-or-nothing while keeping the catch's per-ENTRY granularity: entries applied earlier in the
> same batch survive, and the poison entry is still dead-lettered and the batch still continues.
> - `0.3.2` (2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review
> (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, no `lib_schema_version`
> bump, no on-disk schema change — the keepalive is transport-level and the rest is local policy.
> - **A live session now ends itself when a cap prune eats deltas it has not sent.** The pump reads
> `seq > sentThruSeq`, which resumes SILENTLY above a pruned hole; the peer acks the later seqs
> and the pair is durably divergent for the whole life of that session. `needs_snapshot` (set in
> the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable
> LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares
> the oplog horizon (`MIN(seq)`, one b-tree seek) against its sent watermark and throws
> `LocalDbOplogGapException` when deltas it still owes are gone. The horizon, not the flag,
> because the flag is also raised for reasons that do not invalidate the running session.
> - **The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins.**
> Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash
> or partition of the dialling node left the passive node's handler parked in `MoveNext` on a
> half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted
> peer's reconnects with `AlreadyExists`, indefinitely on an idle pair. The initiator now dials
> with `KeepAlivePingDelay` / `KeepAlivePingTimeout` (new options, 60s/20s, `Always` policy so
> idle connections ping too), and a fresh authenticated `Sync` cancels the incumbent and waits
> for it to unwind before starting — the guard serializes sessions, it was never a fence, and an
> authenticated same-key dialler is by definition the peer. An incumbent that will not yield
> within the takeover timeout still gets `AlreadyExists`, because two live sessions over one
> database is worse than a refused dial.
> - `0.3.1` (2026-08-18) — a faulting session flushes its control lane before cancelling its writer,
> so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to
> reconnect-loop against a generic disconnect.
> - `0.3.0` (2026-08-15) — **the snapshot-resync flag is per-table.** On-disk bookkeeping schema
> goes to **v3** (new `__localdb_snapshot_state`, upgraded in place on open); wire-compatible with
> `0.1.x`/`0.2.x` peers, and `lib_schema_version` deliberately stays `1` because it is compared
> fail-closed for equality.
> - **`RegisterReplicated(table, baselineExistingRows: true)` now flags a resync for THAT TABLE.**
> Through `0.2.x` the flag was one bit for the whole database, so baselining one table
> re-streamed every registered table in full — and on a pair enabling replication late, both
> nodes did that to each other at once. A snapshot now streams only the flagged tables.
> - **The database-wide flag survives, unchanged, for the cases that genuinely need it**: an oplog
> cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of
> every table is sound. What moved is only where a *baseline* records its debt.
> - **Additive wire extension, negotiated by capability, not by version.** `Handshake` gains
> `supports_partial_snapshot`; `SnapshotBegin` gains a `tables` scope where **empty means "all
> tables"** — which is precisely what a pre-`0.3.0` sender emits and what a pre-`0.3.0` receiver
> assumes. A peer that does not advertise the capability is served a **full** snapshot, because a
> scoped one would make it advance `last_applied_remote_seq` past deltas it never received.
> - **A partial snapshot moves no watermark on either side.** The receiver keeps
> `last_applied_remote_seq` (it is one number for the whole peer oplog, and the untouched tables'
> deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The
> overlap re-delivers rows the snapshot already carried, which LWW absorbs.
> - **Snapshot debt now clears on acknowledgement, not on enqueue** (new `SnapshotAck`, exchanged
> only between `0.3.0` peers). A crash part-way through a snapshot re-streams the same scope on
> reconnect instead of losing it. Against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue
> behaviour is kept — there is no ack to wait for.
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
> `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
> - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
> open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was
> written only by graceful disposal — while deregistration deletes the table's oplog AND ledger
> rows. A crash after a deregister boot could therefore recover the clock below stamps the peer
> already holds, and LWW discards those as older. The current clock is now flushed inside the
> deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never
> regress the anchor).
> - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped
> at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a
> mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now
> stops draining the gRPC stream, which closes the peer's flow-control window.
> - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the
> handshake. That is what makes bounding safe: previously neither side drained its inbox while
> sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was
> already safe (snapshot rows and deltas both merge through LWW, the watermark write is
> MAX-guarded) — only the ordering changed.
> - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining
> one table re-streams every registered table, in both directions on a late-enabled pair. Trim
> high-volume tables (event logs) before turning replication on late.
> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
> All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
> the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
> - **`DeregisterReplicated(table)`** — drops a table's capture triggers and prunes its oplog and
> row-version rows. A node that does not replicate should not be paying for capture on every
> write, and until now there was no way to stop once a file had been registered. Idempotent, and
> usable on triggers left behind by an earlier build.
> - **`RegisterReplicated(table, baselineExistingRows: true)`** — closes the late-opt-in hole: rows
> that predate registration were invisible to the peer forever, because the snapshot streamer
> pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at
> an LWW-safe floor (HLC `0`, this node's id) and flags a snapshot resync so they actually
> travel. Default is still off — the documented CDC contract is unchanged unless you ask.
> - **O(1) backlog depth** — `MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)` replaces a
> `COUNT(*)` over the whole unacked backlog that ran once a minute AND on every metrics scrape
> (measured 58 VM steps vs 100,020 on a 50k backlog).
> - **Byte-budget batching** — `MaxBatchBytes` (default 2 MB) bounds delta and snapshot messages by
> summed serialized size, with `MaxBatchSize` demoted to a secondary row cap. Consumers no longer
> have to tune the row count against their widest row to stay under the gRPC message limit.
> Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
> - **`LwwApplier` prepares once per statement** instead of three SQL parses per applied row.
> - **Dropped `__localdb_oplog_hlc`** — an index no query ever used, costing a b-tree insert inside
> every captured consumer write. Existing files lose it on the next open.
> - `0.1.3` (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN
> writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq
> space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's
@@ -107,7 +218,8 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
| `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). |
| `ApiKey` | `null` | Bearer token the initiator presents on its channel; `null` ⇒ no key header. |
| `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. |
| `MaxBatchSize` | `500` | Max oplog entries per outbound batch (also the snapshot page size). |
| `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
| `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
| `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
| `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
@@ -133,8 +245,15 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
delta/snapshot batches), so receiving never blocks on the ability to ack.
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
the snapshot's as-of seq. **Each node decides from its OWN state** whether it owes the
peer a snapshot.
the snapshot's as-of seq. When instead a single table was *baselined*, only that table is
streamed and **no watermark moves on either side** — a partial snapshot speaks for nothing
outside its own tables, so the deltas it overlaps are still pumped and merged by LWW.
**Each node decides from its OWN state** whether it owes the peer a snapshot, and a peer
that does not advertise `supports_partial_snapshot` is sent a full one. The snapshot is
streamed *after* the handshake, alongside a running receive loop, so a node sending one
still applies what the peer sends it — a mutual snapshot exchanges under flow control
instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
and prunes expired tombstones every `MaintenanceInterval`.
@@ -160,14 +279,41 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
`RegisterReplicated`, because `json_object` cannot capture it). A *typeless* column can
still hold a BLOB value at runtime, which then surfaces as a write-time error, not a
registration error.
- **Replicated tables must not be cascade-delete foreign-key parents.** If another table
references the one you are registering with `ON DELETE CASCADE` (or `SET NULL` /
`SET DEFAULT`), `RegisterReplicated` throws. Applying a remote row uses `INSERT OR
REPLACE`, whose implicit DELETE fires that referential action while change capture is
suppressed — so the child rows it deletes or nulls are never captured, the peer keeps
them, and the pair diverges with nothing reporting a fault. Model the parent/child
relationship with `ON DELETE NO ACTION` (or `RESTRICT`) and delete children explicitly, or
leave the parent unreplicated. `NO ACTION` / `RESTRICT` are allowed because they mutate no
child row: the implicit delete just fails, and the applier dead-letters the entry — loud,
not silent. The check runs at **registration time**, so a cascading child table created
*later* is not caught (SQLite offers no DDL hook); register after schema setup, which is
where startup does it anyway.
- **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
and are not snapshotted; only changes after registration are replicated. Re-baselining
pre-existing rows requires re-writing them.
and are not snapshotted; only changes after registration are replicated. **Opt out with
`RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync
**for that table** so they reach the peer. The floor is below every genuine stamp, so a baselined row loses
every comparison against a real remote write and wins only where the peer has no version
of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
the node-id tie-break — convergent, but arbitrary as to which content survives.
- **A snapshot still ships whole tables.** The flag is per-table as of `0.3.0`, so baselining
one table streams that table and no other — but *that* table travels in full, and two
cases still stream **everything**: an oplog cap prune / impossible peer ack (delta
continuity is lost for the oplog as a whole), and any peer older than `0.3.0` (which
cannot be told a snapshot is partial). So **purge or trim high-volume tables such as event
logs before baselining them**, and expect one slow first sync. Bounded either way (batch
budget out, bounded inbox in).
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
SQLite↔SQLite peers this library targets).
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
- A **mutual simultaneous snapshot** buffers the peer's inbound snapshot in memory while
this node is still sending its own (acceptable for the small datasets this lib targets).
- **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so
peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far
less in practice — messages are only that large when the sender has that much backlog).
Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being
drained and the peer is slowed by flow control.
## Observability
@@ -196,8 +342,28 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
## Ops notes
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags
`needs_snapshot` (disk safety over delta continuity); the peer then snapshot-resyncs.
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
**database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
baselining writes.
- **Upgrading a pair to `0.3.0`** needs no coordination: the two nodes negotiate scoped snapshots
by capability, so a `0.3.0` node paired with a `0.2.x` one simply keeps sending full snapshots
until both are upgraded. Rolling the file forward is one-way — a `0.2.x` build reopening a v3
file leaves `__localdb_snapshot_state` in place and ignores it, so any per-table debt recorded
by `0.3.0` is invisible to it (roll forward again, or re-baseline).
- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
registered-table digests fail-closed, so a node that drops a table its peer still
replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
accumulated while capture was off stay invisible to the peer — but trim high-volume
tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's
durable anchor is flushed in the same transaction, so a crash after a deregister boot
cannot bring the clock back below stamps peers already hold.
- **On-disk schema upgrades** are applied in place when the file is opened, in one
transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
keeps syncing.
- The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
rejects.
@@ -208,13 +374,17 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
## Testing
144 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, trigger SQL generation, LWW decision table, watermark/pruning, digest
computation), integration over two real SQLite DBs on an in-memory (and real) gRPC pipe
(bidirectional convergence, delete/update races, offline-accumulate → reconnect,
prune → snapshot resync), plus seeded randomized property tests (random interleaved op
sequences → both DBs identical after quiesce). Family conventions apply
(`TreatWarningsAsErrors`).
199 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
generation, LWW decision table, watermark/pruning, digest computation, schema-version
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
snapshot over a capacity-1 transport, single-table baseline streaming only that table,
mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
Family conventions apply (`TreatWarningsAsErrors`).
Build / test from this directory:
ZB.MOM.WW.LocalDb/build/push.sh
+12 -2
View File
@@ -16,9 +16,19 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked as `bash /path/to/ZB.MOM.WW.LocalDb/build/push.sh` from another
# repo's checkout, the relative form packed and published 25 unrelated packages (recovered
# with `dotnet nuget delete`, but only because the token happened to carry delete rights).
# A publish script must be unambiguous about WHAT it publishes no matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Contracts/Protos/localdb_sync.proto
+19 -1
View File
@@ -16,6 +16,7 @@ message SyncMessage {
SnapshotBegin snapshot_begin = 5; // sender switches to snapshot mode
SnapshotBatch snapshot_batch = 6;
SnapshotComplete snapshot_complete = 7;
SnapshotAck snapshot_ack = 8; // 0.3.0+; only sent when the peer advertised support
}
}
@@ -24,6 +25,13 @@ message Handshake {
uint32 lib_schema_version = 2;
repeated TableDigest tables = 3; // fail-closed on any mismatch
int64 last_applied_remote_seq = 4; // highest of YOUR oplog seqs I have applied
// 0.3.0+ capability, additive: I understand a snapshot scoped to a SUBSET of the registered
// tables (SnapshotBegin.tables) and I will answer a completed snapshot with a SnapshotAck.
// A pre-0.3.0 peer leaves this false — proto3 default — and a 0.3.0 sender then promotes any
// scoped snapshot to a full one and clears its flags on enqueue, exactly as 0.2.x did. This is
// why lib_schema_version stays 1: it is compared fail-closed for EQUALITY, so bumping it would
// stop 0.1.x/0.2.x pairs syncing outright.
bool supports_partial_snapshot = 5;
}
message TableDigest { string table_name = 1; string digest = 2; }
message HandshakeAck { string node_id = 1; bool snapshot_required = 2; }
@@ -40,7 +48,12 @@ message OplogEntry {
message DeltaBatch { repeated OplogEntry entries = 1; }
message DeltaAck { int64 applied_thru_seq = 1; }
message SnapshotBegin { int64 as_of_seq = 1; }
// tables: the SCOPE of this snapshot (0.3.0+). EMPTY means every registered table — a full
// snapshot, which is the only thing a pre-0.3.0 sender ever emits and the only thing a
// pre-0.3.0 receiver can safely assume. A non-empty list means the snapshot is PARTIAL, so the
// receiver must NOT advance last_applied_remote_seq to as_of_seq (deltas below it for the tables
// NOT listed have not been delivered); it merges the rows through LWW and leaves the watermark.
message SnapshotBegin { int64 as_of_seq = 1; repeated string tables = 2; }
message SnapshotBatch { string table_name = 1; repeated SnapshotRow rows = 2; }
message SnapshotRow {
string pk_json = 1;
@@ -50,3 +63,8 @@ message SnapshotRow {
bool is_tombstone = 5;
}
message SnapshotComplete { int64 as_of_seq = 1; }
// Sent by the RECEIVER once a snapshot has been durably applied, so the sender clears the
// per-table snapshot flags only after the peer actually holds the rows (a crash mid-snapshot then
// re-streams the same scope on reconnect). Only exchanged when both peers advertised
// supports_partial_snapshot; a pre-0.3.0 pair keeps the 0.2.x clear-on-enqueue behaviour.
message SnapshotAck { int64 as_of_seq = 1; }
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/GrpcSyncDuplex.cs
+50 -8
View File
@@ -12,19 +12,58 @@ namespace ZB.MOM.WW.LocalDb.Replication.Internal;
internal static class GrpcSyncDuplex
{
/// <summary>
/// Wires <paramref name="reader"/> into an unbounded inbox channel drained by a background loop,
/// and exposes <paramref name="writer"/> as the duplex send. Returns the duplex plus the reader
/// loop's task so the caller can observe it. On clean stream end the inbox channel is completed;
/// on a reader fault it is completed WITH the exception — the session's handshake and receive
/// phases both await the inbox, so terminating it is what unblocks them when the call dies (they
/// would otherwise hang on a dead stream).
/// Inbound messages buffered between the transport and the session's receive loop.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why bounded at all.</b> An unbounded inbox makes this process's memory a function of what
/// the PEER decides to send. The worst case is not hypothetical: a full snapshot streams every
/// row of every registered table, and both sides of a pair that enabled replication late
/// snapshot each other simultaneously — so an unbounded inbox holds an entire peer dataset (an
/// event log at its storage cap, say) in memory on both nodes at once. Bounding it converts that
/// into backpressure: the pump stops draining the gRPC stream, the HTTP/2 flow-control window
/// closes, and the SENDER slows down.
/// </para>
/// <para>
/// <b>Why 64.</b> Memory here is bounded by roughly capacity x
/// <see cref="ReplicationOptions.MaxBatchBytes"/> (the sender's own per-message budget), so the
/// ceiling is a configured number rather than a peer-controlled one. 64 is four times the
/// sender's outbound data-lane depth, so a healthy stream never reaches the bound and pays
/// nothing; it engages only when this node is genuinely slower than the peer for a sustained
/// burst, which is exactly when backpressure is wanted. Lower <c>MaxBatchBytes</c> to lower the
/// ceiling.
/// </para>
/// <para>
/// <b>What makes bounding safe.</b> Backpressure on the inbox deadlocks any design where both
/// peers can stop draining their inbox at the same time — mutual snapshots were exactly that
/// case while the send ran inside the handshake. <see cref="SyncSession"/> therefore starts its
/// receive loop BEFORE streaming a snapshot it owes, so the inbox is drained continuously for
/// the whole life of the session; see the remarks on its snapshot phase.
/// </para>
/// </remarks>
internal const int InboxCapacity = 64;
/// <summary>
/// Wires <paramref name="reader"/> into a bounded inbox channel (see
/// <see cref="InboxCapacity"/>) drained by a background loop, and exposes
/// <paramref name="writer"/> as the duplex send. Returns the duplex plus the reader loop's task
/// so the caller can observe it. On clean stream end the inbox channel is completed; on a reader
/// fault it is completed WITH the exception — the session's handshake and receive phases both
/// await the inbox, so terminating it is what unblocks them when the call dies (they would
/// otherwise hang on a dead stream).
/// </summary>
public static (SyncDuplex Duplex, Task ReaderTask) Create(
IAsyncStreamReader<SyncMessage> reader,
IAsyncStreamWriter<SyncMessage> writer,
CancellationToken ct)
{
var inbox = Channel.CreateUnbounded<SyncMessage>(new UnboundedChannelOptions { SingleWriter = true });
var inbox = Channel.CreateBounded<SyncMessage>(new BoundedChannelOptions(InboxCapacity)
{
// Wait, never Drop: every inbound message is protocol state (a delta the peer will not
// resend in-session, a snapshot frame, an ack). Dropping one is silent divergence.
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = true,
});
var readerTask = PumpInboundAsync(reader, inbox.Writer, ct);
var duplex = new SyncDuplex
{
@@ -44,7 +83,10 @@ internal static class GrpcSyncDuplex
try
{
while (await reader.MoveNext(ct))
inbox.TryWrite(reader.Current);
// WriteAsync, not TryWrite: on a FULL bounded channel TryWrite returns false and the
// message is silently lost. Awaiting here is the backpressure — it stops draining the
// gRPC stream, which closes the peer's flow-control window.
await inbox.WriteAsync(reader.Current, ct);
}
catch (Exception ex)
{
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/LwwApplier.cs
+246 -72
View File
@@ -14,12 +14,28 @@ internal sealed record ApplyResult(long AppliedThruSeq, int Applied, int Discard
/// row-version (hlc, then node_id ordinal tie-break), poison rows are dead-lettered while the
/// batch continues, and the remote watermark advances in the SAME transaction as the data so a
/// crash-replay of the batch is a no-op (LWW's strict-greater compare loses on equal versions).
/// Each entry's row write and its ledger write are additionally scoped by a per-entry SAVEPOINT, so
/// the pair is all-or-nothing even when the poison catch keeps the batch alive around it.
/// This never writes <c>__localdb_oplog</c>: the 2-node topology does not re-forward applied deltas.
/// </summary>
/// <remarks>
/// Every statement is issued through a <see cref="PreparedStatements"/> set held for the life of the
/// batch, so each distinct statement is prepared ONCE and then rebound per row. The previous shape
/// went through the generic <c>ILocalDbTransaction</c> façade, which builds a fresh
/// <see cref="SqliteCommand"/> per call — three SQL parses per applied row (version lookup, the
/// table's delete/upsert, the row-version write), on the path that drains a reconnect backlog.
/// </remarks>
internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow = null)
{
private readonly Func<DateTimeOffset> _utcNow = utcNow ?? (() => DateTimeOffset.UtcNow);
/// <summary>
/// Distinct statements prepared by the most recent <see cref="ApplyBatchAsync"/>. Test-only
/// observation: it is the direct evidence that statement count tracks the number of distinct
/// STATEMENTS rather than the number of rows.
/// </summary>
internal int LastBatchStatementCount { get; private set; }
public async Task<ApplyResult> ApplyBatchAsync(IReadOnlyList<OplogEntryRecord> entries, CancellationToken ct)
{
if (entries.Count == 0)
@@ -42,13 +58,14 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
if (entry.Hlc > maxHlc) maxHlc = entry.Hlc;
}
var sqlByTable = new Dictionary<string, (string Delete, string Upsert)>(StringComparer.Ordinal);
var applied = 0;
var discarded = 0;
var deadLettered = 0;
await using var tx = await db.BeginTransactionAsync(ct);
await tx.ExecuteAsync("UPDATE __localdb_applying SET applying = 1 WHERE id = 1", null, ct);
await using var tx = await db.BeginPreparedTransactionAsync(ct);
using var sql = new PreparedStatements(tx.Connection, tx.Sqlite);
await sql.SetApplyingAsync(1, ct);
foreach (var entry in entries)
{
@@ -56,19 +73,16 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
// KEY table that silently inserts a phantom auto-rowid row — so it is poison, not a win.
if (!entry.IsTombstone && entry.RowJson is null)
{
await DeadLetterAsync(tx, entry, "non-tombstone entry with null row_json", utcNow, ct);
await sql.DeadLetterAsync(entry, "non-tombstone entry with null row_json", utcNow, ct);
deadLettered++;
continue;
}
var local = await tx.QueryAsync(
"SELECT hlc, node_id FROM __localdb_row_version WHERE table_name = @t AND pk_json = @pk",
static r => (Hlc: r.GetInt64(0), NodeId: r.GetString(1)),
new { t = entry.TableName, pk = entry.PkJson }, ct);
var local = await sql.ReadLocalVersionAsync(entry.TableName, entry.PkJson, ct);
var remoteWins = local.Count == 0
|| entry.Hlc > local[0].Hlc
|| (entry.Hlc == local[0].Hlc && string.CompareOrdinal(entry.NodeId, local[0].NodeId) > 0);
var remoteWins = local is null
|| entry.Hlc > local.Value.Hlc
|| (entry.Hlc == local.Value.Hlc && string.CompareOrdinal(entry.NodeId, local.Value.NodeId) > 0);
if (!remoteWins)
{
@@ -76,33 +90,28 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
continue;
}
if (!sqlByTable.TryGetValue(entry.TableName, out var sql))
{
sql = BuildTableSql(db.ReplicatedTables[entry.TableName]);
sqlByTable[entry.TableName] = sql;
}
// The row mutation and its ledger entry are TWO statements and must land together or
// not at all: a failure between them commits a deleted/replaced row carrying a stale
// ledger version, which SnapshotStreamer then warns about and skips — so that key never
// travels again and the pair is silently, permanently divergent for it. A per-entry
// savepoint makes the pair atomic without giving up the poison catch's per-ENTRY
// granularity (the surrounding transaction, and every entry already applied in it,
// survives a rollback to the savepoint). Cost measured on the path that matters — a
// 100 000-row backlog drain — is ~7% (178.6k rows/s → 166.1k rows/s): the two extra
// statements are prepared once like the rest and neither one touches the disk, since
// an inner savepoint's release just merges into the enclosing transaction.
var pairApplied = false;
await sql.SavepointAsync(ct);
try
{
var table = db.ReplicatedTables[entry.TableName];
if (entry.IsTombstone)
await tx.ExecuteAsync(sql.Delete, new { pk = entry.PkJson }, ct);
await sql.DeleteAsync(table, entry.PkJson, ct);
else
await tx.ExecuteAsync(sql.Upsert, new { row = entry.RowJson }, ct);
await sql.UpsertAsync(table, entry.RowJson!, ct);
await tx.ExecuteAsync(
"INSERT OR REPLACE INTO __localdb_row_version " +
"(table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
"VALUES (@t, @pk, @hlc, @node, @tomb, @tut)",
new
{
t = entry.TableName,
pk = entry.PkJson,
hlc = entry.Hlc,
node = entry.NodeId,
tomb = entry.IsTombstone ? 1 : 0,
tut = entry.IsTombstone ? utcNow : null,
}, ct);
applied++;
await sql.WriteRowVersionAsync(entry, utcNow, ct);
pairApplied = true;
}
catch (SqliteException ex)
{
@@ -110,59 +119,224 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
// malformed row_json) are treated as per-entry poison — SQLite aborts just the
// statement, not the transaction, so the batch continues after dead-lettering.
// Any other exception propagates and rolls back the whole batch (fail-closed).
await DeadLetterAsync(tx, entry, ex.Message, utcNow, ct);
// The rollback comes FIRST so the dead-letter row it records is not discarded with
// the half-applied pair.
await sql.RollbackToSavepointAsync(ct);
await sql.DeadLetterAsync(entry, ex.Message, utcNow, ct);
deadLettered++;
}
// Released outside the catch on purpose: a RELEASE that itself fails means the
// transaction is no longer sound, and there is no savepoint state left to reason about,
// so it propagates and rolls the whole batch back rather than being swallowed as poison.
if (pairApplied)
{
await sql.ReleaseSavepointAsync(ct);
applied++;
}
}
await tx.ExecuteAsync(
"UPDATE __localdb_peer_state SET " +
"last_applied_remote_seq = MAX(last_applied_remote_seq, @maxSeq), " +
"last_seen_hlc = MAX(last_seen_hlc, @maxHlc), last_sync_utc = @utc WHERE id = 1",
new { maxSeq, maxHlc, utc = utcNow }, ct);
await sql.AdvancePeerStateAsync(maxSeq, maxHlc, utcNow, ct);
await sql.SetApplyingAsync(0, ct);
LastBatchStatementCount = sql.StatementCount;
await tx.ExecuteAsync("UPDATE __localdb_applying SET applying = 0 WHERE id = 1", null, ct);
await tx.CommitAsync(ct);
db.Clock.Observe(maxHlc);
return new ApplyResult(maxSeq, applied, discarded, deadLettered);
}
private static Task DeadLetterAsync(
ILocalDbTransaction tx, OplogEntryRecord entry, string error, string utcNow, CancellationToken ct) =>
tx.ExecuteAsync(
"INSERT INTO __localdb_dead_letter " +
"(received_utc, table_name, pk_json, row_json, hlc, node_id, error) " +
"VALUES (@utc, @t, @pk, @row, @hlc, @node, @err)",
new
{
utc = utcNow,
t = entry.TableName,
pk = entry.PkJson,
row = entry.RowJson,
hlc = entry.Hlc,
node = entry.NodeId,
err = error,
}, ct);
private static (string Delete, string Upsert) BuildTableSql(ReplicatedTable table)
/// <summary>
/// The statement set one apply-batch runs on: each distinct statement is built and prepared on
/// first use, then rebound and re-executed per row. Statements whose SQL depends on the table
/// (delete / upsert) are cached per table; the rest are shared across all of them.
/// </summary>
/// <remarks>
/// Reusing the same <see cref="SqliteCommand"/> with unchanged <c>CommandText</c> is what makes
/// this prepare-once: Microsoft.Data.Sqlite holds the compiled <c>sqlite3_stmt</c> on the command
/// and only recompiles when the text changes. <see cref="SqliteCommand.Prepare"/> is called
/// explicitly anyway, so the intent is stated rather than inferred from an implementation detail.
/// </remarks>
private sealed class PreparedStatements(SqliteConnection connection, SqliteTransaction transaction) : IDisposable
{
var name = Ident(table.Name);
private readonly List<SqliteCommand> _owned = [];
private readonly Dictionary<string, SqliteCommand> _deletes = new(StringComparer.Ordinal);
private readonly Dictionary<string, SqliteCommand> _upserts = new(StringComparer.Ordinal);
var pkMatch = string.Join(" AND ", table.PkColumns.Select(p => $"{Ident(p)} = json_extract(@pk, {JsonPath(p)})"));
var delete = $"DELETE FROM {name} WHERE {pkMatch}";
private SqliteCommand? _applying;
private SqliteCommand? _lookup;
private SqliteCommand? _rowVersion;
private SqliteCommand? _deadLetter;
private SqliteCommand? _peerState;
private SqliteCommand? _savepoint;
private SqliteCommand? _releaseSavepoint;
private SqliteCommand? _rollbackToSavepoint;
var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
var upsert = $"INSERT OR REPLACE INTO {name} ({columns}) VALUES ({values})";
/// <summary>The per-entry savepoint's name. Reused for every entry — one is open at a time.</summary>
private const string SavepointName = "__localdb_apply_entry";
return (delete, upsert);
/// <summary>How many distinct statements this set has prepared.</summary>
public int StatementCount => _owned.Count;
/// <summary>Opens the per-entry savepoint that scopes one entry's row + ledger writes.</summary>
public Task SavepointAsync(CancellationToken ct)
{
var cmd = _savepoint ??= Create("SAVEPOINT " + SavepointName);
return cmd.ExecuteNonQueryAsync(ct);
}
/// <summary>Commits the entry's savepoint into the surrounding batch transaction.</summary>
public Task ReleaseSavepointAsync(CancellationToken ct)
{
var cmd = _releaseSavepoint ??= Create("RELEASE " + SavepointName);
return cmd.ExecuteNonQueryAsync(ct);
}
/// <summary>
/// Undoes everything the entry did and closes its savepoint. ROLLBACK TO leaves the savepoint
/// OPEN, so the RELEASE is required — without it the names would nest one deeper per poison
/// entry and the next rollback would unwind more than one entry's work.
/// </summary>
public async Task RollbackToSavepointAsync(CancellationToken ct)
{
var cmd = _rollbackToSavepoint ??= Create("ROLLBACK TO " + SavepointName);
await cmd.ExecuteNonQueryAsync(ct);
await ReleaseSavepointAsync(ct);
}
public Task SetApplyingAsync(int value, CancellationToken ct)
{
var cmd = _applying ??= Create(
"UPDATE __localdb_applying SET applying = @applying WHERE id = 1", "@applying");
cmd.Parameters["@applying"].Value = value;
return cmd.ExecuteNonQueryAsync(ct);
}
public async Task<(long Hlc, string NodeId)?> ReadLocalVersionAsync(
string tableName, string pkJson, CancellationToken ct)
{
var cmd = _lookup ??= Create(
"SELECT hlc, node_id FROM __localdb_row_version WHERE table_name = @t AND pk_json = @pk",
"@t", "@pk");
cmd.Parameters["@t"].Value = tableName;
cmd.Parameters["@pk"].Value = pkJson;
await using var reader = await cmd.ExecuteReaderAsync(ct);
if (!await reader.ReadAsync(ct))
return null;
return (reader.GetInt64(0), reader.GetString(1));
}
public Task DeleteAsync(ReplicatedTable table, string pkJson, CancellationToken ct)
{
if (!_deletes.TryGetValue(table.Name, out var cmd))
_deletes[table.Name] = cmd = Create(BuildDelete(table), "@pk");
cmd.Parameters["@pk"].Value = pkJson;
return cmd.ExecuteNonQueryAsync(ct);
}
public Task UpsertAsync(ReplicatedTable table, string rowJson, CancellationToken ct)
{
if (!_upserts.TryGetValue(table.Name, out var cmd))
_upserts[table.Name] = cmd = Create(BuildUpsert(table), "@row");
cmd.Parameters["@row"].Value = rowJson;
return cmd.ExecuteNonQueryAsync(ct);
}
public Task WriteRowVersionAsync(OplogEntryRecord entry, string utcNow, CancellationToken ct)
{
var cmd = _rowVersion ??= Create(
"INSERT OR REPLACE INTO __localdb_row_version " +
"(table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
"VALUES (@t, @pk, @hlc, @node, @tomb, @tut)",
"@t", "@pk", "@hlc", "@node", "@tomb", "@tut");
cmd.Parameters["@t"].Value = entry.TableName;
cmd.Parameters["@pk"].Value = entry.PkJson;
cmd.Parameters["@hlc"].Value = entry.Hlc;
cmd.Parameters["@node"].Value = entry.NodeId;
cmd.Parameters["@tomb"].Value = entry.IsTombstone ? 1 : 0;
cmd.Parameters["@tut"].Value = entry.IsTombstone ? utcNow : DBNull.Value;
return cmd.ExecuteNonQueryAsync(ct);
}
public Task DeadLetterAsync(OplogEntryRecord entry, string error, string utcNow, CancellationToken ct)
{
var cmd = _deadLetter ??= Create(
"INSERT INTO __localdb_dead_letter " +
"(received_utc, table_name, pk_json, row_json, hlc, node_id, error) " +
"VALUES (@utc, @t, @pk, @row, @hlc, @node, @err)",
"@utc", "@t", "@pk", "@row", "@hlc", "@node", "@err");
cmd.Parameters["@utc"].Value = utcNow;
cmd.Parameters["@t"].Value = entry.TableName;
cmd.Parameters["@pk"].Value = entry.PkJson;
cmd.Parameters["@row"].Value = (object?)entry.RowJson ?? DBNull.Value;
cmd.Parameters["@hlc"].Value = entry.Hlc;
cmd.Parameters["@node"].Value = entry.NodeId;
cmd.Parameters["@err"].Value = error;
return cmd.ExecuteNonQueryAsync(ct);
}
public Task AdvancePeerStateAsync(long maxSeq, long maxHlc, string utcNow, CancellationToken ct)
{
var cmd = _peerState ??= Create(
"UPDATE __localdb_peer_state SET " +
"last_applied_remote_seq = MAX(last_applied_remote_seq, @maxSeq), " +
"last_seen_hlc = MAX(last_seen_hlc, @maxHlc), last_sync_utc = @utc WHERE id = 1",
"@maxSeq", "@maxHlc", "@utc");
cmd.Parameters["@maxSeq"].Value = maxSeq;
cmd.Parameters["@maxHlc"].Value = maxHlc;
cmd.Parameters["@utc"].Value = utcNow;
return cmd.ExecuteNonQueryAsync(ct);
}
private SqliteCommand Create(string sql, params string[] parameterNames)
{
var cmd = connection.CreateCommand();
try
{
cmd.Transaction = transaction;
cmd.CommandText = sql;
foreach (var name in parameterNames)
cmd.Parameters.Add(new SqliteParameter(name, DBNull.Value));
cmd.Prepare();
_owned.Add(cmd);
return cmd;
}
catch
{
cmd.Dispose();
throw;
}
}
public void Dispose()
{
foreach (var cmd in _owned)
cmd.Dispose();
_owned.Clear();
}
private static string BuildDelete(ReplicatedTable table)
{
var pkMatch = string.Join(
" AND ", table.PkColumns.Select(p => $"{Ident(p)} = json_extract(@pk, {JsonPath(p)})"));
return $"DELETE FROM {Ident(table.Name)} WHERE {pkMatch}";
}
private static string BuildUpsert(ReplicatedTable table)
{
var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
return $"INSERT OR REPLACE INTO {Ident(table.Name)} ({columns}) VALUES ({values})";
}
// Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
// A single-quoted JSON path with a quoted key member, so a column name with reserved
// characters still resolves: $."col". The key's double quotes are backslash-escaped per
// JSON path grammar.
private static string JsonPath(string column) => "'$.\"" + column.Replace("\"", "\\\"") + "\"'";
}
// Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
// A single-quoted JSON path with a quoted key member, so a column name with reserved characters
// still resolves: $."col". The key's double quotes are backslash-escaped per JSON path grammar.
private static string JsonPath(string column) => "'$.\"" + column.Replace("\"", "\\\"") + "\"'";
}
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/OplogStore.cs
+120 -13
View File
@@ -24,8 +24,9 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
/// <summary>
/// Reads up to <paramref name="maxBatch"/> oplog entries with <c>seq &gt; afterSeq</c> in ascending
/// seq order. Caller contract: check <c>needs_snapshot</c> before trusting incremental reads —
/// after cap-pruning, reads silently resume past a gap.
/// seq order. Caller contract: this read resumes SILENTLY past a cap-pruned gap, so a caller
/// streaming incrementally must first confirm its position is still below the horizon — see
/// <see cref="GetMinSeqAsync"/>, which <c>SyncSession</c>'s pump checks on every iteration.
/// </summary>
public Task<IReadOnlyList<OplogEntryRecord>> ReadBatchAboveAsync(long afterSeq, int maxBatch, CancellationToken ct = default) =>
db.QueryAsync(
@@ -70,12 +71,42 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
"DELETE FROM __localdb_row_version WHERE is_tombstone = 1 AND tombstone_utc < @cutoff",
new { cutoff = FormatUtc(_utcNow() - options.TombstoneRetention) }, ct);
/// <summary>Counts the unacked backlog: oplog entries above the peer's last-acked watermark.</summary>
/// <summary>
/// The unacked backlog — oplog entries above the peer's last-acked watermark — in O(1).
/// </summary>
/// <remarks>
/// <para>
/// <b>Why not <c>COUNT(*)</c>.</b> The obvious query walks every unacked row, and this value is
/// read once a minute by maintenance AND on every metrics scrape. A node whose peer has been
/// down for a day therefore pays a full scan of a million-row backlog, repeatedly, precisely
/// when it is already under stress.
/// </para>
/// <para>
/// <b>The formula.</b> <c>MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)</c>, floored at 0. Both
/// MIN and MAX over <c>seq</c> are single b-tree seeks (it is the rowid, and SQLite's min/max
/// optimization turns each into one seek), and the peer-state read is a one-row lookup — so the
/// whole thing is three seeks regardless of backlog size.
/// </para>
/// <para>
/// <b>Why it is exact.</b> <c>seq</c> is AUTOINCREMENT, so retained rows are a contiguous run
/// wherever nothing has punched a hole in the middle — and no ordinary path punches holes: ack
/// pruning deletes <c>seq &lt;= last_acked_seq</c> (a prefix), the row cap deletes below
/// <c>MAX(seq) - keep + 1</c> (a prefix), and the age cap deletes below an HLC cutoff, which is
/// a prefix because HLCs are stamped in seq order. Clamping the floor to <c>MIN(seq) - 1</c> is
/// what keeps the answer exact after a prune has moved the horizon above the ack watermark;
/// without it a cap-pruned oplog would report its whole seq range forever and re-flag the cap on
/// every tick.
/// </para>
/// <para>
/// <b>The one inexact case</b> is <c>ILocalDb.DeregisterReplicated</c>, which deletes one
/// table's rows from wherever they sit and so can leave holes. The estimate then over-reports by
/// at most the number of holes, until the watermark advances past them. Over-reporting is the
/// safe direction: it can only trigger a snapshot resync earlier, never suppress one.
/// </para>
/// </remarks>
public async Task<long> GetOplogDepthAsync(CancellationToken ct = default)
{
var rows = await db.QueryAsync(
"SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)",
static r => r.GetInt64(0), null, ct);
var rows = await db.QueryAsync(DepthSql, static r => r.GetInt64(0), null, ct);
return rows[0];
}
@@ -84,7 +115,7 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
/// metric-observation callbacks (gauge scrape / status poll); mirrors
/// <see cref="GetOplogDepthAsync"/>'s query. Deliberate tradeoff: synchronous I/O on the
/// metrics-collection thread is acceptable here because WAL-mode readers never block (or are
/// blocked by) writers, the COUNT is indexed-cheap, and the worst case is bounded by
/// blocked by) writers, the query is three b-tree seeks, and the worst case is bounded by
/// <c>BusyTimeoutMs</c> — all at scrape frequency, not on any data path. May throw if the db
/// is disposing — callers on the observation path swallow and drop the sample.
/// </summary>
@@ -92,11 +123,18 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
{
using var conn = db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText =
"SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)";
cmd.CommandText = DepthSql;
return (long)cmd.ExecuteScalar()!;
}
// Both depth accessors run exactly this; keeping it in one place is what makes "mirrors the
// async query" a fact rather than a comment that can rot.
internal const string DepthSql =
"SELECT MAX(0, " +
"COALESCE((SELECT MAX(seq) FROM __localdb_oplog), 0) - " +
"MAX((SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1), " +
"COALESCE((SELECT MIN(seq) FROM __localdb_oplog), 1) - 1))";
/// <summary>
/// The highest seq this node has ever produced: the oplog's max, or — once acked rows have been
/// pruned away — the ack watermark that pruned them. Nothing we emit can exceed it, so a peer
@@ -111,6 +149,24 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
return rows[0];
}
/// <summary>
/// The oldest oplog seq still present — the delta horizon — or 0 when the oplog is empty.
/// </summary>
/// <remarks>
/// One b-tree seek (<c>seq</c> is the rowid and SQLite optimizes MIN into a seek), which is what
/// makes it affordable for the pump to check on every flush interval. Compared against a
/// session's sent watermark it answers the only question the pump has: is everything I have not
/// yet sent still here? Ack-pruning deletes <c>seq &lt;= last_acked_seq</c> and the peer can only
/// ack what it received, so in a healthy session the horizon never climbs past
/// <c>sentThruSeq + 1</c>; a higher one means a cap prune ate unsent deltas.
/// </remarks>
public async Task<long> GetMinSeqAsync(CancellationToken ct = default)
{
var rows = await db.QueryAsync(
"SELECT COALESCE(MIN(seq), 0) FROM __localdb_oplog", static r => r.GetInt64(0), null, ct);
return rows[0];
}
/// <summary>The highest oplog seq present, or 0 when the oplog is empty.</summary>
public async Task<long> GetMaxSeqAsync(CancellationToken ct = default)
{
@@ -133,9 +189,9 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
// Condition reads and prunes share the transaction so the decision cannot go stale (TOCTOU).
await using var tx = await db.BeginTransactionAsync(ct);
var depth = (await tx.QueryAsync(
"SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)",
static r => r.GetInt64(0), null, ct))[0];
// Same O(1) expression as GetOplogDepthAsync — the caps check is the per-minute reader that
// made the old COUNT(*) expensive in the first place.
var depth = (await tx.QueryAsync(DepthSql, static r => r.GetInt64(0), null, ct))[0];
var overRows = depth > options.MaxOplogRows;
var oldest = await tx.QueryAsync(
@@ -196,10 +252,61 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
db.ExecuteAsync(
"UPDATE __localdb_peer_state SET last_applied_remote_seq = 0, last_seen_hlc = 0 WHERE id = 1", null, ct);
/// <summary>Sets or clears the snapshot-resync-required flag.</summary>
/// <summary>
/// Sets or clears the DATABASE-WIDE snapshot-resync flag — the one that means "every registered
/// table". Set by the oplog-horizon paths (cap prune, impossible peer ack), which invalidate
/// delta continuity for all tables at once. Per-table flags live in
/// <c>__localdb_snapshot_state</c>; see <see cref="GetTablesNeedingSnapshotAsync"/>.
/// </summary>
public Task SetNeedsSnapshotAsync(bool value, CancellationToken ct = default) =>
db.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = @v WHERE id = 1", new { v = value ? 1 : 0 }, ct);
/// <summary>
/// The tables individually flagged as owing the peer a snapshot (schema v3), in ordinal name
/// order. Independent of the database-wide flag: baselining one table flags only that table.
/// </summary>
public Task<IReadOnlyList<string>> GetTablesNeedingSnapshotAsync(CancellationToken ct = default) =>
db.QueryAsync(
"SELECT table_name FROM __localdb_snapshot_state WHERE needs_snapshot <> 0 ORDER BY table_name",
static r => r.GetString(0), null, ct);
/// <summary>
/// Clears the snapshot debt a completed snapshot discharged: the named tables when
/// <paramref name="tables"/> is non-empty, otherwise <b>everything</b> — the database-wide flag
/// and every per-table row, because a full snapshot subsumes each of them.
/// </summary>
/// <remarks>
/// <para>
/// Rows are deleted rather than zeroed: absence is the resting state, and keeping a row per
/// table that has ever been baselined would accumulate bookkeeping that means nothing. Runs in
/// one transaction so a full snapshot's two clears cannot half-apply and leave the pair
/// re-snapshotting on every reconnect.
/// </para>
/// <para>
/// A scoped clear deliberately does NOT touch the database-wide flag. If the cap enforcer
/// flagged a whole-database resync while this snapshot was in flight, that debt is real and
/// unrelated — the snapshot only carried some of the tables — so it must survive. The full
/// clear may take it down because a full snapshot carries every row state as of its as-of seq,
/// which is exactly what the pruned deltas would have delivered.
/// </para>
/// </remarks>
public async Task ClearSnapshotDebtAsync(IReadOnlyList<string> tables, CancellationToken ct = default)
{
await using var tx = await db.BeginTransactionAsync(ct);
if (tables.Count == 0)
{
await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 0 WHERE id = 1", null, ct);
await tx.ExecuteAsync("DELETE FROM __localdb_snapshot_state", null, ct);
}
else
{
foreach (var table in tables)
await tx.ExecuteAsync(
"DELETE FROM __localdb_snapshot_state WHERE table_name = @t", new { t = table }, ct);
}
await tx.CommitAsync(ct);
}
/// <summary>
/// Records the highest remote seq applied locally plus the peer HLC observed with it; also stamps
/// <c>last_sync_utc</c>. Both watermarks are MAX-guarded (monotonic): a snapshot completing with a
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/SnapshotStreamer.cs
+102 -23
View File
@@ -19,21 +19,57 @@ internal interface ISnapshotApplier
}
/// <summary>
/// Streams a full snapshot of every registered table to a peer that has fallen behind the pruned
/// oplog horizon. Reads the authoritative per-row version from <c>__localdb_row_version</c> joined
/// Which tables a snapshot covers. An EMPTY <see cref="Tables"/> is the wire's own encoding of
/// "all registered tables" (see <c>SnapshotBegin.tables</c>) and is what every pre-0.3.0 peer both
/// sends and expects, so it is deliberately the same value here rather than a separate flag.
/// </summary>
/// <remarks>
/// The distinction is not cosmetic: a FULL snapshot carries every row state as of its
/// <c>as_of_seq</c>, so the receiver may jump its watermark there and the sender may skip the
/// deltas it covered. A PARTIAL one carries only the listed tables, so neither side may move a
/// watermark that also speaks for the tables it left out.
/// </remarks>
internal sealed record SnapshotScope(IReadOnlyList<string> Tables)
{
/// <summary>Every registered table — the only scope a pre-0.3.0 peer can be sent.</summary>
public static readonly SnapshotScope Full = new([]);
public bool IsPartial => Tables.Count > 0;
public bool Includes(string tableName) =>
Tables.Count == 0 || Tables.Contains(tableName, StringComparer.Ordinal);
}
/// <summary>
/// Streams a snapshot of the tables in the requested <see cref="SnapshotScope"/> — every registered
/// table for a peer that has fallen behind the pruned oplog horizon, or just the baselined ones for
/// a scoped resync. Reads the authoritative per-row version from <c>__localdb_row_version</c> joined
/// to each live table, pages it into <see cref="SnapshotBatch"/> messages through the gated data
/// lane, and brackets them with SnapshotBegin/SnapshotComplete carrying the max oplog seq as-of.
/// </summary>
/// <remarks>
/// Clearing the snapshot debt is deliberately NOT done here: whether the flags may be cleared at all
/// depends on the peer's capability (a 0.3.0 peer acknowledges the snapshot, a pre-0.3.0 one cannot),
/// which is session state. <see cref="SyncSession"/> owns that decision; this type only streams.
/// </remarks>
internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, ReplicationOptions options, ILogger logger)
{
/// <summary>Streams the full snapshot; returns the as-of oplog seq it covers, so the session can skip already-covered tail deltas.</summary>
public async Task<long> SendAsync(Func<SyncMessage, CancellationToken, Task> send, CancellationToken ct)
/// <summary>
/// Streams the snapshot for <paramref name="scope"/>; returns the as-of oplog seq it covers, so
/// the session can skip already-covered tail deltas (a FULL snapshot only — a partial one speaks
/// for nothing outside its tables).
/// </summary>
public async Task<long> SendAsync(SnapshotScope scope, Func<SyncMessage, CancellationToken, Task> send, CancellationToken ct)
{
var asOfSeq = await store.GetMaxSeqAsync(ct);
await send(new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = asOfSeq } }, ct);
var begin = new SnapshotBegin { AsOfSeq = asOfSeq };
begin.Tables.AddRange(scope.Tables);
await send(new SyncMessage { SnapshotBegin = begin }, ct);
var totalRows = 0L;
foreach (var table in db.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
foreach (var table in db.ReplicatedTables.Values
.Where(t => scope.Includes(t.Name))
.OrderBy(t => t.Name, StringComparer.Ordinal))
{
var sql = BuildPageSql(table);
// Keyset cursor (pk_json > last seen), not LIMIT/OFFSET: concurrent inserts/prunes
@@ -57,7 +93,13 @@ internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, Repli
break;
lastPk = rows[^1].PkJson;
// One DB page may span several messages: the page is bounded by ROW COUNT, but a
// message is bounded by BYTES (ReplicationOptions.MaxBatchBytes) — the same budget
// the delta pump honours, and for the same reason. Wide rows are exactly why a
// snapshot is the easiest way to exceed the transport's message limit: it ships
// whole rows, all of them, back to back.
var batch = new SnapshotBatch { TableName = table.Name };
var bytes = 0L;
foreach (var row in rows)
{
if (!row.IsTombstone && row.RowJson is null)
@@ -79,11 +121,25 @@ internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, Repli
};
if (row.RowJson is not null)
snapshotRow.RowJson = row.RowJson;
var size = RowSize(snapshotRow);
// Flush before adding, never after: a message keeps at least one row however
// large, so an oversized row goes out alone rather than stalling the snapshot.
if (batch.Rows.Count > 0 && bytes + size > options.MaxBatchBytes)
{
await send(new SyncMessage { SnapshotBatch = batch }, ct);
totalRows += batch.Rows.Count;
batch = new SnapshotBatch { TableName = table.Name };
bytes = 0;
}
batch.Rows.Add(snapshotRow);
bytes += size;
}
if (batch.Rows.Count > 0)
{
await send(new SyncMessage { SnapshotBatch = batch }, ct);
totalRows += batch.Rows.Count;
totalRows += batch.Rows.Count;
}
if (rows.Count < options.MaxBatchSize)
break;
@@ -91,24 +147,29 @@ internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, Repli
}
await send(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = asOfSeq } }, ct);
// The sender owed this snapshot (its own needs_snapshot); clearing it here — after the whole
// snapshot is on the wire — stops it re-streaming on the next reconnect. Narrow window: the
// messages are enqueued on the data lane, not yet transport-delivered, so a crash/drop after
// this clear could lose the snapshot with the flag already down. The reconnect handshake's
// gap check (peer watermark below the pruned horizon) re-detects that in every case where the
// oplog is non-empty; the sole undetectable corner is an EMPTY oplog (nothing to gap-check),
// where the peer stays behind until new local writes recreate a horizon. Accepted: closing it
// would need transport-level delivery acks for snapshot frames.
await store.SetNeedsSnapshotAsync(false, ct);
logger.LogInformation("Snapshot sent (as_of_seq {AsOfSeq}, {Rows} rows).", asOfSeq, totalRows);
logger.LogInformation(
"Snapshot sent (as_of_seq {AsOfSeq}, {Rows} rows, scope {Scope}).",
asOfSeq, totalRows, scope.IsPartial ? string.Join(",", scope.Tables) : "all");
return asOfSeq;
}
/// <summary>
/// Serialized bytes one row contributes to its batch message: the row's own encoding plus the
/// repeated field's tag and length prefix, as protobuf itself computes them.
/// </summary>
private static long RowSize(SnapshotRow row) =>
1 + Google.Protobuf.CodedOutputStream.ComputeMessageSize(row);
// Pages one table's rows by joining the row-version ledger to the live table on the PK columns
// extracted from pk_json. Rows present in the live table but absent from __localdb_row_version
// (data written BEFORE the table was registered for replication) are intentionally NOT snapshotted:
// they were never captured, consistent with CDC semantics — a full re-registration is required to
// baseline them.
// (data written BEFORE the table was registered for replication) are NOT snapshotted: they were
// never captured, consistent with CDC semantics.
//
// That is a real hole for a node that starts replicating LATE — its whole pre-existing dataset is
// invisible to the peer, silently and permanently — and the supported way to close it is to
// register with `ILocalDb.RegisterReplicated(table, baselineExistingRows: true)`, which seeds the
// ledger at an LWW-safe floor and flags the resync that carries the rows here. Nothing changes on
// this side: baselined rows are ordinary ledger rows and page like any other.
private static string BuildPageSql(ReplicatedTable table)
{
var name = Ident(table.Name);
@@ -149,6 +210,7 @@ internal sealed class SnapshotApplier : ISnapshotApplier
private long _asOfSeq;
private long _rows;
private long _maxHlc;
private bool _partial;
// db is part of the collaborator's canonical shape (mirrors the streamer) even though the merge
// reaches the database through the applier and store; kept for construction symmetry in the factory.
@@ -165,7 +227,13 @@ internal sealed class SnapshotApplier : ISnapshotApplier
_asOfSeq = begin.AsOfSeq;
_rows = 0;
_maxHlc = 0;
_logger.LogInformation("Snapshot receive begun (as_of_seq {AsOfSeq}).", _asOfSeq);
// A non-empty table list scopes the snapshot; empty (which is all a pre-0.3.0 sender can
// emit) means every registered table. Read here rather than on complete because
// SnapshotComplete does not repeat it — the receive loop guarantees a Begin precedes both.
_partial = begin.Tables.Count > 0;
_logger.LogInformation(
"Snapshot receive begun (as_of_seq {AsOfSeq}, scope {Scope}).",
_asOfSeq, _partial ? string.Join(",", begin.Tables) : "all");
return Task.CompletedTask;
}
@@ -187,9 +255,20 @@ internal sealed class SnapshotApplier : ISnapshotApplier
public async Task OnCompleteAsync(SnapshotComplete complete, CancellationToken ct)
{
// A PARTIAL snapshot may not move the inbound seq watermark. That watermark is one number
// for the whole peer oplog, and jumping it to as_of_seq would declare every table applied
// through that point — including the ones this snapshot deliberately left out, whose deltas
// below as_of_seq the sender would then never resend. Only the HLC is carried across (seq 0
// is a no-op against the MAX guard), so the clock still cannot regress below what the peer
// has stamped. The sender makes the mirror-image concession: it does not advance its own
// sent-thru watermark past a partial snapshot, so those deltas are still pumped normally
// and merely re-deliver rows the snapshot already carried, which LWW absorbs.
var seq = _partial ? 0 : complete.AsOfSeq;
// SetLastAppliedRemoteSeqAsync is MAX-guarded, so a delta applied mid-snapshot with a seq
// above the snapshot's as-of (or an HLC above what the snapshot carried) is never regressed.
await _store.SetLastAppliedRemoteSeqAsync(complete.AsOfSeq, _maxHlc, ct);
_logger.LogInformation("Snapshot receive complete (as_of_seq {AsOfSeq}, {Rows} rows).", complete.AsOfSeq, _rows);
await _store.SetLastAppliedRemoteSeqAsync(seq, _maxHlc, ct);
_logger.LogInformation(
"Snapshot receive complete (as_of_seq {AsOfSeq}, {Rows} rows, partial {Partial}).",
complete.AsOfSeq, _rows, _partial);
}
}
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/SyncSession.cs
+378 -48
View File
@@ -10,11 +10,13 @@ namespace ZB.MOM.WW.LocalDb.Replication.Internal;
/// <summary>
/// The symmetric 2-node sync protocol as one state machine: both the gRPC service and the gRPC
/// client run this identical logic over a <see cref="SyncDuplex"/>. It performs the fail-closed
/// handshake, then runs a bidirectional steady state — an outbound change pump and an inbound
/// apply/ack loop — until the token is cancelled or the stream ends. All outbound traffic flows
/// through a single writer task fed by two lanes: an unbounded CONTROL lane (handshake messages,
/// acks) and a bounded DATA lane (delta/snapshot batches). Receiving therefore never blocks on the
/// ability to write — acks always fit — while the bounded data lane is the pump's backpressure.
/// handshake, then runs a bidirectional steady state until the token is cancelled or the stream
/// ends: an outbound phase (the full snapshot this node owes, if any, then the change pump) and an
/// inbound apply/ack loop, STARTED TOGETHER so the inbox keeps draining even while a snapshot is
/// being sent. All outbound traffic flows through a single writer task fed by two lanes: an
/// unbounded CONTROL lane (handshake messages, acks) and a bounded DATA lane (delta/snapshot
/// batches). Receiving therefore never blocks on the ability to write — acks always fit — while the
/// bounded data lane is the pump's backpressure.
/// Local writes never block on the peer; the first fault tears down all loops and propagates.
/// </summary>
internal sealed class SyncSession
@@ -25,6 +27,10 @@ internal sealed class SyncSession
// inbound batch rate (the peer's flow control), so its lane can safely be unbounded.
private const int DataLaneCapacity = 16;
// How long a faulting session's teardown lets the writer keep flushing already-enqueued
// messages before the backstop cancel; only ever reached when the transport itself stalls.
private static readonly TimeSpan FaultFlushGrace = TimeSpan.FromSeconds(2);
private readonly SqliteLocalDb _db;
private readonly OplogStore _store;
private readonly LwwApplier _applier;
@@ -45,6 +51,17 @@ internal sealed class SyncSession
// single receive-loop task, so no synchronization is needed.
private bool _snapshotReceiving;
// Whether the peer speaks the 0.3.0 snapshot extensions: scoped snapshots
// (SnapshotBegin.tables) and the SnapshotAck that answers a completed one. Written once during
// the handshake — which completes before either loop starts — then read-only.
private bool _peerSupportsPartialSnapshot;
// The scope of the snapshot this node is streaming, held until the peer's SnapshotAck discharges
// it; null on every session that owes none. Assigned on the pump task strictly BEFORE the
// snapshot's first message is enqueued and read on the receive loop when the ack lands, so the
// ordering is causal — volatile only to make the cross-thread publication explicit.
private SnapshotScope? _sentSnapshotScope;
public SyncSession(
SqliteLocalDb db, OplogStore store, LwwApplier applier, ReplicationOptions options, ILogger logger,
Func<DateTimeOffset>? utcNow = null, LocalDbMetrics? metrics = null, SyncStatus? status = null)
@@ -61,11 +78,12 @@ internal sealed class SyncSession
/// <summary>
/// Invoked when THIS node owes the peer a snapshot (its own snapshot-required computation, not
/// the peer's) before deltas resume. Receives a gated send delegate that enqueues on the bounded
/// data lane (never the raw transport); returns the as-of oplog seq the snapshot covers so the
/// pump can skip already-covered tail deltas. Wired to <see cref="SnapshotStreamer.SendAsync"/>.
/// the peer's) before deltas resume. Receives the negotiated <see cref="SnapshotScope"/> and a
/// gated send delegate that enqueues on the bounded data lane (never the raw transport); returns
/// the as-of oplog seq the snapshot covers so the pump can skip already-covered tail deltas.
/// Wired to <see cref="SnapshotStreamer.SendAsync"/>.
/// </summary>
public Func<Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
public Func<SnapshotScope, Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
/// <summary>
/// Consumes an inbound snapshot inline on the receive loop: SnapshotBegin opens the receive,
@@ -97,10 +115,19 @@ internal sealed class SyncSession
var writer = WriterLoopAsync(duplex, control.Reader, data.Reader, linked.Token);
try
{
await HandshakeAsync(duplex, control.Writer, GatedDataSend, linked.Token);
var snapshotScope = await HandshakeAsync(duplex, control.Writer, linked.Token);
var pump = PumpLoopAsync(data.Writer, linked.Token);
// The receive loop starts BEFORE any snapshot this node owes is streamed. That ordering
// is load-bearing, not incidental: a snapshot sent from inside the handshake left the
// inbox undrained for its whole duration, so a MUTUAL snapshot (both sides flagged —
// e.g. both baselined an existing dataset) required an unbounded inbox to avoid
// deadlock, and therefore buffered a whole peer dataset in memory. With the receive loop
// running throughout, the peer's snapshot is applied as it arrives, the inbox can be
// bounded (see GrpcSyncDuplex.InboxCapacity), and mutual snapshots exchange under
// ordinary flow control. Interleaving is already safe by design: snapshot rows and
// deltas both merge through LWW, and the snapshot's watermark write is MAX-guarded.
var receive = ReceiveLoopAsync(duplex, control.Writer, linked.Token);
var pump = SnapshotThenPumpLoopAsync(snapshotScope, GatedDataSend, data.Writer, linked.Token);
var loops = new[] { pump, receive, writer };
// The first COMPLETED task is the primary outcome. Cancel the siblings, observe every
@@ -117,15 +144,33 @@ internal sealed class SyncSession
{
// Handshake-phase faults (and the primary rethrow above) land here: the writer may
// still be running, so stop and observe it before the primary exception surfaces.
linked.Cancel();
// It must be allowed to DRAIN, not be cancelled outright: the Handshake enqueued at
// the top of HandshakeAsync can still be sitting in the control lane (the enqueue →
// receive → validate → fault path has no yields, while the writer's wakeup waits in
// the thread-pool queue), and that message is the only thing that lets the peer
// compute its own fail-closed verdict — mismatch detection is symmetric, so this
// node's locally-thrown verdict reaches the peer only as the Handshake it transmits.
// Cancelling first killed it almost every time the peer's handshake was already
// buffered, leaving the healthy peer to misread a schema mismatch as a generic
// disconnect and reconnect-loop at the fast backoff. Completing the lanes lets the
// writer send what is already enqueued and exit on its own; the deferred cancel is
// the backstop that keeps a wedged transport from holding teardown hostage.
control.Writer.TryComplete();
data.Writer.TryComplete();
linked.CancelAfter(FaultFlushGrace);
await DrainSecondaryAsync(writer);
throw;
}
}
private async Task HandshakeAsync(
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane,
Func<SyncMessage, CancellationToken, Task> dataSend, CancellationToken ct)
/// <summary>
/// Runs the two-phase fail-closed handshake and returns the snapshot THIS node owes the peer —
/// null for none, otherwise the <see cref="SnapshotScope"/> to stream (its own computation from
/// its own state; the peer's answer drives the peer's send direction, symmetrically). Sending
/// that snapshot is deliberately NOT part of the handshake; see <see cref="RunAsync"/>.
/// </summary>
private async Task<SnapshotScope?> HandshakeAsync(
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, CancellationToken ct)
{
var peerState = await _store.GetPeerStateAsync(ct);
var localHandshake = BuildHandshake(peerState);
@@ -142,6 +187,11 @@ internal sealed class SyncSession
ValidateHandshake(localHandshake, peerHandshake);
// Read before anything can act on it. A pre-0.3.0 peer leaves the field at its proto3
// default, so "false" covers both "explicitly unsupported" and "older build" without a
// version check — which is the point of advertising a capability rather than a version.
_peerSupportsPartialSnapshot = peerHandshake.SupportsPartialSnapshot;
// A different node id on the far end means the peer was replaced, not merely restarted — a
// rebuilt host, a restored-from-nothing volume. Its oplog numbers from 1 again, so the
// watermark we hold (in the OLD peer's seq space) would silently swallow its first N
@@ -175,34 +225,78 @@ internal sealed class SyncSession
_sentThruSeq = peerAppliedThruSeq;
Interlocked.Exchange(ref _peerAckedSeq, peerAppliedThruSeq);
var snapshotRequired = await ComputeSnapshotRequiredAsync(peerHandshake, peerState, ct);
var snapshotScope = await ComputeSnapshotScopeAsync(peerHandshake, peerState, ct);
await controlLane.WriteAsync(
new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotRequired } }, ct);
new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotScope is not null } }, ct);
var peerAckMsg = await ReceiveOneAsync(duplex, ct);
if (peerAckMsg.MsgCase != SyncMessage.MsgOneofCase.HandshakeAck)
throw new InvalidOperationException(
$"Replication protocol error: expected a HandshakeAck, got {peerAckMsg.MsgCase}.");
// We stream the snapshot when WE owe the peer one (computed from our pruned horizon /
// needs_snapshot), then it lands on the peer's receive loop. The peer's own flag drives ITS
// send direction symmetrically; both are safe to run because LWW merges either way.
// Memory profile of a MUTUAL snapshot: while we send here (receive loop not yet running),
// the peer's inbound frames — potentially its entire snapshot — buffer in the unbounded
// duplex inbox until our send completes. Worst case one full peer snapshot held in memory;
// acceptable for the small LocalDb datasets this library targets.
if (snapshotRequired)
return snapshotScope;
}
/// <summary>
/// The outbound steady state: stream the full snapshot this node owes the peer (if any), then
/// run the incremental delta pump. One task, because the ordering between them is a protocol
/// requirement — the pump must not put deltas on the data lane ahead of the snapshot, and it
/// starts from the as-of seq the snapshot covered.
/// </summary>
/// <remarks>
/// Split out of the handshake so the receive loop is already draining the inbox while this
/// streams (see <see cref="RunAsync"/>). Nothing else about the snapshot changes: it goes
/// through the same gated data lane, so it is still bounded by the writer's flow control rather
/// than racing the transport.
/// </remarks>
private async Task SnapshotThenPumpLoopAsync(
SnapshotScope? snapshotScope, Func<SyncMessage, CancellationToken, Task> dataSend,
ChannelWriter<SyncMessage> dataLane, CancellationToken ct)
{
if (snapshotScope is not null)
{
if (SnapshotSender is null)
throw new NotSupportedException("snapshot required but no snapshot sender configured");
var snapshotAsOfSeq = await SnapshotSender(dataSend, ct);
// Published BEFORE the first frame is enqueued, so the peer's SnapshotAck can never
// arrive on the receive loop while this is still null.
Volatile.Write(ref _sentSnapshotScope, snapshotScope);
var snapshotAsOfSeq = await SnapshotSender(snapshotScope, dataSend, ct);
_metrics?.RecordSnapshotSent();
_status?.SetLastSyncUtc(_utcNow());
// The snapshot already carries every row state up to as-of; skipping the pump past it
// avoids re-sending tail deltas the peer would only discard via LWW.
if (snapshotAsOfSeq > _sentThruSeq)
// A FULL snapshot carries every row state up to as-of, so skipping the pump past it
// avoids re-sending tail deltas the peer would only discard via LWW. A PARTIAL one
// speaks only for its own tables, so the pump must still deliver every delta below
// as-of — the peer has not moved its watermark either (see SnapshotApplier.OnComplete).
if (!snapshotScope.IsPartial && snapshotAsOfSeq > _sentThruSeq)
_sentThruSeq = snapshotAsOfSeq;
// A peer that cannot acknowledge the snapshot leaves only the 0.2.x option: clear once
// the frames are on the data lane. The window that opens (enqueued, not yet delivered —
// a crash there loses the snapshot with the debt already discharged) is why 0.3.0 peers
// wait for the ack instead. The reconnect handshake's gap check re-detects it whenever
// the oplog is non-empty; an EMPTY oplog remains the undetectable corner.
if (!_peerSupportsPartialSnapshot)
await ClearSentSnapshotDebtAsync(ct);
}
await PumpLoopAsync(dataLane, ct);
}
/// <summary>
/// Discharges the snapshot debt this session's outbound snapshot covered, exactly once.
/// </summary>
/// <remarks>
/// Taking the scope out of the field before clearing makes this idempotent, which matters
/// because two paths reach it — the pre-0.3.0 clear-on-enqueue above and the peer's SnapshotAck
/// — and a peer that acknowledges anyway (a newer build than its handshake claimed) must not
/// clear a second time on top of flags a later baseline may already have re-raised.
/// </remarks>
private async Task ClearSentSnapshotDebtAsync(CancellationToken ct)
{
var scope = Interlocked.Exchange(ref _sentSnapshotScope, null);
if (scope is null)
return;
await _store.ClearSnapshotDebtAsync(scope.Tables, ct);
}
// The ONLY caller of duplex.Send. Control (handshake + acks) always preempts data so the peer's
@@ -246,20 +340,54 @@ internal sealed class SyncSession
}
}
/// <summary>
/// Reads oplog deltas above the sent watermark and ships them, splitting each read into as many
/// messages as <see cref="ReplicationOptions.MaxBatchBytes"/> requires.
/// </summary>
/// <remarks>
/// Row count alone cannot bound a message: rows are arbitrarily wide, so any row-count cap is
/// really a guess about the widest row, and guessing wrong produces a message the transport
/// rejects — replication then stops converging with a size error, not a data error. Budgeting by
/// summed serialized bytes bounds the thing that is actually limited, and lets the row cap go
/// back to meaning what it says.
/// </remarks>
private async Task PumpLoopAsync(ChannelWriter<SyncMessage> dataLane, CancellationToken ct)
{
while (true)
{
ct.ThrowIfCancellationRequested();
// BEFORE the read, because the read itself cannot tell a pruned gap from an idle oplog.
ThrowIfPruneGap(await _store.GetMinSeqAsync(ct));
var batch = await _store.ReadBatchAboveAsync(_sentThruSeq, _options.MaxBatchSize, ct);
if (batch.Count > 0)
{
var delta = new DeltaBatch();
foreach (var entry in batch)
delta.Entries.Add(ToProto(entry));
// Blocking here on a full data lane IS the pump's backpressure.
await dataLane.WriteAsync(new SyncMessage { DeltaBatch = delta }, ct);
_sentThruSeq = batch[^1].Seq;
// The exact guard on the rows about to go out, for a prune that landed between the
// probe and the read. Free (the seq is already in hand) and strictly tighter.
ThrowIfPruneGap(batch[0].Seq);
var index = 0;
while (index < batch.Count)
{
var delta = new DeltaBatch();
var bytes = 0L;
while (index < batch.Count)
{
var proto = ToProto(batch[index]);
var size = EntrySize(proto);
// The first entry of a message is taken however large it is: refusing an
// oversized row would stall the stream on it permanently, whereas sending it
// alone either succeeds or fails loudly at the transport with its real size.
if (delta.Entries.Count > 0 && bytes + size > _options.MaxBatchBytes)
break;
delta.Entries.Add(proto);
bytes += size;
index++;
}
// Blocking here on a full data lane IS the pump's backpressure.
await dataLane.WriteAsync(new SyncMessage { DeltaBatch = delta }, ct);
// Advanced per MESSAGE, not per read: a fault mid-split must not re-send what
// already went out, and must not skip what did not.
_sentThruSeq = batch[index - 1].Seq;
}
continue;
}
@@ -267,23 +395,77 @@ internal sealed class SyncSession
}
}
/// <summary>
/// Faults the session when the oplog horizon has climbed above this session's read position —
/// i.e. a cap prune deleted deltas between <see cref="_sentThruSeq"/> and
/// <paramref name="oldestAvailableSeq"/>. An empty oplog (0) is not a gap.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why a live session has to die for this.</b> <c>EnforceCapsAsync</c> sets
/// <c>needs_snapshot</c> in the same transaction as the prune, so a RECONNECT heals — the
/// handshake serves a full snapshot instead of deltas. Nothing, however, made the live session
/// notice: the pump reads <c>seq &gt; _sentThruSeq</c>, which silently resumes above the hole, and
/// the peer acks those later seqs as if the pair were converged. On a stable LAN with a slow but
/// alive peer that session can run for days, and the pair is durably divergent for all of it —
/// the flag that would fix it is only ever read at handshake time. Tearing down is what turns a
/// persisted "needs snapshot" into an actual snapshot.
/// </para>
/// <para>
/// <b>Why the horizon, not the flag.</b> <c>needs_snapshot</c> is also set for reasons that do
/// not invalidate THIS session's position (and a full snapshot this session just streamed clears
/// it only when the peer's <c>SnapshotAck</c> lands, which is after the pump has started).
/// Comparing the horizon against the sent watermark asks the precise question instead, so a
/// session is faulted when — and only when — rows it still owes the peer are gone.
/// </para>
/// </remarks>
private void ThrowIfPruneGap(long oldestAvailableSeq)
{
if (oldestAvailableSeq == 0 || oldestAvailableSeq <= _sentThruSeq + 1)
return;
throw new LocalDbOplogGapException(
$"Oplog deltas above the sent watermark {_sentThruSeq} were pruned (oldest surviving seq " +
$"{oldestAvailableSeq}): this session can no longer stream incrementally without skipping " +
"them. Ending the session so the reconnect handshake serves a snapshot resync.");
}
/// <summary>
/// Serialized bytes one entry contributes to its batch message: the entry's own encoding plus
/// the repeated field's tag and length prefix, as protobuf itself computes them.
/// </summary>
private static long EntrySize(OplogEntry entry) =>
1 + Google.Protobuf.CodedOutputStream.ComputeMessageSize(entry);
private async Task ReceiveLoopAsync(
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, CancellationToken ct)
{
while (await duplex.Inbox.WaitToReadAsync(ct))
// One-message lookahead. Coalescing means reading ahead until a message turns out NOT to be
// of the kind being coalesced, and a Channel cannot un-read — so that message is parked here
// and processed first on the next pass. Its presence also keeps the outer loop from waiting
// on the inbox when there is already work in hand.
SyncMessage? lookahead = null;
while (lookahead is not null || await duplex.Inbox.WaitToReadAsync(ct))
{
while (duplex.Inbox.TryRead(out var msg))
while (true)
{
var msg = lookahead;
lookahead = null;
if (msg is null)
{
if (!duplex.Inbox.TryRead(out var read))
break;
msg = read;
}
switch (msg.MsgCase)
{
case SyncMessage.MsgOneofCase.DeltaBatch:
// Deltas apply normally even mid-snapshot: LWW makes the interleave safe.
await HandleDeltaBatchAsync(controlLane, msg.DeltaBatch, ct);
lookahead = await HandleCoalescedDeltasAsync(duplex, controlLane, msg.DeltaBatch, ct);
break;
case SyncMessage.MsgOneofCase.DeltaAck:
var acked = msg.DeltaAck.AppliedThruSeq;
InterlockedMax(ref _peerAckedSeq, acked);
await _store.RecordPeerAckAsync(acked, ct);
lookahead = await HandleCoalescedAcksAsync(duplex, msg.DeltaAck, ct);
break;
case SyncMessage.MsgOneofCase.SnapshotBegin:
if (SnapshotApplier is null)
@@ -305,6 +487,20 @@ internal sealed class SyncSession
_snapshotReceiving = false;
_metrics?.RecordSnapshotReceived();
_status?.SetLastSyncUtc(_utcNow());
// Acknowledge only to a peer that asked for it. Sent AFTER the applier's
// durable watermark write, so the ack means "I hold these rows", which is
// what lets the sender drop its per-table flags. On the unbounded control
// lane, like DeltaAck: an ack must never queue behind a data backlog.
if (_peerSupportsPartialSnapshot)
await controlLane.WriteAsync(
new SyncMessage { SnapshotAck = new SnapshotAck { AsOfSeq = msg.SnapshotComplete.AsOfSeq } }, ct);
break;
case SyncMessage.MsgOneofCase.SnapshotAck:
// The peer has durably applied the snapshot THIS node streamed, so the
// tables it covered no longer owe one. Nothing to reconcile against the
// ack's as-of seq: only one snapshot is ever sent per session, so the scope
// held in _sentSnapshotScope is unambiguously the one being acknowledged.
await ClearSentSnapshotDebtAsync(ct);
break;
default:
throw new InvalidOperationException(
@@ -314,19 +510,98 @@ internal sealed class SyncSession
}
}
private async Task HandleDeltaBatchAsync(ChannelWriter<SyncMessage> controlLane, DeltaBatch batch, CancellationToken ct)
/// <summary>
/// Applies the given delta batch together with any delta batches ALREADY waiting in the inbox,
/// as one transaction answered by one ack; returns the non-delta message that ended the run (to
/// be processed next), or null if the inbox merely ran dry.
/// </summary>
/// <remarks>
/// <para>
/// Byte-budgeting the sender produces more, smaller messages, and one apply transaction plus one
/// ack round-trip per message is the cost that would otherwise scale with that. Coalescing pays
/// it once per burst instead — and a burst is exactly what a reconnect after an outage delivers.
/// </para>
/// <para>
/// Only messages already queued are absorbed (<c>TryRead</c>, never a wait), so this adds no
/// latency: a trickle of one batch at a time coalesces nothing and behaves exactly as before.
/// The run stops at the sender's own caps, so the extra memory held is bounded by roughly one
/// sender batch beyond the budget — the overshoot of finishing whichever batch crosses it.
/// </para>
/// </remarks>
private async Task<SyncMessage?> HandleCoalescedDeltasAsync(
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, DeltaBatch first, CancellationToken ct)
{
if (batch.Entries.Count == 0)
var entries = new List<OplogEntry>(first.Entries);
var bytes = 0L;
foreach (var entry in first.Entries)
bytes += EntrySize(entry);
SyncMessage? lookahead = null;
while (entries.Count < _options.MaxBatchSize && bytes < _options.MaxBatchBytes
&& duplex.Inbox.TryRead(out var next))
{
if (next.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
{
lookahead = next;
break;
}
foreach (var entry in next.DeltaBatch.Entries)
{
entries.Add(entry);
bytes += EntrySize(entry);
}
}
await ApplyDeltaEntriesAsync(controlLane, entries, ct);
return lookahead;
}
/// <summary>
/// Records the given ack together with any acks ALREADY waiting in the inbox, as a single
/// watermark write; returns the non-ack message that ended the run, or null if the inbox ran dry.
/// </summary>
/// <remarks>
/// Exactly equivalent to recording each in turn, and strictly cheaper: <c>applied_thru_seq</c> is
/// monotonic and <c>RecordPeerAckAsync</c> is MAX-guarded, so every superseded ack would have
/// written a watermark that the final one overwrites and run a DELETE below its own horizon.
/// Unbounded coalescing is safe here — acks collapse into one number, accumulating nothing.
/// </remarks>
private async Task<SyncMessage?> HandleCoalescedAcksAsync(
SyncDuplex duplex, DeltaAck first, CancellationToken ct)
{
var acked = first.AppliedThruSeq;
SyncMessage? lookahead = null;
while (duplex.Inbox.TryRead(out var next))
{
if (next.MsgCase != SyncMessage.MsgOneofCase.DeltaAck)
{
lookahead = next;
break;
}
if (next.DeltaAck.AppliedThruSeq > acked)
acked = next.DeltaAck.AppliedThruSeq;
}
InterlockedMax(ref _peerAckedSeq, acked);
await _store.RecordPeerAckAsync(acked, ct);
return lookahead;
}
private async Task ApplyDeltaEntriesAsync(
ChannelWriter<SyncMessage> controlLane, IReadOnlyList<OplogEntry> batchEntries, CancellationToken ct)
{
if (batchEntries.Count == 0)
return;
var now = _utcNow();
var nowMs = now.ToUnixTimeMilliseconds();
var driftLimitMs = nowMs + (long)_options.MaxHlcDriftAhead.TotalMilliseconds;
var toApply = new List<OplogEntryRecord>(batch.Entries.Count);
var toApply = new List<OplogEntryRecord>(batchEntries.Count);
var maxSeq = 0L;
var driftDeadLettered = 0;
foreach (var proto in batch.Entries)
foreach (var proto in batchEntries)
{
if (proto.Seq > maxSeq) maxSeq = proto.Seq;
var entry = FromProto(proto);
@@ -407,6 +682,9 @@ internal sealed class SyncSession
NodeId = _db.NodeId,
LibSchemaVersion = LibSchemaVersion,
LastAppliedRemoteSeq = peerState.LastAppliedRemoteSeq,
// Unconditional: this build always understands a scoped snapshot and always answers a
// completed one. It is a statement about the code, not about configuration or state.
SupportsPartialSnapshot = true,
};
foreach (var table in _db.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
handshake.Tables.Add(new TableDigest { TableName = table.Name, Digest = table.Digest });
@@ -437,11 +715,63 @@ internal sealed class SyncSession
"Replication handshake rejected (fail-closed): " + string.Join("; ", parts) + ".");
}
private async Task<bool> ComputeSnapshotRequiredAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
/// <summary>
/// Decides what snapshot this node owes the peer: null for none, <see cref="SnapshotScope.Full"/>
/// for every registered table, or a scoped snapshot naming just the tables that need one.
/// </summary>
/// <remarks>
/// <para>
/// The two sources of snapshot debt are genuinely different in reach, which is why they are
/// stored separately. An <b>oplog-horizon</b> gap — the peer's watermark below the oldest delta
/// still streamable, which is what a cap prune or an impossible peer ack leaves behind — means
/// deltas have been lost for the oplog as a whole, so nothing short of every table is sound. A
/// <b>baseline</b> means one table's pre-existing rows were never captured, and says nothing at
/// all about the others.
/// </para>
/// <para>
/// A scope that happens to name every registered table is returned as Full rather than as a
/// list: it covers exactly what a full snapshot covers, so it should also get a full snapshot's
/// watermark advance on both sides, and it needs no promotion for a pre-0.3.0 peer.
/// </para>
/// <para>
/// Flags naming tables that are not registered are ignored (they can only stream nothing).
/// <c>DeregisterReplicated</c> drops them with the table's ledger rows, so the surviving way to
/// hold one is to drop the SQL table without deregistering — which the handshake's fail-closed
/// digest comparison rejects for its own reasons.
/// </para>
/// </remarks>
private async Task<SnapshotScope?> ComputeSnapshotScopeAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
{
if (peerState.NeedsSnapshot)
return true;
if (peerState.NeedsSnapshot || await HasHorizonGapAsync(peerHandshake, peerState, ct))
return SnapshotScope.Full;
var flagged = await _store.GetTablesNeedingSnapshotAsync(ct);
if (flagged.Count == 0)
return null;
var scoped = flagged.Where(_db.ReplicatedTables.ContainsKey).ToList();
if (scoped.Count == 0)
return null;
if (scoped.Count >= _db.ReplicatedTables.Count)
return SnapshotScope.Full;
// A peer that would read SnapshotBegin.tables as "all tables" and then advance its watermark
// past deltas this snapshot did not carry gets the full snapshot instead. Wider than asked
// for, but it is the only shape the older protocol can express correctly — and it is what
// 0.2.x did for every baseline.
if (!_peerSupportsPartialSnapshot)
{
_logger.LogInformation(
"Peer {PeerNodeId} does not support scoped snapshots; promoting the snapshot of {Tables} to a full one.",
PeerNodeId, string.Join(",", scoped));
return SnapshotScope.Full;
}
return new SnapshotScope(scoped);
}
private async Task<bool> HasHorizonGapAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
{
var minRows = await _db.QueryAsync(
"SELECT COALESCE(MIN(seq), 0) FROM __localdb_oplog", static r => r.GetInt64(0), null, ct);
var minSeq = minRows[0];
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/SyncSessionFactory.cs
+1 -1
View File
@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.LocalDb.Replication.Internal;
/// <summary>
/// Shared construction of a <see cref="SyncSession"/> and its dependencies (OplogStore + LwwApplier)
/// from a <see cref="SqliteLocalDb"/> and the bound options. Both the service and client adapters use
/// this so they build an identical session; snapshot hooks are left null (Task 12 fills them in).
/// this so they build an identical session.
/// </summary>
internal static class SyncSessionFactory
{
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/LocalDbOplogGapException.cs
+23
View File
@@ -0,0 +1,23 @@
namespace ZB.MOM.WW.LocalDb.Replication;
/// <summary>
/// Thrown by a live sync session when the oplog horizon has moved ABOVE the position that session
/// is streaming from: a cap prune deleted deltas it had not yet sent, so every later delta it could
/// ship would land on the far side of a hole the peer will never receive.
/// </summary>
/// <remarks>
/// This is not a transport failure and not a data error — it is the session discovering that its own
/// incremental position has become unsound. Nothing can repair it in-session (the deleted rows are
/// gone), so the session tears itself down; the same prune set <c>needs_snapshot</c>, which makes the
/// reconnect handshake serve a snapshot instead of deltas. Catching this and continuing would
/// reintroduce exactly the silent divergence it exists to stop.
/// </remarks>
public sealed class LocalDbOplogGapException : Exception
{
/// <summary>Creates the exception with a message describing the gap.</summary>
public LocalDbOplogGapException(string message) : base(message) { }
/// <summary>Creates the exception with a message and the underlying cause.</summary>
public LocalDbOplogGapException(string message, Exception innerException)
: base(message, innerException) { }
}
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/LocalDbSyncService.cs
+107 -10
View File
@@ -21,8 +21,26 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
private readonly LocalDbMetrics? _metrics;
private readonly SyncStatus? _status;
// 0 = idle, 1 = a stream is active. Interlocked-guarded across concurrent calls (singleton service).
private int _active;
// The stream currently holding the guard, or null when idle. Guarded by _gate (singleton service).
private readonly object _gate = new();
private Registration? _active;
/// <summary>
/// How long a displacing call waits for the incumbent it evicted to actually finish before
/// giving up and refusing itself. Test seam; the default is generous because eviction only has
/// to outrun the operator, not the network.
/// </summary>
internal TimeSpan TakeoverTimeout { get; set; } = TimeSpan.FromSeconds(10);
/// <summary>One in-flight <c>Sync</c> handler: its cancellation handle plus a completion signal
/// a displacing handler can await, so two sessions never run against the database at once.</summary>
private sealed class Registration(CancellationTokenSource cts)
{
public CancellationTokenSource Cts { get; } = cts;
public TaskCompletionSource Finished { get; } =
new(TaskCreationOptions.RunContinuationsAsynchronously);
}
/// <summary>
/// Creates the passive sync service. <paramref name="status"/> is taken as the public
@@ -41,21 +59,47 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
_status = status as SyncStatus;
}
/// <summary>
/// Accepts one inbound sync stream, evicting any stream already holding the guard.
/// </summary>
/// <remarks>
/// <para>
/// <b>Newest wins.</b> The guard exists to SERIALIZE sessions — a node syncs with exactly one
/// peer, and two sessions against one database would fight over the same watermarks — not to
/// fence out an impostor; authentication is the interceptor's job, and an authenticated caller
/// on this key IS the peer by definition. Refusing the newcomer therefore protected nothing and
/// cost everything in the case that actually happens: the dialling node crashes or is
/// partitioned, this handler stays parked forever inside <c>MoveNext</c> on a half-open stream
/// that will never produce another byte, and every reconnect from the rebooted peer is rejected
/// <c>AlreadyExists</c> — potentially for as long as the pair stays idle, while this side keeps
/// reporting itself connected. The newest dialler is the one with a live link, so it takes over.
/// </para>
/// <para>
/// Eviction is cancel-then-await, not cancel-and-go: the incumbent's cancellation unwinds its
/// session and reader pump, and the newcomer waits for that to complete before starting its own,
/// so the "one session at a time" property the guard was built for still holds. An incumbent
/// that cannot be evicted within <see cref="TakeoverTimeout"/> is the one case where the old
/// answer is still right — the newcomer refuses itself rather than double-running.
/// </para>
/// </remarks>
public override async Task Sync(
IAsyncStreamReader<SyncMessage> requestStream,
IServerStreamWriter<SyncMessage> responseStream,
ServerCallContext context)
{
if (Interlocked.CompareExchange(ref _active, 1, 0) != 0)
throw new RpcException(new Status(StatusCode.AlreadyExists, "a sync stream is already active"));
// The linked source exists so the reader pump is deterministically torn down on EVERY exit
// path — including a session fault for a local (non-transport) reason, where the call token
// never fires and the pump would otherwise keep reading a stream whose handler has exited.
// It is also the handle a later call cancels to displace this one.
using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
var mine = new Registration(linkedCts);
try
{
// The linked source exists so the reader pump is deterministically torn down on EVERY
// exit path — including a session fault for a local (non-transport) reason, where the
// call token never fires and the pump would otherwise keep reading a stream whose
// handler has exited.
using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
// Inside the try so the finally below always releases the guard — including when the
// takeover itself is abandoned because THIS caller gave up mid-handover.
await TakeOverAsync(mine, context.CancellationToken);
var (duplex, readerTask) = GrpcSyncDuplex.Create(requestStream, responseStream, linkedCts.Token);
var session = SyncSessionFactory.Create(_db, _options, _loggerFactory, _metrics, _status);
_status?.SessionStarted();
@@ -81,13 +125,66 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
{
// Client disconnect / server shutdown: the normal end of a long-lived streaming RPC.
}
catch (OperationCanceledException)
{
// Displaced by a newer stream: an orderly handover, not a failure of this node.
_logger.LogInformation("Sync stream ended: displaced by a newer stream from the peer.");
}
catch (LocalDbSchemaMismatchException ex)
{
throw new RpcException(new Status(StatusCode.FailedPrecondition, ex.Message));
}
finally
{
Interlocked.Exchange(ref _active, 0);
lock (_gate)
{
if (ReferenceEquals(_active, mine))
_active = null;
}
mine.Finished.TrySetResult();
}
}
/// <summary>
/// Claims the single-stream guard for <paramref name="mine"/>, cancelling and awaiting whatever
/// held it. Throws <c>AlreadyExists</c> only when the incumbent refuses to end.
/// </summary>
private async Task TakeOverAsync(Registration mine, CancellationToken ct)
{
Registration? displaced;
lock (_gate)
{
displaced = _active;
_active = mine;
}
if (displaced is null)
return;
_logger.LogWarning(
"A new sync stream arrived while one was already active; displacing the incumbent (newest wins).");
try
{
// The incumbent may already be disposing its own source between its finally and here.
displaced.Cts.Cancel();
}
catch (ObjectDisposedException) { }
try
{
await displaced.Finished.Task.WaitAsync(TakeoverTimeout, ct);
}
catch (TimeoutException)
{
// Could not evict it: two live sessions over one database is worse than a refused dial,
// so fall back to the historical answer — and hand the guard back to the incumbent.
lock (_gate)
{
if (ReferenceEquals(_active, mine))
_active = displaced;
}
throw new RpcException(new Status(
StatusCode.AlreadyExists,
"a sync stream is already active and did not yield within the takeover timeout"));
}
}
}
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/MaintenanceBackgroundService.cs
+1 -1
View File
@@ -54,7 +54,7 @@ public sealed class MaintenanceBackgroundService : BackgroundService
{
_snapshotFlagWarned = true;
_logger.LogWarning(
"Oplog backlog/age caps exceeded: pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync. Not re-logged until a clean tick.");
"Oplog backlog/age caps exceeded: pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync. A session streaming from below the new horizon ends itself on its next pump tick so the reconnect serves that snapshot. Not re-logged until a clean tick.");
}
else if (!snapshotFlagged)
{
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/ReplicationOptions.cs
+64 -1
View File
@@ -14,9 +14,44 @@ public sealed class ReplicationOptions
/// <summary>How often the change pump flushes pending oplog deltas to the peer.</summary>
public TimeSpan FlushInterval { get; set; } = TimeSpan.FromMilliseconds(250);
/// <summary>Maximum oplog entries sent per outbound batch.</summary>
/// <summary>
/// Secondary, row-count cap on an outbound batch (and the snapshot streamer's DB page size).
/// <see cref="MaxBatchBytes"/> is the primary cap.
/// </summary>
/// <remarks>
/// Historically this was the ONLY cap, which made it a proxy for a byte budget that consumers
/// had to tune by hand against their widest row — ScadaBridge pinned it to 16 because a 70 KB
/// <c>config_json</c> times the 500 default is ~35 MB against a 4 MB gRPC limit. With
/// <see cref="MaxBatchBytes"/> doing that job directly, this is back to being what its name
/// says: a bound on how many rows one message may carry.
/// </remarks>
public int MaxBatchSize { get; set; } = 500;
/// <summary>
/// Primary cap on an outbound batch: the summed serialized size of the entries in one
/// <c>DeltaBatch</c> / <c>SnapshotBatch</c> message, in bytes. Default 2 MB.
/// </summary>
/// <remarks>
/// <para>
/// Sized at half the default 4 MB gRPC receive limit, so the message framing, the oneof wrapper
/// and any per-entry overhead the estimate does not model all fit inside the margin.
/// </para>
/// <para>
/// <b>A single row larger than the budget is still sent, alone.</b> Skipping it would stall the
/// stream permanently on one oversized row; sending it gives the transport the chance to accept
/// it (gRPC limits are configurable) and, if it refuses, a loud transport error naming the size
/// rather than silent non-convergence.
/// </para>
/// <para>
/// <b>Wire compatibility:</b> none required. Batch size is a sender-side choice the protocol
/// already leaves free — <c>DeltaBatch</c>/<c>SnapshotBatch</c> carry a repeated field with no
/// declared count — so a node on this version interoperates with a peer on any earlier one in
/// both directions, with no proto change and no <c>lib_schema_version</c> bump. (ScadaBridge
/// site pairs stop and start together regardless; this simply does not depend on it.)
/// </para>
/// </remarks>
public int MaxBatchBytes { get; set; } = 2 * 1024 * 1024;
/// <summary>Backlog ceiling: exceeding it flags a snapshot resync and prunes the oldest oplog rows.</summary>
public long MaxOplogRows { get; set; } = 1_000_000;
@@ -29,6 +64,34 @@ public sealed class ReplicationOptions
/// <summary>Upper bound on the exponential reconnect backoff.</summary>
public TimeSpan ReconnectBackoffMax { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>
/// How long the initiator's HTTP/2 connection may sit without a keepalive PING before one is
/// sent. Default 60s.
/// </summary>
/// <remarks>
/// <para>
/// A sync stream is long-lived and, on a converged pair, almost entirely silent — which is
/// exactly the shape a half-open TCP connection hides behind. Without pings, a hard crash or a
/// partition of one node leaves the other holding a stream that will never produce another byte
/// and never fail either: the initiator sits in a session it believes is healthy, and the
/// passive node's handler sits inside its single-stream guard, refusing the reconnect that would
/// heal the pair. Pings turn that into a detected failure on both ends.
/// </para>
/// <para>
/// Sent with <c>KeepAlivePingPolicy.Always</c>, so they flow on an idle connection too — an
/// active-streams-only policy would not fire in precisely the silent-pair case above. Safe over
/// h2c: a PING is an HTTP/2 frame, nothing to do with TLS.
/// </para>
/// </remarks>
public TimeSpan KeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>
/// How long the initiator waits for a keepalive PING ack before tearing the connection down.
/// Default 20s. Detection therefore lands within roughly
/// <see cref="KeepAlivePingDelay"/> + this.
/// </summary>
public TimeSpan KeepAlivePingTimeout { get; set; } = TimeSpan.FromSeconds(20);
/// <summary>How often the maintenance service enforces the oplog caps and prunes expired tombstones.</summary>
public TimeSpan MaintenanceInterval { get; set; } = TimeSpan.FromSeconds(60);
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/ReplicationOptionsValidator.cs
+6
View File
@@ -28,6 +28,8 @@ public sealed class ReplicationOptionsValidator : IValidateOptions<ReplicationOp
failures.Add($"LocalDb:Replication:FlushInterval must be greater than zero; got {options.FlushInterval}.");
if (options.MaxBatchSize <= 0)
failures.Add($"LocalDb:Replication:MaxBatchSize must be greater than 0; got {options.MaxBatchSize}.");
if (options.MaxBatchBytes <= 0)
failures.Add($"LocalDb:Replication:MaxBatchBytes must be greater than 0; got {options.MaxBatchBytes}.");
if (options.MaxOplogRows <= 0)
failures.Add($"LocalDb:Replication:MaxOplogRows must be greater than 0; got {options.MaxOplogRows}.");
if (options.MaxOplogAge <= TimeSpan.Zero)
@@ -38,6 +40,10 @@ public sealed class ReplicationOptionsValidator : IValidateOptions<ReplicationOp
failures.Add($"LocalDb:Replication:ReconnectBackoffMax must be greater than zero; got {options.ReconnectBackoffMax}.");
else if (options.ReconnectBackoffMax > ReconnectBackoffCeiling)
failures.Add($"LocalDb:Replication:ReconnectBackoffMax must be at most {ReconnectBackoffCeiling}; got {options.ReconnectBackoffMax}.");
if (options.KeepAlivePingDelay <= TimeSpan.Zero)
failures.Add($"LocalDb:Replication:KeepAlivePingDelay must be greater than zero; got {options.KeepAlivePingDelay}.");
if (options.KeepAlivePingTimeout <= TimeSpan.Zero)
failures.Add($"LocalDb:Replication:KeepAlivePingTimeout must be greater than zero; got {options.KeepAlivePingTimeout}.");
if (options.MaintenanceInterval <= TimeSpan.Zero)
failures.Add($"LocalDb:Replication:MaintenanceInterval must be greater than zero; got {options.MaintenanceInterval}.");
if (options.MaxHlcDriftAhead <= TimeSpan.Zero)
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/SyncBackgroundService.cs
+35 -1
View File
@@ -1,3 +1,4 @@
using System.Net.Http;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Hosting;
@@ -76,7 +77,7 @@ public sealed class SyncBackgroundService : BackgroundService
{
var channel = ChannelFactory is { } factory
? factory()
: ownedChannel = GrpcChannel.ForAddress(_options.PeerAddress);
: ownedChannel = DialPeer();
var client = new LocalDbSync.LocalDbSyncClient(channel);
var headers = new Metadata();
@@ -152,6 +153,39 @@ public sealed class SyncBackgroundService : BackgroundService
}
}
/// <summary>
/// Dials the peer over a channel that sends HTTP/2 keepalive PINGs, so a half-open link is
/// detected instead of being waited on forever.
/// </summary>
/// <remarks>
/// <para>
/// The bare <c>GrpcChannel.ForAddress(address)</c> this replaces had NO keepalive of any kind,
/// and neither end of a sync stream ever times out on silence by design (a converged pair is
/// legitimately quiet). A hard crash or partition of the peer therefore left this node inside a
/// session that could never make progress and could never fail — and its own half of that stream
/// pinned on the peer, where the single-stream guard turns it into a refusal of the very
/// reconnect that heals the pair.
/// </para>
/// <para>
/// <c>DisposeHttpClient</c> hands the handler's lifetime to the channel, which this loop already
/// disposes on every exit path.
/// </para>
/// </remarks>
private GrpcChannel DialPeer() =>
GrpcChannel.ForAddress(_options.PeerAddress, new GrpcChannelOptions
{
HttpHandler = new SocketsHttpHandler
{
KeepAlivePingDelay = _options.KeepAlivePingDelay,
KeepAlivePingTimeout = _options.KeepAlivePingTimeout,
// Always, not WithActiveRequests: the case being closed is a SILENT stream on a dead
// link, and an active-requests policy stops pinging once the peer stops responding.
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
EnableMultipleHttp2Connections = true,
},
DisposeHttpClient = true,
});
private static bool IsSchemaMismatch(Exception? ex) =>
ex is LocalDbSchemaMismatchException
|| (ex is RpcException rpc && rpc.StatusCode == StatusCode.FailedPrecondition);
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/ILocalDb.cs
+124 -1
View File
@@ -38,10 +38,133 @@ public interface ILocalDb
/// schema digest, and installs the three AFTER triggers that capture every change into the
/// oplog + row-version tables within the same transaction as the consumer's write. Idempotent
/// (re-registering regenerates the triggers). Throws <see cref="LocalDbRegistrationException"/>
/// if the name is unsafe, the table is missing, or it has no primary key.
/// if the name is unsafe, the table is missing, it has no primary key, it has a BLOB column, or
/// another table references it with a row-mutating <c>ON DELETE</c> action.
/// </summary>
/// <remarks>
/// <b>Replicated tables must not be cascade-delete foreign-key parents.</b> Applying a remote row
/// uses <c>INSERT OR REPLACE</c>, whose implicit DELETE fires <c>ON DELETE CASCADE</c> /
/// <c>SET NULL</c> / <c>SET DEFAULT</c> on referencing tables while change capture is suppressed
/// — the child rows it touches are never captured, so the peer keeps them and the pair diverges
/// silently. Such a table is refused here, because registration is the only point at which the
/// problem is still visible. <c>NO ACTION</c> / <c>RESTRICT</c> are allowed: they mutate no child
/// row, so the implicit delete simply fails and the applier dead-letters the entry.
/// </remarks>
ReplicatedTable RegisterReplicated(string tableName);
/// <summary>
/// As <see cref="RegisterReplicated(string)"/>, additionally baselining rows that already exist
/// in the table when <paramref name="baselineExistingRows"/> is true.
/// </summary>
/// <param name="tableName">The table to opt into replication.</param>
/// <param name="baselineExistingRows">
/// When true, every row already in the table that has no <c>__localdb_row_version</c> entry is
/// given one at the <b>baseline floor</b> — HLC <c>0</c>, stamped with this node's id — and, if
/// anything was seeded, a snapshot resync is flagged <b>for this table</b> so the peer actually
/// receives them.
/// </param>
/// <remarks>
/// <para>
/// <b>The problem this solves.</b> Capture is change-data-capture: without baselining, a row
/// written before registration exists in neither the oplog nor the row-version ledger, and the
/// snapshot streamer pages from the LEDGER — so a node that starts replicating LATE never sends
/// its pre-existing rows, silently and forever. Opting in seeds the ledger so a snapshot can
/// carry them.
/// </para>
/// <para>
/// <b>Why the floor is HLC 0, and why that is LWW-safe.</b> A seeded version must never beat a
/// genuine write. Every genuine HLC is a UTC millisecond shifted left 16 bits, so every genuine
/// HLC is greater than 0: a baselined row therefore LOSES every comparison against a real remote
/// write of the same key, and wins only where the peer has no version of that key at all
/// (which is precisely the case baselining exists to fix). It cannot perturb the local clock
/// either — clock recovery takes a MAX over existing HLCs, and 0 raises nothing.
/// </para>
/// <para>
/// <b>The one genuinely ambiguous case</b> is both nodes baselining the SAME key (e.g. both
/// restored the same legacy file). Both sides hold HLC 0, so the LWW tie-break falls to the
/// node-id ordinal: deterministic and convergent, but arbitrary as to content. Nothing can do
/// better — two rows of unknown, unordered antiquity carry no information about which is newer.
/// </para>
/// <para>
/// <b>Idempotent.</b> Seeding uses <c>ON CONFLICT DO NOTHING</c>, so a row that already has a
/// ledger entry — captured normally, or seeded on an earlier boot — keeps its genuine version
/// and no snapshot is flagged. Passing true on every boot is therefore free after the first.
/// </para>
/// <para>
/// <b>Cost, and what bounds it (0.3.0).</b> The snapshot flag is <b>per-table</b>
/// (<c>__localdb_snapshot_state</c>, on-disk schema v3), so baselining one table asks the peer
/// for that table alone — the tables around it are not re-streamed. Through 0.2.x the flag was a
/// single per-database bit and baselining anything re-sent everything, in both directions at
/// once on a pair enabling replication late.
/// </para>
/// <para>
/// Two cases still stream every registered table, by design. A <b>pre-0.3.0 peer</b> cannot be
/// told a snapshot is partial (it would advance its watermark past deltas it never received),
/// so a scoped snapshot is promoted to a full one for it. And the <b>oplog-horizon</b> paths —
/// a cap prune, an impossible peer ack — invalidate delta continuity for every table at once
/// and still set the database-wide flag. So the operational advice survives, narrowed: baseline
/// a high-volume table (an event log at its retention cap, say) deliberately, and <b>purge or
/// trim it first</b>, because that ONE table's rows still travel in full. The work is bounded
/// either way — outbound messages respect <c>MaxBatchBytes</c>, the receive side is bounded by
/// the transport's inbox.
/// </para>
/// <para>
/// <b>Crash safety.</b> Between 0.3.0 peers a table's flag is cleared only when the receiver
/// acknowledges the completed snapshot, so an ungraceful exit part-way through re-streams the
/// same scope on reconnect rather than losing it. Against a pre-0.3.0 peer (which sends no such
/// acknowledgement) the 0.2.x behaviour is kept: the flag clears once the snapshot is on the
/// wire, and the handshake's horizon gap check is the net.
/// </para>
/// </remarks>
ReplicatedTable RegisterReplicated(string tableName, bool baselineExistingRows);
/// <summary>
/// Removes a table from replication: drops its three capture triggers and prunes its rows from
/// <c>__localdb_oplog</c> and <c>__localdb_row_version</c>, in one transaction. Idempotent, and
/// deliberately usable on a table this process never registered — cleaning up triggers left
/// behind by an EARLIER build is the primary use.
/// </summary>
/// <param name="tableName">The table to stop capturing. The table's own data is never touched.</param>
/// <returns>True if anything was removed (a trigger dropped or a row pruned); false if there was nothing to clean up.</returns>
/// <remarks>
/// <para>
/// <b>Intended caller:</b> an application that has decided this node no longer replicates (or no
/// longer replicates this table). Capture is not free — the three triggers run inside every
/// consumer write transaction and append to an oplog nothing will ever drain — so a node with no
/// peer should not be paying for it.
/// </para>
/// <para>
/// <b>Pruning the row-version ledger is safe with respect to re-registration</b>, because
/// <see cref="RegisterReplicated(string, bool)"/> can re-baseline. Without baselining, a
/// deregister-then-register cycle would leave the previously-known rows unstreamable — the same
/// late-opt-in hole — so the two APIs are designed as a pair.
/// </para>
/// <para>
/// <b>Deregistration must be symmetric across a replicating pair, and the library enforces it
/// loudly.</b> The sync handshake compares the two nodes' registered-table digests fail-closed,
/// so a node that drops a table its peer still replicates simply stops syncing — with a
/// schema-mismatch error, not silent divergence. Deregister on both nodes, or on neither.
/// </para>
/// <para>
/// <b>The HLC's durable crash anchor is flushed as part of the same transaction</b>, before the
/// prune. The clock recovers on open as the MAX of <c>hlc_high_water</c>, the oplog's HLCs, the
/// row-version ledger's HLCs and the last-seen peer HLC — and <c>hlc_high_water</c> is otherwise
/// written only on graceful disposal. Since this API deletes the deregistered table's oplog and
/// ledger rows, a file with no other registered table would, after an UNGRACEFUL exit (crash,
/// SIGKILL, power loss), have nothing left to recover from: the clock would restart from
/// wall-clock time and could re-issue stamps below HLCs peers already hold, which LWW then
/// discards as older. Flushing here removes that window.
/// </para>
/// <para>
/// <b>Backlog-depth residual.</b> Pruning one table's rows can leave holes in the oplog's seq
/// range, and the O(1) depth estimate assumes the retained range is contiguous, so it can
/// over-report until the peer acks past the holes. Over-reporting is conservative — it can only
/// trigger an earlier snapshot resync, never suppress one — and on the node this API is for
/// (no peer at all) depth is moot.
/// </para>
/// </remarks>
bool DeregisterReplicated(string tableName);
/// <summary>The tables currently opted into replication, keyed by table name.</summary>
IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables { get; }
}
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/LocalDbSchema.cs
+111 -4
View File
@@ -1,3 +1,4 @@
using System.Globalization;
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.LocalDb.Internal;
@@ -7,6 +8,40 @@ namespace ZB.MOM.WW.LocalDb.Internal;
/// </summary>
internal static class LocalDbSchema
{
/// <summary>
/// The on-disk bookkeeping-schema version this build creates and upgrades to. Bumping it means
/// adding a case to <see cref="Upgrade"/>; existing files are upgraded in place on open.
/// </summary>
/// <remarks>
/// Deliberately NOT the same number as <c>SyncSession.LibSchemaVersion</c>, which is the WIRE
/// protocol version and is compared fail-closed against the peer's. This one is local-file-only:
/// two nodes may briefly sit on different file-schema versions during an upgrade and still sync,
/// because nothing about the file layout crosses the wire.
/// <list type="bullet">
/// <item><description>
/// <b>1</b> — the original layout, which also created <c>__localdb_oplog_hlc</c>.
/// </description></item>
/// <item><description>
/// <b>2</b> — drops <c>__localdb_oplog_hlc</c>. That index (<c>table_name, pk_json, hlc</c>) was
/// never read by any query: every oplog access path is by <c>seq</c> (the rowid) or a full
/// <c>hlc</c>-predicate delete. It cost a b-tree insert on the CAPTURE path — i.e. inside every
/// consumer write transaction — to serve nothing.
/// </description></item>
/// <item><description>
/// <b>3</b> — adds <c>__localdb_snapshot_state</c>, the PER-TABLE snapshot-resync flag. Until
/// v3 the only flag was <c>__localdb_peer_state.needs_snapshot</c>, one bit for the whole
/// database, so baselining a single table re-streamed EVERY registered table in full — in both
/// directions at once on a pair enabling replication late. The database-wide bit survives with
/// its original meaning (the oplog-horizon and cap-prune paths genuinely invalidate every
/// table); the new table narrows the baseline case to the tables that actually need it. The
/// transition itself is a version stamp only — <see cref="Ddl"/> creates the table with
/// <c>IF NOT EXISTS</c> on every open, so an older file gains it whatever its stamp says, and
/// an empty flag table means exactly what it meant before: nothing per-table is owed.
/// </description></item>
/// </list>
/// </remarks>
internal const long CurrentSchemaVersion = 3;
private const string Ddl = """
CREATE TABLE IF NOT EXISTS __localdb_meta (
id INTEGER PRIMARY KEY CHECK (id = 1),
@@ -25,7 +60,6 @@ internal static class LocalDbSchema
node_id TEXT NOT NULL,
is_tombstone INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS __localdb_oplog_hlc ON __localdb_oplog(table_name, pk_json, hlc);
CREATE TABLE IF NOT EXISTS __localdb_peer_state (
id INTEGER PRIMARY KEY CHECK (id = 1),
peer_node_id TEXT NULL,
@@ -50,6 +84,14 @@ internal static class LocalDbSchema
id INTEGER PRIMARY KEY CHECK (id = 1),
applying INTEGER NOT NULL DEFAULT 0
);
-- Per-table snapshot-resync flags (schema v3). A row exists only while that table owes the
-- peer a snapshot; the whole-database __localdb_peer_state.needs_snapshot bit still means
-- "every registered table" and is what the cap/horizon paths set.
-- WITHOUT ROWID: table_name IS the only access path (read as a set, cleared by name).
CREATE TABLE IF NOT EXISTS __localdb_snapshot_state (
table_name TEXT NOT NULL PRIMARY KEY,
needs_snapshot INTEGER NOT NULL DEFAULT 0
) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS __localdb_dead_letter (
id INTEGER PRIMARY KEY AUTOINCREMENT,
received_utc TEXT NOT NULL,
@@ -80,14 +122,67 @@ internal static class LocalDbSchema
meta.Transaction = transaction;
meta.CommandText =
"INSERT INTO __localdb_meta (id, node_id, schema_version) " +
"SELECT 1, $nodeId, 1 WHERE NOT EXISTS (SELECT 1 FROM __localdb_meta)";
"SELECT 1, $nodeId, $version WHERE NOT EXISTS (SELECT 1 FROM __localdb_meta)";
meta.Parameters.AddWithValue("$nodeId", Guid.NewGuid().ToString("D"));
meta.Parameters.AddWithValue("$version", CurrentSchemaVersion);
meta.ExecuteNonQuery();
}
Upgrade(connection, transaction);
transaction.Commit();
}
/// <summary>
/// Brings a file created by an OLDER build up to <see cref="CurrentSchemaVersion"/>, in the
/// caller's transaction. A file already at the current version is untouched.
/// </summary>
/// <remarks>
/// Each step is written to be safe to re-run (IF EXISTS / IF NOT EXISTS), so a crash between the
/// DDL and the version stamp costs nothing but a repeat on the next open. Steps run in ascending
/// order and the stamp is written once at the end, inside the same transaction as the DDL — a
/// file is therefore never observed at the new version with an unapplied step.
/// </remarks>
private static void Upgrade(SqliteConnection connection, SqliteTransaction transaction)
{
long current;
using (var read = connection.CreateCommand())
{
read.Transaction = transaction;
read.CommandText = "SELECT schema_version FROM __localdb_meta WHERE id = 1";
current = Convert.ToInt64(read.ExecuteScalar(), CultureInfo.InvariantCulture);
}
if (current >= CurrentSchemaVersion)
return;
var steps = new List<string>();
// 1 -> 2: drop the write-amplifying, never-read oplog index (see CurrentSchemaVersion).
if (current < 2)
steps.Add("DROP INDEX IF EXISTS __localdb_oplog_hlc;");
// 2 -> 3: __localdb_snapshot_state. No step: the shared Ddl above already ran
// CREATE TABLE IF NOT EXISTS for it in this same transaction, and an empty flag table is
// the correct v3 state for a file that has never baselined a table. Repeating the DDL here
// would be a second source of truth for the same layout. The stamp still moves, because
// schema_version records what the file's layout IS, not which statements produced it.
if (steps.Count > 0)
{
using var ddl = connection.CreateCommand();
ddl.Transaction = transaction;
ddl.CommandText = string.Concat(steps);
ddl.ExecuteNonQuery();
}
using var stamp = connection.CreateCommand();
stamp.Transaction = transaction;
stamp.CommandText = "UPDATE __localdb_meta SET schema_version = $version WHERE id = 1";
stamp.Parameters.AddWithValue("$version", CurrentSchemaVersion);
stamp.ExecuteNonQuery();
}
internal static (string NodeId, long HlcHighWater) ReadMeta(SqliteConnection connection)
{
using var cmd = connection.CreateCommand();
@@ -99,10 +194,22 @@ internal static class LocalDbSchema
return (reader.GetString(0), reader.GetInt64(1));
}
internal static void FlushHlcHighWater(SqliteConnection connection, long value)
/// <summary>
/// Writes the clock's durable crash anchor. <paramref name="transaction"/> must be supplied when
/// the connection has one open (SQLite commands are rejected otherwise), which is what lets a
/// caller flush the anchor atomically with a change that would otherwise erase the clock's other
/// recovery evidence — see <see cref="ILocalDb.DeregisterReplicated"/>.
/// </summary>
/// <remarks>
/// MAX-guarded rather than an assignment: every writer of this column is flushing a monotonic
/// clock, so a lower value can only be a stale one, and the anchor exists precisely to never
/// regress.
/// </remarks>
internal static void FlushHlcHighWater(SqliteConnection connection, long value, SqliteTransaction? transaction = null)
{
using var cmd = connection.CreateCommand();
cmd.CommandText = "UPDATE __localdb_meta SET hlc_high_water = $value WHERE id = 1";
cmd.Transaction = transaction;
cmd.CommandText = "UPDATE __localdb_meta SET hlc_high_water = MAX(hlc_high_water, $value) WHERE id = 1";
cmd.Parameters.AddWithValue("$value", value);
cmd.ExecuteNonQuery();
}
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs
+234 -3
View File
@@ -138,7 +138,15 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
return await ReadAllAsync(cmd, map, ct);
}
public async Task<ILocalDbTransaction> BeginTransactionAsync(CancellationToken ct = default)
public async Task<ILocalDbTransaction> BeginTransactionAsync(CancellationToken ct = default) =>
await BeginPreparedTransactionAsync(ct);
/// <summary>
/// As <see cref="BeginTransactionAsync"/>, but typed so internal callers reach the underlying
/// connection and transaction — which is what lets a hot loop hold <see cref="SqliteCommand"/>s
/// across many rows and rebind them, instead of re-preparing statements per statement per row.
/// </summary>
internal async Task<Transaction> BeginPreparedTransactionAsync(CancellationToken ct = default)
{
var conn = await CreateConnectionAsync(ct);
try
@@ -155,8 +163,11 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
public IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables => _replicatedTables;
public ReplicatedTable RegisterReplicated(string tableName) =>
RegisterReplicated(tableName, baselineExistingRows: false);
// Thread-safe: all master-connection work runs under _masterLock.
public ReplicatedTable RegisterReplicated(string tableName)
public ReplicatedTable RegisterReplicated(string tableName, bool baselineExistingRows)
{
if (string.IsNullOrWhiteSpace(tableName) || !TableNameRegex.IsMatch(tableName))
throw new LocalDbRegistrationException(
@@ -188,6 +199,8 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
throw new LocalDbRegistrationException(
$"Cannot register table '{tableName}': replication requires an explicit primary key.");
EnsureNoRowMutatingDeleteAction(tableName);
var allColumns = columns.Select(c => (c.Name, c.Type)).ToList();
var digest = ReplicatedTable.ComputeDigest(tableName, pkColumns, allColumns);
var table = new ReplicatedTable(tableName, pkColumns, allColumns, digest);
@@ -201,6 +214,10 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
cmd.CommandText = script;
cmd.ExecuteNonQuery();
}
if (baselineExistingRows)
BaselineExistingRows(tx, table);
tx.Commit();
}
@@ -209,6 +226,214 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
}
}
// Thread-safe: all master-connection work runs under _masterLock.
public bool DeregisterReplicated(string tableName)
{
if (string.IsNullOrWhiteSpace(tableName) || !TableNameRegex.IsMatch(tableName))
throw new LocalDbRegistrationException(
$"Cannot deregister table '{tableName}': name must be a plain identifier matching {TableNameRegex}.");
lock (_masterLock)
{
using var tx = _master.BeginTransaction();
// Counted BEFORE the drop: DROP TRIGGER reports no row count, and "did anything change?"
// is the caller's cue to log a one-time cleanup rather than a no-op every boot.
long triggersPresent;
using (var probe = _master.CreateCommand())
{
probe.Transaction = tx;
probe.CommandText =
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'trigger' AND name IN ($ai, $au, $ad)";
foreach (var suffix in TriggerSqlGenerator.Suffixes)
probe.Parameters.AddWithValue("$" + suffix, TriggerSqlGenerator.TriggerName(tableName, suffix));
triggersPresent = Convert.ToInt64(probe.ExecuteScalar(), CultureInfo.InvariantCulture);
}
using (var drop = _master.CreateCommand())
{
drop.Transaction = tx;
drop.CommandText = TriggerSqlGenerator.BuildUninstallScript(tableName);
drop.ExecuteNonQuery();
}
// Flushed BEFORE the prune below, in the SAME transaction, because that prune destroys
// the clock's crash evidence. Recovery on open takes a MAX over hlc_high_water, the
// oplog's HLCs, the row-version ledger's HLCs and last_seen_hlc — and hlc_high_water is
// otherwise written only by Dispose. Deregistering a table deletes that table's oplog
// AND ledger rows, so on a file whose registered tables are all deregistered, an
// UNGRACEFUL exit (crash, SIGKILL, power loss — no Dispose) afterwards leaves nothing
// above zero to recover from: the reopened clock would restart at wall-clock time and
// could re-issue HLCs below stamps peers have already observed, which LWW reads as
// "older" and discards. One row-level UPDATE on an administrative one-shot buys the
// anchor back; it is a separate command from the prune so it cannot inflate prunedRows.
LocalDbSchema.FlushHlcHighWater(_master, _clock.Current, tx);
// Full scan of the oplog by table_name — the index that would have served it is exactly
// the one schema v2 drops, and paying a write on every capture to speed up a one-shot
// administrative prune is the wrong trade.
int prunedRows;
using (var prune = _master.CreateCommand())
{
prune.Transaction = tx;
prune.CommandText =
"DELETE FROM __localdb_oplog WHERE table_name = $t;" +
"DELETE FROM __localdb_row_version WHERE table_name = $t;" +
// A pending per-table snapshot flag for a table nothing streams any more would
// never clear — the streamer skips unregistered tables, so the flag would sit
// in the file forever, and re-registering later would find a stale "already
// owed" it did not ask for. Dropped with the ledger it refers to.
"DELETE FROM __localdb_snapshot_state WHERE table_name = $t;";
prune.Parameters.AddWithValue("$t", tableName);
prunedRows = prune.ExecuteNonQuery();
}
tx.Commit();
_replicatedTables.TryRemove(tableName, out _);
return triggersPresent > 0 || prunedRows > 0;
}
}
/// <summary>
/// Refuses to register <paramref name="tableName"/> when another table references it with an
/// <c>ON DELETE</c> action that MUTATES the referencing rows (<c>CASCADE</c>, <c>SET NULL</c>,
/// <c>SET DEFAULT</c>).
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this is a replication correctness rule and not a schema opinion.</b> Applying a peer's
/// row runs <c>INSERT OR REPLACE</c>, and REPLACE resolves a primary-key conflict by DELETING the
/// existing row first. Connections run with <c>PRAGMA foreign_keys=ON</c>, so that implicit delete
/// fires the referential action on every child table — while the apply path holds the capture
/// guard raised, precisely so its own writes are not re-captured. The child rows are therefore
/// deleted (or nulled) locally and produce NO oplog entry: the peer never hears about them, keeps
/// them, and the pair is durably divergent with nothing anywhere reporting a fault.
/// </para>
/// <para>
/// Refusing at registration is the only place the problem is visible. At apply time the damage is
/// already done and indistinguishable from a normal write, and no runtime signal exists to detect
/// it. <b>NO ACTION / RESTRICT are deliberately allowed</b>: they mutate no child row — the
/// implicit delete simply fails, which the applier surfaces as a dead-lettered entry. Loud, not
/// silent, so there is nothing to guard.
/// </para>
/// <para>
/// The check is registration-time, so a cascading child table CREATED LATER is not caught (SQLite
/// has no DDL hook to catch it on). Registration normally runs at startup after schema setup,
/// which is where the constraint belongs.
/// </para>
/// </remarks>
private void EnsureNoRowMutatingDeleteAction(string tableName)
{
var candidates = new List<string>();
using (var cmd = _master.CreateCommand())
{
// Every user table is a candidate CHILD; the internal sqlite_* tables carry no
// foreign keys. The registered table itself is included: a self-referencing cascade
// is the same hazard.
cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'";
using var reader = cmd.ExecuteReader();
while (reader.Read())
candidates.Add(reader.GetString(0));
}
foreach (var child in candidates)
{
using var cmd = _master.CreateCommand();
cmd.CommandText = $"PRAGMA foreign_key_list({TriggerSqlGenerator.Ident(child)})";
using var reader = cmd.ExecuteReader();
while (reader.Read())
{
// foreign_key_list cols: 0=id, 1=seq, 2=table (the PARENT), 3=from, 4=to,
// 5=on_update, 6=on_delete, 7=match.
var parent = reader.IsDBNull(2) ? "" : reader.GetString(2);
// SQLite identifiers are case-insensitive, and the REFERENCES clause need not
// match the CREATE TABLE spelling.
if (!string.Equals(parent, tableName, StringComparison.OrdinalIgnoreCase))
continue;
var onDelete = reader.IsDBNull(6) ? "" : reader.GetString(6);
if (!RowMutatingDeleteActions.Contains(onDelete))
continue;
throw new LocalDbRegistrationException(
$"Cannot register table '{tableName}': table '{child}' has a foreign key referencing it " +
$"with ON DELETE {onDelete}. Replicated tables must not be cascade-delete foreign-key " +
"parents: applying a remote row uses INSERT OR REPLACE, whose implicit DELETE fires the " +
"referential action while change capture is suppressed — the child rows it touches are " +
"never captured, so the peer never learns of them and the two nodes diverge silently. " +
$"Drop the referential action on '{child}' (ON DELETE NO ACTION, removing children " +
"explicitly instead), or do not replicate this table.");
}
}
}
/// <summary>
/// The <c>ON DELETE</c> actions that mutate referencing rows, as SQLite spells them in
/// <c>PRAGMA foreign_key_list</c>. NO ACTION / RESTRICT are absent on purpose — see
/// <see cref="EnsureNoRowMutatingDeleteAction"/>.
/// </summary>
private static readonly HashSet<string> RowMutatingDeleteActions =
new(["CASCADE", "SET NULL", "SET DEFAULT"], StringComparer.OrdinalIgnoreCase);
/// <summary>
/// Seeds <c>__localdb_row_version</c> at the baseline floor for every row of
/// <paramref name="table"/> that has no ledger entry, and flags a snapshot resync if it seeded
/// anything. Runs in the registration transaction so triggers and baseline commit together.
/// </summary>
/// <remarks>
/// <para>
/// The floor is HLC <c>0</c> — see <see cref="ILocalDb.RegisterReplicated(string, bool)"/> for
/// why that is the only LWW-safe choice.
/// </para>
/// <para>
/// <b>The <c>needs_snapshot</c> flag is what makes this actually deliver.</b> Seeding the ledger
/// alone changes nothing on the wire: a node whose oplog starts at seq 1 presents no gap at the
/// handshake, so the peer is told it needs no snapshot and only ever receives post-registration
/// deltas. Flagging the resync is what turns the seeded ledger into rows the peer receives.
/// Deliberately NOT done by writing oplog entries instead: that would dump the entire table into
/// the oplog on the first boot after opting in, likely blowing <c>MaxOplogRows</c> and forcing
/// the same snapshot anyway, via a cap-prune.
/// </para>
/// <para>
/// <b>The flag is PER-TABLE</b> (<c>__localdb_snapshot_state</c>, schema v3): baselining one
/// table asks for a snapshot of that table alone, not of every registered table. The
/// database-wide <c>__localdb_peer_state.needs_snapshot</c> bit is untouched here and keeps its
/// original meaning — the oplog-horizon and cap-prune paths invalidate every table's delta
/// continuity at once, and only those paths set it. A peer too old to understand a scoped
/// snapshot is served a full one instead; the negotiation lives in the replication package.
/// </para>
/// </remarks>
private void BaselineExistingRows(SqliteTransaction tx, ReplicatedTable table)
{
int seeded;
using (var cmd = _master.CreateCommand())
{
cmd.Transaction = tx;
// The trailing WHERE 1 disambiguates the upsert parse on a SELECT source — the same
// reason TriggerSqlGenerator's row-version statements carry one.
cmd.CommandText =
"INSERT INTO __localdb_row_version (table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
$"SELECT $t, {TriggerSqlGenerator.JsonObject("t", table.PkColumns)}, 0, $node, 0, NULL " +
$"FROM {TriggerSqlGenerator.Ident(table.Name)} t WHERE 1 " +
"ON CONFLICT(table_name, pk_json) DO NOTHING";
cmd.Parameters.AddWithValue("$t", table.Name);
cmd.Parameters.AddWithValue("$node", _nodeId);
seeded = cmd.ExecuteNonQuery();
}
if (seeded == 0)
return;
using var flag = _master.CreateCommand();
flag.Transaction = tx;
flag.CommandText =
"INSERT INTO __localdb_snapshot_state (table_name, needs_snapshot) VALUES ($t, 1) " +
"ON CONFLICT(table_name) DO UPDATE SET needs_snapshot = 1";
flag.Parameters.AddWithValue("$t", table.Name);
flag.ExecuteNonQuery();
}
public void Dispose()
{
if (_disposed)
@@ -329,8 +554,14 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
return Convert.ToInt64(cmd.ExecuteScalar(), CultureInfo.InvariantCulture);
}
private sealed class Transaction(SqliteConnection connection, SqliteTransaction transaction) : ILocalDbTransaction
internal sealed class Transaction(SqliteConnection connection, SqliteTransaction transaction) : ILocalDbTransaction
{
/// <summary>The pinned connection, for internal callers holding their own commands.</summary>
internal SqliteConnection Connection => connection;
/// <summary>The underlying transaction, to assign to internally-held commands.</summary>
internal SqliteTransaction Sqlite => transaction;
public async Task<int> ExecuteAsync(string sql, object? parameters = null, CancellationToken ct = default)
{
await using var cmd = connection.CreateCommand();
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Registration/TriggerSqlGenerator.cs
+23 -4
View File
@@ -27,14 +27,16 @@ internal static class TriggerSqlGenerator
"ON CONFLICT(table_name, pk_json) DO UPDATE SET hlc = excluded.hlc, node_id = excluded.node_id, " +
"is_tombstone = excluded.is_tombstone, tombstone_utc = excluded.tombstone_utc";
/// <summary>The three trigger suffixes, in the order the install/uninstall scripts emit them.</summary>
internal static readonly string[] Suffixes = ["ai", "au", "ad"];
public static string TriggerName(string table, string suffix) => $"__localdb_{table}_{suffix}";
/// <summary>DROP IF EXISTS + CREATE for all three triggers, ready to run in one transaction.</summary>
public static string BuildInstallScript(ReplicatedTable table)
{
var sb = new StringBuilder();
foreach (var suffix in new[] { "ai", "au", "ad" })
sb.Append("DROP TRIGGER IF EXISTS ").Append(Ident(TriggerName(table.Name, suffix))).Append(";\n");
sb.Append(BuildUninstallScript(table.Name));
sb.Append(BuildInsert(table)).Append('\n');
sb.Append(BuildUpdate(table)).Append('\n');
@@ -42,6 +44,20 @@ internal static class TriggerSqlGenerator
return sb.ToString();
}
/// <summary>
/// DROP IF EXISTS for all three triggers of <paramref name="tableName"/>. Takes the bare name
/// rather than a <see cref="ReplicatedTable"/> because uninstall must work for a table this
/// process never registered — the whole point of the cleanup API is stale triggers left by an
/// EARLIER build, whose column list (and therefore digest) is unavailable and irrelevant.
/// </summary>
public static string BuildUninstallScript(string tableName)
{
var sb = new StringBuilder();
foreach (var suffix in Suffixes)
sb.Append("DROP TRIGGER IF EXISTS ").Append(Ident(TriggerName(tableName, suffix))).Append(";\n");
return sb.ToString();
}
private static string BuildInsert(ReplicatedTable t) =>
Wrap(t, "ai", "INSERT", NewRowCapture(t));
@@ -112,13 +128,16 @@ internal static class TriggerSqlGenerator
}
// json_object('col', ALIAS."col", ...) — natively handles NULLs and SQLite scalar types.
private static string JsonObject(string alias, IEnumerable<string> columns)
// Internal rather than private: baseline seeding must build a pk_json identical to what the
// capture triggers emit, character for character, or a seeded ledger key would never match the
// key a later capture writes for the same row.
internal static string JsonObject(string alias, IEnumerable<string> columns)
{
var args = columns.Select(c => $"{Literal(c)}, {alias}.{Ident(c)}");
return $"json_object({string.Join(", ", args)})";
}
private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
internal static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
private static string Literal(string value) => "'" + value.Replace("'", "''") + "'";
}
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/BatchBudgetTests.cs
+444
View File
@@ -0,0 +1,444 @@
using System.Threading.Channels;
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.LocalDb.Contracts;
using ZB.MOM.WW.LocalDb.Internal;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.LocalDb.Replication.Internal;
namespace ZB.MOM.WW.LocalDb.Tests;
/// <summary>
/// Outbound batching by summed serialized BYTES (with the row count kept as a secondary cap), and
/// the inbound coalescing that keeps one transaction + one ack per burst rather than per message.
/// </summary>
public sealed class BatchBudgetTests : IDisposable
{
private static readonly TimeSpan RunTimeout = TimeSpan.FromSeconds(15);
private readonly List<string> _paths = [];
private readonly List<IDisposable> _disposables = [];
public void Dispose()
{
foreach (var d in _disposables)
d.Dispose();
SqliteConnection.ClearAllPools();
foreach (var p in _paths)
{
if (File.Exists(p)) File.Delete(p);
if (File.Exists(p + "-wal")) File.Delete(p + "-wal");
if (File.Exists(p + "-shm")) File.Delete(p + "-shm");
}
}
private async Task<SqliteLocalDb> NewOrdersDbAsync()
{
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
_paths.Add(path);
var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
_disposables.Add(db);
await db.ExecuteAsync("CREATE TABLE orders (id INTEGER PRIMARY KEY, sku TEXT, qty INTEGER)");
db.RegisterReplicated("orders");
return db;
}
private static (SyncDuplex Duplex, ChannelWriter<SyncMessage> ToSession, ChannelReader<SyncMessage> FromSession) ScriptedPeer()
{
var toSession = Channel.CreateUnbounded<SyncMessage>();
var fromSession = Channel.CreateUnbounded<SyncMessage>();
var duplex = new SyncDuplex
{
Send = (m, ct) => fromSession.Writer.WriteAsync(m, ct).AsTask(),
Inbox = toSession.Reader,
};
return (duplex, toSession.Writer, fromSession.Reader);
}
private static Handshake MatchingHandshake(SqliteLocalDb sessionDb, string peerNodeId)
{
var h = new Handshake { NodeId = peerNodeId, LibSchemaVersion = 1, LastAppliedRemoteSeq = 0 };
foreach (var t in sessionDb.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
h.Tables.Add(new TableDigest { TableName = t.Name, Digest = t.Digest });
return h;
}
private static async Task<SyncMessage> NextAsync(ChannelReader<SyncMessage> reader, CancellationToken ct)
{
await reader.WaitToReadAsync(ct);
reader.TryRead(out var msg);
return msg!;
}
// Reads the next scripted-peer message while also observing the session task: a session that
// faults would otherwise leave this wait to run out the whole test timeout, reporting an
// opaque cancellation instead of the session's real exception.
private static async Task<SyncMessage> NextAsync(Task run, ChannelReader<SyncMessage> reader, CancellationToken ct)
{
var next = NextAsync(reader, ct);
if (ReferenceEquals(await Task.WhenAny(run, next), run))
{
await run;
throw new InvalidOperationException("The sync session ended before the expected message arrived.");
}
return await next;
}
private static async Task SwallowAsync(Task task)
{
try { await task; }
catch (OperationCanceledException) { }
}
// Queues a complete peer handshake AHEAD of the session's own, so anything queued after it is
// guaranteed to be sitting in the inbox by the time the receive loop starts. That is what makes
// the coalescing assertions deterministic rather than a race with the reader.
private static async Task PreloadHandshakeAsync(
SqliteLocalDb db, ChannelWriter<SyncMessage> toSession, CancellationToken ct)
{
await toSession.WriteAsync(new SyncMessage { Handshake = MatchingHandshake(db, "peer") }, ct);
await toSession.WriteAsync(
new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = "peer", SnapshotRequired = false } }, ct);
}
private static long BatchBytes(DeltaBatch batch) =>
batch.Entries.Sum(e => 1L + Google.Protobuf.CodedOutputStream.ComputeMessageSize(e));
private static long BatchBytes(SnapshotBatch batch) =>
batch.Rows.Sum(r => 1L + Google.Protobuf.CodedOutputStream.ComputeMessageSize(r));
private static SyncSession NewSession(SqliteLocalDb db, ReplicationOptions options)
{
var store = new OplogStore(db, options);
var applier = new LwwApplier(db);
return new SyncSession(db, store, applier, options, NullLogger.Instance);
}
private static OplogEntry ProtoEntry(long seq, long id, long hlc) =>
new()
{
Seq = seq,
TableName = "orders",
PkJson = $"{{\"id\":{id}}}",
RowJson = $"{{\"id\":{id},\"sku\":\"S\",\"qty\":1}}",
Hlc = hlc,
NodeId = "peer",
IsTombstone = false,
};
// ---- Outbound: delta pump --------------------------------------------------------------
[Fact]
public async Task Pump_SplitsOneReadIntoSeveralMessages_EachWithinTheByteBudget()
{
const int budget = 4_000;
var db = await NewOrdersDbAsync();
var wide = new string('x', 900);
for (var i = 1; i <= 20; i++)
await db.ExecuteAsync(
"INSERT INTO orders (id, sku, qty) VALUES (@id, @sku, @id)", new { id = i, sku = wide });
// Row count alone would put all 20 wide rows (~18 KB) in ONE message; the byte budget is
// what actually bounds the message the transport has to carry.
var options = new ReplicationOptions
{
FlushInterval = TimeSpan.FromMilliseconds(20), MaxBatchSize = 500, MaxBatchBytes = budget,
};
var session = NewSession(db, options);
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = session.RunAsync(duplex, cts.Token);
await PreloadHandshakeAsync(db, toSession, cts.Token);
var seen = new List<long>();
while (seen.Count < 20)
{
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
continue;
Assert.True(BatchBytes(msg.DeltaBatch) <= budget,
$"a {BatchBytes(msg.DeltaBatch)}-byte message exceeded the {budget}-byte budget");
Assert.True(msg.DeltaBatch.Entries.Count > 1, "the budget should still fit several of these rows");
seen.AddRange(msg.DeltaBatch.Entries.Select(e => e.Seq));
}
// Split, not dropped or duplicated: every captured seq travelled exactly once, in order.
Assert.Equal(20, seen.Count);
Assert.Equal(seen.OrderBy(x => x).ToList(), seen);
Assert.Equal(20, seen.Distinct().Count());
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream()
{
var db = await NewOrdersDbAsync();
await db.ExecuteAsync(
"INSERT INTO orders (id, sku, qty) VALUES (1, @sku, 1)", new { sku = new string('x', 5_000) });
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2, 'small', 2)");
var options = new ReplicationOptions
{
FlushInterval = TimeSpan.FromMilliseconds(20), MaxBatchSize = 500, MaxBatchBytes = 100,
};
var session = NewSession(db, options);
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = session.RunAsync(duplex, cts.Token);
await PreloadHandshakeAsync(db, toSession, cts.Token);
var batches = new List<DeltaBatch>();
while (batches.Sum(b => b.Entries.Count) < 2)
{
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase == SyncMessage.MsgOneofCase.DeltaBatch)
batches.Add(msg.DeltaBatch);
}
// The oversized row goes out on its own — refusing it would wedge the stream on one row
// forever, and everything behind it with it.
Assert.Equal(2, batches.Count);
Assert.Single(batches[0].Entries);
Assert.Equal(1, batches[0].Entries[0].Seq);
Assert.True(BatchBytes(batches[0]) > options.MaxBatchBytes);
Assert.Single(batches[1].Entries);
Assert.Equal(2, batches[1].Entries[0].Seq);
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task Pump_RowCountRemainsASecondaryCap()
{
var db = await NewOrdersDbAsync();
for (var i = 1; i <= 12; i++)
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'S', @id)", new { id = i });
// Tiny rows, enormous byte budget: only the row cap can be doing the splitting here.
var options = new ReplicationOptions
{
FlushInterval = TimeSpan.FromMilliseconds(20), MaxBatchSize = 5, MaxBatchBytes = 8 * 1024 * 1024,
};
var session = NewSession(db, options);
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = session.RunAsync(duplex, cts.Token);
await PreloadHandshakeAsync(db, toSession, cts.Token);
var seen = 0;
while (seen < 12)
{
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
continue;
Assert.True(msg.DeltaBatch.Entries.Count <= 5);
seen += msg.DeltaBatch.Entries.Count;
}
cts.Cancel();
await SwallowAsync(run);
}
// ---- Outbound: snapshot --------------------------------------------------------------
[Fact]
public async Task Snapshot_SplitsPagesIntoMessagesWithinTheByteBudget()
{
const int budget = 4_000;
var db = await NewOrdersDbAsync();
var wide = new string('x', 900);
for (var i = 1; i <= 20; i++)
await db.ExecuteAsync(
"INSERT INTO orders (id, sku, qty) VALUES (@id, @sku, @id)", new { id = i, sku = wide });
// A snapshot is the easiest way to blow a message limit: it ships whole rows, all of them,
// back to back — so its page size has to be bounded in bytes too, not just in rows.
var options = new ReplicationOptions { MaxBatchSize = 500, MaxBatchBytes = budget };
var store = new OplogStore(db, options);
var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
var sent = new List<SyncMessage>();
await streamer.SendAsync(SnapshotScope.Full, (m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
var batches = sent.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
.Select(m => m.SnapshotBatch).ToList();
Assert.True(batches.Count > 1, "20 wide rows must not have travelled as one message");
Assert.All(batches, b => Assert.True(BatchBytes(b) <= budget,
$"a {BatchBytes(b)}-byte snapshot message exceeded the {budget}-byte budget"));
// Bracketed and complete: every row, exactly once.
Assert.Equal(SyncMessage.MsgOneofCase.SnapshotBegin, sent[0].MsgCase);
Assert.Equal(SyncMessage.MsgOneofCase.SnapshotComplete, sent[^1].MsgCase);
var pks = batches.SelectMany(b => b.Rows).Select(r => r.PkJson).ToList();
Assert.Equal(20, pks.Count);
Assert.Equal(20, pks.Distinct().Count());
}
[Fact]
public async Task Snapshot_RowLargerThanTheBudget_IsSentAlone()
{
var db = await NewOrdersDbAsync();
await db.ExecuteAsync(
"INSERT INTO orders (id, sku, qty) VALUES (1, @sku, 1)", new { sku = new string('x', 5_000) });
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2, 'small', 2)");
var options = new ReplicationOptions { MaxBatchSize = 500, MaxBatchBytes = 100 };
var store = new OplogStore(db, options);
var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
var sent = new List<SyncMessage>();
await streamer.SendAsync(SnapshotScope.Full, (m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
var batches = sent.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
.Select(m => m.SnapshotBatch).ToList();
Assert.Equal(2, batches.Count);
Assert.All(batches, b => Assert.Single(b.Rows));
Assert.Equal("{\"id\":1}", batches[0].Rows[0].PkJson);
Assert.Equal("{\"id\":2}", batches[1].Rows[0].PkJson);
}
// ---- Inbound: coalescing -------------------------------------------------------------
[Fact]
public async Task Receive_CoalescesQueuedDeltaBatches_IntoOneAck()
{
var db = await NewOrdersDbAsync();
var options = new ReplicationOptions { FlushInterval = TimeSpan.FromSeconds(30) };
var session = NewSession(db, options);
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = session.RunAsync(duplex, cts.Token);
// Queued ahead of the receive loop, so all three are demonstrably in the inbox together.
await PreloadHandshakeAsync(db, toSession, cts.Token);
for (var i = 1; i <= 3; i++)
{
var batch = new DeltaBatch();
batch.Entries.Add(ProtoEntry(i, i, 1_000_000 + i));
await toSession.WriteAsync(new SyncMessage { DeltaBatch = batch }, cts.Token);
}
SyncMessage ack;
do { ack = await NextAsync(run, fromSession, cts.Token); }
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
// The discriminator: uncoalesced, the FIRST ack could only cover seq 1.
Assert.Equal(3, ack.DeltaAck.AppliedThruSeq);
var rows = await db.QueryAsync("SELECT COUNT(*) FROM orders", r => r.GetInt64(0));
Assert.Equal(3, rows[0]);
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task Receive_DeltaCoalescing_DoesNotSwallowTheMessageThatEndsTheRun()
{
// The risk the one-message lookahead exists to handle: coalescing reads ahead, and whatever
// it reads that is NOT a delta must still be processed, not dropped on the floor.
var db = await NewOrdersDbAsync();
var options = new ReplicationOptions { FlushInterval = TimeSpan.FromSeconds(30) };
var session = NewSession(db, options);
var captured = new CapturingSnapshotApplier();
session.SnapshotApplier = captured;
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = session.RunAsync(duplex, cts.Token);
await PreloadHandshakeAsync(db, toSession, cts.Token);
var batch = new DeltaBatch();
batch.Entries.Add(ProtoEntry(1, 1, 1_000_001));
await toSession.WriteAsync(new SyncMessage { DeltaBatch = batch }, cts.Token);
await toSession.WriteAsync(
new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 77 } }, cts.Token);
SyncMessage ack;
do { ack = await NextAsync(run, fromSession, cts.Token); }
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
Assert.Equal(1, ack.DeltaAck.AppliedThruSeq);
// The SnapshotBegin that terminated the coalescing run still reached its handler.
await WaitForAsync(() => captured.BeginAsOfSeq == 77, RunTimeout);
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task Receive_CoalescesQueuedAcks_ToTheHighestWatermark_WithoutLosingTheNextMessage()
{
var db = await NewOrdersDbAsync();
for (var i = 1; i <= 9; i++)
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'S', @id)", new { id = i });
var options = new ReplicationOptions { FlushInterval = TimeSpan.FromSeconds(30) };
var session = NewSession(db, options);
var captured = new CapturingSnapshotApplier();
session.SnapshotApplier = captured;
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = session.RunAsync(duplex, cts.Token);
await PreloadHandshakeAsync(db, toSession, cts.Token);
// Out of order on purpose: collapsing acks must take the MAX, never the last.
foreach (var seq in new long[] { 4, 9, 2 })
await toSession.WriteAsync(
new SyncMessage { DeltaAck = new DeltaAck { AppliedThruSeq = seq } }, cts.Token);
await toSession.WriteAsync(
new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 55 } }, cts.Token);
await WaitForAsync(async () =>
{
var acked = await db.QueryAsync(
"SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1", r => r.GetInt64(0));
return acked[0] == 9;
}, RunTimeout);
Assert.Equal(9, session.PeerAckedSeq);
await WaitForAsync(() => captured.BeginAsOfSeq == 55, RunTimeout);
cts.Cancel();
await SwallowAsync(run);
}
private sealed class CapturingSnapshotApplier : ISnapshotApplier
{
private long _beginAsOfSeq = -1;
public long BeginAsOfSeq => Interlocked.Read(ref _beginAsOfSeq);
public Task OnBeginAsync(SnapshotBegin begin, CancellationToken ct)
{
Interlocked.Exchange(ref _beginAsOfSeq, begin.AsOfSeq);
return Task.CompletedTask;
}
public Task OnBatchAsync(SnapshotBatch batch, CancellationToken ct) => Task.CompletedTask;
public Task OnCompleteAsync(SnapshotComplete complete, CancellationToken ct) => Task.CompletedTask;
}
private static Task WaitForAsync(Func<bool> predicate, TimeSpan timeout) =>
WaitForAsync(() => Task.FromResult(predicate()), timeout);
private static async Task WaitForAsync(Func<Task<bool>> predicate, TimeSpan timeout)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (await predicate()) return;
await Task.Delay(20);
}
throw new TimeoutException("Condition not reached within " + timeout);
}
}
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/GrpcAdapterTests.cs
+80 -3
View File
@@ -70,8 +70,12 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
}
[Fact]
public async Task SecondConcurrentStream_Rejected()
public async Task SecondConcurrentStream_DisplacesTheIncumbent()
{
// Newest wins (0.3.2). The guard serializes sessions; it was never a fence — the auth
// interceptor is, and an authenticated same-key dialler IS the peer. Refusing the newcomer
// bought nothing and cost the pair its only route back from a half-open link, where the
// incumbent handler can hold the guard indefinitely on a dead stream.
var serverHost = await BuildServerAsync(NewDbPath(), apiKey: null, sink: null);
var serverDb = serverHost.Services.GetRequiredService<ILocalDb>();
var handler = serverHost.GetTestServer().CreateHandler();
@@ -85,11 +89,15 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
// Convergence proves the first stream's server handler is active and holding the single-stream guard.
await WaitForAsync(async () => (await ReadOrders(serverDb)).Count == 1, RunTimeout);
var attemptsBefore = bg.ConnectionAttempts;
var client = new LocalDbSync.LocalDbSyncClient(ChannelOver(handler));
using var call2 = client.Sync(cancellationToken: cts.Token);
var ex = await Assert.ThrowsAsync<RpcException>(() => call2.ResponseStream.MoveNext(cts.Token));
Assert.Equal(StatusCode.AlreadyExists, ex.StatusCode);
Assert.True(await call2.ResponseStream.MoveNext(cts.Token));
Assert.Equal(SyncMessage.MsgOneofCase.Handshake, call2.ResponseStream.Current.MsgCase);
// The incumbent really ended: the displaced client notices and dials again.
await WaitForAsync(() => Task.FromResult(bg.ConnectionAttempts > attemptsBefore), RunTimeout);
await bg.StopAsync(CancellationToken.None);
}
@@ -173,6 +181,71 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(3), $"teardown took {sw.Elapsed}");
}
// ---- inbox backpressure ------------------------------------------------------------------
[Fact]
public async Task Inbox_SlowReader_StopsDrainingTheStream_AndLosesNothing()
{
// The pump must STOP pulling from the gRPC stream once the inbox is full — that stall is
// what closes the peer's flow-control window and bounds this process's memory at
// InboxCapacity x MaxBatchBytes instead of "whatever the peer decides to send" (a full
// snapshot of a large dataset, on both nodes at once, when a pair baselines late).
const int capacity = Replication.Internal.GrpcSyncDuplex.InboxCapacity;
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(15));
var reader = new CountingReader();
var (duplex, readerTask) = Replication.Internal.GrpcSyncDuplex.Create(
reader, new HangingWriter(), cts.Token);
// Nothing reads the inbox: the pump fills it and blocks.
await WaitForAsync(() => Task.FromResult(reader.Produced >= capacity), TimeSpan.FromSeconds(5));
await Task.Delay(200, cts.Token);
// capacity in the channel + at most one in the pump's hand, blocked on WriteAsync.
Assert.True(reader.Produced <= capacity + 1,
$"the reader pump kept draining an already-full inbox: {reader.Produced} messages read for a {capacity}-message bound");
// Draining releases it, and every message arrives exactly once in stream order — TryWrite
// on a full bounded channel would have dropped them silently instead.
for (long expected = 1; expected <= capacity + 1; expected++)
{
var msg = await ReadInboxAsync(duplex, cts.Token);
Assert.Equal(expected, msg.DeltaAck.AppliedThruSeq);
}
await WaitForAsync(() => Task.FromResult(reader.Produced > capacity + 1), TimeSpan.FromSeconds(5));
cts.Cancel();
try { await readerTask.WaitAsync(TimeSpan.FromSeconds(5)); }
catch (OperationCanceledException) { }
}
private static async Task<SyncMessage> ReadInboxAsync(
Replication.Internal.SyncDuplex duplex, CancellationToken ct)
{
await duplex.Inbox.WaitToReadAsync(ct);
duplex.Inbox.TryRead(out var message);
return message!;
}
// Produces messages as fast as the inbox will take them, numbering each one so the consumer can
// prove nothing was dropped or reordered.
private sealed class CountingReader : IAsyncStreamReader<SyncMessage>
{
private long _produced;
public long Produced => Interlocked.Read(ref _produced);
public SyncMessage Current { get; private set; } = new();
public Task<bool> MoveNext(CancellationToken cancellationToken)
{
cancellationToken.ThrowIfCancellationRequested();
Current = new SyncMessage
{
DeltaAck = new DeltaAck { AppliedThruSeq = Interlocked.Increment(ref _produced) },
};
return Task.FromResult(true);
}
}
// ---- validator -------------------------------------------------------------------------
[Fact]
@@ -197,9 +270,13 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
TombstoneRetention = TimeSpan.Zero,
ReconnectBackoffMax = TimeSpan.Zero,
MaxHlcDriftAhead = TimeSpan.Zero,
KeepAlivePingDelay = TimeSpan.Zero,
KeepAlivePingTimeout = TimeSpan.Zero,
});
Assert.True(result.Failed);
Assert.Contains(result.Failures, f => f.Contains("KeepAlivePingDelay", StringComparison.Ordinal));
Assert.Contains(result.Failures, f => f.Contains("KeepAlivePingTimeout", StringComparison.Ordinal));
Assert.Contains(result.Failures, f => f.Contains("FlushInterval", StringComparison.Ordinal));
Assert.Contains(result.Failures, f => f.Contains("MaxBatchSize", StringComparison.Ordinal));
Assert.Contains(result.Failures, f => f.Contains("MaxOplogRows", StringComparison.Ordinal));
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/LocalDbSchemaTests.cs
+107 -1
View File
@@ -51,12 +51,118 @@ public sealed class LocalDbSchemaTests : IDisposable
foreach (var expected in new[]
{
"__localdb_meta", "__localdb_oplog", "__localdb_peer_state",
"__localdb_row_version", "__localdb_applying", "__localdb_dead_letter"
"__localdb_row_version", "__localdb_applying", "__localdb_dead_letter",
"__localdb_snapshot_state",
})
Assert.Contains(expected, tables);
// __localdb_oplog_hlc is deliberately NOT created: schema v2 removed it. It indexed
// (table_name, pk_json, hlc), which no query has ever used — every oplog access path is by
// seq or a full hlc-predicate delete — while costing a b-tree insert inside every captured
// consumer write.
Assert.Equal(0, ScalarLong(conn,
"SELECT count(*) FROM sqlite_master WHERE type='index' AND name='__localdb_oplog_hlc'"));
Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
}
[Fact]
public void EnsureCreated_UpgradesLegacyFile_DropsOplogHlcIndex()
{
using var conn = Open();
// A file exactly as schema v1 left it: the tables, the index, and a v1 stamp.
LocalDbSchema.EnsureCreated(conn);
Exec(conn, "CREATE INDEX __localdb_oplog_hlc ON __localdb_oplog(table_name, pk_json, hlc);" +
"UPDATE __localdb_meta SET schema_version = 1 WHERE id = 1;");
Assert.Equal(1, ScalarLong(conn,
"SELECT count(*) FROM sqlite_master WHERE type='index' AND name='__localdb_oplog_hlc'"));
LocalDbSchema.EnsureCreated(conn);
Assert.Equal(0, ScalarLong(conn,
"SELECT count(*) FROM sqlite_master WHERE type='index' AND name='__localdb_oplog_hlc'"));
Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
}
[Fact]
public void EnsureCreated_UpgradesV2File_AddsPerTableSnapshotState()
{
using var conn = Open();
// A file exactly as schema v2 left it: everything except __localdb_snapshot_state, stamped 2.
LocalDbSchema.EnsureCreated(conn);
Exec(conn, "DROP TABLE __localdb_snapshot_state;" +
"UPDATE __localdb_meta SET schema_version = 2 WHERE id = 1;");
LocalDbSchema.EnsureCreated(conn);
Assert.Equal(1, ScalarLong(conn,
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='__localdb_snapshot_state'"));
// Empty is the right v3 state for a file that never baselined anything: no table owes a
// snapshot, and the database-wide flag it was carrying keeps whatever value it had.
Assert.Equal(0, ScalarLong(conn, "SELECT COUNT(*) FROM __localdb_snapshot_state"));
Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
}
[Fact]
public void EnsureCreated_Upgrade_PreservesTheDatabaseWideSnapshotFlag()
{
// A v2 file mid-resync (cap prune flagged a full snapshot, the peer was down) must still owe
// that snapshot after the upgrade. v3 narrows where NEW debt is recorded; it does not
// reinterpret debt already on disk.
using var conn = Open();
LocalDbSchema.EnsureCreated(conn);
Exec(conn, "DROP TABLE __localdb_snapshot_state;" +
"UPDATE __localdb_meta SET schema_version = 2 WHERE id = 1;" +
"UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1;");
LocalDbSchema.EnsureCreated(conn);
Assert.Equal(1, ScalarLong(conn, "SELECT needs_snapshot FROM __localdb_peer_state WHERE id=1"));
}
[Fact]
public void EnsureCreated_Upgrade_PreservesNodeIdentityAndOplogRows()
{
using var conn = Open();
LocalDbSchema.EnsureCreated(conn);
var nodeId = ScalarString(conn, "SELECT node_id FROM __localdb_meta WHERE id=1");
Exec(conn,
"CREATE INDEX __localdb_oplog_hlc ON __localdb_oplog(table_name, pk_json, hlc);" +
"UPDATE __localdb_meta SET schema_version = 1 WHERE id = 1;" +
"INSERT INTO __localdb_oplog (table_name, pk_json, row_json, hlc, node_id, is_tombstone) " +
"VALUES ('orders', '{\"id\":1}', '{\"id\":1}', 42, 'n', 0);" +
"UPDATE __localdb_peer_state SET last_acked_seq = 7 WHERE id = 1;");
LocalDbSchema.EnsureCreated(conn);
// The upgrade is DDL only — identity, replication watermarks and captured data survive it.
Assert.Equal(nodeId, ScalarString(conn, "SELECT node_id FROM __localdb_meta WHERE id=1"));
Assert.Equal(1, ScalarLong(conn, "SELECT COUNT(*) FROM __localdb_oplog"));
Assert.Equal(42, ScalarLong(conn, "SELECT hlc FROM __localdb_oplog"));
Assert.Equal(7, ScalarLong(conn, "SELECT last_acked_seq FROM __localdb_peer_state WHERE id=1"));
}
[Fact]
public void EnsureCreated_AlreadyCurrent_LeavesVersionAlone()
{
using var conn = Open();
LocalDbSchema.EnsureCreated(conn);
LocalDbSchema.EnsureCreated(conn);
LocalDbSchema.EnsureCreated(conn);
Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
}
private static void Exec(SqliteConnection conn, string sql)
{
using var cmd = conn.CreateCommand();
cmd.CommandText = sql;
cmd.ExecuteNonQuery();
}
[Fact]
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/LwwApplierTests.cs
+127
View File
@@ -404,6 +404,133 @@ public sealed class LwwApplierTests : IDisposable
Assert.Equal(("NEW", (long?)7), await ReadOrder(db, 2));
}
[Fact]
public async Task Apply_PreparesEachStatementOnce_NotOncePerRow()
{
// The regression this guards: the old shape went through the generic transaction façade,
// which builds a fresh SqliteCommand — and therefore re-parses the SQL — per call. Three
// statements per applied row means a 200-row batch parsed 600 statements to run 6 distinct
// ones. Statement count must track distinct SQL, not rows.
using var db = await NewOrdersDb();
await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
db.RegisterReplicated("widgets");
var entries = new List<OplogEntryRecord>();
for (var i = 1; i <= 200; i++)
entries.Add(Entry(i, $"{{\"id\":{i}}}", $"{{\"id\":{i},\"sku\":\"S\",\"qty\":{i}}}", 1_000_000 + i, "remote"));
// A second table and a tombstone, so every statement kind in the set is exercised.
entries.Add(new OplogEntryRecord(
201, "widgets", "{\"id\":1}", "{\"id\":1,\"name\":\"W\"}", 1_000_300, "remote", false));
entries.Add(Entry(202, "{\"id\":1}", null, 1_000_400, "remote", tombstone: true));
var applier = new LwwApplier(db);
var result = await applier.ApplyBatchAsync(entries, default);
Assert.Equal(202, result.Applied);
// Six table-independent statements (applying guard, version lookup, row-version write,
// peer state, plus the per-entry savepoint's SAVEPOINT and RELEASE), an upsert per table
// touched, and a delete for the one table tombstoned. The savepoint's ROLLBACK TO and the
// dead-letter insert are absent because this batch has no poison entry — statements are
// built on first use, so the count tracks the statement KINDS the batch actually ran.
Assert.Equal(9, applier.LastBatchStatementCount);
// ...and the batch is still correct, statement reuse notwithstanding.
Assert.Null(await ReadOrder(db, 1));
Assert.Equal(("S", (long?)200), await ReadOrder(db, 200));
var widget = await db.QueryAsync("SELECT name FROM widgets WHERE id = 1", x => x.GetString(0));
Assert.Equal("W", widget[0]);
}
[Fact]
public async Task Apply_ReusedStatements_SurviveAPoisonRowMidBatch()
{
// A statement that faults is reset and reused for the following rows; if it were not, one
// poison row would silently corrupt every row after it in the batch.
using var db = await NewOrdersDb();
var applier = new LwwApplier(db);
var result = await applier.ApplyBatchAsync(
[
Entry(1, "{\"id\":1}", "{\"id\":1,\"sku\":\"BEFORE\",\"qty\":1}", 1_000_000, "remote"),
Entry(2, "{\"id\":2}", "NOT VALID JSON", 1_000_001, "remote"),
Entry(3, "{\"id\":3}", "{\"id\":3,\"sku\":\"AFTER\",\"qty\":3}", 1_000_002, "remote"),
Entry(4, "{\"id\":4}", "ALSO NOT JSON", 1_000_003, "remote"),
Entry(5, "{\"id\":5}", "{\"id\":5,\"sku\":\"LAST\",\"qty\":5}", 1_000_004, "remote"),
], default);
Assert.Equal(3, result.Applied);
Assert.Equal(2, result.DeadLettered);
Assert.Equal(("BEFORE", (long?)1), await ReadOrder(db, 1));
Assert.Equal(("AFTER", (long?)3), await ReadOrder(db, 3));
Assert.Equal(("LAST", (long?)5), await ReadOrder(db, 5));
}
[Fact]
public async Task Apply_LedgerWriteFails_LeavesNeitherTheDeleteNorTheLedgerEntry()
{
// The hazard (P4-7): applying a tombstone is TWO statements — delete the row, then write the
// ledger's tombstone version — and the poison catch commits whatever the first one already
// did. A failure landing between them therefore committed a DELETED ROW carrying a STALE LIVE
// ledger entry, which is exactly the shape SnapshotStreamer warns about and skips: the key
// then never travels again and the pair is divergent for that row, permanently and silently.
//
// The failure is injected with a trigger that aborts the ledger write (in production the
// realistic causes are disk-full / IO error / corruption — none of them reproducible here),
// and RAISE(ABORT) aborts only the statement, so the applier's narrow SqliteException catch
// treats it as per-entry poison exactly as it would a real one.
using var db = await NewOrdersDb();
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
var localHlc = await RowVersionHlc(db, Pk1);
await db.ExecuteAsync(
"CREATE TRIGGER poison_ledger BEFORE INSERT ON __localdb_row_version " +
"WHEN NEW.is_tombstone = 1 BEGIN SELECT RAISE(ABORT, 'injected ledger failure'); END");
var result = await new LwwApplier(db).ApplyBatchAsync(
[Entry(90, Pk1, null, localHlc + 1000, "remote-node", tombstone: true)], default);
// The entry is poison, so it is dead-lettered and the batch continues — unchanged semantics.
Assert.Equal(1, result.DeadLettered);
Assert.Equal(0, result.Applied);
// BOTH or NEITHER. Neither, here: the row must still be present, because its ledger entry
// still says "live at the local version".
Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
var rv = await db.QueryAsync(
"SELECT hlc, is_tombstone FROM __localdb_row_version WHERE table_name = 'orders' AND pk_json = @pk",
x => (Hlc: x.GetInt64(0), Tomb: x.GetInt64(1)), new { pk = Pk1 });
Assert.Single(rv);
Assert.Equal(localHlc, rv[0].Hlc);
Assert.Equal(0, rv[0].Tomb);
}
[Fact]
public async Task Apply_LedgerWriteFails_MidBatch_LeavesTheOtherEntriesApplied()
{
// The atomicity must be per-ENTRY, not per-batch: rolling back to the savepoint may not
// discard the rows applied before the poison one, nor block the ones after it.
using var db = await NewOrdersDb();
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2, 'DOOMED', 2)");
var doomedHlc = await RowVersionHlc(db, "{\"id\":2}");
await db.ExecuteAsync(
"CREATE TRIGGER poison_ledger BEFORE INSERT ON __localdb_row_version " +
"WHEN NEW.is_tombstone = 1 BEGIN SELECT RAISE(ABORT, 'injected ledger failure'); END");
var result = await new LwwApplier(db).ApplyBatchAsync(
[
Entry(91, Pk1, "{\"id\":1,\"sku\":\"BEFORE\",\"qty\":1}", doomedHlc + 1, "remote-node"),
Entry(92, "{\"id\":2}", null, doomedHlc + 1000, "remote-node", tombstone: true),
Entry(93, "{\"id\":3}", "{\"id\":3,\"sku\":\"AFTER\",\"qty\":3}", doomedHlc + 3, "remote-node"),
], default);
Assert.Equal(2, result.Applied);
Assert.Equal(1, result.DeadLettered);
Assert.Equal(("BEFORE", (long?)1), await ReadOrder(db, 1));
Assert.Equal(("DOOMED", (long?)2), await ReadOrder(db, 2));
Assert.Equal(("AFTER", (long?)3), await ReadOrder(db, 3));
}
[Fact]
public async Task Apply_ReplaceExistingRow_DoesNotFireCaptureTriggers()
{
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/OplogStoreTests.cs
+131
View File
@@ -212,6 +212,137 @@ public sealed class OplogStoreTests : IDisposable
Assert.Equal("{\"id\":2}", remaining[0].PkJson);
}
// ---- O(1) backlog depth --------------------------------------------------------------
// The definition the O(1) formula has to reproduce. Every scenario below is asserted against
// this, not against a hand-computed number, so the two can never drift apart.
private static async Task<long> CountingDepth(SqliteLocalDb db) =>
(await db.QueryAsync(
"SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)",
r => r.GetInt64(0)))[0];
private static async Task AssertDepthMatchesCount(SqliteLocalDb db, OplogStore store)
{
var expected = await CountingDepth(db);
Assert.Equal(expected, await store.GetOplogDepthAsync(default));
Assert.Equal(expected, store.GetOplogDepthSync());
}
[Fact]
public async Task OplogDepth_Empty_IsZero()
{
using var db = await NewOrdersDb();
var store = new OplogStore(db, new ReplicationOptions());
await AssertDepthMatchesCount(db, store);
Assert.Equal(0, await store.GetOplogDepthAsync(default));
}
[Fact]
public async Task OplogDepth_AllAcked_EmptiesToZero_NotToTheSeqRange()
{
using var db = await NewOrdersDb();
await WriteRows(db, 5);
var store = new OplogStore(db, new ReplicationOptions());
await store.RecordPeerAckAsync(await store.GetMaxSeqAsync(default), default);
await AssertDepthMatchesCount(db, store);
Assert.Equal(0, await store.GetOplogDepthAsync(default));
}
[Fact]
public async Task OplogDepth_AfterCapPrune_ReportsRetainedRows_NotTheWholeSeqRange()
{
// The reason the formula clamps its floor to MIN(seq) - 1. A cap prune leaves the retained
// rows far above an ack watermark that never moved, so a naive MAX(seq) - last_acked would
// report the entire seq range forever and re-flag the cap on every maintenance tick.
using var db = await NewOrdersDb();
await WriteRows(db, 40);
var store = new OplogStore(db, new ReplicationOptions { MaxOplogRows = 5 });
Assert.True(await store.EnforceCapsAsync(default));
await AssertDepthMatchesCount(db, store);
Assert.Equal(5, await store.GetOplogDepthAsync(default));
// And the next tick is clean: depth is not over cap any more.
Assert.False(await store.EnforceCapsAsync(default));
}
[Fact]
public async Task OplogDepth_AfterAgePrune_MatchesCount()
{
using var db = await NewOrdersDb();
await WriteRows(db, 6);
// Backdate the oldest three past the age cap; HLCs rise with seq, so this is a prefix.
var expired = DateTimeOffset.UtcNow.AddDays(-30).ToUnixTimeMilliseconds() << 16;
await db.ExecuteAsync(
"UPDATE __localdb_oplog SET hlc = @h WHERE seq IN (SELECT seq FROM __localdb_oplog ORDER BY seq LIMIT 3)",
new { h = expired });
var store = new OplogStore(db, new ReplicationOptions());
Assert.True(await store.EnforceCapsAsync(default));
await AssertDepthMatchesCount(db, store);
Assert.Equal(3, await store.GetOplogDepthAsync(default));
}
[Fact]
public async Task OplogDepth_PartialAck_MatchesCount()
{
using var db = await NewOrdersDb();
await WriteRows(db, 9);
var store = new OplogStore(db, new ReplicationOptions());
var batch = await store.ReadBatchAboveAsync(0, 100, default);
await store.RecordPeerAckAsync(batch[3].Seq, default);
await AssertDepthMatchesCount(db, store);
Assert.Equal(5, await store.GetOplogDepthAsync(default));
}
[Fact]
public async Task OplogDepth_AckPruneThenMoreWrites_MatchesCount()
{
using var db = await NewOrdersDb();
await WriteRows(db, 4);
var store = new OplogStore(db, new ReplicationOptions());
await store.RecordPeerAckAsync(await store.GetMaxSeqAsync(default), default);
// The steady state of a converged pair: oplog pruned empty, watermark high, new writes above it.
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (91,'S',1),(92,'S',2)");
await AssertDepthMatchesCount(db, store);
Assert.Equal(2, await store.GetOplogDepthAsync(default));
}
[Fact]
public async Task OplogDepth_LargeBacklog_IsConstantTime()
{
// 200k rows: a COUNT(*) over the backlog runs ~200k VM steps per call (measured: 100,020 at
// 50k rows), the seek-based formula runs 58 regardless of size. The loop below costs tens of
// milliseconds at seek shape and seconds at scan shape, so the budget separates them by a
// wide margin in both directions.
using var db = await NewOrdersDb();
await db.ExecuteAsync(
"INSERT INTO __localdb_oplog (table_name, pk_json, row_json, hlc, node_id, is_tombstone) " +
"WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x + 1 FROM c WHERE x < 200000) " +
"SELECT 'orders', '{\"id\":' || x || '}', '{\"id\":' || x || '}', x << 16, 'n', 0 FROM c");
var store = new OplogStore(db, new ReplicationOptions());
Assert.Equal(200_000, await store.GetOplogDepthAsync(default));
using var conn = db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = OplogStore.DepthSql;
var sw = System.Diagnostics.Stopwatch.StartNew();
for (var i = 0; i < 2_000; i++)
Assert.Equal(200_000L, (long)cmd.ExecuteScalar()!);
sw.Stop();
Assert.True(sw.ElapsedMilliseconds < 1_500,
$"2000 depth reads over a 200k backlog took {sw.ElapsedMilliseconds} ms — that is scan-shaped, not seek-shaped.");
}
[Fact]
public async Task PeerState_RoundTrips()
{
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/RegistrationTests.cs
+496
View File
@@ -70,6 +70,97 @@ public sealed class RegistrationTests : IDisposable
Assert.Contains("BLOB", ex.Message);
}
// ---- Cascading-delete foreign-key parents (P4-7) --------------------------------------
[Fact]
public async Task Register_CascadeDeleteFkParent_Throws()
{
// The hazard: applying a remote row uses INSERT OR REPLACE, whose implicit DELETE fires
// ON DELETE CASCADE while the capture guard is RAISED — so the cascaded child rows are
// deleted locally and never captured, and the peer keeps them forever.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync(
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, order_id INTEGER " +
"REFERENCES orders(id) ON DELETE CASCADE)");
var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("orders"));
Assert.Contains("orders", ex.Message);
Assert.Contains("order_lines", ex.Message);
Assert.Contains("ON DELETE CASCADE", ex.Message);
// Refused means refused: no triggers installed, nothing in the registry.
Assert.Empty(await Triggers(db));
Assert.False(db.ReplicatedTables.ContainsKey("orders"));
}
[Fact]
public async Task Register_CascadeDeleteFkChild_IsAllowed()
{
// Only the PARENT is refused. The child side is safe: replacing a child row cascades
// nothing, and the child's own deletes are captured normally.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync(
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, order_id INTEGER " +
"REFERENCES orders(id) ON DELETE CASCADE)");
db.RegisterReplicated("order_lines");
Assert.True(db.ReplicatedTables.ContainsKey("order_lines"));
}
[Theory]
[InlineData("ON DELETE SET NULL", "ON DELETE SET NULL")]
[InlineData("ON DELETE SET DEFAULT", "ON DELETE SET DEFAULT")]
public async Task Register_OtherRowMutatingDeleteActions_AlsoThrow(string clause, string expected)
{
// SET NULL / SET DEFAULT mutate child rows on the same implicit DELETE and are captured
// just as little — same divergence, so the same refusal.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync(
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, order_id INTEGER DEFAULT 0 " +
$"REFERENCES orders(id) {clause})");
var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("orders"));
Assert.Contains(expected, ex.Message);
}
[Fact]
public async Task Register_NonCascadingFkParent_IsAllowed()
{
// The default (NO ACTION / RESTRICT) mutates no child row: the implicit DELETE simply
// fails the statement, which the applier already surfaces as a dead-lettered entry. Noisy,
// not silent — so there is nothing to refuse.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync(
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, order_id INTEGER REFERENCES orders(id))");
await db.ExecuteAsync("CREATE TABLE restricted (id INTEGER PRIMARY KEY, " +
"order_id INTEGER REFERENCES orders(id) ON DELETE RESTRICT)");
db.RegisterReplicated("orders");
Assert.Equal(3, (await Triggers(db)).Count);
}
[Fact]
public async Task Register_UnrelatedTablesWithCascades_DoNotBlockRegistration()
{
// The scan must key on the table being registered, not on "the database contains a cascade".
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
await db.ExecuteAsync(
"CREATE TABLE widget_parts (id INTEGER PRIMARY KEY, widget_id INTEGER " +
"REFERENCES widgets(id) ON DELETE CASCADE)");
db.RegisterReplicated("orders");
Assert.True(db.ReplicatedTables.ContainsKey("orders"));
}
[Fact]
public async Task Register_IsIdempotent_RegeneratesTriggers()
{
@@ -128,4 +219,409 @@ public sealed class RegistrationTests : IDisposable
Assert.Equal(new[] { "id" }, table.PkColumns);
Assert.Equal(new[] { "id", "sku", "qty" }, table.Columns.Select(c => c.Name).ToArray());
}
// ---- Deregistration ------------------------------------------------------------------
[Fact]
public async Task Deregister_DropsTriggers_PrunesOplogAndLedger_LeavesDataAlone()
{
using var db = Create();
await CreateOrders(db);
db.RegisterReplicated("orders");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1),(2,'B',2)");
var removed = db.DeregisterReplicated("orders");
Assert.True(removed);
Assert.Empty(await Triggers(db));
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name='orders'"));
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version WHERE table_name='orders'"));
Assert.False(db.ReplicatedTables.ContainsKey("orders"));
// The consumer's own data is never touched — this removes CAPTURE, not rows.
Assert.Equal(2, await CountAsync(db, "SELECT COUNT(*) FROM orders"));
}
[Fact]
public async Task Deregister_StopsCapture()
{
using var db = Create();
await CreateOrders(db);
db.RegisterReplicated("orders");
db.DeregisterReplicated("orders");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
await db.ExecuteAsync("UPDATE orders SET qty = 9 WHERE id = 1");
await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog"));
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
}
[Fact]
public async Task Deregister_IsIdempotent_SecondCallReportsNothingToDo()
{
using var db = Create();
await CreateOrders(db);
db.RegisterReplicated("orders");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
Assert.True(db.DeregisterReplicated("orders"));
Assert.False(db.DeregisterReplicated("orders"));
Assert.False(db.DeregisterReplicated("orders"));
Assert.Empty(await Triggers(db));
}
[Fact]
public async Task Deregister_NeverRegisteredTable_IsANoOp()
{
using var db = Create();
await CreateOrders(db);
Assert.False(db.DeregisterReplicated("orders"));
Assert.False(db.DeregisterReplicated("ghost"));
}
[Fact]
public async Task Deregister_ClearsTriggersLeftByAnEarlierProcess()
{
// The primary use: a database file registered by an OLDER build, opened by one that has
// decided this node does not replicate. Nothing in memory says the table was ever
// registered — the evidence is entirely on disk.
using (var first = Create())
{
await CreateOrders(first);
first.RegisterReplicated("orders");
await first.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
}
SqliteConnection.ClearAllPools();
using var second = Create();
Assert.Equal(3, (await Triggers(second)).Count);
Assert.False(second.ReplicatedTables.ContainsKey("orders"));
Assert.True(second.DeregisterReplicated("orders"));
Assert.Empty(await Triggers(second));
Assert.Equal(0, await CountAsync(second, "SELECT COUNT(*) FROM __localdb_oplog"));
}
[Fact]
public async Task Deregister_LeavesOtherTablesCapturing()
{
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
db.RegisterReplicated("orders");
db.RegisterReplicated("widgets");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W')");
db.DeregisterReplicated("orders");
await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (2,'W2')");
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name='orders'"));
Assert.Equal(2, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name='widgets'"));
Assert.True(db.ReplicatedTables.ContainsKey("widgets"));
}
[Fact]
public async Task Deregister_ThenRegisterWithBaseline_RestoresTheLedger()
{
// Pruning the ledger is only safe because re-registration can re-baseline; this is that
// round trip.
using var db = Create();
await CreateOrders(db);
db.RegisterReplicated("orders");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1),(2,'B',2)");
db.DeregisterReplicated("orders");
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
db.RegisterReplicated("orders", baselineExistingRows: true);
Assert.Equal(2, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version WHERE table_name='orders'"));
}
// ---- Baselining ----------------------------------------------------------------------
[Fact]
public async Task Register_WithoutBaseline_LeavesPreexistingRowsUnledgered()
{
// The documented CDC contract, unchanged: opting in is what changes behaviour, not upgrading.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
db.RegisterReplicated("orders");
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
Assert.False(await NeedsSnapshot(db));
}
[Fact]
public async Task Register_WithBaseline_SeedsAtHlcZero_AndFlagsSnapshot()
{
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1),(2,'B',2)");
db.RegisterReplicated("orders", baselineExistingRows: true);
var seeded = await db.QueryAsync(
"SELECT pk_json, hlc, node_id, is_tombstone FROM __localdb_row_version ORDER BY pk_json",
r => (Pk: r.GetString(0), Hlc: r.GetInt64(1), Node: r.GetString(2), Tomb: r.GetInt64(3)));
Assert.Equal(2, seeded.Count);
Assert.Equal(["{\"id\":1}", "{\"id\":2}"], seeded.Select(s => s.Pk));
// HLC 0 is the floor: below every genuine stamp (physical ms << 16), so it can never beat one.
Assert.All(seeded, s => Assert.Equal(0, s.Hlc));
Assert.All(seeded, s => Assert.Equal(db.NodeId, s.Node));
Assert.All(seeded, s => Assert.Equal(0, s.Tomb));
// Without the flag the seeding would be inert: no gap exists at the handshake, so the peer
// would be told it needs no snapshot and the baselined rows would never travel.
Assert.True(await NeedsSnapshot(db));
// ...and it is flagged for THIS TABLE ONLY. Through 0.2.x this set the one database-wide
// bit, so baselining `orders` re-streamed every other registered table with it.
Assert.False(await NeedsFullSnapshot(db));
}
[Fact]
public async Task Register_WithBaseline_FlagsOnlyTheBaselinedTable()
{
// The per-table flag's reason to exist: opting one table into replication late must not
// schedule a full re-stream of its unrelated neighbours.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W')");
db.RegisterReplicated("widgets");
db.RegisterReplicated("orders", baselineExistingRows: true);
Assert.True(await NeedsSnapshot(db, "orders"));
Assert.False(await NeedsSnapshot(db, "widgets"));
Assert.False(await NeedsFullSnapshot(db));
}
[Fact]
public async Task Deregister_DropsThePendingSnapshotFlag()
{
// A flag naming a table nothing streams any more can never clear itself — the streamer skips
// unregistered tables — so it would sit in the file and re-appear as unasked-for debt if the
// table were registered again.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
db.RegisterReplicated("orders", baselineExistingRows: true);
Assert.True(await NeedsSnapshot(db));
db.DeregisterReplicated("orders");
Assert.False(await NeedsSnapshot(db));
}
[Fact]
public async Task Register_WithBaseline_IsIdempotent_AndDoesNotReflagSnapshot()
{
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
db.RegisterReplicated("orders", baselineExistingRows: true);
// Stand in for the snapshot having been delivered and acknowledged.
await db.ExecuteAsync("DELETE FROM __localdb_snapshot_state");
db.RegisterReplicated("orders", baselineExistingRows: true);
Assert.Equal(1, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
// Nothing new was seeded, so no resync is owed: passing true every boot is free after the first.
Assert.False(await NeedsSnapshot(db));
}
[Fact]
public async Task Register_WithBaseline_NeverOverwritesACapturedVersion()
{
// The LWW-safety core: a row with a real, captured version must keep it. A floor that
// clobbered genuine versions would make every baselined node look newer than its peer.
using var db = Create();
await CreateOrders(db);
db.RegisterReplicated("orders");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
var captured = (await db.QueryAsync(
"SELECT hlc FROM __localdb_row_version WHERE pk_json='{\"id\":1}'", r => r.GetInt64(0)))[0];
Assert.True(captured > 0);
// A second, pre-existing row that capture never saw (inserted with triggers removed).
db.DeregisterReplicated("orders");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2,'B',2)");
await db.ExecuteAsync(
"INSERT INTO __localdb_row_version (table_name, pk_json, hlc, node_id, is_tombstone) " +
"VALUES ('orders', '{\"id\":1}', @h, 'genuine', 0)", new { h = captured });
db.RegisterReplicated("orders", baselineExistingRows: true);
var versions = await db.QueryAsync(
"SELECT pk_json, hlc, node_id FROM __localdb_row_version ORDER BY pk_json",
r => (Pk: r.GetString(0), Hlc: r.GetInt64(1), Node: r.GetString(2)));
Assert.Equal(2, versions.Count);
Assert.Equal(captured, versions[0].Hlc);
Assert.Equal("genuine", versions[0].Node);
Assert.Equal(0, versions[1].Hlc);
}
[Fact]
public async Task Register_WithBaseline_SeedsPkJsonIdenticalToWhatCaptureWrites()
{
// A seeded key that differs by so much as a space from the capture triggers' json_object
// output would be a DIFFERENT ledger row, and the baseline would silently never merge.
using var db = Create();
await db.ExecuteAsync("CREATE TABLE kv (a INTEGER, b TEXT, v TEXT, PRIMARY KEY (a, b))");
await db.ExecuteAsync("INSERT INTO kv (a, b, v) VALUES (1, 'x', 'seeded')");
db.RegisterReplicated("kv", baselineExistingRows: true);
await db.ExecuteAsync("UPDATE kv SET v = 'captured' WHERE a = 1 AND b = 'x'");
// One ledger row, not two: the capture upsert landed on the seeded key.
var rows = await db.QueryAsync(
"SELECT pk_json, hlc FROM __localdb_row_version", r => (Pk: r.GetString(0), Hlc: r.GetInt64(1)));
Assert.Single(rows);
Assert.Equal("{\"a\":1,\"b\":\"x\"}", rows[0].Pk);
Assert.True(rows[0].Hlc > 0, "the genuine capture must have replaced the floor");
}
[Fact]
public async Task Register_WithBaseline_FloorLosesToEveryGenuineRemoteWrite()
{
// Proved through the real LWW path rather than by reasoning about the comparison: an
// absurdly small remote HLC of 1 still beats the baseline floor of 0.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'BASELINE',1)");
db.RegisterReplicated("orders", baselineExistingRows: true);
var result = await new Replication.Internal.LwwApplier(db).ApplyBatchAsync(
[new Replication.Internal.OplogEntryRecord(
1, "orders", "{\"id\":1}", "{\"id\":1,\"sku\":\"REMOTE\",\"qty\":9}", 1, "peer", false)],
default);
Assert.Equal(1, result.Applied);
var sku = await db.QueryAsync("SELECT sku FROM orders WHERE id = 1", r => r.GetString(0));
Assert.Equal("REMOTE", sku[0]);
}
[Fact]
public async Task Register_WithBaseline_TieBetweenTwoBaselinedNodes_ResolvesByNodeIdOrdinal()
{
// Two nodes that baselined the same key both hold HLC 0 — genuinely unordered history. The
// node-id tie-break is what keeps that convergent instead of oscillating.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'MINE',1)");
db.RegisterReplicated("orders", baselineExistingRows: true);
var applier = new Replication.Internal.LwwApplier(db);
var lower = await applier.ApplyBatchAsync(
[new Replication.Internal.OplogEntryRecord(
1, "orders", "{\"id\":1}", "{\"id\":1,\"sku\":\"LOWER\",\"qty\":9}", 0, "!" + db.NodeId, false)],
default);
Assert.Equal(1, lower.Discarded);
var higher = await applier.ApplyBatchAsync(
[new Replication.Internal.OplogEntryRecord(
2, "orders", "{\"id\":1}", "{\"id\":1,\"sku\":\"HIGHER\",\"qty\":9}", 0, db.NodeId + "z", false)],
default);
Assert.Equal(1, higher.Applied);
var sku = await db.QueryAsync("SELECT sku FROM orders WHERE id = 1", r => r.GetString(0));
Assert.Equal("HIGHER", sku[0]);
}
[Fact]
public async Task Register_WithBaseline_EmptyTable_SeedsNothing_FlagsNothing()
{
using var db = Create();
await CreateOrders(db);
db.RegisterReplicated("orders", baselineExistingRows: true);
Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
Assert.False(await NeedsSnapshot(db));
}
// ---- Deregistration and the HLC crash anchor ------------------------------------------
[Fact]
public async Task Deregister_FlushesHlcAnchor_SoAnUngracefulReopenCannotRegress()
{
// The clock recovers as MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc), and
// deregistering the only registered table empties both ledgers. Without the in-transaction
// flush, an ungraceful exit right afterwards leaves hlc_high_water at 0 and the reopened
// clock restarts from wall-clock time — below stamps the peer already holds.
var crashed = Create();
try
{
await CreateOrders(crashed);
crashed.RegisterReplicated("orders");
await crashed.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
// A peer HLC an hour ahead, merged as the applier would merge it: this is what makes the
// clock outrun wall-clock time, and therefore what a wall-clock restart would regress.
var peerHlc = DateTimeOffset.UtcNow.AddHours(1).ToUnixTimeMilliseconds() << 16;
crashed.Clock.Observe(peerHlc);
var preDereg = crashed.Clock.Current;
Assert.True(preDereg >= peerHlc);
crashed.DeregisterReplicated("orders");
Assert.Equal(preDereg, await CountAsync(crashed, "SELECT hlc_high_water FROM __localdb_meta WHERE id = 1"));
// The evidence the clock would otherwise have recovered from is gone.
Assert.Equal(0, await CountAsync(crashed, "SELECT COUNT(*) FROM __localdb_oplog"));
Assert.Equal(0, await CountAsync(crashed, "SELECT COUNT(*) FROM __localdb_row_version"));
Assert.Equal(0, await CountAsync(crashed, "SELECT last_seen_hlc FROM __localdb_peer_state WHERE id = 1"));
// Ungraceful reopen: a second instance over the same file while the first was NEVER
// disposed, so nothing but the deregistration itself can have written the anchor.
using var reopened = Create();
Assert.True(reopened.Clock.Current >= preDereg,
$"recovered clock {reopened.Clock.Current} regressed below the pre-deregistration stamp {preDereg}");
Assert.True(reopened.Clock.Next() > preDereg);
}
finally
{
crashed.Dispose();
}
}
[Fact]
public async Task Deregister_HlcFlush_DoesNotCountAsWorkDone()
{
// The flush is an UPDATE on every call, including the ones that clean up nothing; it must
// not turn the "was there anything to do?" answer into a permanent true.
using var db = Create();
await CreateOrders(db);
Assert.False(db.DeregisterReplicated("orders"));
Assert.False(db.DeregisterReplicated("ghost"));
}
private static async Task<long> CountAsync(SqliteLocalDb db, string sql) =>
(await db.QueryAsync(sql, r => r.GetInt64(0)))[0];
/// <summary>Whether <paramref name="table"/> individually owes the peer a snapshot (schema v3).</summary>
private static async Task<bool> NeedsSnapshot(SqliteLocalDb db, string table = "orders") =>
(await db.QueryAsync(
"SELECT COUNT(*) FROM __localdb_snapshot_state WHERE table_name = @t AND needs_snapshot <> 0",
r => r.GetInt64(0), new { t = table }))[0] != 0;
/// <summary>
/// The DATABASE-WIDE flag — "every registered table". Baselining must never set it: that is the
/// whole point of the per-table flags, and setting it would restore the 0.2.x behaviour silently.
/// </summary>
private static async Task<bool> NeedsFullSnapshot(SqliteLocalDb db) =>
(await db.QueryAsync(
"SELECT needs_snapshot FROM __localdb_peer_state WHERE id = 1", r => r.GetInt64(0)))[0] != 0;
}
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/SnapshotResyncTests.cs
+434 -7
View File
@@ -67,10 +67,15 @@ public sealed class SnapshotResyncTests : IDisposable
return new Side(db, store, applier, session);
}
private static (SyncDuplex A, SyncDuplex B) DuplexPair()
// capacity models a transport that applies backpressure (which the gRPC duplex now is, via its
// bounded inbox); null is the historical unbounded pipe.
private static (SyncDuplex A, SyncDuplex B) DuplexPair(int? capacity = null)
{
var aToB = Channel.CreateUnbounded<SyncMessage>();
var bToA = Channel.CreateUnbounded<SyncMessage>();
Channel<SyncMessage> New() => capacity is int c
? Channel.CreateBounded<SyncMessage>(c)
: Channel.CreateUnbounded<SyncMessage>();
var aToB = New();
var bToA = New();
var a = new SyncDuplex { Send = (m, ct) => aToB.Writer.WriteAsync(m, ct).AsTask(), Inbox = bToA.Reader };
var b = new SyncDuplex { Send = (m, ct) => bToA.Writer.WriteAsync(m, ct).AsTask(), Inbox = aToB.Reader };
return (a, b);
@@ -84,9 +89,19 @@ public sealed class SnapshotResyncTests : IDisposable
return (duplex, toSession.Writer, fromSession.Reader);
}
private static Handshake MatchingHandshake(SqliteLocalDb sessionDb, string peerNodeId, long lastAppliedRemoteSeq = 0)
// supportsPartialSnapshot defaults to FALSE, which is exactly what a 0.1.x/0.2.x peer puts on
// the wire (the field does not exist in its proto, so proto3 decodes the default). Scripted
// peers are therefore old peers unless a test says otherwise.
private static Handshake MatchingHandshake(
SqliteLocalDb sessionDb, string peerNodeId, long lastAppliedRemoteSeq = 0, bool supportsPartialSnapshot = false)
{
var h = new Handshake { NodeId = peerNodeId, LibSchemaVersion = 1, LastAppliedRemoteSeq = lastAppliedRemoteSeq };
var h = new Handshake
{
NodeId = peerNodeId,
LibSchemaVersion = 1,
LastAppliedRemoteSeq = lastAppliedRemoteSeq,
SupportsPartialSnapshot = supportsPartialSnapshot,
};
foreach (var t in sessionDb.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
h.Tables.Add(new TableDigest { TableName = t.Name, Digest = t.Digest });
return h;
@@ -109,14 +124,34 @@ public sealed class SnapshotResyncTests : IDisposable
// Completes the two-phase handshake against a real session: the peer never requests a snapshot.
private static async Task DriveHandshakeAsync(
SqliteLocalDb sessionDb, ChannelWriter<SyncMessage> toSession, ChannelReader<SyncMessage> fromSession, CancellationToken ct)
SqliteLocalDb sessionDb, ChannelWriter<SyncMessage> toSession, ChannelReader<SyncMessage> fromSession,
CancellationToken ct, bool supportsPartialSnapshot = false, long lastAppliedRemoteSeq = 0)
{
await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.Handshake, ct);
await toSession.WriteAsync(new SyncMessage { Handshake = MatchingHandshake(sessionDb, "peer") }, ct);
await toSession.WriteAsync(
new SyncMessage
{
Handshake = MatchingHandshake(
sessionDb, "peer", lastAppliedRemoteSeq, supportsPartialSnapshot),
}, ct);
await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.HandshakeAck, ct);
await toSession.WriteAsync(new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = "peer", SnapshotRequired = false } }, ct);
}
// Reads outbound messages until (and including) the first of the given kind.
private static async Task<List<SyncMessage>> DrainThroughAsync(
ChannelReader<SyncMessage> reader, SyncMessage.MsgOneofCase last, CancellationToken ct)
{
var all = new List<SyncMessage>();
while (true)
{
var msg = await NextAsync(reader, ct);
all.Add(msg);
if (msg.MsgCase == last)
return all;
}
}
private static async Task SwallowAsync(Task task)
{
try { await task; }
@@ -154,6 +189,12 @@ public sealed class SnapshotResyncTests : IDisposable
return r[0];
}
/// <summary>The tables individually flagged as owing the peer a snapshot (schema v3).</summary>
private static async Task<List<string>> PendingSnapshotTables(SqliteLocalDb db) =>
(await db.QueryAsync(
"SELECT table_name FROM __localdb_snapshot_state WHERE needs_snapshot <> 0 ORDER BY table_name",
x => x.GetString(0))).ToList();
private static async Task<long> OplogCount(SqliteLocalDb db)
{
var r = await db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", x => x.GetInt64(0));
@@ -170,6 +211,50 @@ public sealed class SnapshotResyncTests : IDisposable
IsTombstone = false,
};
[Fact]
public async Task MutualSnapshot_OverABoundedTransport_ConvergesWithoutDeadlock()
{
// BOTH sides owe a snapshot — the ordinary outcome of enabling replication late on a pair,
// since baselining flags needs_snapshot on whichever node baselined and both nodes do.
//
// The snapshot used to be streamed from INSIDE the handshake, before the receive loop
// existed, so neither side drained its inbox while sending. That exchange completed only
// because the inbox was unbounded and could hold the peer's ENTIRE snapshot in memory —
// which is the memory profile this design gave up on. Over a transport that pushes back
// (which the gRPC duplex now is, with its bounded inbox) the old shape deadlocks outright.
// Streaming the snapshot alongside an already-running receive loop is what makes both the
// bound and this test possible.
var options = new ReplicationOptions { MaxBatchSize = 2, FlushInterval = TimeSpan.FromMilliseconds(20) };
var a = await NewSideAsync(options);
var b = await NewSideAsync(options);
const int rowsPerSide = 60;
for (var i = 1; i <= rowsPerSide; i++)
{
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
await b.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'B', @id)", new { id = 100 + i });
}
await a.Store.SetNeedsSnapshotAsync(true, default);
await b.Store.SetNeedsSnapshotAsync(true, default);
// Capacity 1 each way: the tightest possible bound, so a side that stops draining its inbox
// to send stalls at once instead of getting away with it on buffer slack.
var (da, db) = DuplexPair(capacity: 1);
using var cts = new CancellationTokenSource(RunTimeout);
var runA = a.Session.RunAsync(da, cts.Token);
var runB = b.Session.RunAsync(db, cts.Token);
await WaitForAsync(
async () => (await ReadOrders(a.Db)).Count == 2 * rowsPerSide
&& (await ReadOrders(b.Db)).Count == 2 * rowsPerSide,
RunTimeout);
Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
cts.Cancel();
await SwallowAsync(runA);
await SwallowAsync(runB);
}
[Fact]
public async Task WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot()
{
@@ -292,6 +377,93 @@ public sealed class SnapshotResyncTests : IDisposable
await SwallowAsync(runB);
}
[Fact]
public async Task CapPrune_IntoALiveSessionsUnsentRange_FaultsIt_AndTheReconnectSnapshots()
{
// P4-1. The pump reads `seq > sentThruSeq`, which resumes SILENTLY above a hole a cap prune
// punched — so it keeps shipping later deltas, the peer acks them, and the pair is durably
// divergent. `needs_snapshot` (set in the prune's own transaction) heals only a RECONNECT,
// and nothing forced one: on a stable LAN with a slow-but-alive peer the session, and the
// divergence, could last for days. Faulting is what turns the persisted flag into a snapshot.
var a = await NewSideAsync();
for (var i = 1; i <= 6; i++)
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = a.Session.RunAsync(duplex, cts.Token);
// The peer holds everything so far: no snapshot is owed, and the pump idles at seq 6.
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, lastAppliedRemoteSeq: 6);
// Write three more rows and prune all but the newest, in ONE transaction — which is what a
// cap prune is, and what makes this deterministic: the pump's reader can only ever see the
// before state (nothing above seq 6) or the after state (seqs 7 and 8 already gone). It can
// never catch rows 7/8 in between and legitimately advance past them.
await using (var tx = await a.Db.BeginTransactionAsync(cts.Token))
{
await tx.ExecuteAsync(
"INSERT INTO orders (id, sku, qty) VALUES (7,'PRUNED',7),(8,'PRUNED',8),(9,'KEPT',9)", null, cts.Token);
await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1", null, cts.Token);
await tx.ExecuteAsync(
"DELETE FROM __localdb_oplog WHERE seq < (SELECT MAX(seq) FROM __localdb_oplog)", null, cts.Token);
await tx.CommitAsync(cts.Token);
}
var fault = await Assert.ThrowsAsync<LocalDbOplogGapException>(() => run);
Assert.Contains("6", fault.Message, StringComparison.Ordinal);
// And it faulted BEFORE shipping anything from above the hole: a delta for seq 9 would have
// moved the peer's watermark past rows 7 and 8, which is the divergence itself.
while (fromSession.TryRead(out var stray))
Assert.NotEqual(SyncMessage.MsgOneofCase.DeltaBatch, stray.MsgCase);
// The reconnect is what heals: the prune's needs_snapshot makes the handshake owe a full
// snapshot, which carries the pruned rows' current state to a real peer.
var a2 = NewSessionOver(a.Db);
var b = await NewSideAsync();
var (da, db) = DuplexPair();
var run2A = a2.Session.RunAsync(da, cts.Token);
var run2B = b.Session.RunAsync(db, cts.Token);
await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 9, RunTimeout);
Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
cts.Cancel();
await SwallowAsync(run2A);
await SwallowAsync(run2B);
}
[Fact]
public async Task AckPruningToEmpty_DoesNotFaultALiveSession()
{
// The other half of the horizon check: ordinary ack-pruning deletes seq <= last_acked_seq,
// which by construction sits at or below the sent watermark, so a converged pair that has
// pruned its whole oplog must keep streaming. A gap test written against `needs_snapshot`,
// or one that treated any risen horizon as a fault, would kill healthy sessions here.
var a = await NewSideAsync();
var b = await NewSideAsync();
for (var i = 1; i <= 4; i++)
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
var (da, db) = DuplexPair();
using var cts = new CancellationTokenSource(RunTimeout);
var runA = a.Session.RunAsync(da, cts.Token);
var runB = b.Session.RunAsync(db, cts.Token);
await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 4, RunTimeout);
await WaitForAsync(async () => await OplogCount(a.Db) == 0, RunTimeout);
// Still live, still incremental: a post-prune write reaches B as an ordinary delta.
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (5, 'POST', 5)");
await WaitForAsync(async () => (await ReadOrders(b.Db)).Any(r => r.Id == 5), RunTimeout);
Assert.False(runA.IsFaulted, runA.Exception?.ToString());
cts.Cancel();
await SwallowAsync(runA);
await SwallowAsync(runB);
}
[Fact]
public async Task Snapshot_MergesNotTruncates()
{
@@ -536,6 +708,261 @@ public sealed class SnapshotResyncTests : IDisposable
await SwallowAsync(run);
}
// ---- Per-table snapshot scoping (0.3.0) ----------------------------------------------
private const string WidgetsSql = "CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)";
/// <summary>
/// A node with two registered tables where exactly one — <c>orders</c> — was baselined.
/// <c>widgets</c> is registered BEFORE its rows are written, so ordinary capture ledgers them;
/// <c>orders</c> is registered after, so only baselining can. That asymmetry is what makes the
/// scope observable: a full snapshot carries both tables' ledgers, a scoped one carries orders
/// alone — and both tables hold rows either way, so an empty widgets stream cannot be mistaken
/// for "there was nothing to send".
/// </summary>
private async Task<Side> NewBaselinedSideAsync(ReplicationOptions? options = null)
{
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
_paths.Add(path);
var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
_disposables.Add(db);
await db.ExecuteAsync(OrdersSql);
await db.ExecuteAsync(WidgetsSql);
// Predates registration: invisible to CDC, so only baselining puts it in the ledger.
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'PRE',1),(2,'PRE',2)");
db.RegisterReplicated("widgets");
await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W1'),(2,'W2')");
db.RegisterReplicated("orders", baselineExistingRows: true);
return NewSessionOver(db, options);
}
/// <summary>The peer of <see cref="NewBaselinedSideAsync"/>: the same two tables, empty, owing nothing.</summary>
private async Task<Side> NewTwoTableSideAsync(ReplicationOptions? options = null)
{
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
_paths.Add(path);
var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
_disposables.Add(db);
await db.ExecuteAsync(OrdersSql);
await db.ExecuteAsync(WidgetsSql);
db.RegisterReplicated("orders");
db.RegisterReplicated("widgets");
return NewSessionOver(db, options);
}
private static List<string> SnapshotTablesIn(IEnumerable<SyncMessage> messages) =>
messages.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
.Select(m => m.SnapshotBatch.TableName)
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToList();
[Fact]
public async Task Baselining_OneTable_StreamsOnlyThatTable()
{
// The whole point of the per-table flag. Through 0.2.x the flag was one bit for the whole
// database, so baselining `orders` re-streamed `widgets` (and everything else) in full — in
// both directions at once on a pair enabling replication late, which on a database holding
// an event log at its retention cap is the largest transfer this library performs.
var a = await NewBaselinedSideAsync();
Assert.Equal(["orders"], await PendingSnapshotTables(a.Db));
Assert.Equal(0, await NeedsSnapshot(a.Db)); // NOT the database-wide flag
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = a.Session.RunAsync(duplex, cts.Token);
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
var begin = frames.Single(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBegin).SnapshotBegin;
Assert.Equal(["orders"], begin.Tables);
Assert.Equal(["orders"], SnapshotTablesIn(frames));
// The baselined rows really are in there — the scope narrowed the snapshot, it did not empty it.
var pks = frames.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
.SelectMany(m => m.SnapshotBatch.Rows).Select(r => r.PkJson).Order(StringComparer.Ordinal);
Assert.Equal(["{\"id\":1}", "{\"id\":2}"], pks);
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task Baselining_AgainstAPeerWithoutPartialSupport_StreamsEveryTable()
{
// Mixed-version interop, the hard constraint. A pre-0.3.0 peer ignores SnapshotBegin.tables
// and would advance last_applied_remote_seq to as_of_seq on complete — which for a SCOPED
// snapshot silently discards every delta below as_of for the tables it left out. There is no
// way to express "partial" to that peer, so it is served the 0.2.x answer: everything.
var a = await NewBaselinedSideAsync();
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = a.Session.RunAsync(duplex, cts.Token);
// supportsPartialSnapshot omitted == what a 0.1.x/0.2.x handshake decodes to.
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
var begin = frames.Single(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBegin).SnapshotBegin;
Assert.Empty(begin.Tables); // empty == full, the only scope the old wire can express
Assert.Equal(["orders", "widgets"], SnapshotTablesIn(frames));
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task PartialSnapshot_DebtClearsOnlyWhenThePeerAcknowledges()
{
// Crash safety: an ungraceful exit part-way through a snapshot must re-stream it, so the
// flag cannot come down when the frames are merely enqueued. The peer's SnapshotAck is
// written after its own durable watermark write, so it means "I hold these rows".
var a = await NewBaselinedSideAsync();
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = a.Session.RunAsync(duplex, cts.Token);
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
var complete = frames[^1].SnapshotComplete;
// The entire snapshot is on the wire and unacknowledged: the debt must still stand.
Assert.Equal(["orders"], await PendingSnapshotTables(a.Db));
await toSession.WriteAsync(new SyncMessage { SnapshotAck = new SnapshotAck { AsOfSeq = complete.AsOfSeq } }, cts.Token);
await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task Snapshot_ToAPeerWithoutPartialSupport_ClearsOnEnqueue()
{
// The 0.2.x behaviour, kept deliberately: a peer that sends no SnapshotAck would otherwise
// leave the debt standing forever and re-stream the snapshot on every reconnect.
var a = await NewBaselinedSideAsync();
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = a.Session.RunAsync(duplex, cts.Token);
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task PartialSnapshot_DoesNotAdvanceTheReceiversWatermark()
{
// The receiver's half of the same invariant. last_applied_remote_seq is ONE number for the
// whole peer oplog; moving it to a partial snapshot's as_of would declare the untouched
// tables applied through that point, and their deltas below it would never be resent.
var a = await NewSideAsync();
await a.Store.SetLastAppliedRemoteSeqAsync(10, 0);
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = a.Session.RunAsync(duplex, cts.Token);
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
var begin = new SnapshotBegin { AsOfSeq = 900 };
begin.Tables.Add("orders");
await toSession.WriteAsync(new SyncMessage { SnapshotBegin = begin }, cts.Token);
await toSession.WriteAsync(new SyncMessage
{
SnapshotBatch = new SnapshotBatch { TableName = "orders", Rows = { SnapRow(1, "SNAP", 6_000_000) } },
}, cts.Token);
await toSession.WriteAsync(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 900 } }, cts.Token);
// The ack proves the complete was processed — and that a 0.3.0 receiver answers one at all.
var ack = await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotAck, cts.Token);
Assert.Equal(900, ack.SnapshotAck.AsOfSeq);
Assert.Equal(10, await LastAppliedRemoteSeq(a.Db)); // NOT 900
Assert.Single(await ReadOrders(a.Db)); // the rows still merged
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task FullSnapshot_ToAnOldPeer_DrawsNoSnapshotAck()
{
// Symmetry check on the negotiation: a 0.3.0 receiver must not put a message a pre-0.3.0
// sender has never heard of on the wire back to it.
var a = await NewSideAsync();
var (duplex, toSession, fromSession) = ScriptedPeer();
using var cts = new CancellationTokenSource(RunTimeout);
var run = a.Session.RunAsync(duplex, cts.Token);
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
await toSession.WriteAsync(new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 300 } }, cts.Token);
await toSession.WriteAsync(new SyncMessage
{
SnapshotBatch = new SnapshotBatch { TableName = "orders", Rows = { SnapRow(1, "SNAP", 6_000_000) } },
}, cts.Token);
await toSession.WriteAsync(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 300 } }, cts.Token);
// A full snapshot from an old peer still advances the watermark — that is the 0.2.x contract.
await WaitForAsync(async () => await LastAppliedRemoteSeq(a.Db) == 300, RunTimeout);
// Barrier: an ack for this delta proves everything before it was processed, so an absent
// SnapshotAck is absent rather than merely late.
await toSession.WriteAsync(new SyncMessage
{
DeltaBatch = new DeltaBatch { Entries = { new OplogEntry
{
Seq = 301, TableName = "orders", PkJson = "{\"id\":2}",
RowJson = "{\"id\":2,\"sku\":\"BARRIER\",\"qty\":1}", Hlc = 6_000_001, NodeId = "peer", IsTombstone = false,
} } },
}, cts.Token);
await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.DeltaAck, cts.Token);
Assert.False(fromSession.TryPeek(out var stray) && stray.MsgCase == SyncMessage.MsgOneofCase.SnapshotAck);
cts.Cancel();
await SwallowAsync(run);
}
[Fact]
public async Task BaselinedTable_ReachesARealPeer_WithoutRestreamingTheOthers()
{
// End to end over two real databases: the baselined table's pre-existing rows arrive via the
// scoped snapshot, the untouched table's rows arrive as ordinary deltas, and the debt clears.
var options = new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
var a = await NewBaselinedSideAsync(options);
var b = await NewTwoTableSideAsync(options);
var (da, db) = DuplexPair();
using var cts = new CancellationTokenSource(RunTimeout);
var runA = a.Session.RunAsync(da, cts.Token);
var runB = b.Session.RunAsync(db, cts.Token);
async Task<bool> Converged() =>
(await ReadOrders(b.Db)).Count == 2
&& (await b.Db.QueryAsync("SELECT COUNT(*) FROM widgets", x => x.GetInt64(0)))[0] == 2;
await WaitForAsync(Converged, RunTimeout);
Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
// A partial snapshot speaks for nothing outside its tables, so neither side may skip the
// deltas it covered: B's watermark tracks the oplog, not the snapshot's as-of.
Assert.Equal(0, await NeedsSnapshot(a.Db));
cts.Cancel();
await SwallowAsync(runA);
await SwallowAsync(runB);
}
[Fact]
public async Task Sender_ClearsNeedsSnapshotFlag()
{
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/SyncSessionTests.cs
+46 -1
View File
@@ -208,6 +208,50 @@ public sealed class SyncSessionTests : IDisposable
Assert.Empty(await ReadOrders(b.Db));
}
[Fact]
public async Task Handshake_FaultTeardown_StillTransmitsTheLocalHandshake()
{
// Schema-mismatch detection is symmetric: each side computes its verdict FROM the other's
// Handshake. So a session that faults on the peer's handshake must still flush its own —
// otherwise the healthy peer learns nothing and waits out its receive cap (tests) or
// misclassifies the mismatch as a generic disconnect and reconnect-loops (production).
//
// The race being pinned: with the peer's handshake already buffered, the path from
// enqueueing our Handshake on the control lane to the validation throw has no yields,
// while the writer's wakeup sits in the thread-pool queue — a teardown that cancels the
// writer can therefore kill the Handshake before it ever transmits. Preloading makes the
// sprint as tight as possible, and running the scenarios concurrently starves the pool
// the way a busy test suite does.
const int Iterations = 40;
var sides = new List<Side>(Iterations);
for (var i = 0; i < Iterations; i++)
sides.Add(await NewSideAsync());
using var cts = new CancellationTokenSource(RunTimeout);
var runs = new List<(Task Run, ChannelReader<SyncMessage> FromSession)>(Iterations);
foreach (var side in sides)
{
var (duplex, toSession, fromSession) = ScriptedPeer();
var mismatched = MatchingHandshake(side.Db, "peer");
mismatched.Tables[0].Digest = "not-the-local-digest";
await toSession.WriteAsync(new SyncMessage { Handshake = mismatched }, cts.Token);
runs.Add((side.Session.RunAsync(duplex, cts.Token), fromSession));
}
var lost = 0;
foreach (var (run, fromSession) in runs)
{
await Assert.ThrowsAsync<LocalDbSchemaMismatchException>(() => run);
var sent = false;
while (fromSession.TryRead(out var msg))
sent |= msg.MsgCase == SyncMessage.MsgOneofCase.Handshake;
if (!sent)
lost++;
}
Assert.True(lost == 0, $"{lost}/{Iterations} sessions tore down without transmitting their own Handshake");
}
[Fact]
public async Task Handshake_SchemaVersionMismatch_FailsClosed()
{
@@ -553,8 +597,9 @@ public sealed class SyncSessionTests : IDisposable
await a.Store.SetNeedsSnapshotAsync(true, default);
var (duplex, toSession, fromSession) = ScriptedPeer();
var sent = new TaskCompletionSource();
a.Session.SnapshotSender = async (send, token) =>
a.Session.SnapshotSender = async (scope, send, token) =>
{
Assert.False(scope.IsPartial); // the database-wide flag always means every table
sent.TrySetResult();
await send(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 7 } }, token);
return 7;
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/SyncStreamGuardTests.cs
+181
View File
@@ -0,0 +1,181 @@
using System.Threading.Channels;
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.LocalDb.Contracts;
using ZB.MOM.WW.LocalDb.Replication;
namespace ZB.MOM.WW.LocalDb.Tests;
/// <summary>
/// The passive service's single-stream guard under a HALF-OPEN link: the dialling node is gone, but
/// nothing on this side knows it, so the handler sits inside <c>MoveNext</c> forever holding the
/// guard. Through 0.3.1 that made the pair unrecoverable without a restart — every reconnect from
/// the rebooted peer was refused <c>AlreadyExists</c>, indefinitely on an idle pair, while this node
/// went on reporting itself connected. The guard is there to serialize sessions, not to fence out a
/// caller the auth interceptor already vouched for, so the newest dialler wins.
/// </summary>
public sealed class SyncStreamGuardTests : IDisposable
{
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(10);
private readonly List<string> _paths = [];
private readonly List<IDisposable> _disposables = [];
public void Dispose()
{
foreach (var d in _disposables)
d.Dispose();
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
foreach (var p in _paths)
{
if (File.Exists(p)) File.Delete(p);
if (File.Exists(p + "-wal")) File.Delete(p + "-wal");
if (File.Exists(p + "-shm")) File.Delete(p + "-shm");
}
}
private LocalDbSyncService NewService()
{
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
_paths.Add(path);
var db = new Internal.SqliteLocalDb(new LocalDbOptions { Path = path });
_disposables.Add(db);
db.ExecuteAsync("CREATE TABLE orders (id INTEGER PRIMARY KEY, sku TEXT)").GetAwaiter().GetResult();
db.RegisterReplicated("orders");
return new LocalDbSyncService(
db,
Options.Create(new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) }),
NullLoggerFactory.Instance);
}
private static ServerCallContext ContextFor(CancellationToken ct) => new FakeCallContext(ct);
[Fact]
public async Task WedgedIncumbent_IsDisplacedByANewStream()
{
var service = NewService();
// Eviction is normally generous; the test only needs it to not be the thing under test.
service.TakeoverTimeout = Timeout;
// The incumbent's peer crashed: its stream will never deliver another message, and neither
// end times out on silence — a converged pair is legitimately quiet. The handler parks in
// the handshake's first read, holding the guard.
using var incumbentCts = new CancellationTokenSource();
var incumbentOut = new RecordingWriter();
var incumbent = service.Sync(new SilentReader(), incumbentOut, ContextFor(incumbentCts.Token));
// It really is running and really did claim the guard (its handshake is on the wire).
await incumbentOut.NextAsync(Timeout);
Assert.False(incumbent.IsCompleted);
// The rebooted peer dials again. Pre-fix this threw AlreadyExists and the pair stayed wedged.
using var newcomerCts = new CancellationTokenSource();
var newcomerOut = new RecordingWriter();
var newcomer = service.Sync(new SilentReader(), newcomerOut, ContextFor(newcomerCts.Token));
// The incumbent is torn down — gracefully, not as a fault of this node.
await incumbent.WaitAsync(Timeout);
// ...and the new stream is genuinely served: its own handshake goes out.
var handshake = await newcomerOut.NextAsync(Timeout);
Assert.Equal(SyncMessage.MsgOneofCase.Handshake, handshake.MsgCase);
Assert.False(newcomer.IsCompleted);
newcomerCts.Cancel();
await SwallowAsync(newcomer);
}
[Fact]
public async Task DisplacedIncumbent_ReleasesTheGuard_SoAThirdStreamIsAlsoServed()
{
// Takeover must leave the guard in a state that can be taken over AGAIN — a handover that
// leaked the slot would simply move the wedge one reconnect down the line.
var service = NewService();
service.TakeoverTimeout = Timeout;
using var firstCts = new CancellationTokenSource();
var firstOut = new RecordingWriter();
var first = service.Sync(new SilentReader(), firstOut, ContextFor(firstCts.Token));
await firstOut.NextAsync(Timeout);
using var secondCts = new CancellationTokenSource();
var secondOut = new RecordingWriter();
var second = service.Sync(new SilentReader(), secondOut, ContextFor(secondCts.Token));
await first.WaitAsync(Timeout);
await secondOut.NextAsync(Timeout);
using var thirdCts = new CancellationTokenSource();
var thirdOut = new RecordingWriter();
var third = service.Sync(new SilentReader(), thirdOut, ContextFor(thirdCts.Token));
await second.WaitAsync(Timeout);
Assert.Equal(SyncMessage.MsgOneofCase.Handshake, (await thirdOut.NextAsync(Timeout)).MsgCase);
thirdCts.Cancel();
await SwallowAsync(third);
}
private static async Task SwallowAsync(Task task)
{
try { await task.WaitAsync(Timeout); }
catch (OperationCanceledException) { }
catch (RpcException) { }
}
/// <summary>
/// The minimum <see cref="ServerCallContext"/> the handler actually reads: its cancellation
/// token. (Grpc.Core.Api ships no test double, and the ASP.NET Core one is not reachable outside
/// a live call.)
/// </summary>
private sealed class FakeCallContext(CancellationToken ct) : ServerCallContext
{
protected override string MethodCore => "/localdb_sync.v1.LocalDbSync/Sync";
protected override string HostCore => "test";
protected override string PeerCore => "test";
protected override DateTime DeadlineCore => DateTime.MaxValue;
protected override Metadata RequestHeadersCore => [];
protected override CancellationToken CancellationTokenCore => ct;
protected override Metadata ResponseTrailersCore { get; } = [];
protected override Status StatusCore { get; set; }
protected override WriteOptions? WriteOptionsCore { get; set; }
protected override AuthContext AuthContextCore => new(null, new Dictionary<string, List<AuthProperty>>());
protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options) =>
throw new NotSupportedException();
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) => Task.CompletedTask;
}
/// <summary>A stream that never delivers anything and never ends — a half-open link.</summary>
private sealed class SilentReader : IAsyncStreamReader<SyncMessage>
{
public SyncMessage Current => throw new InvalidOperationException("no message");
public async Task<bool> MoveNext(CancellationToken cancellationToken)
{
await Task.Delay(System.Threading.Timeout.Infinite, cancellationToken);
return false;
}
}
private sealed class RecordingWriter : IServerStreamWriter<SyncMessage>
{
private readonly Channel<SyncMessage> _written = Channel.CreateUnbounded<SyncMessage>();
public WriteOptions? WriteOptions { get; set; }
public Task WriteAsync(SyncMessage message) => WriteAsync(message, CancellationToken.None);
public Task WriteAsync(SyncMessage message, CancellationToken cancellationToken)
{
_written.Writer.TryWrite(message);
return Task.CompletedTask;
}
public async Task<SyncMessage> NextAsync(TimeSpan timeout)
{
using var cts = new CancellationTokenSource(timeout);
return await _written.Reader.ReadAsync(cts.Token);
}
}
}
ZB.MOM.WW.Secrets/build/push.sh
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Secrets.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
ZB.MOM.WW.Theme/build/pack.sh
+7 -1
View File
@@ -6,4 +6,10 @@
set -euo pipefail
dotnet pack -c Release -o ./artifacts
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
# neither a project argument nor a fixed working directory packs whatever solution happens
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet pack ZB.MOM.WW.Theme.slnx -c Release -o "$REPO_ROOT/artifacts"
ZB.MOM.WW.Theme/build/push.sh
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Theme.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
deployment.md
+35 -15
View File
@@ -50,7 +50,7 @@ collide (see [§7](#7-consolidated-host-port-map)).
| **fixture host** | `10.100.0.35` | Debian/Linux + Docker | OtOpcUa driver **integration-test fixtures** + a test SQL Server | 22 |
| **VD03 (DEV)** | `wonder-app-vd03.zmr.zimmer.com` | Windows | **DEV** all-in-one box: ScadaBridge (single-node) + OtOpcUa + MxGateway + HistorianGateway + Overview | **2222** |
| **VQ03 (QA)** | `wonder-app-vq03.zmr.zimmer.com` (`10.220.158.2`) | Windows | **QA node A** — ScadaBridge central-a + site-a, OtOpcUa node-a, **plus the singleton services**: MxGateway, HistorianGateway, Overview | **2222** |
| **VQ04 (QA)** | `wonder-app-vq04.zmr.zimmer.com` (`10.220.158.3`) | Windows | **QA node B** — ScadaBridge central-b + site-b, OtOpcUa node-b. **No** MxGateway/HistorianGateway/Overview | **2222** |
| **VQ04 (QA)** | `wonder-app-vq04.zmr.zimmer.com` (`10.220.158.3`) | Windows | **QA node B** — ScadaBridge central-b + site-b, OtOpcUa node-b, **plus (since 2026-08-14) its own MxGateway, HistorianGateway and Overview** (cloned from VQ03 incl. API-key stores; Galaxy SQL points at VQ03 — no local SQL Server) | **2222** |
| **wonder-sql-vq03** | `wonder-sql-vq03.zmr.zimmer.com` | Windows | **QA data tier** — SQL Server (`1433`) + AVEVA Historian gRPC TLS (`32565`); mirrors `wonder-sql-vd03` | _(not probed)_ |
| **gitea** | `gitea.dohertylan.com` (`10.100.0.228`) | Linux | Git remotes + NuGet feed (`/api/packages/dohertj2/nuget`) | 22 |
@@ -85,7 +85,7 @@ Host windev
| **windev** (Win host) | `ssh windev` | Configured alias; user `dohertj2`, key `id_ed25519`, port 22 |
| **fixture host** | `ssh dohertj2@10.100.0.35` | In `known_hosts`; **no** config alias — pass user explicitly; port 22, key-based |
| **VD03 (DEV)** | `ssh dohertj2@wonder-app-vd03.zmr.zimmer.com -p 2222` | In `known_hosts` on **port 2222** (servecli, not OpenSSH); user/key not pinned in config — confirm before use. Durable key `~/.ssh/servecli_wonder`. **servecli does not survive a reboot** — if it's down, use RDP |
| **VQ03 / VQ04 (QA)** | `ssh dohertj2@wonder-app-vq0{3,4}.zmr.zimmer.com -p 2222` | ⚠️ **Not working 2026-08-13** — see the banner note below |
| **VQ03 / VQ04 (QA)** | `psbridge --server wonder-app-vq0{3,4}.zmr.zimmer.com:6150 <cmd>` | **The supported path since 2026-08-14** (servecli `2222` is retired on these boxes — see the banner note below). Root jail `E:\psrem`; see the `psbridge` skill |
| **local Linux VMs** | `ssh orb` / `orb` | OrbStack-managed |
> ⚠️ **A TCP-open `2222` does NOT mean SSH works.** On 2026-08-13 all three wonder boxes
@@ -281,9 +281,11 @@ dotnet build src/MxGateway.Worker/MxGateway.Worker.csproj -p:Platform=x86
- **Worker model:** the Server spawns one `ZB.MOM.WW.MxGateway.Worker.exe` (x86) **per gRPC session**;
IPC over a named pipe (`\\.\pipe\mxgateway-<session>` + a per-session `MXGATEWAY_WORKER_NONCE`);
heartbeat 5s / grace 15s; max 64 concurrent sessions. The worker exits when the session closes.
- **Deployed hosts:** `10.100.0.48` (windev), `wonder-app-vd03` (DEV) and `wonder-app-vq03` (QA)
all serve gRPC on `:5120` (per `docs/GatewayConfiguration.md`; the VQ03 port was probed
2026-08-13). **QA runs MxGateway on VQ03 only** — VQ04 has no gateway.
- **Deployed hosts:** `10.100.0.48` (windev), `wonder-app-vd03` (DEV), `wonder-app-vq03` and
(since 2026-08-14) `wonder-app-vq04` (QA) all serve gRPC on `:5120` (per
`docs/GatewayConfiguration.md`). **QA runs MxGateway on BOTH nodes** — VQ04's was cloned from
VQ03 (same API-key store + pepper, so the same tokens verify on either; its Galaxy SQL and its
alarm subscription `\\wonder-app-vq03\Galaxy!NAM` both point at VQ03).
### 5.3 Who connects to it
@@ -354,14 +356,18 @@ DEV gRPC on `wonder-app-vd03:5120`, QA on `wonder-app-vq03:5120`. SSH: windev/fi
### 7.1 QA port map: VQ03 and VQ04
Established **2026-08-13** from two independent live sources, neither of them a config file — treat
connection strings and service names as unverified. Rows are marked with how each was established:
Established **2026-08-13** from two independent live sources, neither of them a config file, and
**updated 2026-08-14** when the three singleton services were cloned onto VQ04 (that update IS from
the boxes' live NSSM config + health probes, so its rows are verified). Rows are marked with how
each was established:
- **`scan`** — TCP port scan of both hosts, plus a `/health/ready` GET where one was served.
- **`dash`** — read off the **QA Overview dashboard** (`vq03:5320`), which was already polling the
QA fleet and reports each instance's endpoint, health entries and active/standby state. Stronger
evidence than a scan for these rows: the dashboard is successfully talking to the service, not
merely finding an open socket. These ports were never scanned by me.
- **`svc` / `live`** — added 2026-08-14: read from the boxes' own service registry (NSSM
`AppEnvironmentExtra` / `Win32_Service`) or exercised end-to-end during the VQ04 singleton setup.
| Port | VQ03 | VQ04 | How | Service |
|---|---|---|---|---|
@@ -372,21 +378,35 @@ connection strings and service names as unverified. Rows are marked with how eac
| 8084 | ✅ | ✅ | dash | ScadaBridge **Site** health listener (no UI; `/` → 404) |
| 8081 | ✅ | ✅ | scan | ScadaBridge Central Akka remoting |
| 8082 / 8083 | ✅ | ✅ | scan | ScadaBridge Site Akka / gRPC control plane (h2c — rejects an HTTP/1.1 GET) |
| 5120 / 5130 | ✅ | ✗ | scan | **MxAccessGateway** gRPC / dashboard |
| 5220 / 5221 | ✅ | ✗ | scan | **HistorianGateway** dashboard / gRPC h2c |
| 5320 | ✅ | ✗ | scan | **Overview** dashboard (the QA fleet's own single pane) |
| 1433 | ✅ | ✗ | scan | SQL Server |
| 2222 | ⚠️ | ⚠️ | scan | servecli SSH — **TCP-open but NOT serving SSH** (see below) |
| 5120 / 5130 | ✅ | ✅ | scan | **MxAccessGateway** gRPC / dashboard (VQ04 added 2026-08-14, cloned from VQ03) |
| 5220 / 5221 | ✅ | ✅ | scan | **HistorianGateway** dashboard / gRPC h2c (VQ04 added 2026-08-14) |
| 5320 | ✅ | ✅ | scan | **Overview** dashboard (one per node since 2026-08-14; both watch the same 10-instance registry) |
| 1433 | ✅ | ✗ | scan | SQL Server (**ZB Galaxy repo DB — VQ03 only**; VQ04's gateways point their Galaxy connection strings here) |
| 3893 | ✅ | ✅ | svc | glauth LDAP (`ScadaBridge-LDAP` service, loopback-only, `dc=scadabridge,dc=local` — one per box, configs identical) |
| **6150** | ✅ | ✅ | live | **psbridge** remote-admin bridge (`server.exe`, root jail `E:\psrem`) — **the supported access path**; see §2.1 |
| 2222 | ⚠️ | ⚠️ | scan | servecli SSH — **retired; TCP-open but NOT serving SSH** (see below) |
**Live state at capture** (`8 instances across 4 applications`, all Up):
**Live state at the 2026-08-13 capture** (`8 instances across 4 applications`, all Up; since
2026-08-14 the registry is **10 instances** — MxAccessGateway and HistorianGateway run on both
nodes, verified Healthy at setup):
| Application | VQ03 | VQ04 |
|---|---|---|
| OtOpcUa (cluster `MAIN`) | node-a — **Standby** (`active 503`) | node-b — **Active** (`active 200`) |
| ScadaBridge `central` | central-a — **Active** | central-b — Standby |
| ScadaBridge `site` | site-a — **Active** | site-b — Standby |
| MxAccessGateway | Active | — |
| HistorianGateway | Up | — |
| MxAccessGateway | Active | Active (since 2026-08-14; independent gateways, no cluster) |
| HistorianGateway | Up | Up (since 2026-08-14) |
| Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 10-instance registry) |
**The 2026-08-14 VQ04 clone, in one paragraph:** MxAccessGw + HistorianGateway + ZbOverview were
copied from VQ03 (same `E:\ApiInstall\...` layout, NSSM `AppEnvironmentExtra` config, LocalSystem)
**including the API-key stores and peppers, so the same `mxgw_…`/`histgw_…` tokens verify on both
boxes** — proven with positive + negative controls. The only config deltas: both apps' Galaxy SQL
connection strings point at VQ03 (`Server=wonder-app-vq03…;Database=ZB` — VQ04 has no local SQL
Server) and mxgw's alarm subscription stays `\\wonder-app-vq03\Galaxy!NAM`. **So VQ04's gateways
depend on VQ03 for Galaxy browse + alarms**; Historian/Runtime-DB traffic goes to `wonder-sql-vq03`
independently. Deploy record: `E:\ApiInstall\_DEPLOY-NOTES-20260814-singletons-added.txt` on VQ04.
Three things worth carrying forward:
docs/plans/2026-08-13-vd03-nam-ldap-cutover.md
+261
View File
@@ -0,0 +1,261 @@
# VD03 NAM AD LDAP cutover — plan (NOT executed)
**Written 2026-08-13. Status: PLAN ONLY.** Swap every auth-bearing service on
`wonder-app-vd03` from its current login posture (mostly login-disabled) to the real
Zimmer NAM Active Directory, with the service-account password held as a
`${secret:}` reference in each app's encrypted Secrets store — never in config.
Grounded in: [`ldap_setup.md`](../../ldap_setup.md) (directory facts + the 2026-08-13
end-to-end verification via the HistorianGateway docker container),
[`ldap_mapping.md`](../../ldap_mapping.md) (the `UG_WW_*` → role mapping, §4),
and the wonder deploy records (memory: `wonder-upgrade-program`).
## Scope
| Service on VD03 | In scope | LDAP config surface | Login state today |
|---|---|---|---|
| OtOpcUa AdminUI | ✅ | `Security:Ldap` (+ `GroupToRole` subkey, merged with a DB overlay) | LDAP enabled; target = TBD in preflight (data plane is API-key, **not** LDAP — untouched) |
| ScadaBridge (central UI) | ✅ | `ScadaBridge:Security:Ldap` in `appsettings.Central.json`; roles in the **`LdapGroupMapping` DB table**, not config | **Login DISABLED** (`DisableLogin=true` + `AllowOutsideDevelopment=true` — the two-key set) |
| MxAccessGateway dashboard | ✅ | `MxGateway:Ldap` + `MxGateway:Dashboard:GroupToRole` (gRPC stays API-key, untouched) | LDAP enabled per 08-12 notes; target = TBD in preflight |
| HistorianGateway dashboard | ✅ | `Ldap` (root section) + `Dashboard:GroupToRole` (gRPC stays API-key, untouched) | **Login DISABLED** (`Dashboard:DisableLogin=true`, deliberately open) |
| Overview | ❌ exempt | — | No login **by design** (holds no credentials; references no Auth package) |
Out of scope: windev's OtOpcUa/mxgw instances, the QA pair (VQ03/VQ04 — do it as a
follow-on once VD03 has soaked; QA is where the 2-node/failover behaviour actually gets
exercised), and OtOpcUa's OPC UA data plane (API-key auth since Gitea #520).
> ⚠️ **ignitionoee (router): LDAP setup ON HOLD** — explicit decision 2026-08-13. Its role
> vocabulary is not the canonical set (its admin role is literally `"Admin"`, not
> `"Administrator"`), so no existing `UG_WW_*` group maps onto it — a NAM cutover there
> would first require **minting new AD groups**. Do not wire its LDAP as part of this plan
> or its follow-ons; it keeps its current auth until the groups exist and are decided.
## The target LDAP block (identical shape in all four apps)
```jsonc
{
"Enabled": true,
"Server": "uswap-vdc-nam1.nam.zimmer.com",
"Port": 636,
"Transport": "Ldaps",
"AllowInsecure": false, // see TLS decision below
"FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com:636" ],
"SearchBase": "DC=nam,DC=zimmer,DC=com",
"ServiceAccountDn": "svc_igntadmin@nam.zimmer.com",
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
"UserNameAttribute": "sAMAccountName",
"DisplayNameAttribute": "name",
"GroupAttribute": "memberOf"
}
```
Facts already proven live (2026-08-13, HG docker → real NAM AD): the UPN bind as
`svc_igntadmin` works (**not** `svc_igntnapp` — that is Ignition's account), the
domain-root SearchBase works **on Auth 0.2.1 only**, `memberOf` DNs collapse to bare
CNs so `GroupToRole` keys are plain `UG_WW_*` names, and `FallbackServers` walks both
DCs on system-side failures. Users log in with the **bare sAMAccountName**.
## Hard prerequisite: every deployed build must carry Auth 0.2.1
**None of the four deployed builds has it** (pins landed 2026-08-13, after the last
deploy round): OtOpcUa runs `c51c5998`, ScadaBridge `20f2ae05`, mxgw the 08-12 build
(Auth 0.2.0), HG `e9c31da` (Auth **0.1.5**). On anything below 0.2.1 a domain-root
SearchBase fails **every** login as `ServiceAccountBindFailed` (AD continuation
referrals), and on 0.1.5 there is no `FallbackServers` at all. Narrowing SearchBase
below the root is NOT an acceptable workaround — user objects span multiple OUs.
So **Phase 1 is a redeploy of all four apps at their current default-branch tips**
(each already pins 0.2.1 and is pushed: OtOpcUa `08e33fc7`, ScadaBridge `d28824d9`,
mxgw `1cb14d2`, HG `e42ff31`). This folds into the standing wonder-upgrade practice —
HG is behind `main` anyway. Run the standard per-app preflight (migration-delta check,
appsettings SHA capture, env backup) from the `wonder-upgrade-program` memory.
## Decisions to confirm before executing
1. **Service account.** Use `svc_igntadmin` (proven), or request a dedicated
`svc_zbmomww` via ServiceNow first (the `UG_WW_*` groups are ticket-provisioned, so
the process exists). Reusing the Ignition admin account couples our uptime to their
rotations. **Recommended: start with `svc_igntadmin`, file the dedicated-account
ticket in parallel, swap later by rotating one secret per store.**
2. **Enabling login on services that are deliberately open.** ScadaBridge and HG run
login-disabled on this box today, on purpose. This cutover flips both to
authenticated. Confirm that is wanted on the DEV box (it is the point of the
exercise, but it changes who can reach those dashboards).
3. **Who gets in: VD03 = DEV tier ⇒ map only `_DEV` groups** (ldap_mapping §3 — a
deployment maps only its own tier; mapping PROD groups here would make every DEV
grant a cross-tier grant). Verify the intended users are actually members:
`UG_WW_APP_ADMIN_DEV` (7), `UG_WW_HIST_ADMIN_DEV` (7), `UG_WW_APP_IDE_DEV` (6),
`UG_WW_APP_IDE_RO_DEV` (2). If someone who needs access is in none, that is a
ServiceNow group-membership request **before** cutover, not a mapping widen.
(`BTMGR_RO` has no DEV group — omit.) Check with
`ldapprobe --insecure --svc svc_igntadmin@nam.zimmer.com --self-bind <sAMAccountName>`.
4. **TLS.** Try `AllowInsecure: false` first — the box is on the corporate network and
Windows machine stores on managed boxes usually carry `zimmer-rootca` (installing it
is sufficient; the only chain error observed is `UntrustedRoot`, and the RSA-1024
hazard is OpenSSL-container-specific, not Windows). If the preflight cert check
fails: install `zimmer-rootca` into LocalMachine\Root (preferred) or fall back to
`AllowInsecure: true` (DEV-acceptable; transport still encrypted).
5. **ScadaBridge plant-floor/operator rows** (`*_OPERATOR`, `SITE_ADMIN_*`,
`SUPERVISOR`/`ENGINEER`): ldap_mapping flags these ⚠ DECIDE and they are not
env-scoped. **Recommended: leave them out of this cutover** — seed only the
platform rows (below) and add operator rows when a real site use-case lands.
## Phase 0 — preflight (read-only, no changes)
From the box (servecli :2222; note servecli never survives a reboot — restart it first
if dead):
- **Network:** `Test-NetConnection uswap-vdc-nam1.nam.zimmer.com -Port 636` (and nam2).
- **TLS trust:** from PowerShell, open an `SslStream` to nam1:636 (or
`certutil -verify` the exported chain) to learn whether `zimmer-rootca` is trusted →
settles decision 4.
- **Service-account credential, once, without lockout risk** (bad binds count on both
DCs per app attempt; policy is 5/15 min): a single bind test from the Mac —
`LDAPTLS_REQCERT=never ldapsearch -H ldaps://uswap-vdc-nam1.nam.zimmer.com:636 -x -D
svc_igntadmin@nam.zimmer.com -W -b "DC=nam,DC=zimmer,DC=com" -s base dn`.
- **Capture current state per app:** SHA the live appsettings files; record the NSSM
registry env (`Parameters\AppEnvironmentExtra` for mxgw/HG; the OtOpcUa/ScadaBridge
service envs) — **check no `Ldap__*` keys lurk in service env, which would silently
override the JSON edits**; record current `DisableLogin` posture.
- **Secrets plumbing:** confirm the four stores + file-based KEKs are healthy (they
were authored + round-trip-verified 2026-08-12; KEK backups at
`E:\ApiInstall\_kek-backup-20260812T070057`). Stores live outside content roots
(OtOpcUa `E:\ApiInstall\OtOpcUa-data\`, mxgw + HG under their `data\`,
ScadaBridge per its overlays).
## Phase 1 — redeploy all four apps at Auth-0.2.1 tips
Standard wonder deploy recipe per app (memory: `wonder-app-vd03-mxgateway-deploy` +
per-app records). Self-contained win-x64 publishes (box has no .NET 10 runtime).
Expected deltas are small (each app's tip is its 08-12 deployed SHA + the 0.2.1 pin
± UI-sweep commits for HG); still run the migration-delta check — HG's jump
(`e9c31da → e42ff31`) carries the Theme 0.4.1/Health 0.3.0/Secrets 0.6.2 sweep, so
give it the full preflight (its Secrets 0.6.2 store-path guard will validate the
box's paths — they were fixed 08-12, so this should pass, but it is the guard that
bit OtOpcUa last round).
**Gate:** all services healthy on their current (pre-LDAP-swap) config before touching
auth. Do not combine the redeploy and the auth swap in one restart — two variables.
## Phase 2 — seed the secret into each app's store
Secret name: **`nam-ldap-svc-password`** (same name in every store, so the config
reference is identical everywhere and rotation is a mechanical sweep).
Use the on-box CLI `E:\ApiInstall\_tools\secret` (0.6.2, self-contained). **Trap
(proven 08-12): bare CLI verbs read the CLI's own env, NOT the target app's config** —
target each store per-invocation:
```powershell
$env:Secrets__SqlitePath = "<that app's store path>"
$env:Secrets__MasterKey__Source = "File"
$env:Secrets__MasterKey__FilePath = "<that app's KEK file>"
E:\ApiInstall\_tools\secret\secret.exe set nam-ldap-svc-password # prompts; never a literal arg
```
Stores to seed (verify paths from the live NSSM/registry env captured in Phase 0):
| Store | Needed by |
|---|---|
| OtOpcUa (`E:\ApiInstall\OtOpcUa-data\...`) | AdminUI LDAP login |
| ScadaBridge **central** | central UI LDAP login (the `${secret:}` ref lives in `appsettings.Central.json`) |
| ScadaBridge **site** | not strictly needed (site does no LDAP) — seed anyway for symmetry; the two stores share one KEK but are separate stores |
| MxGateway | dashboard LDAP login |
| HistorianGateway | dashboard LDAP login |
Verify each with `secret.exe get nam-ldap-svc-password` (value round-trip) under the
same per-invocation targeting. **Rotation note for the runbook:** one password now
lives in 4–5 stores; a rotation is N `set` operations plus zero config changes.
## Phase 3 — config edits (one app at a time, restart, verify, then next)
Order: **HG → mxgw → OtOpcUa → ScadaBridge** (proven-pattern first, DB-mapping app
last). Back up every file edited; edits go in the box's live overlay files, and the
next deploy must re-apply them (add to the deploy-notes convention — the deploy
process restores box appsettings, so once these edits ARE the box appsettings they
survive; the risk is a fresh-install overwrite, same as every other box-local key).
### 3a. HistorianGateway
- `Ldap` section ← the target block above.
- `Dashboard:DisableLogin` → `false`; `Dashboard:GroupToRole` ←
`UG_WW_HIST_ADMIN_DEV: Administrator`, `UG_WW_APP_ADMIN_DEV: Administrator`,
`UG_WW_APP_IDE_RO_DEV: Viewer`.
- Restart `ZbHistorianGateway`; run the Phase 4 checks.
### 3b. MxAccessGateway
- `MxGateway:Ldap` ← target block.
- `MxGateway:Dashboard:GroupToRole` ← `UG_WW_APP_ADMIN_DEV: Administrator`,
`UG_WW_APP_IDE_DEV: Viewer`, `UG_WW_APP_IDE_RO_DEV: Viewer` (only two roles exist;
Designer collapses to Viewer per ldap_mapping §4).
- Restart; verify. (gRPC/API-key surface untouched — do NOT disturb the restored
API-key store in `data\`.)
### 3c. OtOpcUa (AdminUI only)
- `Security:Ldap` ← target block (keep any existing `GroupToRole` DB-overlay rows in
mind — config merges with the DB overlay; check the RoleGrants page after).
- `Security:Ldap:GroupToRole` ← `UG_WW_APP_ADMIN_DEV: Administrator`,
`UG_WW_APP_IDE_DEV: Designer`, `UG_WW_APP_IDE_RO_DEV: Viewer`,
`UG_WW_HIST_ADMIN_DEV: Viewer` (historization targets are authored here — see
ldap_mapping §4 rationale).
- Restart the AdminUI/central service. Driver-only nodes are unaffected (no LDAP).
- The OPC UA data plane keeps its `otua_` API keys — no consumer impact.
### 3d. ScadaBridge (central)
- `appsettings.Central.json`: `ScadaBridge:Security:Ldap` ← target block; **remove the
two disable-login keys** (`DisableLogin` + `AllowOutsideDevelopment`) — note the trap:
removing only `AllowOutsideDevelopment` while leaving `DisableLogin=true`
**crash-loops** (`DisableLoginGuard`); remove both.
- Role rows in the central DB (`LdapGroupMapping` — the only app whose mapping is data,
and the only one that can grant two roles per group):
```sql
INSERT INTO LdapGroupMapping (LdapGroupName, Role) VALUES
('UG_WW_APP_ADMIN_DEV', 'Administrator'),
('UG_WW_APP_IDE_DEV', 'Designer'),
('UG_WW_APP_IDE_DEV', 'Deployer'), -- two rows on purpose (§2.1)
('UG_WW_APP_IDE_RO_DEV', 'Viewer');
```
(Operator/site rows deliberately deferred — decision 5.)
- Restart central; site service untouched.
## Phase 4 — verification (per app, immediately after its restart)
The log **failure stage** is the diagnostic (proven live, recorded in ldap_setup.md):
1. **Negative control first** (bogus user + bogus password): expect a clean
"invalid/not authorized" with **zero** `WRN ... failed system-side` lines. A
`service-bind`-stage WRN = wrong secret value; a `search`-stage WRN = empty
password (the `${secret:}` ref did not resolve — check the store/KEK targeting);
connect-stage = network/TLS.
2. **Real login** with an account in a mapped `_DEV` group → correct role visible in
the UI. And one with **no** mapped group → denied (fail-closed zero-group rule).
3. **`${secret:}` positive control:** the value must come from the store, not residue —
e.g. confirm no `Ldap__ServiceAccountPassword` in service env, and that
`secret.exe get` matches what auth uses (a wrong-value canary set+revert is the
strong form).
4. **Failover spot-check (once, on HG):** block/blackhole nam1:636 briefly →
login still works, WRN for nam1 + `sticky preference moved` to nam2; restore.
Do not drill on every app — it is the same library and it doubles bad-bind
exposure against lockout policy if anything is misconfigured.
5. Mind the lockout arithmetic throughout: every failed app attempt = a failed bind
on **both** DCs (endpoint walk). Stop and diagnose after 2 failures, don't retry.
## Rollback
Per app, in reverse: restore the backed-up appsettings (which restores the previous
login posture, including ScadaBridge's two-key disable and HG's `DisableLogin=true`),
restart. The seeded secrets are inert once nothing references them — leave them.
ScadaBridge: `DELETE FROM LdapGroupMapping WHERE LdapGroupName LIKE 'UG_WW_%'` only if
a clean slate is wanted; unreferenced rows grant nothing while login is disabled.
Phase 1's redeploys are independent and should NOT be rolled back for an auth problem.
## Follow-ups this creates
- QA pair (VQ03/VQ04) cutover with `_QA` groups — the same plan, plus real
failover/pair semantics; QA is where `FallbackServers` + sticky actually matter.
- Dedicated service account (decision 1) → rotate `nam-ldap-svc-password` in N stores.
- Revert the Mac's HG docker container from its temporary NAM flip back to the
glauth-redundant pair (uncommitted working-tree change, independent of this plan).
- ldap_mapping.md is still untracked in scadaproj — commit it when its DECIDE items
settle; this plan depends on its §4 tables.
- ignitionoee LDAP: **held** (see the Scope flag) until new `UG_WW_*` groups matching its
role vocabulary are minted and mapped — a group-creation request, not a config task.
infra/glauth-redundant/README.md
+58
View File
@@ -0,0 +1,58 @@
# Redundant LDAP test pair (primary + backup "DC")
Two GLAuth containers on the **dev Mac** that simulate the NAM two-DC topology
(`uswap-vdc-nam1`/`nam2`) for exercising `ZB.MOM.WW.Auth.Ldap` **0.2.x
`FallbackServers` sticky failover**. Deliberately local — the shared
`zb-shared-glauth` on `10.100.0.35:3893` serves every other dev stack and must
never be stopped for a drill.
| | container | host port | from a container |
|---|---|---|---|
| primary | `zb-ldap-primary` | `3893` | `host.docker.internal:3893` |
| backup | `zb-ldap-backup` | `3894` | `host.docker.internal:3894` |
Both mount the same `config.toml` — identical directory contents, like a real
replicated pair. It is a copy of [`../glauth/config.toml`](../glauth/config.toml)
with one difference: the serviceaccount password here is the well-known local
value `serviceaccount123` (the shared instance's is rotated out-of-band, SEC-36).
Same users otherwise (`multi-role`/`password`, etc.).
```bash
docker compose up -d # start the pair
```
## Consumer wiring (any of the four apps)
```
Ldap__Server: host.docker.internal # primary
Ldap__Port: "3893"
Ldap__FallbackServers__0: host.docker.internal:3894
Ldap__Transport: None
Ldap__AllowInsecure: "true"
```
Wired consumers (both drilled PASS 2026-08-13):
- `HistorianGateway/docker/docker-compose.yml` (Auth `0.2.1`, dashboard login
enabled). Note its Production-env container needs
`Serilog__WriteTo__0__Name: Console` or it logs **nothing** — the drill's
failover lines would be invisible.
- ScadaBridge docker rig (`docker/central-node-{a,b}/appsettings.Central.json`,
commit `b71fbae3`) — this pair REPLACED the shared `10.100.0.35` GLAuth there,
whose serviceaccount password rotation (SEC-36) had silently broken rig login.
ScadaBridge's `AuthFlowTests` also target `localhost:3893` and pass 5/5 with
the pair up (they skip when it is down).
## Failover drill (verified PASS 2026-08-13 against HistorianGateway)
1. login → served by primary, no warnings
2. `docker stop zb-ldap-primary` → login OK; app logs
`WRN LDAP endpoint …:3893 failed system-side at connect` +
`INF LDAP sticky preference moved to …:3894`
3. `docker start zb-ldap-primary` → login OK, **still served by the backup**
(sticky by design — verified by bind counts: primary got zero new binds)
4. `docker stop zb-ldap-backup` → login OK; warn on `:3894`, sticky moves back
to `:3893`
The sticky preference lives in the app's singleton `LdapAuthService`, so an app
restart resets it to the configured primary.
infra/glauth-redundant/config.toml
+169
View File
@@ -0,0 +1,169 @@
[ldap]
enabled = true
listen = "0.0.0.0:3893"
[ldaps]
enabled = false
[backend]
datastore = "config"
baseDN = "dc=zb,dc=local"
[behaviors]
# Dev: do not lock out on failed binds (avoids surprises during testing).
LimitFailedBinds = false
# ── Groups ───────────────────────────────────────────────────────────
# ScadaBridge role groups (55xx) — DB-mapped (LdapGroupMappings)
[[groups]]
name = "SCADA-Admins"
gidnumber = 5501
[[groups]]
name = "SCADA-Designers"
gidnumber = 5502
[[groups]]
name = "SCADA-Deploy-All"
gidnumber = 5503
[[groups]]
name = "SCADA-Deploy-SiteA"
gidnumber = 5504
[[groups]]
name = "SCADA-Viewers"
gidnumber = 5505
# OPC-UA permission groups (560x) — OtOpcUa + MxGateway OPC write model
[[groups]]
name = "ReadOnly"
gidnumber = 5601
[[groups]]
name = "WriteOperate"
gidnumber = 5602
[[groups]]
name = "WriteTune"
gidnumber = 5603
[[groups]]
name = "WriteConfigure"
gidnumber = 5604
[[groups]]
name = "AlarmAck"
gidnumber = 5605
# MxGateway dashboard groups (561x) — config-mapped (GroupToRole)
[[groups]]
name = "GwAdmin"
gidnumber = 5610
[[groups]]
name = "GwReader"
gidnumber = 5611
# OtOpcUa AdminUI role groups (57xx) — DB-mapped (LdapGroupRoleMapping)
[[groups]]
name = "OtOpcUa-Admins"
gidnumber = 5701
[[groups]]
name = "OtOpcUa-Designers"
gidnumber = 5702
[[groups]]
name = "OtOpcUa-Viewers"
gidnumber = 5703
# ── Users ────────────────────────────────────────────────────────────
# All passwords are "password". The serviceaccount password is rotated
# out-of-band (SEC-36) and is not recorded here.
# sha256("password") = 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8
# The single bind account every app uses (search-then-bind).
[[users]]
name = "serviceaccount"
uidnumber = 5999
primarygroup = 5601
passsha256 = "af29d0e5c9801ae98a999ed3915e1cf428a64b4b62b3cf221b6336cce0398419"
[[users.capabilities]]
action = "search"
object = "*"
# Cross-app: member of EVERY group → all roles in all three apps.
[[users]]
name = "multi-role"
givenname = "Multi"
sn = "Role"
mail = "multi-role@zb.local"
uidnumber = 5005
primarygroup = 5501
othergroups = [5502, 5503, 5504, 5505, 5601, 5602, 5603, 5604, 5605, 5610, 5611, 5701, 5702, 5703]
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
# Administrator everywhere (admin-equivalent of each app).
[[users]]
name = "admin"
uidnumber = 5001
primarygroup = 5501
othergroups = [5610, 5701]
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
# ScadaBridge single-role testers
[[users]]
name = "designer"
uidnumber = 5002
primarygroup = 5502
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
[[users]]
name = "deployer"
uidnumber = 5003
primarygroup = 5503
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
[[users]]
name = "site-deployer"
uidnumber = 5004
primarygroup = 5504
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
# MxGateway dashboard Viewer tester.
# NOTE: tester usernames must NOT case-collide with a group name — glauth exposes
# each group as cn=<Group> under ou=users, so a case-insensitive (cn=X) search
# would match both the user and the group (ambiguous → auth fails). Hence gw-viewer
# (not "gwreader" which collides with the GwReader group), opc-* below, etc.
[[users]]
name = "gw-viewer"
uidnumber = 5106
primarygroup = 5611
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
# OPC-UA permission testers
[[users]]
name = "opc-readonly"
uidnumber = 5101
primarygroup = 5601
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
[[users]]
name = "opc-writeop"
uidnumber = 5102
primarygroup = 5602
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
[[users]]
name = "opc-writetune"
uidnumber = 5103
primarygroup = 5603
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
[[users]]
name = "opc-writeconfig"
uidnumber = 5104
primarygroup = 5604
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
[[users]]
name = "opc-alarmack"
uidnumber = 5105
primarygroup = 5605
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
# OtOpcUa single-role testers (admin covers OtOpcUa-Admins)
[[users]]
name = "otdesigner"
uidnumber = 5202
primarygroup = 5702
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
[[users]]
name = "otviewer"
uidnumber = 5203
primarygroup = 5703
passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
infra/glauth-redundant/docker-compose.yml
+36
View File
@@ -0,0 +1,36 @@
# Redundant LDAP test pair — two GLAuth instances simulating a primary + backup DC,
# for exercising ZB.MOM.WW.Auth.Ldap 0.2.x `FallbackServers` sticky failover.
#
# Runs LOCALLY on the dev Mac (not on 10.100.0.35) precisely so the "primary DC"
# can be stopped/started during a failover drill without touching the shared
# zb-shared-glauth that every other dev stack authenticates against.
#
# Both instances mount the SAME config.toml — identical directory contents, like a
# real replicated DC pair. The config is a copy of ../glauth/config.toml with ONE
# difference: the serviceaccount password here is the well-known local-dev value
# `serviceaccount123` (the shared instance's is rotated out-of-band, SEC-36).
#
# primary : ldap://localhost:3893 (containers reach it at host.docker.internal:3893)
# backup : ldap://localhost:3894 (containers reach it at host.docker.internal:3894)
#
# Drill: docker stop zb-ldap-primary → logins fail over to the backup (sticky)
# docker start zb-ldap-primary → preference STAYS on the backup by design
# docker stop zb-ldap-backup → next login walks back to the primary
name: zb-ldap-redundant
services:
ldap-primary:
image: glauth/glauth:latest
container_name: zb-ldap-primary
restart: unless-stopped
ports:
- "3893:3893"
volumes:
- ./config.toml:/app/config/config.cfg:ro
ldap-backup:
image: glauth/glauth:latest
container_name: zb-ldap-backup
restart: unless-stopped
ports:
- "3894:3893"
volumes:
- ./config.toml:/app/config/config.cfg:ro
ldap_setup.md
+40 -3
View File
@@ -142,6 +142,40 @@ Two refinements to the TLS notes above, from the actual chain a .NET client sees
Cert observed on nam1: `CN=USWAP-VDC-NAM1.nam.zimmer.com`, SAN matches, sha256RSA,
valid 2026-03-20 → 2027-03-20, chain `leaf ← zimmer-issueca1 ← zimmer-rootca`.
### End-to-end app login verified 2026-08-13 (HistorianGateway docker → real NAM AD)
A real ZB.MOM.WW app authenticated a real user against the production directory for the
first time: the local HistorianGateway container (Auth `0.2.1`) with its dashboard LDAP
temporarily flipped from the glauth-redundant pair to the two NAM DCs — LDAPS 636,
`AllowInsecure` (the container trust store lacks `zimmer-rootca`), domain-root
`SearchBase`, `sAMAccountName` login, `memberOf` → bare-CN `UG_WW_*` keys in
`Dashboard:GroupToRole`. Login succeeded and the group mapping resolved to a dashboard
role. The flip is an **uncommitted working-tree change** in
`HistorianGateway/docker/docker-compose.yml` (revert: `git checkout -- docker/docker-compose.yml`
+ remove the `Ldap__ServiceAccountPassword` line from `histgw.secrets.env`).
Three findings worth keeping:
- **The service account for our use is `svc_igntadmin@nam.zimmer.com` — not
`svc_igntnapp`.** `svc_igntnapp` is *Ignition's* account (the "Bind identity" section
above describes Ignition's config); binding it with our held password fails
`data 52e` (bad credential). ldapprobe's built-in default is `svc_igntnapp` — pass
`--svc svc_igntadmin@nam.zimmer.com` when probing with the held password.
- **The failover log's failure *stage* is a precise diagnostic.** Seen live:
`failed system-side at search` on both DCs = the service password is **empty**
(AD treats a passwordless simple bind as anonymous, accepts it, then refuses the
search); `failed system-side at service-bind` = the credential is **wrong** (52e);
a clean "invalid or not authorized" with **no** WRN lines = the directory answered
and the user is unknown or matched zero mapped groups (fail-closed).
- **Failed logins double against lockout policy** (5 attempts / 15 min): a bad
*service* credential costs one failed bind on **each** DC per attempt, because the
endpoint walk retries system-side failures on the fallback. Validate a candidate
service password with a single `ldapsearch -H ldaps://uswap-vdc-nam1… -x -D <upn> -W`
before letting an app loop on it.
The Debian-based .NET 10 container handshook LDAPS 636 to both DCs without the
`SECLEVEL` workaround mentioned above being needed.
### `UG_WW_*` — an existing Wonderware group convention (observed 2026-08-12)
**This partly overturns the "Groups / roles" section above.** That section is accurate
@@ -242,9 +276,12 @@ Three gaps to settle before this is a design rather than a sketch:
**Found and fixed 2026-08-12.** All four Auth packages are **published to the Gitea feed at
`0.2.1`** and restore-verified from a scratch consumer (the assembly restored *from the
feed* carries the fix; `0.2.0` does not). **The four apps still pin `0.1.5`, so nothing is
fixed for them until each bumps** — a pin bump is the remaining step, and until it happens
every app still fails a domain-root search with `ServiceAccountBindFailed`.
feed* carries the fix; `0.2.0` does not). **ADOPTED by all four apps 2026-08-13** —
OtOpcUa `master @ 08e33fc7`, ScadaBridge `main @ d28824d9`, MxAccessGateway
`main @ 1cb14d2`, HistorianGateway `e42ff31` (merged to `main`) — each built + tested green, all pushed to origin. `FallbackServers`
failover was live-gated the same day against the local redundant GLAuth pair
([`infra/glauth-redundant/`](infra/glauth-redundant/)): HistorianGateway dashboard login
drill PASS (sticky failover both directions).
Authenticating with `SearchBase = DC=nam,DC=zimmer,DC=com` fails on **both DCs** with
`ServiceAccountBindFailed` in ~546 ms. It is not connectivity, not the service account,