41bdc178c2bc7e149cf7bfe41bc552c0876ab689
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
41bdc178c2 |
fix(localdb): refuse cascade-delete FK parents; make one entry's row+ledger writes atomic (0.3.3, ScadaBridge arch-review P4-7)
Two halves of the same INSERT OR REPLACE hazard in the LWW apply path. Wire-compatible with every earlier peer: no proto change, no lib_schema_version bump, no on-disk schema change. 1. RegisterReplicated now REFUSES a table that another table references with a row-mutating ON DELETE action (CASCADE / SET NULL / SET DEFAULT). Applying a remote row uses INSERT OR REPLACE, whose implicit DELETE fires that referential action — while the apply path holds the capture guard raised, precisely so its own writes are not re-captured. The child rows it deletes or nulls therefore produce no oplog entry: the peer keeps them and the pair diverges with nothing reporting a fault. Registration is the only point at which the problem is still visible, so it is refused there with an actionable LocalDbRegistrationException naming the referencing table and its action. NO ACTION / RESTRICT stay allowed: they mutate no child row, so the implicit delete just fails and the applier dead-letters the entry — loud, not silent. Registration-time only; a cascading child table created LATER is not caught (SQLite offers no DDL hook). No current consumer has a foreign key on a replicated table (ScadaBridge's ten tables have none), so this is a guard against a future foot-gun, not a live fix. 2. One entry's row write and its ledger write are now atomic. They were two statements under the poison catch, so a failure landing between them committed a deleted/replaced row carrying a STALE LIVE row-version entry — exactly the shape SnapshotStreamer warns about and skips, so that key never travels again and the pair is permanently divergent for it. A per-entry SAVEPOINT makes the pair all-or-nothing while keeping the catch's per-ENTRY granularity: entries applied earlier in the same batch survive the rollback, the poison entry is still dead-lettered, and the batch still continues. The RELEASE runs outside the catch on purpose, so a failure there rolls the whole batch back rather than being swallowed as poison. Measured cost on a 100k-row backlog drain: ~7% (178.6k rows/s -> 166.1k rows/s). Tests: both hazards proven against the unfixed code first (FK parent registered silently; an injected ledger-write abort left the row deleted with a live ledger entry). Adds FK guard coverage (parent refused, child allowed, SET NULL / SET DEFAULT refused, NO ACTION / RESTRICT allowed, unrelated cascades ignored) and two applier atomicity tests. The statement-count lock-in moves 7 -> 9 for the two new savepoint statements. Version 0.3.2 -> 0.3.3; README limitations + release notes document the constraint. |
||
|
Joseph Doherty
|
f62b017cc7 | fix(localdb): fault live sync session on cap-prune; keepalive + newest-wins sync guard (0.3.2, ScadaBridge arch-review P4-1 + P4-3) | ||
|
Joseph Doherty
|
3e5a5fde81 |
fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1)
A session that faulted during the handshake cancelled its writer loop immediately, which could kill its OWN Handshake while it still sat in the control lane: the enqueue -> receive -> validate -> fault path has no yields, while the writer's wakeup waits in the thread-pool queue, so with the peer's handshake already buffered the cancel won the race almost every time (39/40 in the new pinning test). Mismatch detection is symmetric — the peer computes its fail-closed verdict FROM our Handshake — so the healthy peer never learned anything: in tests it waited out the full 15 s receive cap (the register-row-41 flake on SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it would misclassify a permanent schema mismatch as a generic disconnect and reconnect-loop at the fast backoff instead of taking the classified FailedPrecondition -> LogError + max-backoff path. RunAsync's fault teardown now completes both lanes so the writer drains what is already enqueued and exits on its own, with a 2 s deferred cancel as the backstop for a wedged transport. Caller-initiated cancellation is unchanged (the linked token is already cancelled, so shutdown is not delayed), and nothing on the wire changes — a 0.3.0 peer simply receives the Handshake it was always owed. New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green post-fix). BatchBudgetTests hardened separately: the register's other flaky test (Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the scripted-peer channel without ever observing the session task, so ANY session fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now races the session task at every read site in the class, so the next flake reports the session's real exception instead. (The handshake-race root cause does not structurally apply to this test — no handshake fault occurs there — so its intermittent timeout is attributed to the fault-hiding wait, now removed.) Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo). |
||
|
Joseph Doherty
|
cea3930b94 |
feat(localdb): per-table needs_snapshot (0.3.0, on-disk schema v3)
Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a snapshot streams every registered table in full — so opting one table into replication late re-sent all of them, in both directions at once on a pair enabling replication late. On a database holding a high-volume table (an event log at its retention cap) that is the largest transfer this library performs. 0.2.1 documented the cost and deferred the fix as needing an on-disk change. Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3 transition is a version stamp only. RegisterReplicated(..., baselineExistingRows: true) now flags that table alone. The database-wide flag survives unchanged for the paths that genuinely reach every table — a cap prune or an impossible peer ack loses delta continuity for the oplog as a whole. DeregisterReplicated drops a table's pending flag with its ledger rows, since nothing would ever stream (and so clear) it afterwards. Wire compatibility is additive and negotiated by CAPABILITY, not version: lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope (field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits and what a pre-0.3.0 receiver assumes. A peer that does not advertise the capability is served a full snapshot: a scoped one would make it advance last_applied_remote_seq to the snapshot's as-of and silently discard every delta below it for the tables left out. For the same reason a partial snapshot moves no watermark on either side. The receiver keeps last_applied_remote_seq (carrying only the HLC across, which is MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot overlaps are still pumped and merged by LWW. Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather than on enqueue, so an ungraceful exit part-way through re-streams the same scope on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot. Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is no ack to wait for — with the handshake's horizon gap check as the net. Tests 199/199 (was 188): single-table baseline streams only that table, mixed- version interop (peer without the capability gets a full snapshot), debt standing until SnapshotAck, partial snapshot not advancing the receiver's watermark, no SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3 upgrade and per-table flag isolation. README, ILocalDb remarks (which named this as the follow-up) and the umbrella CLAUDE.md index updated. |
||
|
Joseph Doherty
|
522cfbfda6 |
fix(localdb): flush the HLC anchor on deregister, bound the sync inbox (0.2.1)
Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers: no proto change, no lib_schema_version bump, no on-disk schema change. F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc), last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the table's rows from BOTH ledgers. On a file whose registered tables have all been deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above zero to recover from: the reopened clock restarts from wall-clock time and can re-issue stamps below HLCs peers already hold, which LWW discards as older. The current clock is now flushed inside the deregistration transaction, before the prune, as a separate command so it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional transaction (SQLite rejects a command with no transaction on a connection that has one) and is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower value can only be stale. F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every registered table, needs_snapshot is per-database, and both sides of a pair that enables replication late baseline unconditionally — so a mutual full snapshot of a large dataset (an event log near its storage cap) was buffered whole, in memory, on both nodes at once. The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is ~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full inbox stops draining the gRPC stream, which closes the peer's flow-control window. The pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the message, and every inbound message is protocol state. Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node owes was streamed from INSIDE the handshake, before the receive loop existed, so neither side drained its inbox while sending; a mutual snapshot converged only because the inbox was unbounded. The snapshot now runs as the first phase of the outbound task, started alongside the receive loop, so the inbox is drained for the whole life of the session. Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved (the pump still starts only after the snapshot, from its as-of seq). Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in __localdb_peer_state and narrowing it needs an on-disk schema change, which this fix deliberately avoids. Documented as a follow-up, and the cost is now written down where it is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that baselining one table re-streams every registered table in both directions, and that high-volume tables should be trimmed before enabling replication late. 188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an ungraceful reopen (a second instance over the same file with no Dispose — fails without the flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that last one deadlocks and times out against the old handshake-phase snapshot ordering. Version 0.2.1. |
||
|
Joseph Doherty
|
9377fa1678 |
perf(localdb): dereg API, O(1) backlog depth, byte-budget batching, prepare-once apply (0.2.0)
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened. DeregisterReplicated(table). There was no way to stop capturing once a file had been registered, so a node that does not replicate kept paying three triggers inside every write transaction, appending to an oplog nothing would ever drain. Drops the triggers and prunes the table's oplog + row-version rows in one transaction; idempotent, and deliberately usable on a table this process never registered, since stale triggers left by an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and already is enforced: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing loudly rather than diverging. RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot, and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly greater, which makes a baselined row lose every comparison against a real remote write and win only where the peer has no version of that key at all. Two nodes baselining the same key both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is unchanged unless asked. Pruning the ledger on deregister is safe precisely because re-registration can re-baseline; the two APIs are a pair. O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at 0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a stalled watermark — without it the caps would re-flag on every tick forever. Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed serialized size, with the row count demoted to a secondary cap. A row larger than the budget is sent alone rather than stalling the stream on it. Since more, smaller messages would otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY queued are coalesced — one transaction and one ack per burst, bounded by the sender's own caps, and a no-op under a trickle. LwwApplier prepares each statement once. It went through the generic transaction façade, which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path that drains a reconnect backlog. Statements are now held for the batch and rebound. Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever used — every oplog access path is by seq or a full hlc-predicate delete — while costing a b-tree insert inside every captured consumer write. 184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves identity/watermarks/data, deregistration idempotency and trigger absence and cross-process cleanup, baseline seeding at the floor with LWW safety proved through the real applier, depth asserted equal to the counting query across every prune scenario plus a constant-time assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both the delta and snapshot paths, prepare-once statement counting, and coalescing that must not swallow the message that ends its run. Version 0.2.0. |
||
|
Joseph Doherty
|
3b7489a7b0 |
fix(localdb): a rebuilt peer's own writes were silently dropped
The other half of the rebuilt-peer story, and only reachable once 0.1.2 made back-fill work: with the wiped node repopulated and writing again, its writes never reached its peer. last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt peer is a new database whose oplog numbers from 1. Two things then went wrong, and both had to be fixed — the first alone leaves data stuck: - The healthy node kept advertising the OLD peer's watermark, and the sending side seeds its pump from exactly that number, so the rebuilt node skipped its entire oplog. The watermark (and the observed peer clock) is now reset when the peer's node id changes. last_acked_seq is deliberately NOT reset: it describes our own oplog's pruned horizon and is what tells a rebuilt peer it needs a snapshot — clearing it would turn 0.1.2's back-fill back off. - A peer claiming to have applied more of our stream than we have ever produced can only be remembering a previous incarnation of us. That claim is now rejected in favour of starting from the beginning; LWW makes the re-send harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks. Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10 while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the peer never accepted a single one. Covered by a test that converges with one peer, replaces it with a fresh database, and requires a write on the new peer to arrive: RED before this fix, green after. 149/149 pass. Version 0.1.3. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
cad3bcb6bf |
fix(localdb): snapshot a wiped peer whose deltas were already acked and pruned
A converged pair prunes its oplog on ack, so "no oplog rows at all" is the steady state of a healthy pair — not an anomaly. ComputeSnapshotRequiredAsync measured the peer's gap against the oldest surviving oplog row and treated an empty oplog as "no gap possible", which made that steady state the one state from which a wiped peer could never be healed: it reconnected with an empty database and a zero watermark, was told it needed no snapshot, and stayed empty indefinitely. With an empty oplog the oldest seq we could still stream is one past last_acked_seq — ack-pruning is the only thing that empties the oplog without also flagging needs_snapshot, and it deletes exactly seq <= last_acked_seq. Measuring the gap against that heals the wiped peer and leaves every other case identical: a converged peer's watermark equals last_acked_seq (no snapshot), and a pair that has never written anything has both at zero. Found by the OtOpcUa LocalDb Phase 1 live gate (check 4), which recorded it as a documented limitation. Covered by a test that reproduces the live-gate scenario through the real session stack: RED before this fix (the wiped side never converges, 15 s timeout), green after. 148/148 pass. Version 0.1.2. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
12e6f0d2c5 |
fix(localdb): create the database file's parent directory (0.1.1)
SQLite creates the database FILE on demand but never its parent DIRECTORY, and
SqliteLocalDb opens the file eagerly in its constructor — so a configured path
under a not-yet-existing directory was a hard startup failure ("SQLite Error
14: unable to open database file") rather than a first run that just worked.
Relative defaults like ./data/app.db hit it routinely; a containerized consumer
escapes only when a volume mount happens to create the directory first.
ScadaBridge found this during LocalDb Phase 2 and worked around it app-side
(SiteLocalDbDirectory.Ensure). Every other consumer still had the gap, so the
guarantee belongs here.
The standing objection to a library that creates directories is that it can
mask a mis-typed path by silently starting an empty database. That does not
apply: SQLite ALREADY does exactly that for a mis-typed file NAME. Refusing to
create the directory protects nothing — it only makes the two halves of one
path behave inconsistently and turns the directory half into an opaque error.
A creation failure (permissions, read-only mount) now throws an
InvalidOperationException naming the directory and the cause, instead of
surfacing as SQLite error 14. That is a change of exception type, but only on a
path that already failed hard.
Both behaviours are pinned and verified to fail with the call removed.
147 tests pass, 0 warnings. Published to the Gitea feed at 0.1.1.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
|
||
|
Joseph Doherty
|
61e5564c26 |
feat(localdb): scaffold ZB.MOM.WW.LocalDb (core/contracts/replication + tests)
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |