docs(localdb): phase 2 truth pass across both repos
Normative first: Component-StoreAndForward.md:83 specified the whole chunked, ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It is rewritten rather than removed — the failure modes it reasoned about still exist, they are just bounded differently — and it now states the duplicate-delivery bound explicitly: a message can be delivered twice only when the OLD primary delivered it and the status change had not yet replicated when the gate flipped. One flush interval plus the in-flight ack, and unlike the old model it does NOT grow with backlog depth or with how long a node was absent. The N1 directional-authority and N5 orphan-row hazards are recorded as structurally gone, not merely unguarded. Frame-size known-issue amended: its 2026-06-26 resolution replaced the intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch itself, so the 128 KB Akka frame constraint no longer applies to that hop in any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which batches by ROW COUNT), including that the failure mode differs — an oversized gRPC message is rejected, not silently dropped. Deployment docs gain the two operational constraints that have no home in code: a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat handler that made a mixed-version pair converge went with the replicator, and a mixed pair now diverges silently), and a node offline beyond TombstoneRetention can resurrect deleted rows on rejoin. Both CLAUDE.md files corrected — each still said Phase 2 was NOT started. Definition of done closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures, 0 skips). Two DoD items needed amending rather than ticking: the stale-symbol grep still matches 4 lines, all deliberate comment prose recording what was deleted (a literal zero would delete the explanations that stop the old design coming back), and the live gate has 10 evidence items, not the 9 the checklist claimed. Deletes the phase2 resume-state scratch doc, which said to delete it on landing. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -120,12 +120,19 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- DCL write failures returned synchronously to calling script.
|
||||
- Tag path resolution retried periodically for devices still booting.
|
||||
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
|
||||
- **Consolidated site database (LocalDb Phase 1, 2026-07-19).** `OperationTracking` and `site_events` now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file, configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
|
||||
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
||||
- `ScadaBridge:OperationTracking:ConnectionString` and `ScadaBridge:SiteEventLog:DatabasePath` are **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
|
||||
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started — see `docs/plans/2026-07-19-localdb-phase2-gate.md`.
|
||||
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them:
|
||||
|
||||
- `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing.
|
||||
- `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now.
|
||||
- `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
|
||||
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
|
||||
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
|
||||
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
|
||||
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
|
||||
|
||||
@@ -104,7 +104,6 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon
|
||||
`SiteServiceRegistration.Configure` registers the site-only components. `AkkaHostedService.RegisterSiteActorsAsync` creates:
|
||||
- `DeploymentManagerActor` — cluster singleton scoped to `"site-{SiteId}"`.
|
||||
- `SiteCommunicationActor` — registered with `ClusterClientReceptionist`; creates a `ClusterClient` to configured central contact points.
|
||||
- `SiteReplicationActor` — one per node (not a singleton); handles best-effort S&F replication to the standby.
|
||||
- `EventLogHandlerActor` — cluster singleton scoped to `"site-{SiteId}"`.
|
||||
- `ParkedMessageHandlerActor` — bridges Akka to `StoreAndForwardService`.
|
||||
- `SiteAuditTelemetryActor` — created on a dedicated `audit-telemetry-dispatcher` (2-thread `ForkJoinDispatcher`) so SQLite reads and gRPC pushes never contend with hot-path actors.
|
||||
|
||||
@@ -8,7 +8,7 @@ Site Runtime (#3) operates exclusively on site clusters. Its entry point is the
|
||||
|
||||
The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
|
||||
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`, `SiteReplicationActor`.
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`.
|
||||
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
|
||||
- `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream).
|
||||
- `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`.
|
||||
@@ -79,7 +79,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
|
||||
|
||||
1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped).
|
||||
2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`.
|
||||
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync`, clears static overrides and native alarm state, and — if `_replicationActor` is non-null (it is optional and null in isolated deployments/tests) — tells `SiteReplicationActor` to push to the peer node.
|
||||
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
|
||||
4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
|
||||
|
||||
For redeployment (instance already running), the existing actor is stopped and watched:
|
||||
@@ -216,7 +216,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
|
||||
|
||||
### Standby replication
|
||||
|
||||
`SiteReplicationActor` runs on every site node (not a singleton). The active node's `DeploymentManagerActor` tells it `ReplicateConfigDeploy`, `ReplicateConfigRemove`, `ReplicateConfigSetEnabled`, `ReplicateArtifacts`, or `ReplicateStoreAndForward`. The replication actor tracks the peer node via Akka cluster membership events and forwards each command to `/user/site-replication` on the peer via `ActorSelection`. Replication is fire-and-forget (no ack wait per design), so a failed write to the standby is logged but does not fail the primary operation.
|
||||
Config replication has no actor. `SiteReplicationActor` — which received `ReplicateConfigDeploy` / `ReplicateConfigRemove` / `ReplicateConfigSetEnabled` / `ReplicateArtifacts` / `ReplicateStoreAndForward` from the active node's `DeploymentManagerActor`, tracked the peer through cluster membership events, and forwarded each command to `/user/site-replication` via `ActorSelection` — was deleted in LocalDb Phase 2, together with its notify-and-fetch exchange (the standby was told a deploy had happened and then HTTP-fetched the config itself). The site's config tables are now replicated by LocalDb CDC, so a deploy on either node reaches the other as an ordinary row change. `SiteReconciliationActor` still fetches over HTTP at node startup when central reports gaps; that is a different path and it survives.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
@@ -4,13 +4,12 @@ The Store-and-Forward Engine buffers site-originated outbound messages when a ta
|
||||
|
||||
## Overview
|
||||
|
||||
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` SQLite store and an optional `ReplicationService` that fans each buffer mutation to the standby.
|
||||
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` store. As of LocalDb Phase 2 that store writes to the consolidated LocalDb database, and the buffer reaches the peer as the replicated `sf_messages` table — the `ReplicationService` that used to fan each mutation to the standby by hand was deleted.
|
||||
|
||||
The component code lives in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`:
|
||||
|
||||
- `StoreAndForwardService` — the core buffer: enqueue, retry sweep, park/retry/discard, and the `ICachedCallLifecycleObserver` audit hook.
|
||||
- `StoreAndForwardStorage` — the SQLite layer; all reads and writes against `sf_messages`.
|
||||
- `ReplicationService` — fire-and-forget buffer replication to the standby.
|
||||
- `ParkedMessageHandlerActor` — Akka actor bridge that exposes parked-message query/retry/discard to the `SiteCommunicationActor`.
|
||||
- `NotificationForwarder` — the delivery handler for the `Notification` category; forwards buffered notifications to central via the ClusterClient transport and interprets the ack.
|
||||
- `StoreAndForwardOptions` — options class bound from the `StoreAndForward` configuration section.
|
||||
@@ -129,7 +128,9 @@ else
|
||||
|
||||
### Async replication to standby
|
||||
|
||||
`ReplicationService` wraps each buffer mutation — add, remove, park, requeue — in a `Task.Run` fire-and-forget. The active node does not wait for standby acknowledgment. The standby applies each `ReplicationOperation` via `ApplyReplicatedOperationAsync`, which calls the same `StoreAndForwardStorage` methods. Replication failures are logged at Debug and discarded; the standby may be slightly behind the active at any moment, producing at-most a few duplicate deliveries or missed retries after a failover — an accepted trade-off for zero added latency on the enqueue path.
|
||||
Replication is no longer the buffer's own concern. `sf_messages` is registered with LocalDb (`SiteLocalDbSetup.OnReady`), so every insert and status change is captured by a CDC trigger and shipped to the peer on the shared sync stream. The four hand-written operations — add, remove, park, requeue — and the `Task.Run` fan-out that carried them are gone, along with `ApplyReplicatedOperationAsync` and `ReplaceAllAsync`.
|
||||
|
||||
The trade-off is unchanged in shape: replication is still asynchronous, so the peer may be slightly behind at any instant. What changed is the bound. Convergence is now per row under last-writer-wins with HLC-ordered tombstones, so a lagging peer converges rather than diverging, and duplicate delivery after a failover is limited to messages the old primary delivered whose status change had not yet replicated. See `Component-StoreAndForward.md` for the normative statement of that bound.
|
||||
|
||||
The four `ReplicationOperationType` values are `Add`, `Remove`, `Park`, and `Requeue` (requeue was added to cover the operator-initiated `Parked→Pending` transition so the standby preserves retry intent after failover).
|
||||
|
||||
|
||||
@@ -187,6 +187,14 @@ ALTER ROLE db_owner ADD MEMBER scadabridge_svc;
|
||||
|
||||
Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions.
|
||||
|
||||
## Upgrading a Site Pair
|
||||
|
||||
**Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
|
||||
not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
|
||||
mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
|
||||
silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
|
||||
`TombstoneRetention` bound on how long one node may stay offline.
|
||||
|
||||
## Post-Installation Verification
|
||||
|
||||
1. Start the service: `sc.exe start ScadaBridge-Central`
|
||||
|
||||
@@ -150,7 +150,27 @@ Each site has its own two-node cluster:
|
||||
- Same split-brain resolver as central (keep-oldest).
|
||||
- Singleton actors: Site Deployment Manager migrates on failover.
|
||||
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
|
||||
- SQLite persistence: Both nodes access the same SQLite files (or each has its own copy with async replication).
|
||||
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
|
||||
asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
|
||||
share a SQLite file.
|
||||
|
||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||
|
||||
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
||||
the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby
|
||||
could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler
|
||||
along with the replicator, so a mixed-version pair has no common replication path: the two nodes
|
||||
will run, but they will not converge, and the divergence is silent.
|
||||
|
||||
Stop both nodes of a site pair, upgrade both, then start both.
|
||||
|
||||
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
|
||||
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
|
||||
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
|
||||
nothing left to suppress the stale row the returning node still holds. Within the retention window
|
||||
a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
|
||||
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
|
||||
database from its peer rather than letting it rejoin.
|
||||
|
||||
### Central-Site Communication
|
||||
|
||||
|
||||
@@ -10,6 +10,27 @@ Fixed via the **notify-and-fetch** rework (the primary recommendation below), no
|
||||
- **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md)
|
||||
- **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps.
|
||||
|
||||
## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely
|
||||
|
||||
The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb
|
||||
Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's
|
||||
`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an
|
||||
ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any
|
||||
more, so the 128 000-byte frame constraint does not apply to it in any form.
|
||||
|
||||
The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site
|
||||
still fetches over HTTP, so that half of the original fix stands.
|
||||
|
||||
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
|
||||
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
|
||||
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
|
||||
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
|
||||
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
|
||||
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
|
||||
|
||||
Note the failure mode differs from the one documented below: an oversized gRPC message is
|
||||
**rejected**, not silently dropped.
|
||||
|
||||
The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
|
||||
|
||||
## Summary
|
||||
|
||||
@@ -1186,10 +1186,22 @@ Specific corrections:
|
||||
|
||||
## Definition of done
|
||||
|
||||
- [ ] Solution builds with **0 warnings** (`TreatWarningsAsErrors` is on)
|
||||
- [ ] Every suite green: Host, CentralUI, Commons, SiteRuntime, AuditLog, Communication,
|
||||
StoreAndForward, HealthMonitoring, IntegrationTests, SiteEventLogging
|
||||
- [ ] `grep -rn "SiteReplicationActor\|ReplicationService\|SfBufferSnapshot" src/` → no matches
|
||||
- [ ] Live gate PASS with all 9 evidence items captured
|
||||
- [ ] Convergence suite verified non-vacuous
|
||||
- [ ] Both `CLAUDE.md` files updated
|
||||
All met, 2026-07-20.
|
||||
|
||||
- [x] Solution builds with **0 warnings** (`TreatWarningsAsErrors` is on)
|
||||
- [x] Every suite green — Host 330, CentralUI 925, Commons 684, SiteRuntime 512, AuditLog 355,
|
||||
Communication 312, StoreAndForward 130, HealthMonitoring 97, SiteEventLogging 70,
|
||||
IntegrationTests 94. 0 failures, 0 skips.
|
||||
- [x] `grep -rn "SiteReplicationActor\|ReplicationService\|SfBufferSnapshot" src/` → **no code
|
||||
references.** Four matches remain and are all deliberate: prose inside `///` and `//` comments
|
||||
in `ActiveNodeEvaluator`, `SiteLocalDbSetup` and `AkkaHostedService` that records what was
|
||||
deleted and why. **The check as written is too blunt** — a literal "no matches" would force
|
||||
deleting the very explanations that stop someone re-introducing the old design. Read it as
|
||||
"no references from code".
|
||||
- [x] Live gate PASS — **10** evidence items, not the 9 this checklist says (Task 20 itself lists
|
||||
ten). All captured in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`, with two scope
|
||||
caveats recorded there rather than glossed.
|
||||
- [x] Convergence suite verified non-vacuous — with the eight `RegisterReplicated` calls commented
|
||||
out, all four Task 18 scenarios go red (4 failed / 0 passed); restored, 20/20 across the three
|
||||
LocalDb suites.
|
||||
- [x] Both `CLAUDE.md` files updated
|
||||
|
||||
@@ -88,8 +88,8 @@
|
||||
{"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"},
|
||||
{"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"},
|
||||
{"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"},
|
||||
{"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Sample DBs via throwaway copies (snap() helper), never host-side sqlite3 on live files; metrics on :8084 in-container."},
|
||||
{"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]}
|
||||
{"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "completed", "classification": "high-risk", "blockedBy": [18, 19], "note": "ALL 10 CHECKS PASS. Evidence: docs/plans/2026-07-19-localdb-phase2-live-gate.md. Key blocker found and fixed mid-run: external systems reach a site ONLY via ArtifactDeploymentService, which `instance deploy` never invokes - `deploy artifacts` was needed both to deliver the probe harness and to propagate the owed ExternalSystemDefinitions restore. THREE METHOD CORRECTIONS to the plan: (1) its instruction to run DB checks host-side against the bind mounts is UNSAFE - host sqlite3 poisons the container WAL; copy the db/-wal/-shm triplet and query the copy. (2) `docker exec ... curl` cannot scrape metrics (no curl in aspnet:10.0) and with 2>/dev/null the failure is silent - it nearly became a false 'metrics missing' finding; use a network-sharing curl sidecar. (3) checks needing S&F load need CachedCall, not Call. CAVEATS recorded not glossed: check 2's zero-count is vacuous alone (legacy source was also empty) and rests on the ABSENCE of CDC triggers; check 7's native_alarm_state leg was empty live and is covered only offline; check 10's sampling is coarse and the real rise/drain evidence comes from check 6's 0->4->0.", "commit": "158e79bb"},
|
||||
{"id": 21, "subject": "Task 21: Documentation truth pass", "status": "completed", "classification": "standard", "blockedBy": [20], "note": "Both CLAUDE.md files + Component-StoreAndForward.md:83 (normative resync paragraph rewritten for CDC, stating the duplicate-delivery bound explicitly: limited to messages the OLD primary delivered whose status change had not yet replicated when the gate flipped - one flush interval plus in-flight ack, and it does NOT grow with backlog depth or absence duration) + components/{StoreAndForward,SiteRuntime,Host}.md + the frame-size known-issue (amended: Phase 2 deleted notify-and-fetch itself, so the 128KB Akka frame constraint is gone from the intra-site hop entirely; successor ceiling is the 4MB gRPC cap via MaxBatchSize, and note the failure mode differs - oversized gRPC is REJECTED, not silently dropped) + deployment topology-guide.md and installation-guide.md (D5 stop-both-together, D2 TombstoneRetention resurrection bound). DoD closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures). DoD grep nuance recorded in the plan: 4 matches remain in src/ and are all deliberate COMMENT prose explaining what was deleted - a literal 'no matches' would delete the explanations that stop someone re-introducing the old design.", "commit": null}
|
||||
],
|
||||
"knownFlakes": [
|
||||
{
|
||||
|
||||
@@ -1,198 +0,0 @@
|
||||
# LocalDb Phase 2 — resume state (updated 2026-07-20, pre-compaction #2)
|
||||
|
||||
Scratch handoff for picking the plan back up. Delete once Phase 2 lands.
|
||||
|
||||
**Plan:** [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
|
||||
(execute with `superpowers-extended-cc:executing-plans`; task state in
|
||||
`…-phase2.md.tasks.json`, which carries a per-task deviation note)
|
||||
**Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged)
|
||||
**Tree:** clean @ `0ad11d6b`. Build 0 warnings. 25 commits ahead of `origin/feat/localdb-phase2`.
|
||||
|
||||
---
|
||||
|
||||
## Where we are
|
||||
|
||||
**Tasks 1–16 are DONE. The cutover has landed. Next up: Task 17.**
|
||||
|
||||
| Task | State |
|
||||
|---|---|
|
||||
| 1–6 | done (waves 0–2) — see the git log |
|
||||
| 7 `SiteLocalDbSetup` DDL | done |
|
||||
| 8 `sf_messages` migrator | done |
|
||||
| 9 config-table migrator | done |
|
||||
| 10 S&F CDC convergence specs | done |
|
||||
| 11 resync + config convergence specs | done |
|
||||
| 12 active-node purge pin | done |
|
||||
| 13 guarded-write scope check | done |
|
||||
| **14 + 15 + 16 the CUTOVER** | **done — landed as ONE commit `037798b3`** |
|
||||
| **17–21** | **not started** |
|
||||
|
||||
**Verification at the cutover boundary:** build 0 warnings; SiteRuntime 512, StoreAndForward 130,
|
||||
Host 330, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb integration 16 —
|
||||
**all pass, 0 failures.** (SiteRuntime/StoreAndForward fell from 533/153 as 6 obsolete test files
|
||||
were deleted.)
|
||||
|
||||
### The bespoke replicators are gone
|
||||
|
||||
`SiteReplicationActor`, `ReplicationMessages.cs`, `ReplicationService` and
|
||||
`StoreAndForwardStorage.ReplaceAllAsync` are deleted. Ten tables now replicate via LocalDb CDC:
|
||||
the Phase 1 pair plus `sf_messages` and the 7 site config tables.
|
||||
|
||||
`notification_lists` and `smtp_configurations` are created but **never registered** — permanently
|
||||
empty by design, and registering them would open a standing replication channel whose only
|
||||
historical payload was plaintext SMTP passwords. That decision is pinned by its own
|
||||
security-named test and stated in `OnReady`.
|
||||
|
||||
---
|
||||
|
||||
## Next: Task 17 (config-key cleanup)
|
||||
|
||||
Partly **required**, not optional: `SiteRuntimeOptions.cs:66` still documents
|
||||
`ConfigFetchRetryCount` as "Consumed by `SiteReplicationActor`", which no longer exists.
|
||||
|
||||
Per the plan: relax `StartupValidator.cs:113` (`SiteDbPath`) and
|
||||
`StoreAndForwardOptionsValidator.cs:19-21` (`SqliteDbPath`) to migration-only; delete
|
||||
`ReplicationEnabled` and `ConfigFetchRetryCount` outright with their validator rules and config
|
||||
entries; **leave the two path keys present in configs** with a comment — the legacy migrator reads
|
||||
them, and removing them would strand un-migrated data on a node that has not yet started once.
|
||||
**10** `appsettings.Site.json` files, including `deploy/wonder-app-vd03/`.
|
||||
|
||||
Then 18 (two-node convergence suite), 19 (rig config — `MaxBatchSize = 16`), 20 (live gate),
|
||||
21 (docs truth pass).
|
||||
|
||||
---
|
||||
|
||||
## Findings that change the plan (carry these forward)
|
||||
|
||||
### From Task 1 (unchanged)
|
||||
|
||||
- **D6's premise is wrong, its risk is real.** Largest known production `config_json` is ~60–70 KB,
|
||||
not >128 KB → no stop condition. But `MaxBatchSize=500` × 70 KB ≈ 35 MB vs a 4 MB gRPC cap →
|
||||
**Task 19 must set `LocalDb:Replication:MaxBatchSize = 16`.** The one firmly evidence-backed number.
|
||||
- **D4 is wrong.** `native_alarm_state` is bounded by per-`SourceReference` coalescing on a 100 ms
|
||||
flush (`NativeAlarmActor.cs:473-502`).
|
||||
- **`sf_messages` has a hard 50 rows/sec ceiling** — `SweepBatchLimit` (500) ÷ `RetryTimerInterval` (10 s).
|
||||
- **Task 1 step 7's stop condition is weaker than written.** Exceeding the oplog caps prunes and
|
||||
flags `needs_snapshot` → graceful resync, not data loss.
|
||||
|
||||
Measured (six 60 s intervals, copy-based `snap()`): `sf_messages` **0.80 rows/sec** insert, **0/sec**
|
||||
retry-UPDATE, oplog 0, max `payload_json` 76 B, max `config_json` 721 B (rig — *not* representative),
|
||||
zero SQLite errors in 30 min.
|
||||
|
||||
### From waves 1–2
|
||||
|
||||
- **LATENT PHASE-1 DEFECT, fixed in-app only.** The LocalDb library never creates the parent
|
||||
directory of `LocalDb:Path`, and `SqliteLocalDb` opens the file **eagerly in its constructor**, so
|
||||
a missing directory is a **hard boot failure** (`SQLite Error 14`). The default site config uses
|
||||
the **relative** `./data/site-localdb.db`; the docker rig escapes only because its volume mount
|
||||
creates `/app/data`. Fixed via `SiteLocalDbDirectory.Ensure(config)` before `AddZbLocalDb`, pinned
|
||||
by `Host.Tests/SiteLocalDbDirectoryTests`.
|
||||
**OPEN DECISION FOR THE USER:** the real fix arguably belongs in the `ZB.MOM.WW.LocalDb` library
|
||||
— **every other consumer still has this gap.** Not done here: a library change means a version
|
||||
bump and is outside this plan's scope.
|
||||
- **`SiteStorageService.CreateConnection()` contract flipped** — returns an **already-open**
|
||||
connection; calling `Open`/`OpenAsync` on one **throws**.
|
||||
- **LocalDb has NO in-memory mode.** Every `Mode=Memory;Cache=Shared` fixture became a real temp file.
|
||||
- **A store constructor change's test blast radius is ~7× the plan's estimate** (Task 5: 40 files,
|
||||
7 projects).
|
||||
- **`tests/ZB.MOM.WW.ScadaBridge.TestSupport`** holds `TestLocalDb` (dispose **before** deleting —
|
||||
the master connection anchors the WAL). Use it; don't copy fixtures.
|
||||
- **Where behaviour moved owner, retarget tests — don't delete them with the code.** WAL is genuinely
|
||||
LocalDb's; directory creation is **not**. Both directions occurred.
|
||||
|
||||
### From waves 3–5 (new)
|
||||
|
||||
- **PLAN DEFECT — Tasks 14/15/16 cannot compile separately.** `SiteReplicationActor` takes a
|
||||
`ReplicationService` and calls `ReplaceAllAsync`; `DeploymentManagerActor` Tells types from
|
||||
`ReplicationMessages.cs`; `AkkaHostedService` constructs the actor. Landed as one commit, which
|
||||
also strengthens Task 14's own invariant (never both mechanisms, never neither).
|
||||
- **The legacy-copy column intersection.** The plan said "migrate all 16 columns"; that would be a
|
||||
data-loss bug. An older legacy file lacks `execution_id`/`parent_execution_id`/
|
||||
`last_attempt_at_ms`, naming a missing column throws `no such column`, and the existing reader
|
||||
treats that as an unrecognised shape and **silently discards every row**. The copy intersects the
|
||||
legacy and current column sets, with a required-column (PK) guard so the tolerance cannot degrade
|
||||
into copying NULL-keyed rows.
|
||||
- **`MigratorColumnLists_MatchTheLiveSchema`** (added beyond the plan). A column-list mismatch is
|
||||
invisible at runtime in BOTH directions — a typo is silently dropped by the intersection, a
|
||||
missing column silently leaves data behind. `SiteStorageTables`/`LegacyTable` are `internal` for it.
|
||||
- **Absence assertions need a positive control.** `MessageAddedThenRemoved_NeverConvergesToPresent`
|
||||
PASSED with `sf_messages` unregistered entirely — an absent row is also what a pair replicating
|
||||
nothing looks like. Fixed with a control row that must converge in the same window.
|
||||
- **The positional-argument hazard was REAL.** Removing `DeploymentManagerActor`'s optional
|
||||
`IActorRef? replicationActor` shifted 6 trailing optionals and 4 test call sites bound the wrong
|
||||
arguments, some with no compile error. **`Props.Create` builds an expression tree, which rejects
|
||||
OUT-OF-POSITION named arguments** — so named args work only if in position, and the rest must be
|
||||
padded positionally.
|
||||
- **`ReplaceAllAsync` was unsafe to keep, not merely unused.** A mass DELETE on a now-replicated
|
||||
table would be captured and shipped to the peer.
|
||||
- **`LocalDbSitePairHarness`** is the shared two-node fixture (Phase 1 + both Phase 2 suites; Task 18
|
||||
makes four). Its `Phase2ReplicatedTables` list is **literal, not derived from production code**, so
|
||||
a registration that drifts fails the tests instead of agreeing with itself.
|
||||
- **Ordinal sort gotcha:** `_` (0x5F) sorts before `b` (0x62), so `data_connection_definitions`
|
||||
precedes `database_connections`.
|
||||
- **Never `git checkout <file>` to undo a non-vacuity experiment** — it reverts to HEAD and takes
|
||||
in-progress work with it (cost one redo of Task 8). Use `cp` backups.
|
||||
|
||||
---
|
||||
|
||||
## Rig operational gotchas (hard-won; not obvious from the repo)
|
||||
|
||||
- **`ExternalSystem.Call` does NOT buffer to store-and-forward — only `ExternalSystem.CachedCall`
|
||||
does.** The single most important detail for generating S&F load.
|
||||
- **Never** query the bind-mounted SQLite files from the macOS host while containers run. Use the
|
||||
plan's `snap` helper, or `docker run --rm -v <dir>:/d alpine/sqlite3 …`.
|
||||
- **No `curl` in the `aspnet:10.0` image**, and metrics port 8084 is not published. Use a sidecar:
|
||||
`docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics`
|
||||
- **Seeded template 4 (`Motor Controller`) is not deployable** — 34 validation errors. The soak used
|
||||
a purpose-built `SoakGenerator` template.
|
||||
- CLI: `template script update` requires `--name` and `--trigger-type` even to change only `--code`.
|
||||
In zsh, don't put auth flags in an unquoted variable.
|
||||
- Rig auth: `--url http://localhost:9000 --username multi-role --password password`.
|
||||
`LdapGroupMappings` roles must be canonical `Designer`/`Deployer`; central caches them at startup.
|
||||
|
||||
## Rig state as left
|
||||
|
||||
Reseeded and healthy; both site-a nodes restarted after the poisoning incident.
|
||||
|
||||
**Needs cleanup before the Task 20 live gate:**
|
||||
- `ExternalSystemDefinitions` id 1 is still repointed to `http://127.0.0.1:9` — restore to
|
||||
`http://scadabridge-restapi:5200`.
|
||||
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are still deployed on
|
||||
site-a and **still generating load**.
|
||||
|
||||
---
|
||||
|
||||
## Commits (this branch, newest last)
|
||||
|
||||
| Commit | What |
|
||||
|---|---|
|
||||
| `25463d52` … `d512572d` | plan review, soak, root-cause + hardening, first resume doc |
|
||||
| `12eb97e0` | **Task 2** — gate CLOSED |
|
||||
| `9e3239c5` | **Task 3** — `StoreAndForwardSchema.Apply` |
|
||||
| `ac5eb12c` | **Task 4** — `SiteStorageSchema.Apply` |
|
||||
| `f2efeb37` | **Tasks 5+6** — both stores take `ILocalDb`; latent directory defect fixed; `TestSupport` added |
|
||||
| `fefbbb31` | resume state through wave 2 |
|
||||
| `f8aa02e2` | **Task 7** — Phase 2 DDL in the consolidated DB (not yet registered) |
|
||||
| `bdc0dffe` | **Task 8** — migrate legacy `store-and-forward.db` |
|
||||
| `5ddc7eed` | **Task 9** — migrate legacy `scadabridge.db` config tables |
|
||||
| `2bbe6631` | **Task 10** — S&F CDC convergence specs (+ `LocalDbSitePairHarness` extraction) |
|
||||
| `c56bf4ae` | **Task 11** — resync + config convergence specs |
|
||||
| `79ce5161` | **Tasks 12+13** — purge pin; guarded-write scope |
|
||||
| `037798b3` | **Tasks 14+15+16 — THE CUTOVER** |
|
||||
| `0ad11d6b` | task state for the cutover |
|
||||
|
||||
Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under
|
||||
full-suite load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL
|
||||
fixture, ~1 s warm). Neither reproduced during waves 1–5.
|
||||
|
||||
## Execution notes
|
||||
|
||||
- Waves ran **sequentially in the main worktree**, not as the plan's parallel worktree agents — the
|
||||
tasks were small enough that worktree setup + merge cost more than it saved, and sequential
|
||||
execution removes the git race the isolation rule exists to prevent.
|
||||
- Subagents were used only in waves 1–2, for bulk mechanical fixture rewiring. **Their reported
|
||||
results were re-verified independently** — one agent's partial run had missed 2 real failures that
|
||||
a full-suite run caught. Re-run the suites yourself.
|
||||
- Every new test was checked for **non-vacuity** by breaking the thing it claims to pin (removing
|
||||
DDL, removing the `Migrate` call, unregistering a table, commenting out the purge). This caught one
|
||||
genuinely vacuous test. Keep doing it.
|
||||
@@ -80,7 +80,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
|
||||
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
|
||||
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
|
||||
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
|
||||
- **Peer-join anti-entropy resync (chunked, ack-confirmed).** Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was **down for an extended period** (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node **(re)tracks its peer**, a **standby** requests a full-buffer snapshot (`RequestSfBufferResync`); the **active** node loads up to `MaxResyncRows` (10 000) of its oldest rows and answers with a **sequence of byte-budgeted chunks** (`SfBufferSnapshotChunk`), each carrying a shared `ResyncId`, a 1-based `Sequence`, and the `TotalChunks` count. Chunking is mandatory because the monolithic snapshot exceeds Akka remoting's **default 128 000-byte frame** for any realistic backlog and `BuildHocon` sets no override — a single oversized message is silently undeliverable (review 02 round 2, **N2 High**). Rows accumulate into a chunk until the estimated payload budget (`MaxResyncChunkBytes` = 64 000, ≈50% frame headroom) or the row cap (`MaxResyncChunkRows` = 200) is hit; a single row whose payload alone exceeds the budget ships solo with a Warning. The standby **assembles all chunks of one `ResyncId`** (a new `ResyncId` discards any stale partial assembly; a partial that never completes is dropped after `resyncAssemblyTimeout`, default 30 s, and counted a replication failure), then **replaces its entire local buffer** with the assembled snapshot (`ReplaceAllAsync`, one transaction) and returns a delivery confirmation (`SfBufferResyncAck`). The active node arms an ack window (`resyncAckTimeout`, default 60 s): an acknowledged resync increments `scadabridge.store_and_forward.resync.completed`; an unacknowledged one (lost chunks / dead peer) logs a Warning and increments `scadabridge.store_and_forward.resync.ack_missing` — closing N2's silent-loss mode (previously nothing counted a lost snapshot and nothing retried until the next peer-track). Only the active node answers; only a standby applies, and both sides re-check at apply time (a mid-flight active-flip aborts the wipe) — each side checks the repo-standard **oldest-Up member** active-node predicate (singleton-host semantics via the shared `ActiveNodeEvaluator`, the **same predicate as the S&F delivery gate**; review 02 round 2, **N1 Critical** — using cluster *leadership* here let a rolling restart of the lower-address node make the delivering node wipe its own live buffer). Because replicated applies are **upserts** (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta; the one accepted exception is the rare **N5** orphan-row race (a replicated `Remove` ordered before the snapshot chunks can re-add the removed row, re-delivered once and self-corrected at the next resync — inherent to no-ack replication). If the buffer exceeds the 10 000-row cap the snapshot is flagged `Truncated` and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers. The legacy monolithic `SfBufferSnapshot` message + standby handler are **retained** for rolling-upgrade compatibility (an old active node's monolithic snapshot is still applied by a new standby).
|
||||
- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed by bounding `LocalDb:Replication:MaxBatchSize` (set to 16 on the rig, sized against a ~70 KB worst-case `config_json`; see the Phase 2 plan, D6). Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
|
||||
|
||||
### Operation Tracking Table (lives in Site Runtime, not here)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user