docs(localdb): Phase 2 gate + final Phase 1 task record
Task 14. A GATE document, not an implementation plan - the plan explicitly wants Phase 2 planned against post-Phase-1 reality, which means after this record exists rather than from the original design alone. Records what Phase 1 actually established (ordering constraints, where registration lives and why, the fail-closed interceptor, the silent meter allowlist), what must be ported before anything is deleted (both bespoke replicator test files, treated as specifications, plus the accepted N5 bounded-duplicate race), and why Phase 2 is a harder risk class than Phase 1: it REPLACES a working mechanism rather than adding one where none existed, sf_messages has external side effects, config tables drive deployment, there is no dual-mechanism period, and the migration is real this time rather than the usual no-op. The most important line is an open question, not an answer: choosing MaxOplogRows / MaxOplogAge / snapshot-resync thresholds for sf_messages write rates needs a Phase-1 rig SOAK that has not been run. The live gate proved correctness, not steady-state behaviour over time. Phase 2 should not be planned without it. tasks.json finalized: all 15 tasks completed, every deviation from the plan recorded with its reason. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -125,7 +125,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- `ScadaBridge:OperationTracking:ConnectionString` and `ScadaBridge:SiteEventLog:DatabasePath` are **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
|
||||
- Design: `docs/plans/2026-07-19-scadabridge-localdb-design.md`; Phase 1 plan `docs/plans/2026-07-19-localdb-adoption-phase1.md`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started and needs its own plan.
|
||||
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started — see `docs/plans/2026-07-19-localdb-phase2-gate.md`.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
|
||||
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
|
||||
|
||||
@@ -30,21 +30,155 @@
|
||||
}
|
||||
],
|
||||
"tasks": [
|
||||
{"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "completed", "commit": "7370b331"},
|
||||
{"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "completed", "blockedBy": [1], "commit": "727fa48c"},
|
||||
{"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "completed", "blockedBy": [2], "commit": "2977e6c4", "note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup."},
|
||||
{"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "completed", "blockedBy": [3], "commit": "0b5e9b44", "note": "Ctor takes ILocalDb, not IOptions<OperationTrackingOptions>; both ad-hoc reader paths switched too and their OpenAsync calls removed (CreateConnection returns an OPEN connection). Three DI fixtures needed LocalDb:Path added because resolving IOperationTrackingStore now forces ILocalDb construction."},
|
||||
{"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "completed", "blockedBy": [3], "commit": "727fa48c + 0d8a80aa", "note": "GUID minting landed early with Task 2 (727fa48c) to avoid committing a writer that violated site_events.id NOT NULL. The ILocalDb.CreateConnection() swap is 0d8a80aa. connectionStringOverride DELETED, not preserved: site_events is replicated, so a raw connection lacks zb_hlc_next() and fails closed."},
|
||||
{"id": 15, "subject": "Task 5b: Migrate the event-log read path off integer ids", "status": "completed", "blockedBy": [5], "commit": "727fa48c"},
|
||||
{"id": 6, "subject": "Task 6: One-time legacy data migrator", "status": "pending", "blockedBy": [4, 5]},
|
||||
{"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "completed", "blockedBy": [3], "commit": "a560e9ea", "note": "DEVIATION: AddZbLocalDbReplication moved from Program.cs (where the plan put it) into SiteServiceRegistration, because the composition-root tests build that graph and not Program.cs - in Program.cs a missing registration would still show green. MapZbLocalDbSync stays in Program.cs (needs the WebApplication). Endpoint is UNAUTHENTICATED until Task 8; no config in this repo sets a peer."},
|
||||
{"id": 8, "subject": "Task 8: Bearer auth interceptor for the sync endpoint", "status": "pending", "blockedBy": [7]},
|
||||
{"id": 9, "subject": "Task 9: Surface ISyncStatus as site health signal", "status": "pending", "blockedBy": [7]},
|
||||
{"id": 10, "subject": "Task 10: Two-node in-process convergence test", "status": "pending", "blockedBy": [6, 8, 15]},
|
||||
{"id": 11, "subject": "Task 11: Docker rig enablement (site-a pair)", "status": "pending", "blockedBy": [8]},
|
||||
{"id": 12, "subject": "Task 12: Live gate on the docker rig", "status": "pending", "blockedBy": [9, 10, 11]},
|
||||
{"id": 13, "subject": "Task 13: Documentation truth pass", "status": "pending", "blockedBy": [12]},
|
||||
{"id": 14, "subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration", "status": "pending", "blockedBy": [12]}
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Add LocalDb packages to the build",
|
||||
"status": "completed",
|
||||
"commit": "7370b331"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
1
|
||||
],
|
||||
"commit": "727fa48c"
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: Register LocalDb in the site composition root",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
2
|
||||
],
|
||||
"commit": "2977e6c4",
|
||||
"note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
3
|
||||
],
|
||||
"commit": "0b5e9b44",
|
||||
"note": "Ctor takes ILocalDb, not IOptions<OperationTrackingOptions>; both ad-hoc reader paths switched too and their OpenAsync calls removed (CreateConnection returns an OPEN connection). Three DI fixtures needed LocalDb:Path added because resolving IOperationTrackingStore now forces ILocalDb construction."
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
3
|
||||
],
|
||||
"commit": "727fa48c + 0d8a80aa",
|
||||
"note": "GUID minting landed early with Task 2 (727fa48c) to avoid committing a writer that violated site_events.id NOT NULL. The ILocalDb.CreateConnection() swap is 0d8a80aa. connectionStringOverride DELETED, not preserved: site_events is replicated, so a raw connection lacks zb_hlc_next() and fails closed."
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"subject": "Task 5b: Migrate the event-log read path off integer ids",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
5
|
||||
],
|
||||
"commit": "727fa48c"
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: One-time legacy data migrator",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
4,
|
||||
5
|
||||
],
|
||||
"commit": "98b94771",
|
||||
"note": "Runs AFTER RegisterReplicated (pinned by an oplog assertion). Deterministic mig-{NodeName}-{legacyId} event ids, not fresh GUIDs, so a crash between commit and rename cannot duplicate. DEVIATION: sync transaction on a CreateConnection() connection instead of BeginTransactionAsync - OnReady is a synchronous Action<ILocalDb>."
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: Replication registration + sync endpoint (default OFF)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
3
|
||||
],
|
||||
"commit": "a560e9ea",
|
||||
"note": "DEVIATION: AddZbLocalDbReplication moved from Program.cs (where the plan put it) into SiteServiceRegistration, because the composition-root tests build that graph and not Program.cs - in Program.cs a missing registration would still show green. MapZbLocalDbSync stays in Program.cs (needs the WebApplication). Endpoint is UNAUTHENTICATED until Task 8; no config in this repo sets a peer."
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: Bearer auth interceptor for the sync endpoint",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
7
|
||||
],
|
||||
"commit": "59c69519",
|
||||
"note": "Fail-closed: no configured ApiKey rejects ALL sync streams. All four handler shapes gated, not just unary (the real RPC is a duplex stream). DEVIATION: Grpc.Core.Testing does not exist on grpc-dotnet; hand-rolled FakeServerCallContext instead of adding the retired native package."
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: Surface ISyncStatus as site health signal",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
7
|
||||
],
|
||||
"commit": "59c69519",
|
||||
"note": "Additive init properties on SiteHealthReport so the Akka DTO ctor is untouched. Both fields nullable: null = no data, false/0 = a real reading (which is the healthy default-OFF state). Collector stores one tuple, read once, so a torn read cannot pair fresh Connected with stale backlog."
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Two-node in-process convergence test",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
6,
|
||||
8,
|
||||
15
|
||||
],
|
||||
"commit": "3c395d79",
|
||||
"note": "5 scenarios over real loopback gRPC + the real interceptor, initialized via SiteLocalDbSetup.OnReady. Verified NON-vacuous: mismatching the two ApiKeys turns all 5 red with 2m30s of timeouts."
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Docker rig enablement (site-a pair)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
8
|
||||
],
|
||||
"commit": "(with Task 11 commit)",
|
||||
"note": "site-a replicated, site-b/site-c deliberately not, so default-OFF is proven side-by-side on one rig."
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"subject": "Task 12: Live gate on the docker rig",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
9,
|
||||
10,
|
||||
11
|
||||
],
|
||||
"commit": "95c108f1",
|
||||
"note": "PASS. Found one REAL defect: ZbTelemetryOptions.Meters is a silent allowlist, so localdb_* was absent from /metrics - the plan's 'flows through automatically' claim was wrong. Fixed + pinned. Evidence: identical row_version/HLC/originating node_id across the pair, 0 dead letters, oplog drained, site failover + rejoin catch-up, localdb_* scraped, site-b inert. NOTE: docker/failover-drill.sh targets CENTRAL nodes and does NOT exercise the site-pair claim; the site flip was run directly."
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"subject": "Task 13: Documentation truth pass",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
12
|
||||
],
|
||||
"commit": "(docs commit)",
|
||||
"note": "Both CLAUDE.md files + the false 'starts a fresh log' XML doc + the two now-migration-only option properties."
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
12
|
||||
],
|
||||
"commit": "(gate commit)",
|
||||
"note": "Gate document only, per the plan - Phase 2 gets its own implementation plan. docs/plans/2026-07-19-localdb-phase2-gate.md. KEY FINDING: the rig SOAK that Phase 2's oplog-cap tuning needs has NOT been run; the live gate proved correctness, not steady-state behaviour."
|
||||
}
|
||||
],
|
||||
"knownFlakes": [
|
||||
{
|
||||
@@ -56,5 +190,6 @@
|
||||
"note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out on a cold fixture, ~1s and passes warm. Baselined against a stashed tree to confirm it is not a LocalDb regression."
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-19"
|
||||
}
|
||||
"lastUpdated": "2026-07-19",
|
||||
"phase1Status": "COMPLETE - all 15 tasks done, live gate PASS. Branch feat/localdb-phase1 NOT merged/pushed."
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
# LocalDb Phase 2 — Gate Document
|
||||
|
||||
> **Status: NOT STARTED.** This is the gate, not the plan. Phase 2 work must not begin
|
||||
> until an implementation plan is written against the facts below and the open questions
|
||||
> in §5 are answered.
|
||||
|
||||
**Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence.
|
||||
|
||||
**Phase 2 goal (from the design doc, scadaproj `docs/plans/2026-07-19-scadabridge-localdb-design.md` §1):**
|
||||
move `scadabridge.db`'s config tables and StoreAndForward's `sf_messages` into the same
|
||||
consolidated LocalDb file, then **delete** `SiteReplicationActor` and StoreAndForward's
|
||||
`ReplicationService` outright. CDC triggers replace hand-shipped ops; the library's
|
||||
snapshot resync replaces the bespoke chunked anti-entropy. The design explicitly chose
|
||||
**replace + delete, no dual-mechanism period** — recoverable from git.
|
||||
|
||||
---
|
||||
|
||||
## 1. What Phase 1 actually established
|
||||
|
||||
Facts a Phase 2 plan can rely on, all verified rather than assumed:
|
||||
|
||||
- One `ILocalDb` per site process at `LocalDb:Path`, **required** (`ValidateOnStart`).
|
||||
A site config missing it fails to boot.
|
||||
- `SiteLocalDbSetup.OnReady` is the single place tables are created and registered.
|
||||
Ordering is load-bearing: **DDL → `RegisterReplicated` → writes**, then the legacy
|
||||
migrator. Rows written before registration are invisible to the peer forever, silently.
|
||||
- Replication is registered in `SiteServiceRegistration` (not `Program.cs`) so the
|
||||
composition-root tests cover it; only `MapZbLocalDbSync` lives in `Program.cs`.
|
||||
- Inbound auth is `LocalDbSyncAuthInterceptor`, fail-closed, scoped to
|
||||
`/localdb_sync.v1.LocalDbSync/`, sharing the existing 8083 h2c listener.
|
||||
- Default-OFF is real and proven side-by-side: rig site-a replicates, site-b/site-c do not.
|
||||
- `ZbTelemetryOptions.Meters` is a **silent allowlist** (`SiteServiceRegistration.ObservedMeters`).
|
||||
- Bidirectional sync works with only one node configured as initiator.
|
||||
|
||||
## 2. What Phase 2 must port before deleting anything
|
||||
|
||||
The bespoke replicator's tests encode behaviour that the design's CDC replacement must
|
||||
still satisfy. **Port the test intents first, delete second** — the plan should treat the
|
||||
existing tests as the specification, not as code to be removed alongside the implementation.
|
||||
|
||||
- `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs`
|
||||
- `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs`
|
||||
|
||||
Implementation under deletion:
|
||||
|
||||
- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs`
|
||||
- `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs`
|
||||
|
||||
Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-duplicate
|
||||
race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC
|
||||
inherits the same bound, a tighter one, or a different failure shape.
|
||||
|
||||
## 3. Substantially harder than Phase 1
|
||||
|
||||
Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no
|
||||
worse than today". Phase 2 replaces a working mechanism, which makes it a different risk
|
||||
class:
|
||||
|
||||
- **`sf_messages` is an outbound buffer with side effects.** Phase 1's tables are
|
||||
read-mostly records; a store-and-forward row that resurrects or double-delivers sends
|
||||
real traffic to an external system. Park/requeue/remove becoming ordinary captured
|
||||
row updates/deletes needs its own analysis of what LWW does to an in-flight send.
|
||||
- **Config tables drive deployment.** A converged-but-wrong config row is a deployed
|
||||
instance behaving incorrectly, not a missing history entry.
|
||||
- **No dual-mechanism period.** The chosen posture means the cutover is the test. That
|
||||
raises the bar on the offline convergence suite and the live gate considerably.
|
||||
- **Migration is not a no-op this time.** Phase 1's migrator usually finds nothing,
|
||||
because the legacy files sat outside the volume. `scadabridge.db` and
|
||||
`store-and-forward.db` are real, populated, on the volume, and actively replicated
|
||||
while the migration runs.
|
||||
|
||||
## 4. Required inputs before writing the plan
|
||||
|
||||
1. The design doc §1 Phase 2 and §2 schema/migration sections, re-read against
|
||||
post-Phase-1 reality (several Phase 1 assumptions changed during execution — see the
|
||||
`amendments` array in `2026-07-19-localdb-adoption-phase1.md.tasks.json`).
|
||||
2. The two test files in §2, read as specifications.
|
||||
3. **Rig soak observations that do not exist yet** — see §5.
|
||||
|
||||
## 5. Open questions — answer these first
|
||||
|
||||
- **Oplog growth and churn under real config/S&F write rates.** Phase 1's tables are
|
||||
low-volume; `sf_messages` is not. `MaxOplogRows` / `MaxOplogAge` / snapshot-resync
|
||||
thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that
|
||||
has not been run.* The live gate proved correctness, not steady-state behaviour over
|
||||
time.
|
||||
- **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do
|
||||
when one node parks a message the other is mid-delivery on?
|
||||
- **Migration-under-load.** How does the one-time copy of an actively-replicating
|
||||
`scadabridge.db` interact with the bespoke replicator still running during cutover?
|
||||
- **Rollback story.** With no dual-mechanism period, what is the recovery path if the
|
||||
cutover fails in production — beyond "revert the commit"?
|
||||
- **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly
|
||||
because Phase 1's tables were small. Adding config + S&F changes the size and write
|
||||
profile of that single file.
|
||||
|
||||
## 6. Not in scope (unchanged from the design)
|
||||
|
||||
`auditlog.db` (diverges per node by design — central pulls the union; replicating it would
|
||||
double-forward), the secrets store (has its own answer in `Secrets.Replicator.SqlServer`
|
||||
hub mode), and central nodes (SQL-Server-first, no LocalDb use case).
|
||||
Reference in New Issue
Block a user