2ec0e6367176c0c74b85145a7faffb830e2ef28d
24 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
f47ceb55b1 |
docs: inbound API key store — shared SQL Server provider on central pairs (spec, runbook + central-pairs procedure, DBA DDL copy, topology, env inventory, CLAUDE.md, register row 108 closed with rig proof, row 109 Secrets backoff follow-up, Delmia spec §8)
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB |
||
|
Joseph Doherty
|
25c15dd164 |
ops(docker)+test(host)+docs: close register rows 104 + 105 — offline VACUUM recipe applied to the rig; build-time pin that no replicated table is a cascading-FK parent; record row 106 (LocalDb as_of_seq=0 full-snapshot loop)
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after, then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok, ~11% smaller; site-a replication reconnected with backlog 0 and no watermark change — VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY KEY AUTOINCREMENT). Topology guide gains the recipe. Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady (unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is enforced at build instead of at the first replicated boot. Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty; a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4: as_of_seq = max(MAX(seq), last_acked_seq). |
||
|
Joseph Doherty
|
324ba369d9 |
infra(docker)+test(siteeventlog): close register tripwires 79 + 93 — rig data to named volumes; pin site_events schema to the cap-model calibration
Row 79 executed its own prescribed remedy without waiting for a second corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site nodes) move from virtiofs bind mounts to named docker volumes (scadabridge-<node>-data), seeded in place with both rigs stopped and live-verified (site-a localDbReplicationConnected=true / backlog 0 over the migrated files; env2 stable, 0 restarts). The feared inspectability loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/. The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2 now carries the x-secrets-store-env posture + central data volumes. Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests pins the site_events columns/indexes the EventLogPurgeService size model is calibrated against, failing with a pointer to the memo §4 recalibration on any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire) annotated accordingly. |
||
|
Joseph Doherty
|
ddb3c32e06 |
fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was compared against page_count-freelist_count over the WHOLE consolidated LocalDb file, which since Phase 1/2 holds ten tables plus __localdb_oplog / __localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed ~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the same file, so the measured size did not fall, the "size did not decrease" guard fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present. A node whose peer was down could not enforce its cap at all, and the event log's budget was in practice shared with bookkeeping it does not own. The cap now measures site_events AND ITS FOUR INDEXES only, so replication bookkeeping can neither consume nor block the budget — and the cap stops being a function of the peer's health, which was the wrong direction of coupling: an inflated oplog is a replication fault, and letting it drive event deletion would destroy operational history in response to one. dbstat — the exact per-table measurement the register proposed — is NOT usable here, verified before implementing. It is a compile-time option and the bundled SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails with "no such table". Switching SQLite builds was rejected — the provider is shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy for fleet-wide version skew. The footprint is therefore modelled from sampled row width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted) through a page-layout model; whole pages rather than a flat rate, because page fill collapses once rows approach half a page and a linear model underestimates that by ~40%. Clamped by whole-file logical size, a true upper bound that can only correct an over-estimate and never binds on a replicated node. Index pages count against the cap deliberately: measured at ~34% of the table's footprint, and the 1 GB default was sized against a 393 B/row figure that already included them. Guard semantics restated — two backstops replace the single on-disk check: (1) row width unsampleable => refuse to trim (checked before the cap comparison, since a zero width makes "over cap" unanswerable rather than false; the old loop's answer to an unanswerable cap was to keep deleting); (2) estimated size not falling across a batch => stop. (2) is the old guard re-based onto the table-scoped metric, so it can no longer be tripped by replication bookkeeping and now means exactly one thing: events are arriving as fast as the trim removes them. For that to be observable the loop re-reads the row count each batch, while bytes-per-row is sampled once per tick and held fixed — re-sampling per batch would jitter the estimate as the trim eats the head of the table and could stall the loop through guard (2), the same self-defeating measurement as row 81 itself. Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven by the shared PurgeBudget. A tick where retention AND the cap trim both deleted used to emit two identical Information notes; the tick total is also the more useful number beside an oplog spike. Measured model error: -0.8% / +8.4% / +12.8% across representative workloads, biased high on purpose (an early cap forfeits budget; a low one fails at its only job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row. Fail-before: reverting the estimator to whole-file semantics fails 5 of the new tests and reproduces the original signature exactly (remaining=2000 estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring per-phase notes fails the row-84 test with "collection contained 2 matching items". No config keys added or renamed, so no rig/deploy changes. Decision record: docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean. |
||
|
Joseph Doherty
|
9f418a6aea | Merge row 34: site_events purge burst measured (2.73M-row worst case = 2.7x MaxOplogRows) and paced — shared 250k/tick budget, 5-min catch-up, ceiling-hit counter | ||
|
Joseph Doherty
|
d9a35846d9 |
perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
|
||
|
Joseph Doherty
|
851b528ca3 |
docs(deployment): row 79 — OrbStack/virtiofs rig hazard + proven recovery runbook
Documents the host-level hazard behind row 69's LocalDb corruption, all live-verified 2026-08-15. Scope is the macOS docker rigs only — this is a property of the filesystem the rig's data files sit on, not a ScadaBridge defect and not something a production Linux deployment is exposed to. Every rig data file (site-localdb.db, auditlog.db, secrets) rides a relative bind mount out of the repo tree, which on macOS crosses virtiofs — and virtiofs does not give SQLite true fsync durability. `--force-recreate`, which deploy.sh uses, is exactly the abrupt mid-write kill that turns that into a torn file. Records both observed failure modes, and specifically how differently they present, which is the part that makes this hard to spot: site-a-b failed loudly while running (every sync call throwing, connected:false, backlog 892), while site-a-a was corrupt LATENTLY and only became fatal on the next upgrade, when a new build's boot-time re-registration read pages the old build never touched (SIGABRT crash-loop x12). A clean-looking node proves nothing about its file. Second mode is the VM idle-sleeping mid-operation — `vmgr ... msg=sleep` in ~/.orbstack/log/vmgr.log, then virtiofs `refresh_nodeid: race with new lookup` errors and a hung docker until `orb restart docker` (data intact, containers auto-restart). Written so a future session checks vmgr.log BEFORE investigating ScadaBridge, which is where ~20 minutes went last time. Recovery runbook is the one that actually worked: stop the pair together, quarantine site-localdb.db AND its -wal/-shm sidecars with a .corrupt-<date> suffix (never delete — forensics; and a stale WAL beside a fresh database is its own corruption), restart together, then VERIFY convergence rather than assuming it. Notes the precondition that this is a replicated pair's recovery — it leans on the startup reconcile plus CDC snapshot resync, so an unreplicated node has no peer to reseed from and needs a backup instead. Mitigation options tabulated with their real costs (named volumes = native ext4 but loses the host-side inspectability the debugging sessions lean on; disabling OrbStack idle-pause = a host setting that cannot be version-controlled and silently does not exist on a new machine; accept + runbook = current), so the status quo reads as a choice rather than an oversight. Docs only — no compose change, no host config change, rig untouched. |
||
|
Joseph Doherty
|
e8a90f18dc | merge: external-system retry policy end-to-end (register row 27) — MaxRetries/RetryDelay now travel artifact→gRPC→site schema→S&F enqueue, CLI --max-retries/--retry-delay-seconds, ArtifactDiff compares them | ||
|
Joseph Doherty
|
8da60f57ac |
feat(health): KPI history + sustained-backlog badge for LocalDb replication (row 33)
LocalDbOplogBacklog was reported and Prometheus-exported with nothing consuming it: not sampled into KPI history, so a purge-burst spike could not be reviewed after the fact, and no threshold treatment anywhere. Both halves shipped. KPI SAMPLING. SiteHealthKpiSampleSource now emits localDbOplogBacklog and localDbReplicationConnected (0/1), and the backlog is charted on the Health dashboard's per-site trend panel. Charting is the point: a point-in-time tile can never answer 'did that spike drain?', which is the only thing separating an expected purge burst from a stalled peer. Both are CONDITIONALLY emitted rather than coerced with ?? 0 like the auditBacklogPending metric beside them. Both health-report fields are nullable and null means 'no reading' — replication not configured (rig site-b/site-c) or a failed poll — never zero. A fabricated 0 would render a pair that cannot read its own oplog, or one deliberately unreplicated, as perfectly healthy once a minute forever, burying the real series. So a site emits 12 metrics when unreplicated and 14 when replicated, and a GAP in these two series truthfully means 'not replicated / not reporting'. Note false and null differ for the connected flag: a configured link that is DOWN samples as 0; no link samples as nothing. Both fold as Gauge (summing an oplog depth, or a 0/1 flag, per hour is meaningless). THRESHOLD SEMANTICS — sustained, not instantaneous. This is the design, not a refinement. The site_events retention purge is CDC-captured like any other write, so the backlog jumps by the deleted batch and drains as the peer acks it; a bare 'backlog > threshold' badge would fire on every replicated site every day and be ignored within a week. Healthy is defined by the backlog RETURNING to ~0, which only a dwell can express. CentralHealthAggregator holds LocalDbBacklogAboveSince — the instant the backlog first rose above LocalDbBacklogThreshold (default 1000) — and the badge appears only after LocalDbBacklogSustainedFor (default 15 min). A report at or below the threshold clears the anchor; one above leaves the ORIGINAL instant intact, so the window is sustained rather than sliding. The null case is deliberately a third branch: a null backlog leaves the anchor UNCHANGED rather than clearing it. Null is evidence neither of a drain nor of a backlog, so a node going blind mid-climb must not silently reset its own dwell, and an unreplicated site (null forever) never accumulates an anchor and can never be badged. Validation rejects a non-positive threshold (0 would badge every replicated site permanently, since an idle backlog already sits at it) and a dwell shorter than one ReportInterval (satisfiable by a single sample, which collapses the check back into the bare threshold it replaces). DOC CORRECTION worth flagging: this row's premise was 'mirror the SiteAuditBacklog configurable threshold badge'. That badge does not exist — Component-HealthMonitoring.md described it in two places but no option, no comparison and no markup were ever implemented. The two stale claims are corrected here rather than left to send the next reader looking for it; the LocalDb badge is built fresh on the Metrics-stale badge's pattern, which is a real configurable-threshold badge. Tests: HealthMonitoring 115 (9 new aggregator tests incl. purge-burst-drains, unreplicated-never-anchors, null-mid-climb, exact-threshold, stale-report; 5 new validator tests; 4 new sampler tests incl. the null-semantics and false-vs-null cases); CentralUI 1042 (5 new badge/chart tests incl. the purge-burst negative control); Commons Kpi 93; KpiHistory 54. |
||
|
Joseph Doherty
|
41a5ecf14a |
feat(external-systems): deploy per-system retry policy to sites + CLI/management surface (register row 27)
An external system's MaxRetries/RetryDelay existed on the central entity but never reached a site and had no management surface, so every ExternalSystem.CachedCall buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval 30s ~= 25 min to park) whatever central had configured. The sibling DatabaseConnectionArtifact had always carried its equivalents. The consumption end was already wired: ExternalSystemClient.CachedCallAsync already read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset convention. Only the transport and persistence were missing. Commons - ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params (0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going nullable, so it maps 1:1 onto the entity and the proto3 scalars. - Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following the existing TimeoutSeconds "omit to preserve" convention. Both additive. Central - ManagementActor create maps both; update preserves-if-null. - ArtifactDeploymentService projects them onto the artifact. Wire - site_command.proto ExternalSystemArtifactDto gains max_retries=7 / retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto. Additive: a pre-existing site decodes them as 0/absent, which IS the unset value, so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh sitecommand; csproj Protobuf item re-commented, no message types added or removed. Site - external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing, DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7 appended so existing ones are untouched), apply path, and the legacy migrator's explicit column list all carry them. Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables, so this was checked against the AckTime/native_alarm_state precedent. Real columns are correct here, not a JSON bag: the table has no general-purpose config column (auth_configuration and method_definitions are both semantically specific), timeout_seconds already set the additive-column precedent on this exact table, and sibling database_connections stores the same two values as plain columns. Verified in LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its install script is DROP-then-CREATE, so capture triggers rebuild over the new column set automatically, and SiteStorageSchema.Apply runs before registration on the same boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead of diverging silently. Both-nodes-together is already the standing site-pair rule, so no new operational constraint; recorded in topology-guide.md. Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay, so a bundle import that changed a system's retry budget previewed as "Identical" - harmless while the values were inert, a live behaviour change now. No EF migration: the central ExternalSystemDefinitions columns already exist (InitialSchema). Central UI already had both form fields. Tests: site storage round-trip + unset-default, site apply path, central artifact projection, proto round-trip (via the shared sample), management create/preserve, CLI option parse + alias, and diff reporting. Fail-before verified on the two core tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path). Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI, components/StoreAndForward, components/SiteRuntime, deployment/topology-guide, CLI README (which was also missing --timeout-seconds entirely). |
||
|
Joseph Doherty
|
7b3e455c25 | Merge branch 'localdb-per-table-snapshot' — LocalDb 0.3.0 pin: per-table needs_snapshot baselining (residual #2 / R1) | ||
|
Joseph Doherty
|
a5ac309a94 |
chore(deps): bump ZB.MOM.WW.LocalDb to 0.3.0 — per-table snapshot resync
LocalDb 0.3.0 (scadaproj branch localdb-per-table-snapshot) narrows the snapshot-resync flag from one per-database bit to one per table, closing the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to v3 (__localdb_snapshot_state, upgraded in place on open); the wire stays compatible with 0.1.x/0.2.x, negotiated by capability rather than by a lib_schema_version bump — which is not available, since the handshake compares that field fail-closed for equality. SiteLocalDbSetup needed no restructuring: it already calls RegisterReplicated once per table, which is exactly what per-table flagging keys off. The comment now records that the loop shape is load-bearing, and what it buys — adding an eleventh table to ReplicatedTables on an already-replicating site snapshots that one table, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First boot is unchanged (all ten seed at once, so the flagged set is every registered table and the library sends an ordinary full snapshot), and upgrading the rig in place is a no-op (the ten are already ledgered, so nothing seeds and nothing is flagged). Suites: Host 490/490, SiteRuntime 604/604, StoreAndForward 134/134, SiteEventLogging 76/76. Solution build clean. topology-guide + CLAUDE.md LocalDb bullet updated; the umbrella scadaproj CLAUDE.md travels in that repo's commit. |
||
|
Joseph Doherty
|
9d2834e30a |
docs+log(siteeventlogging): explain the site_events purge oplog-backlog burst (R7)
The daily site_events retention purge (and the storage-cap trim) is CDC-captured on a replication-enabled site node exactly like any other write — correct by design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog / localdb_oplog_depth spike, drain, and an operator watching the gauge with no context reads it as a replication fault. Documentation + one log line, no behaviour change: - topology-guide.md gains "Reading the replication backlog — the daily site_events purge burst": when it fires (PurgeInterval 24h, anchored to the active node's PROCESS START, not a wall-clock hour, so it moves after every failover), where it shows (replicated nodes only — not rig site-b/site-c), the healthy signature (LocalDbReplicationConnected stays true, backlog returns to ~0) and what a genuine fault looks like instead. - Component-SiteEventLogging.md Storage records the same under retention/purge; Component-HealthMonitoring.md gains the two previously-undocumented LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the caveat, with cross-references both ways. - EventLogPurgeService emits one Information line naming the row count and the expected transient backlog when a purge deleted rows on a replication-enabled node, so the spike is correlatable in the log. Replication-awareness comes in as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured goes internal so the PeerAddress-OR-ApiKey rule stays in one place and SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note, matching the default that replication is opt-in and off. Both delete paths carry the note (a cap trim is usually the larger burst); the predicate is try/caught since a log-wording check must never break the purge. Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated does not, zero-rows does not, cap purge logs it, throwing predicate still purges and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green, Host 490/490 green, full solution build clean (0 warnings). |
||
|
Joseph Doherty
|
cca7f1786d | chore(deps): LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication | ||
|
Joseph Doherty
|
7ebdcd370a |
perf(host): install CDC capture only when replication is configured
SiteLocalDbSetup.OnReady registered all ten replicated tables unconditionally, so a deliberately unreplicated site node (site-b and site-c on the rig) carried the full 30-trigger CDC set forever. Every write to those tables paid two extra INSERTs plus a json_object serialization of the whole row, inside the caller's own transaction, and appended to an oplog nothing ever drains. Arch-review finding #5 (High), repo half; the library half — trigger cleanup API and O(1) backlog — is WP3.3. The ten RegisterReplicated calls are now behind a guard on whether the node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey. Either key counts, and the OR is load-bearing rather than defensive: replication is one bidirectional stream that exactly one side dials, so only the initiator sets PeerAddress. Verified against the rig — site-a node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey alone, site-b/site-c have no Replication section at all. Keying on PeerAddress alone would have stripped capture from every passive node and silently made each pair converge in one direction only. The load-bearing ordering documented in the file is preserved: DDL still precedes registration, and the legacy migrator still runs unconditionally after it — an unreplicated node must still absorb its pre-Phase-1 files, and it has no peer for those rows to be invisible to. Known residual, documented in-file and in the topology guide: a database file first created by an older build keeps its stale __localdb_* triggers. The guard decides whether triggers are installed, not whether existing ones are removed, and the library has no removal API until WP3.3. Moot on the docker rig, where a schema-change redeploy recreates the volumes. The inverse is also now documented: enabling replication on a site that has run without it does not baseline existing rows, since CDC never recorded them in __localdb_row_version and the snapshot resync streams from that ledger. Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and absence via sqlite_master across all four config shapes (none, ApiKey only, PeerAddress + ApiKey, and the notification-table exclusion), plus DDL-still-runs and migrator-still-runs on the unreplicated branch. SiteLocalDbWiringTests and the integration site-pair harness now configure an ApiKey — mirroring the rig's passive node — so their registration and convergence assertions still describe a replicating node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests still pass. |
||
|
Joseph Doherty
|
697be0ce43 | docs(topology): fix stale keep-oldest reference in Site Cluster Behavior — clusters run auto-down since 2026-07-21 | ||
|
Joseph Doherty
|
7fd5cb2b56 |
feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath. Deleted: - AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests) - ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it) - ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in AkkaHostedService; the RegisterCentralClient message + receive block - CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport coexistence flags; the CentralTransportMode/SiteTransportKind enums gRPC is now the only site↔central transport (site→central CentralControlService via GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default so TestKit command-dispatch suites still construct the site actor without a wired transport; production always injects GrpcCentralTransport. Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator rejects blank entries (role-agnostic), and StartupValidator requires a Site node to list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default, deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used). Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted the audit-push integration relay to an in-process bridge transport. Docs: Component-Communication/Host/StoreAndForward, components/Communication, topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired amendment), and CLAUDE.md transport decisions. Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles. |
||
|
Joseph Doherty
|
2ee84af1c0 |
feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
|
||
|
Joseph Doherty
|
4a6341d871 |
feat(cluster): self-first seed ordering closes the boot-alone outage gap
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting central-b never came Up (the "registered outage gap"), and self-first ordering closes it using Akka's own protocol. - 6 node appsettings swapped (the *-node-b configs; the -a nodes were already self-first). All 14 shipped node configs now satisfy the invariant. - StartupValidator enforces it at boot, comparing host AND port -- the invariant fails silently when broken, so it is enforced loudly. NOTE: the gitignored deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or that node will refuse to boot. - SelfFirstSeedBootstrapTests: real in-process clusters at production failure-detection timings, incl. a falsifiability control proving the OLD peer-first ordering never forms. Rejected alternative (implemented, measured, discarded): an external self-form timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's join handshake and so cannot tell "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the node's own stale incarnation; a Join(self) during TryingToJoin abandons the in-flight join and forms a second cluster at the same address -- still split after 90s. Docs that claimed self-first ordering was unsafe for simultaneous cold start are corrected: while mutually reachable the InitJoin handshake converges them to one cluster (measured). |
||
|
Joseph Doherty
|
7b5a5a6f34 |
docs(localdb): phase 2 truth pass across both repos
Normative first: Component-StoreAndForward.md:83 specified the whole chunked, ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It is rewritten rather than removed — the failure modes it reasoned about still exist, they are just bounded differently — and it now states the duplicate-delivery bound explicitly: a message can be delivered twice only when the OLD primary delivered it and the status change had not yet replicated when the gate flipped. One flush interval plus the in-flight ack, and unlike the old model it does NOT grow with backlog depth or with how long a node was absent. The N1 directional-authority and N5 orphan-row hazards are recorded as structurally gone, not merely unguarded. Frame-size known-issue amended: its 2026-06-26 resolution replaced the intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch itself, so the 128 KB Akka frame constraint no longer applies to that hop in any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which batches by ROW COUNT), including that the failure mode differs — an oversized gRPC message is rejected, not silently dropped. Deployment docs gain the two operational constraints that have no home in code: a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat handler that made a mixed-version pair converge went with the replicator, and a mixed pair now diverges silently), and a node offline beyond TombstoneRetention can resurrect deleted rows on rejoin. Both CLAUDE.md files corrected — each still said Phase 2 was NOT started. Definition of done closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures, 0 skips). Two DoD items needed amending rather than ticking: the stale-symbol grep still matches 4 lines, all deliberate comment prose recording what was deleted (a literal zero would delete the explanations that stop the old design coming back), and the live gate has 10 evidence items, not the 9 the checklist claimed. Deletes the phase2 resume-state scratch doc, which said to delete it on landing. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
43228185b4 |
docs: convert standard diagrams from draw.io PNGs to inline Mermaid
Gitea renders mermaid inline, so the flow/state/hierarchy/DAG diagrams move to text-in-markdown: auto-layout (removes the manual overlap-prone draw.io step), diffable source, no committed binaries, and a dark-text theme so labels stay legible. Keep draw.io PNGs only for the two complex bespoke diagrams (logical architecture, env2 topology) where pixel control still wins. All 24 mermaid blocks validated by rendering. |
||
|
Joseph Doherty
|
bdee12f4e9 |
docs: render architecture & flow diagrams as draw.io charts
Replace ASCII-art diagrams across the README and docs/ with editable .drawio sources plus exported PNGs, so the diagrams render clearly in rendered markdown and can be maintained/regenerated instead of being hand-edited as fragile text art. Non-diagram blocks (code, folder trees, UI wireframes) were left as text. |
||
|
Joseph Doherty
|
7b0b9c7365 |
refactor: rename ScadaLink → ZB.MOM.WW.ScadaBridge (code + projects + namespaces)
Solution + 23 src projects + 26 test projects renamed; folders, csproj, namespaces, and ScadaLinkDbContext/ScadaBridgeDbContext class updated. ActorSystem "scadalink" → "scadabridge", Akka seed-node URLs migrated. SQL roles/logins, LDAP domains, CLI command name, and CLI config dir (~/.scadalink → ~/.scadabridge) also renamed. Build green; 5 Host.Tests fail awaiting SQL login rename in next commit. Pre-existing StaleTagMonitor timing flakes unchanged. Rename script committed at tools/rename-to-scadabridge.sh. |
||
|
Joseph Doherty
|
b659978764 |
Phase 8: Production readiness — failover tests, security hardening, sandboxing, deployment docs
- WP-1-3: Central/site failover + dual-node recovery tests (17 tests) - WP-4: Performance testing framework for target scale (7 tests) - WP-5: Security hardening (LDAPS, JWT key length, no secrets in logs) (11 tests) - WP-6: Script sandboxing adversarial tests (28 tests, all forbidden APIs) - WP-7: Recovery drill test scaffolds (5 tests) - WP-8: Observability validation (structured logs, correlation IDs, metrics) (6 tests) - WP-9: Message contract compatibility (forward/backward compat) (18 tests) - WP-10: Deployment packaging (installation guide, production checklist, topology) - WP-11: Operational runbooks (failover, troubleshooting, maintenance) 92 new tests, all passing. Zero warnings. |