Commit Graph

7 Commits

Author SHA1 Message Date
Joseph Doherty 3169429c3f fix(site-event-log): queries on own read connection off the actor thread, bounded keyword window; cap trim counts/vacuums rarely (review 2026-10-21 02/P1, 02/P4) 2026-10-22 08:32:37 -04:00
Joseph Doherty 0ffd4bdbea docs(src): fix CommentChecker findings across src/ — 1,032 → 5 (all false positives)
Comment/XML-doc changes only, no code logic:
- strip project-tracking references (WP*, P*-*, row-NN, #NN, Task N) from
  comments, keeping the explanatory rationale
- add missing summaries, <param>, <returns>, <typeparam> and <inheritdoc/>
- replace misused <inheritdoc/> with real docs; where <inheritdoc/> was
  combined with useful remarks, move the rationale into // comments
- drop stale replicationActor <param> on DeploymentManagerActor ctor

Remaining 5 are false positives (CWE-1236 x2, failures-1, year-0001, M365).
Build 0 errors / 0 warnings.
2026-10-07 14:12:25 -04:00
Joseph Doherty ddb3c32e06 fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was
compared against page_count-freelist_count over the WHOLE consolidated LocalDb
file, which since Phase 1/2 holds ten tables plus __localdb_oplog /
__localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed
~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the
same file, so the measured size did not fall, the "size did not decrease" guard
fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present.
A node whose peer was down could not enforce its cap at all, and the event log's
budget was in practice shared with bookkeeping it does not own.

The cap now measures site_events AND ITS FOUR INDEXES only, so replication
bookkeeping can neither consume nor block the budget — and the cap stops being a
function of the peer's health, which was the wrong direction of coupling: an
inflated oplog is a replication fault, and letting it drive event deletion would
destroy operational history in response to one.

dbstat — the exact per-table measurement the register proposed — is NOT usable
here, verified before implementing. It is a compile-time option and the bundled
SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA
compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten
modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails
with "no such table". Switching SQLite builds was rejected — the provider is
shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy
for fleet-wide version skew. The footprint is therefore modelled from sampled row
width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted)
through a page-layout model; whole pages rather than a flat rate, because page
fill collapses once rows approach half a page and a linear model underestimates
that by ~40%. Clamped by whole-file logical size, a true upper bound that can only
correct an over-estimate and never binds on a replicated node.

Index pages count against the cap deliberately: measured at ~34% of the table's
footprint, and the 1 GB default was sized against a 393 B/row figure that already
included them.

Guard semantics restated — two backstops replace the single on-disk check:
(1) row width unsampleable => refuse to trim (checked before the cap comparison,
since a zero width makes "over cap" unanswerable rather than false; the old loop's
answer to an unanswerable cap was to keep deleting); (2) estimated size not
falling across a batch => stop. (2) is the old guard re-based onto the
table-scoped metric, so it can no longer be tripped by replication bookkeeping and
now means exactly one thing: events are arriving as fast as the trim removes them.
For that to be observable the loop re-reads the row count each batch, while
bytes-per-row is sampled once per tick and held fixed — re-sampling per batch
would jitter the estimate as the trim eats the head of the table and could stall
the loop through guard (2), the same self-defeating measurement as row 81 itself.

Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven
by the shared PurgeBudget. A tick where retention AND the cap trim both deleted
used to emit two identical Information notes; the tick total is also the more
useful number beside an oplog spike.

Measured model error: -0.8% / +8.4% / +12.8% across representative workloads,
biased high on purpose (an early cap forfeits budget; a low one fails at its only
job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row.

Fail-before: reverting the estimator to whole-file semantics fails 5 of the new
tests and reproduces the original signature exactly (remaining=2000
estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring
per-phase notes fails the row-84 test with "collection contained 2 matching items".

No config keys added or renamed, so no rig/deploy changes. Decision record:
docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md

Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
2026-08-17 03:58:41 -04:00
Joseph Doherty d9a35846d9 perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.

MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):

  393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
  Expiring all of them in ONE tick:  285 s, 2,731,947 oplog rows,
  284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
  i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
  the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
  snapshot of all ten replicated tables, bought by a routine retention purge.

  The resident backlog is produced-minus-drained, so a healthy peer blunts it:
  on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
  the problem, not the reassurance — the old safety margin was a function of
  peer throughput and vanished with the peer down. A bounded tick is safe at
  any peer speed.

  Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).

SHIPPED:
  - SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
    escape hatch, negatives rejected). One SHARED budget across the retention
    purge and the cap trim — two independently-capped phases would let a tick
    emit twice the intended burst. 4x margin under MaxOplogRows, which is also
    shared with every other replicated table's ordinary traffic.
  - SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
    <= PurgeInterval). Without it pacing would make retention converge in 11
    DAYS at the daily cadence; with it the worst case converges in ~11 ticks
    (~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
    the peer's drain and re-accumulate the backlog.
  - A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
    so an expired set landing exactly on the budget does not arm a no-op retry.
  - Visibility: rate-limited (15 min) Warning naming the ceiling and the
    accelerated interval, plus an unthrottled counter
    scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
    backlog converging from a site whose event rate outruns one paced tick.
  - The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
    now anchors to the END of the previous purge. Documented.

ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.

Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.

Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.

Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
2026-08-16 04:05:21 -04:00
Joseph Doherty 568735a41e docs(localdb): Phase 1 truth pass
Task 13.

SiteEventLogger's XML doc said 'Not replicated to standby. On failover, the new
active node starts a fresh log.' That is now false and was the exact behaviour
Phase 1 set out to fix.

OperationTrackingOptions.ConnectionString and SiteEventLogOptions.DatabasePath
marked MIGRATION-ONLY: nothing reads them but SiteLocalDbLegacyMigrator, and
they are safe to delete from a config once a node has migrated.

ScadaBridge CLAUDE.md gains a consolidated-site-database entry covering the
GUID/opaque-token contract change, the migration-only keys, the incidental
CWD-outside-the-volume data-loss fix, the fail-closed default-OFF replication
posture, and the fact that ZbTelemetryOptions.Meters is a silent allowlist.

scadaproj CLAUDE.md's LocalDb row goes from 'no app adoption yet' to the honest
current state, per the component-status-honesty convention: ScadaBridge Phase 1
only, on an UNMERGED branch, replication live-proven on the rig's site-a pair
ONLY, default-OFF everywhere else, Phase 2 not started, no other app adopted, no
production deployment replicating.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:51:55 -04:00
Joseph Doherty 9cff87fe85 docs(comments): strip internal task/milestone/bundle bookkeeping from code comments
Remove project bookkeeping citations from shipped code comments across the
solution: hyphenated task IDs (WP-14, StoreAndForward-025), milestone/task/
issue refs (M3, Task 4, Audit Log #23, #21), Bundle X task-bundle labels,
and C/D/K/S/T phase labels.

Comment text only — no code logic, string/log literals, or XML-doc structure
changed. Genuine descriptions are preserved (only the citation is stripped),
and technical lookalikes are retained (UTF-8, SHA-256, T00:00:00, M365,
UTC-5, pre-C4/pre-C5 schema versions). Flagged by the new CommentChecker
TaskReferenceInComment / TrackingReferenceInComment checks plus targeted
grep passes; full solution builds clean, append-only guard tests pass.
2026-07-07 11:03:26 -04:00
Joseph Doherty 7b0b9c7365 refactor: rename ScadaLink → ZB.MOM.WW.ScadaBridge (code + projects + namespaces)
Solution + 23 src projects + 26 test projects renamed; folders, csproj,
namespaces, and ScadaLinkDbContext/ScadaBridgeDbContext class updated.
ActorSystem "scadalink" → "scadabridge", Akka seed-node URLs migrated.
SQL roles/logins, LDAP domains, CLI command name, and CLI config dir
(~/.scadalink → ~/.scadabridge) also renamed.

Build green; 5 Host.Tests fail awaiting SQL login rename in next commit.
Pre-existing StaleTagMonitor timing flakes unchanged.

Rename script committed at tools/rename-to-scadabridge.sh.
2026-05-28 09:37:45 -04:00