Commit Graph

11 Commits

Author SHA1 Message Date
Joseph Doherty 9d2834e30a docs+log(siteeventlogging): explain the site_events purge oplog-backlog burst (R7)
The daily site_events retention purge (and the storage-cap trim) is CDC-captured
on a replication-enabled site node exactly like any other write — correct by
design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the
backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog /
localdb_oplog_depth spike, drain, and an operator watching the gauge with no
context reads it as a replication fault.

Documentation + one log line, no behaviour change:

- topology-guide.md gains "Reading the replication backlog — the daily
  site_events purge burst": when it fires (PurgeInterval 24h, anchored to the
  active node's PROCESS START, not a wall-clock hour, so it moves after every
  failover), where it shows (replicated nodes only — not rig site-b/site-c),
  the healthy signature (LocalDbReplicationConnected stays true, backlog
  returns to ~0) and what a genuine fault looks like instead.
- Component-SiteEventLogging.md Storage records the same under retention/purge;
  Component-HealthMonitoring.md gains the two previously-undocumented
  LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the
  caveat, with cross-references both ways.
- EventLogPurgeService emits one Information line naming the row count and the
  expected transient backlog when a purge deleted rows on a replication-enabled
  node, so the spike is correlatable in the log. Replication-awareness comes in
  as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the
  existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured
  goes internal so the PeerAddress-OR-ApiKey rule stays in one place and
  SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note,
  matching the default that replication is opt-in and off.

Both delete paths carry the note (a cap trim is usually the larger burst); the
predicate is try/caught since a log-wording check must never break the purge.

Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated
does not, zero-rows does not, cap purge logs it, throwing predicate still purges
and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green,
Host 490/490 green, full solution build clean (0 warnings).
2026-08-15 03:26:30 -04:00
Joseph Doherty 11efe4be05 fix(health): flag metrics-stale for never-reported sites via FirstSeenAt anchor — null LastReportReceivedAt no longer skips the check (plan R2-01 T8) 2026-07-13 10:45:22 -04:00
Joseph Doherty c73b7faa11 feat(health): metrics-stale signal + status-transition timestamps; spec now matches heartbeat-liveness code 2026-07-08 16:21:30 -04:00
Joseph Doherty fd618cf1dc fix(review): full code-review remediation — 5 High + Medium/Low across 16 modules
Remediation from the full per-module code review at 4307c381 (findings recorded
separately in code-reviews/).

Highs fixed:
- DeploymentManager-025/SiteRuntime-031: stop broadcasting notification lists + SMTP
  configs (incl. credentials) to sites; site purges already-persisted rows on apply
  (enforces the central-only delivery design; clears plaintext SMTP creds at rest).
- DataConnectionLayer-023: guard the native-alarm subscribe path against the
  mid-flight-unsubscribe adapter-feed leak (mirrors the DCL-021 tag-path fix).
- SiteEventLogging-024: normalize From/To query bounds to UTC (the -016 fix the
  audit trail claimed but never committed).
- KpiHistory-001: add an in-flight guard to the recorder sample tick.
- ScriptAnalysis-001: harden the trust analyzer's TPA-absent fallback (resolve
  forbidden anchors in the minimal reference set; warn on degraded mode) — anchors
  added to validation references only, never the compile gate.
(InboundAPI-026 left to the feat/ipsen-movein effort per owner decision.)

Medium/Low: DM-026 deterministic deploy-status tiebreaker; SR-027/028/029/030
native-alarm leak/phantom-active/delete-during-redeploy fixes; AL-013/014/016;
TE-024 (folder-mutation audit rows now persisted)/025; SF-025 gauge-provider
clear-on-stop; ESG-025/026; SEC-023/024/025; SCA-007/008/009; plus doc/test
accuracy COM-023/024, HOST-025/026, HM-024/025, NS-027/028.

Full-solution build 0 warnings; ~3560 tests across 18 touched suites green.
2026-06-20 17:55:12 -04:00
Joseph Doherty 241a792e7b docs(kpi): K17 — #26 KpiHistory component doc + README/CLAUDE + cross-component interactions + completion-design update 2026-06-17 20:52:12 -04:00
Joseph Doherty 487859bff0 docs+code: close Theme 1 — 24 design-doc / XML-doc drift findings
Doc/XML-comment drift + small adherence fixes across 17 modules. Highlights:
- Host-017: site CoordinatedShutdown ordering — SiteStreamGrpcServer gains
  CancelAllStreams() (refuse new streams, cancel active), wired into
  Program.cs site branch via ApplicationStopping.
- InboundAPI-021: ParentExecutionId now travels on RouteToGet/SetAttributes
  symmetric with RouteToCallRequest; RouteHelper stamps from _parentExecutionId.
- ClusterInfra-012: ClusterOptionsValidator now requires both seed nodes.
- Comm-018: SiteCommunicationActor.HeartbeatMessage.IsActive derived from
  cluster leader check (was hardcoded true).
- DM-020: reconciliation audit row attributes the current user, not prior deployer.
- SEL-019: EventLogPurgeService early-exits on standby via active-node check.
- Plus comment/XML-doc accuracy fixes across AuditLog, ConfigurationDatabase,
  NotificationOutbox, SiteRuntime, SiteCallAudit; doc refreshes for Component-
  Commons / -ManagementService / -CLI / -ExternalSystemGateway / -HealthMonitoring
  / -Transport / -ConfigurationDatabase; CD-023 index-name doc alignment.

11 new regression tests (RouteHelper x4, SiteStreamGrpcServer x2,
ClusterOptionsValidator x1, SiteCommunicationActor x1, DeploymentService x1,
EventLogPurgeService x3). Build clean (0 warnings); InboundAPI/Communication/
Host suites all green. README regenerated: 112 open (was 136).
2026-05-28 06:28:31 -04:00
Joseph Doherty 72388a7616 docs(audit): fix Audit error rate semantics and CLI permission split
Bundle D code-review feedback on 0ae1a25 and e6f7a7f:

- Audit error rate (HealthMonitoring tile) was described as a combined
  view of CentralAuditWriteFailures + AuditRedactionFailure (writer
  health). Per alog.md §10.3 / §14.1 it is the operational error rate
  of audited operations: % of central AuditLog rows with Status not
  in (Success/Delivered/Enqueued) over a rolling 5-min window. Audit
  writer issues surface separately via the dedicated metrics.

- Audit volume description gains the spec-mandated 'events/min, global
  + per-site sparkline' shape.

- CLI: scadalink audit was claiming all three subcommands need both
  OperationalAudit and AuditExport. Per alog.md §11.2 / §15.1, read
  (query, verify-chain) needs OperationalAudit; bulk export
  additionally requires AuditExport. Restored the spec's split.
2026-05-20 08:30:42 -04:00
Joseph Doherty 0ae1a254d7 docs(audit): add Audit Log health metrics and dashboard tiles 2026-05-20 08:26:03 -04:00
Joseph Doherty e681a1f0e1 docs(requirements): add Site Call Audit KPIs to Health Monitoring 2026-05-19 11:58:46 -04:00
Joseph Doherty 6ccf3766dc docs(notification-outbox): add central-computed outbox KPIs to Health Monitoring 2026-05-18 23:17:07 -04:00
Joseph Doherty d91aa83665 refactor(docs): move requirements and test infra docs into docs/ subdirectories
Organize documentation by moving requirements (HighLevelReqs, Component-*,
lmxproxy_protocol) to docs/requirements/ and test infrastructure docs to
docs/test_infra/. Updates all cross-references in README, CLAUDE.md,
infra/README, component docs, and 23 plan files.
2026-03-21 01:11:35 -04:00