Commit Graph

2083 Commits

Author SHA1 Message Date
Joseph Doherty 774e3918db merge(r2): PLAN-R2-07 UI/Management/Security 2026-07-13 11:08:10 -04:00
Joseph Doherty b7f5470666 merge(r2): PLAN-R2-01 Cluster/Host/Failover 2026-07-13 11:08:10 -04:00
Joseph Doherty 9b8ce0f51e merge(r2): PLAN-R2-02 Communication & S&F (Critical N1) 2026-07-13 11:08:10 -04:00
Joseph Doherty 2751a6dba9 test(management): freeze scrubber fragment coverage over DataConnection config types + document array-merge limit (plan R2-07 T12) 2026-07-13 11:05:19 -04:00
Joseph Doherty 1b88ca1296 fix(security): amortize LoginThrottle.Prune off the failed-bind hot path (plan R2-07 T11) 2026-07-13 11:03:26 -04:00
Joseph Doherty 5b16429635 fix(sitestream): deathwatch resets live-cache liveness on aggregator termination (plan R2-07 T10) 2026-07-13 11:01:54 -04:00
Joseph Doherty 56d39b5fce fix(host): site singletons (deployment-manager, event-log-handler) via SingletonRegistrar — PhaseClusterLeave drains for site SQLite writes (plan R2-01 T11) 2026-07-13 11:01:44 -04:00
Joseph Doherty 76ef97f729 fix(ui): disposal guard on Alarm Summary live callback, matching the DebugView pattern (plan R2-07 T9) 2026-07-13 10:58:50 -04:00
Joseph Doherty 925c869826 fix(ui): Alarm Summary poll defers row rebuilds to the live path while IsLive (plan R2-07 T8) 2026-07-13 10:57:09 -04:00
Joseph Doherty d33337a834 fix(ui): drop stale-site poll results on Alarm Summary site switch (plan R2-07 T7) 2026-07-13 10:55:25 -04:00
Joseph Doherty 1fcfa4fb16 fix(security): scope-filter secured-write listing + align spec with enforced site scoping (plan R2-07 T6) 2026-07-13 10:53:16 -04:00
Joseph Doherty 80e5b36852 refactor(host): CentralSingletonRegistrar -> SingletonRegistrar with optional role scope — unblocks site-singleton drains (plan R2-01 T10) 2026-07-13 10:49:49 -04:00
Joseph Doherty 92936747b7 fix(security): additive permitted-sites filter on secured-write query/count (plan R2-07 T5) 2026-07-13 10:48:45 -04:00
Joseph Doherty c2707fdcf8 docs(health): CentralHealthReportLoop comments say oldest-member, not cluster leader — stop re-teaching the S2 conflation (plan R2-01 T9) 2026-07-13 10:46:18 -04:00
Joseph Doherty 9b9222d223 fix(security): enforce LDAP-mapping site scope on secured-write submit/approve/reject (plan R2-07 T4) 2026-07-13 10:46:08 -04:00
Joseph Doherty 11efe4be05 fix(health): flag metrics-stale for never-reported sites via FirstSeenAt anchor — null LastReportReceivedAt no longer skips the check (plan R2-01 T8) 2026-07-13 10:45:22 -04:00
Joseph Doherty e4765b2eed docs(tracker): correct the wonder-app-vd03 deferral record — artifact is present (gitignored); overlay edits applied on-disk by R2-01 T5/T6 (plan R2-01 T7) 2026-07-13 10:42:38 -04:00
Joseph Doherty 24c2097ec3 fix(security): ship trusted-proxy ForwardedHeaders config for Traefik topologies + document lockout-DoS trade-off (plan R2-07 T3) 2026-07-13 10:42:37 -04:00
Joseph Doherty b532a1e30e fix(security): trusted-proxy ForwardedHeaders so LoginThrottle keys on the real client IP (plan R2-07 T2) 2026-07-13 10:38:41 -04:00
Joseph Doherty 65e0f770d1 test(failover): unskip FailoverTimingTests on the two-node rig at production timings — measured 33.7s vs ~25s design envelope (plan R2-01 T4; covers report-08 NF2) 2026-07-13 10:38:19 -04:00
Joseph Doherty e1a692016e fix(security): throttle DebugStreamHub LDAP bind via ManagementAuthenticator + correct coverage doc (plan R2-07 T1) 2026-07-13 10:36:32 -04:00
Joseph Doherty 30356803e7 docs(docker): mark failover-drill results table PENDING — live drill run (plan R2-01 T3) deferred to avoid disrupting the running docker cluster 2026-07-13 10:34:24 -04:00
Joseph Doherty d536450611 docs(cluster): per-direction failover truth — first-seed bootstrap constraint + operator recovery actions; drop the undeliverable ~25s active-crash promise (plan R2-01 T2) 2026-07-13 10:33:56 -04:00
Joseph Doherty badc97afe8 fix(docker): failover drill kills the STANDBY by default; active mode measures the registered keep-oldest outage instead of pretending recovery (plan R2-01 T1) 2026-07-13 10:32:54 -04:00
Joseph Doherty c83b51fe9c fix(comm): single-gRPC-endpoint sites can go live (flip degenerates to same-node reconnect) (plan R2-02 T15) 2026-07-13 10:26:43 -04:00
Joseph Doherty 41ffe42de3 docs(comm): accept + document standby aggregators and deleted-site viewer behavior; fix aggregator e2e ctor call for new params (plan R2-02 T14) 2026-07-13 10:25:10 -04:00
Joseph Doherty 4e228110ec fix(comm): site delete disposes the site's cached gRPC channels via RemoveSiteAsync (plan R2-02 T13) 2026-07-13 10:22:56 -04:00
Joseph Doherty b7f8632db8 chore(comm): ctor-inject aggregator reconnect/stability tuning, remove process-global statics (plan R2-02 T12) 2026-07-13 10:19:51 -04:00
Joseph Doherty c29e588905 fix(comm): queue failover re-seed behind in-flight fan-out; stamp stream generation on gRPC errors (plan R2-02 T11) 2026-07-13 10:18:16 -04:00
Joseph Doherty bb8be55382 perf(comm): coalesce live-alarm delta publishes (250ms window, 0 = legacy) (plan R2-02 T10) 2026-07-13 10:14:54 -04:00
Joseph Doherty cc4ff7029d chore(saf): eagerly validate SweepBatchLimit/SweepTargetParallelism (plan R2-02 T9) 2026-07-13 10:10:34 -04:00
Joseph Doherty 61442b5f81 fix(saf): publish _sweepTask only on CAS win so StopAsync drains the real in-flight sweep (plan R2-02 T8) 2026-07-13 10:09:40 -04:00
Joseph Doherty 6e1ab62d27 feat(saf): resync ack confirmation + completed/ack-missing telemetry; doc resync rewrite (plan R2-02 T7) 2026-07-13 10:05:33 -04:00
Joseph Doherty 84bc2380f3 fix(saf): standby assembles chunked resync, applies atomically with re-check, acks; N5 race documented (plan R2-02 T6) 2026-07-13 10:00:47 -04:00
Joseph Doherty 3f937b399b fix(saf): chunk the resync snapshot to fit Akka remoting frames (additive protocol, byte-budgeted) (plan R2-02 T5) 2026-07-13 09:55:32 -04:00
Joseph Doherty 26dce8b69f fix(comm): heartbeat IsActive uses the shared oldest-Up predicate; Host wires one delegate everywhere (plan R2-02 T4) 2026-07-13 09:50:44 -04:00
Joseph Doherty 3a5b885a44 fix(saf): resync authority uses the shared oldest-Up predicate + delivery-gate delegate; apply-time re-check guard (plan R2-02 T3) 2026-07-13 09:48:37 -04:00
Joseph Doherty 6e0fa21307 test(saf): failing two-node repro — leader-vs-oldest resync divergence wipes delivering node buffer (plan R2-02 T2) 2026-07-13 09:46:16 -04:00
Joseph Doherty 91209070d0 fix(comm): shared oldest-Up ActiveNodeEvaluator in Communication; Host evaluator delegates (plan R2-02 T1) 2026-07-13 09:43:22 -04:00
Joseph Doherty 1429ddaa0e docs(plans): add MES alarm-status API draft plan (2026-06-30)
Ports the legacy WWSupport/APIServer MES alarm-status endpoints onto the
Inbound API, mirroring the MesMoveIn/MesMoveOut pattern; draft for review,
not yet executed. Committed ahead of PLAN-R2-08 T13's hygiene pass.
2026-07-13 01:45:50 -04:00
Joseph Doherty 5bbd7689fa docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)
Re-ran all 8 domain reviews at HEAD 8c888f13 against the b910f5eb baseline:
every round-1 finding source-verified (168 fixed, 0 regressions, 0 false
claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low),
concentrated in post-baseline code (anti-entropy resync, KPI rollup
backfill, live alarm stream) and seams the fixes exposed.

Headliners: S&F resync predicate inversion can wipe the delivering node's
buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame
size (02-N2); failover drill kills the one node keep-oldest can't survive
(01-N1); unbounded rollup backfill per failover (04-R1); live production
API key in untracked test.txt (08-NF1).

Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board,
P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
2026-07-12 23:52:10 -04:00
Joseph Doherty 8c888f13a0 merge(deferred-10): aggregated live alarm stream for Alarm Summary
Merges plan #10 (feat/live-alarm-stream) into main alongside plan #22.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 13:01:04 -04:00
Joseph Doherty 5078d25302 docs(sitestream): document aggregated live alarm stream + resolve deferred #10 (plan #10 T8)
Alarm Summary is now live-cache-driven via a transient per-site central
live alarm cache (ISiteAlarmLiveCache) fed by a site-wide, alarm-only
SubscribeSite gRPC stream (seed-then-stream), with the 15s poll retained
as fallback + NotReporting authority. No persisted central alarm store
([PERM]). Updated Component-CentralUI, Component-Communication (new §6.1
+ SubscribeSite RPC), M7 design (T13 delivered note), native-alarms
[PERM] note, stillpending follow-up note, CLAUDE.md, and moved register
row #10 to Resolved (left #22 in place for the sibling branch).

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:49:44 -04:00
Joseph Doherty a8278f65d0 test(sitestream): end-to-end trace for site-wide alarm stream → live cache (plan #10 T7)
Adds SiteAlarmStreamEndToEndTests wiring the whole feature pipe with only the
gRPC HTTP/2 transport mocked: real SiteStreamManager.SubscribeSiteAlarms → real
SiteStreamGrpcServer.SubscribeSite → real StreamRelayActor → proto → wire
round-trip (ToByteArray/ParseFrom) → real SiteStreamGrpcClient.ConvertToAlarmEvent
→ real SiteAlarmAggregatorActor cache.

Closes the cross-cutting gap the per-task T1–T6 unit tests leave (each mocks its
neighbour): proves AlarmKey identity (instance, name, sourceRef) + full native
enrichment survive every boundary mapping, attributes and placeholder rows are
dropped end-to-end, the single site-wide stream carries alarms for multiple
instances (no per-instance filter), and a snapshot-seed row and a live delta for
the same native alarm — both mapped through the real pipe — collapse onto one
cache row.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:44:11 -04:00
Joseph Doherty c4f97d0e87 feat(sitestream): validate live-alarm-cache options + active-aggregator/reconnect telemetry (plan #10 T6)
Extend CommunicationOptionsValidator with eager bounds for the four T4
live-alarm-cache options (linger >= 0, reconcile > 0, seed concurrency 1..64,
subscribers-per-site >= 1). Enforce the per-site viewer cap fail-safe in
SiteAlarmLiveCacheService.Subscribe (reject excess viewers with a no-op
disposable rather than growing the list or throwing into the Blazor render
path). Surface two telemetry instruments on the existing ScadaBridgeTelemetry
meter: an active-aggregator observable gauge and a reconnect counter, wired from
the aggregator actor's PreStart/PostStop and its NodeA<->NodeB flip /
reconcile-driven reopen.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:33:40 -04:00
Joseph Doherty b91ed3c840 feat(alarm-summary): live-cache-driven updates with poll fallback (plan #10 T5)
Wire the operator Alarm Summary page to the transient per-site live alarm
cache (ISiteAlarmLiveCache, T4). Live-cache-first: on site select the page
subscribes and rebuilds rows/rollup from near-real-time onChanged deltas; the
15s poll is kept untouched as the authority for NotReporting and as the
safety net whenever the cache is not live (pre-seed or degraded stream). Both
paths mutate shared state only via the Blazor dispatcher, so they never race,
and each rebuild is an idempotent snapshot.

- IAlarmSummaryService.BuildFromLiveAlarms: flattens a live AlarmStateChanged
  snapshot to AlarmSummaryRows with the same deterministic instance-then-name
  sort as GetSiteAlarmsAsync; NotReporting always empty on the live path.
- AlarmSummary.razor: inject ISiteAlarmLiveCache; subscribe on select,
  re-subscribe on site change, unsubscribe on leave + Dispose.
- Tests: service BuildFromLiveAlarms flatten/sort/empty; page subscribe,
  poll-fallback render, onChanged rebuild, unsubscribe on leave/change/dispose.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:25:36 -04:00
Joseph Doherty c9fa3b07f4 docs(kpihistory): document hourly rollups + resolve deferred #22 (plan #22 T8)
Update Component-KpiHistory.md with the KpiRollupHourly schema, the third
recorder rollup tick + one-shot backfill, dual raw/rollup purge retention,
raw-vs-rollup query routing by RollupThresholdHours, per-metric gauge-vs-rate
aggregation (KpiMetricAggregationCatalog), the four new options + bounds, and
30d/90d trend windows. Append a Delivered-2026-07-10 note to the m6 design
plan, move register row #22 from Deferred to Resolved, and note rollups in the
CLAUDE.md KPI History bullet. Docs-only; no code changed.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:18:46 -04:00
Joseph Doherty 696a4ffea2 feat(sitestream): central transient per-site live alarm cache w/ seed-then-stream + failover (plan #10 T4)
Adds the active-central-node, in-memory, reference-counted per-site live alarm
cache backing the operator Alarm Summary page. No persisted central alarm store
([PERM]) — no EF entity/table/migration/DbSet.

- ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) ->
  IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive.
- SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied
  from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first,
  buffer live deltas while the snapshot fan-out runs, flush with per-key dedup
  (AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through
  into an in-memory dict. Placeholder rows seeded from the snapshot and never
  expected on the stream; a real-alarm delta (distinct key) never wipes them.
  NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds,
  periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift,
  and a budget-exhausted stream self-heals on the next reconcile tick.
- SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting,
  linger-delayed last-out stop (version + TryRemove(ref) race guards), the
  snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped),
  bounded start-retry self-heal on transient start failure, and the immutable
  published-snapshot store the page reads. Cache mutated only on the actor thread;
  viewer callbacks invoked outside the lock.
- CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval
  (60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite
  (200). Task 6 formalizes eager validation + telemetry.
- DI registration + AkkaHostedService SetActorSystem wiring on the active central node.
- Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder
  coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service
  (shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown
  site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical;
  both Important findings addressed.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:18:02 -04:00
Joseph Doherty bbfc1b3278 feat(kpihistory): add 30d/90d trend windows to Audit/SiteCalls/Notifications/Health (plan #22 T7)
Adds 30d (720h) and 90d (2160h) toggle buttons alongside the existing
24h/7d controls on all four KPI trend surfaces. No fetch-logic changes:
the setters already recompute fromUtc and re-fetch, and the query service
(T5) transparently routes windows > RollupThresholdHours to hourly rollups
— so these windows are what actually exercise the rollup path. bUnit tests
extend the Audit and SiteCalls trend harnesses to cover the new buttons.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:12:17 -04:00
Joseph Doherty bedf3c55f0 feat(kpihistory): one-shot rollup backfill of existing samples on recorder start (plan #22 T6)
Fold the whole raw-retention window of already-recorded KpiSample rows into
KpiRollupHourly once shortly after the recorder singleton starts, so long-range
(30 d/90 d) trend charts aren't blank until enough wall-clock passes after
deploy. Reuses Task 3's idempotent FoldHourlyRollupsAsync over
[TruncateToHour(now) - RetentionDays, TruncateToHour(now)); runs at most once
per actor lifetime (_backfillDone latch) and defers past / blocks the periodic
lookback fold via a dedicated _backfillInFlight guard so the two idempotent
upserts never race on overlapping rows. Best-effort: no exception escapes; logs
start, window, and elapsed on completion. No Host or options change.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 12:07:47 -04:00