Adds the active-central-node, in-memory, reference-counted per-site live alarm
cache backing the operator Alarm Summary page. No persisted central alarm store
([PERM]) — no EF entity/table/migration/DbSet.
- ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) ->
IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive.
- SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied
from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first,
buffer live deltas while the snapshot fan-out runs, flush with per-key dedup
(AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through
into an in-memory dict. Placeholder rows seeded from the snapshot and never
expected on the stream; a real-alarm delta (distinct key) never wipes them.
NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds,
periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift,
and a budget-exhausted stream self-heals on the next reconcile tick.
- SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting,
linger-delayed last-out stop (version + TryRemove(ref) race guards), the
snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped),
bounded start-retry self-heal on transient start failure, and the immutable
published-snapshot store the page reads. Cache mutated only on the actor thread;
viewer callbacks invoked outside the lock.
- CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval
(60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite
(200). Task 6 formalizes eager validation + telemetry.
- DI registration + AkkaHostedService SetActorSystem wiring on the active central node.
- Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder
coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service
(shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown
site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical;
both Important findings addressed.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
WhenTerminated watchdog calls IHostApplicationLifetime.StopApplication() when
the ActorSystem dies outside StopAsync (SBR self-down + run-coordinated-shutdown-when-down),
so the service supervisor restarts the node as a fresh incarnation. Optional
ctor param keeps every existing construction site compiling.
Plan's deploy/wonder-app-vd03/install.ps1 service-recovery-actions edit is
skipped: that production deploy artifact is not tracked in this repo (Task 16
skipped its appsettings.Central.json for the same reason).
Remove project bookkeeping citations from shipped code comments across the
solution: hyphenated task IDs (WP-14, StoreAndForward-025), milestone/task/
issue refs (M3, Task 4, Audit Log #23, #21), Bundle X task-bundle labels,
and C/D/K/S/T phase labels.
Comment text only — no code logic, string/log literals, or XML-doc structure
changed. Genuine descriptions are preserved (only the citation is stripped),
and technical lookalikes are retained (UTF-8, SHA-256, T00:00:00, M365,
UTC-5, pre-C4/pre-C5 schema versions). Flagged by the new CommentChecker
TaskReferenceInComment / TrackingReferenceInComment checks plus targeted
grep passes; full solution builds clean, append-only guard tests pass.
Notify-and-fetch follow-ups:
- PendingDeploymentPurgeActor: a central cluster singleton (not
readiness-gated, best-effort) that sweeps expired PendingDeployment
staging rows on CommunicationOptions.PendingDeploymentPurgeInterval
(default 1h). Modeled on the kpi-history-recorder pattern: self-scheduling
timer, per-tick DI scope -> IDeploymentManagerRepository, continue-on-error.
Wired in AkkaHostedService.RegisterCentralActors (manager + proxy + drain);
resolves the deferred TODO in DeploymentService. Correctness never depends
on it (supersession bounds rows to <=1/instance; the fetch endpoint enforces
the TTL), so it is deliberately absent from RequiredSingletonsHealthCheck.
- SQL Server integration test for StagePendingIfAbsentAsync re-staging an
instance's OWN DeploymentId over an expired row against the real UNIQUE
index on DeploymentId — confirms EF orders DELETE before INSERT in one
SaveChanges (SQLite's constraint timing differs from SQL Server's). Plus
a same-instance supersession variant on real SQL Server.
Tests: 2 TestKit actor tests + 2 SQL Server integration tests (both ran
green against the infra MSSQL container); 235 Communication + 15
PendingDeployment tests pass; Host builds 0 warnings.
Wire the M6 KPI History recorder into the central composition path:
- Program.cs: call services.AddKpiHistory(configuration) on the central-only
branch alongside AddNotificationOutbox/AddAuditLog/AddSiteCallAudit.
- AkkaHostedService.cs: register KpiHistoryRecorderActor as a central,
non-role-scoped ClusterSingletonManager + ClusterSingletonProxy + a
PhaseClusterLeave CoordinatedShutdown graceful-stop drain (singleton name
'kpi-history-recorder'), copied/adapted from the audit-log-purge block.
- appsettings.Central.json (Host + docker + docker-env2 central nodes): add a
ScadaBridge:KpiHistory section (SampleInterval 00:01:00, RetentionDays 90,
PurgeInterval 1.00:00:00, DefaultMaxSeriesPoints 200).
KPI history is observability/best-effort and MUST NOT gate readiness: the
recorder is deliberately NOT added to RequiredSingletonsHealthCheck or any
other readiness gate.
Site Call Audit (#22): build the documented periodic reconciliation PULL
self-heal path for the eventually-consistent central SiteCalls mirror, as a
dedicated PullSiteCalls gRPC RPC kept separate from the audit pull. This is the
pull PLUMBING only; the central reconciliation tick is a separate follow-up.
- IOperationTrackingStore.ReadChangedSinceAsync(sinceUtc, batchSize): inclusive
UpdatedAtUtc cursor, oldest-first, batch-capped; SQLite impl projects tracking
rows onto SiteCallOperational (Kind->Channel, TargetSummary->Target, SourceSite
left empty - the store has no site-id column).
- sitestream.proto: rpc PullSiteCalls + PullSiteCallsRequest/Response, mirroring
PullAuditEvents; regenerated checked-in SiteStreamGrpc/*.cs.
- SiteCallDtoMapper.ToDto(SiteCallOperational): inverse of FromDto for the handler.
- SiteStreamGrpcServer.PullSiteCalls handler + SetOperationTrackingStore seam;
Host wires the seam alongside SetSiteAuditQueue (site roles only).
- Central IPullSiteCallsClient + GrpcPullSiteCallsClient (home: AuditLog/Central to
reuse ISiteEnumerator; SiteCallAudit does not reference AuditLog). Re-stamps
SourceSite from the dialed siteId; no-throw on tolerable transport faults;
SpecifyKind (not ToUniversalTime) cursor handling. Central-only DI registration.
Tests: ReadChangedSinceAsync (4), PullSiteCalls handler (6), GrpcPullSiteCallsClient
(8). Full solution build 0 warnings/0 errors (TreatWarningsAsErrors).
Per-probe health-check child scopes were disposing the AddTransient-bridged
ActorSystem (IDisposable), terminating the live cluster node ~4s after boot and
leaving every singleton-proxy Ask to hang the full 30s QueryTimeout — the central
report pages (/notifications, /site-calls, /monitoring/health) loaded in ~30s.
Bridge it as a singleton via a new lazy AkkaHostedService.GetOrCreateActorSystem()
so child-scope disposal never touches it. Verified: 0 post-startup terminates,
healthy active/standby, report pages ~0.05s, Playwright 68 passed / 0 failed.
Joseph Doherty