Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.
- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
SiteCommunicationActor cannot reference Host, and the two definitions must not
drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
site singletons are placed on the former, so the base role would move the wrong
node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
never fail over a site the operator did not select), refuses when there is no
peer, and reports a fault as an ack rather than throwing into supervision --
a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
confirmation deliberately does NOT claim the admin's page will disconnect --
it won't, and crying wolf there devalues the central warning that is real. A
site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
dead-letters and the Ask times out, reported as "site did not respond". That
is the honest outcome; documented on the contract.
Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
Extend CommunicationOptionsValidator with eager bounds for the four T4
live-alarm-cache options (linger >= 0, reconcile > 0, seed concurrency 1..64,
subscribers-per-site >= 1). Enforce the per-site viewer cap fail-safe in
SiteAlarmLiveCacheService.Subscribe (reject excess viewers with a no-op
disposable rather than growing the list or throwing into the Blazor render
path). Surface two telemetry instruments on the existing ScadaBridgeTelemetry
meter: an active-aggregator observable gauge and a reconnect counter, wired from
the aggregator actor's PreStart/PostStop and its NodeA<->NodeB flip /
reconcile-driven reopen.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Adds the active-central-node, in-memory, reference-counted per-site live alarm
cache backing the operator Alarm Summary page. No persisted central alarm store
([PERM]) — no EF entity/table/migration/DbSet.
- ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) ->
IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive.
- SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied
from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first,
buffer live deltas while the snapshot fan-out runs, flush with per-key dedup
(AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through
into an in-memory dict. Placeholder rows seeded from the snapshot and never
expected on the stream; a real-alarm delta (distinct key) never wipes them.
NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds,
periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift,
and a budget-exhausted stream self-heals on the next reconcile tick.
- SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting,
linger-delayed last-out stop (version + TryRemove(ref) race guards), the
snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped),
bounded start-retry self-heal on transient start failure, and the immutable
published-snapshot store the page reads. Cache mutated only on the actor thread;
viewer callbacks invoked outside the lock.
- CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval
(60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite
(200). Task 6 formalizes eager validation + telemetry.
- DI registration + AkkaHostedService SetActorSystem wiring on the active central node.
- Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder
coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service
(shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown
site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical;
both Important findings addressed.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Add SiteStreamGrpcClient.SubscribeSiteAsync mirroring SubscribeAsync but for
the site-wide, alarm-only SubscribeSite RPC: builds a SiteStreamRequest, opens
the server stream, and delivers each event via a typed Action<AlarmStateChanged>
callback (this stream is alarm-only by contract, so Task 4's per-site cache
consumes an alarm delta with no downstream type test). Reuses the shared
enrichment mapping via a new internal ConvertToAlarmEvent helper that returns
null for any non-alarm event, defensively filtering anything that should never
appear on the stream. Factory unchanged - it already caches a client per
(site, endpoint). Adds focused unit tests for the alarm-only filter and the
test-only-client guard.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Adds an additive, site-wide, alarm-only gRPC stream backing the aggregated
Alarm Summary. Proto: new SiteStreamRequest { correlation_id } message + rpc
SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent) on
SiteStreamService — purely additive, no field renumbering. Regenerated the
checked-in SiteStreamGrpc/*.cs.
Server: SubscribeInstance and the new SubscribeSite now delegate to a shared
RunSubscriptionStreamAsync helper (readiness/shutdown guards, correlation-id
validation, duplicate replacement, concurrency cap, bounded DropOldest channel,
relay actor, SiteConnectionOpened/Closed telemetry, guaranteed cleanup). The
only variation is the subscribe delegate: SubscribeSite calls
ISiteStreamSubscriber.SubscribeSiteAlarms (no per-instance filter). Added
SubscribeSiteAlarms to the ISiteStreamSubscriber contract (SiteStreamManager
already implements it from T1). StreamRelayActor reused unchanged — it already
drops IsConfiguredPlaceholder rows and maps the enriched AlarmStateUpdate.
Tests: SubscribeSite subscribes site alarms + removes on cancel, rejects unsafe
correlation ids, and relays a domain AlarmStateChanged as a proto
AlarmStateUpdate on the stream.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Site side: additive proto after_id field, ReadChangedSinceAsync gains an optional
afterId cursor and deterministic (UpdatedAtUtc, TrackedOperationId) ordering — a
batch fully inside one UpdatedAtUtc instant no longer re-reads the same page
forever. Absent/empty afterId preserves the exact legacy inclusive >= contract, so
an older central is unaffected. Regenerated the checked-in gRPC C#.
Notify-and-fetch follow-ups:
- PendingDeploymentPurgeActor: a central cluster singleton (not
readiness-gated, best-effort) that sweeps expired PendingDeployment
staging rows on CommunicationOptions.PendingDeploymentPurgeInterval
(default 1h). Modeled on the kpi-history-recorder pattern: self-scheduling
timer, per-tick DI scope -> IDeploymentManagerRepository, continue-on-error.
Wired in AkkaHostedService.RegisterCentralActors (manager + proxy + drain);
resolves the deferred TODO in DeploymentService. Correctness never depends
on it (supersession bounds rows to <=1/instance; the fetch endpoint enforces
the TTL), so it is deliberately absent from RequiredSingletonsHealthCheck.
- SQL Server integration test for StagePendingIfAbsentAsync re-staging an
instance's OWN DeploymentId over an expired row against the real UNIQUE
index on DeploymentId — confirms EF orders DELETE before INSERT in one
SaveChanges (SQLite's constraint timing differs from SQL Server's). Plus
a same-instance supersession variant on real SQL Server.
Tests: 2 TestKit actor tests + 2 SQL Server integration tests (both ran
green against the infra MSSQL container); 235 Communication + 15
PendingDeployment tests pass; Host builds 0 warnings.
Add CommunicationService.RefreshDeploymentAsync — the typed send method
for the small notify-and-fetch wire message (RefreshDeploymentCommand).
Mirrors DeployInstanceAsync exactly: SiteEnvelope + Ask<DeploymentStatusResponse>
bounded by DeploymentTimeout. CentralCommunicationActor needs no change
(HandleSiteEnvelope is fully generic — all SiteEnvelope messages forward
to /user/site-communication without a per-type switch). Adds a parallel
routing test asserting the envelope reaches the site ClusterClient.
Placeholder AlarmStateChanged rows are a DebugView snapshot-only concept emitted
by InstanceActor.BuildAlarmStatesSnapshot; they are never a real alarm transition.
Their timestamp may be DateTimeOffset.MinValue (the Protobuf Timestamp lower boundary),
which can throw when packed via Timestamp.FromDateTimeOffset.
Added early-return guard at the top of HandleAlarmStateChanged before any timestamp
pack or channel write. Updated the existing NativeBindingLinkage round-trip test to
use a real (non-placeholder) native alarm; added DropsAlarmStateChanged_WhenIsConfiguredPlaceholder
to assert placeholders are silently dropped (15/15 pass).
Add two additive init-only fields to AlarmStateChanged so the Debug View can
nest live native conditions under their configured source-binding node:
- NativeSourceCanonicalName (binding canonical name, e.g. "Motor1.MotorAlarms")
- IsConfiguredPlaceholder (quiet-binding placeholder flag; default false)
Flow on BOTH cross-process paths:
- Live: proto AlarmStateUpdate fields 22/23 -> StreamRelayActor packs ->
SiteStreamGrpcClient unpacks (regenerated SiteStreamGrpc/Sitestream.cs).
- Snapshot (Newtonsoft): record defaults carry through; no special handling.
NativeAlarmActor.Emit now stamps NativeSourceCanonicalName = _source.CanonicalName.
Additive-only: no existing positional constructor or wire frame changed.
Tests: StreamRelayActorTests round-trips both fields pack->unpack;
NativeAlarmActorTests asserts the emitted event carries the binding canonical name.
Replace ValueFormatter.FormatDisplayValue with AttributeValueCodec.Encode
in StreamRelayActor so List<T> attribute values cross the gRPC wire as a
JSON array (e.g. ["a","b"]) rather than a comma-joined display string.
Scalars and null values are unaffected. Tests cover List→JSON, scalar
string pass-through, and null→empty-string.
- MockSiteStreamGrpcClient.SubscribeCalls and UnsubscribedCorrelationIds
switched from bare List<T> to lock-guarded backing fields with snapshot
accessors, eliminating the actor-thread/test-thread data race (matches
the existing lock(events) pattern for ReceivedEvents)
- AttributeKey and AlarmKey null-guard each component with ?? string.Empty
so a null SourceReference/AlarmName/etc. cannot silently collide with an
empty-string component in the dedup dictionary
- On_Snapshot_Opens_GrpcStream renamed to
On_Snapshot_Does_Not_Open_Additional_GrpcStream; assertion updated to
confirm exactly one subscribe (the PreStart stream-first open) with no
second subscribe after snapshot delivery
- _stopped ordering in InstanceNotFound path moved after CleanupGrpc()
for consistency with DebugStreamTerminated and ReceiveTimeout handlers
Re-architect DebugStreamBridgeActor from snapshot-first to stream-first so no
attribute/alarm event occurring during the snapshot-build + network-transit
window is lost (#26).
Lifecycle change:
- PreStart now opens the gRPC subscription FIRST (alongside sending the
SubscribeDebugViewRequest), so live events start flowing immediately.
- Phase model via a single _snapshotDelivered flag (mutated only on the actor
thread). While buffering (snapshot not yet delivered), AttributeValueChanged/
AlarmStateChanged are appended to an ordered _preSnapshotBuffer instead of
being delivered. After snapshot+flush, the same handlers pass through directly.
- On DebugViewSnapshot: deliver snapshot, then flush the buffer in arrival order
with per-entity dedup, then set _snapshotDelivered=true (pass-through).
Dedup rule (exactly-once):
- Identity: attributes by (InstanceUniqueName, AttributePath, AttributeName);
alarms by (InstanceUniqueName, AlarmName, SourceReference) so native
per-condition alarms are not conflated. Keys joined with a NUL delimiter
(declared as an escaped char constant; no raw NUL in source) so distinct
identities never collide on a space within a name.
- Boundary: a buffered event whose timestamp is <= the snapshot's timestamp for
the same entity is already reflected -> DROP; strictly-newer (>) -> DELIVER;
entity absent from the snapshot -> DELIVER (genuine gap-window event).
Preserved paths:
- M2.11 InstanceNotFound: with stream-first the gRPC stream is already open, so
the not-found path now tears it down (CleanupGrpc) + clears the buffer, does
NOT enter pass-through, delivers the not-found snapshot, and stops cleanly.
- Reconnect (ReconnectGrpcStream -> OpenGrpcStream) does not touch the phase
flag: a mid-session reconnect resumes pass-through; a reconnect during the
buffering phase stays buffering until the snapshot arrives.
- Communication-008 retry/stability/stop/terminate + ReceiveTimeout orphan net
unchanged. Duplicate/late snapshot after delivery is ignored defensively.
Tests: 10 new M2.18 tests (stream-first ordering, gap-window buffering, dedup
drop/deliver for attrs + alarms, ordering, pass-through, InstanceNotFound
teardown, reconnect-during-buffering, reconnect-after-snapshot) + revised the
M2.11 not-found test to assert stream teardown. Full DebugStreamBridgeActor
class green: 23/23.
- Add DebugStreamBridgeActorTests: On_InstanceNotFound_Snapshot_Forwards_To_OnEvent_Does_Not_Open_Stream_And_Terminates — asserts _onEvent receives the not-found snapshot, SubscribeCalls remains empty, and the actor terminates cleanly via Watch/ExpectTerminated.
- Add comment in DebugStreamBridgeActor near Context.Stop(Self) explaining that the subsequent StopDebugStream Tell from DebugStreamService.StopStream produces a benign expected dead-letter.
- Reword not-found toast in DebugView.razor to "Instance not found on the selected site — check the deployment target." (accurate when the instance may be deployed to a different site).
Site Call Audit (#22): build the documented periodic reconciliation PULL
self-heal path for the eventually-consistent central SiteCalls mirror, as a
dedicated PullSiteCalls gRPC RPC kept separate from the audit pull. This is the
pull PLUMBING only; the central reconciliation tick is a separate follow-up.
- IOperationTrackingStore.ReadChangedSinceAsync(sinceUtc, batchSize): inclusive
UpdatedAtUtc cursor, oldest-first, batch-capped; SQLite impl projects tracking
rows onto SiteCallOperational (Kind->Channel, TargetSummary->Target, SourceSite
left empty - the store has no site-id column).
- sitestream.proto: rpc PullSiteCalls + PullSiteCallsRequest/Response, mirroring
PullAuditEvents; regenerated checked-in SiteStreamGrpc/*.cs.
- SiteCallDtoMapper.ToDto(SiteCallOperational): inverse of FromDto for the handler.
- SiteStreamGrpcServer.PullSiteCalls handler + SetOperationTrackingStore seam;
Host wires the seam alongside SetSiteAuditQueue (site roles only).
- Central IPullSiteCallsClient + GrpcPullSiteCallsClient (home: AuditLog/Central to
reuse ISiteEnumerator; SiteCallAudit does not reference AuditLog). Re-stamps
SourceSite from the dialed siteId; no-throw on tolerable transport faults;
SpecifyKind (not ToUniversalTime) cursor handling. Central-only DI registration.
Tests: ReadChangedSinceAsync (4), PullSiteCalls handler (6), GrpcPullSiteCallsClient
(8). Full solution build 0 warnings/0 errors (TreatWarningsAsErrors).
Joseph Doherty