feat: complete gRPC streaming channel — site host, docker config, docs, integration tests
Switch site host to WebApplicationBuilder with Kestrel HTTP/2 gRPC server, add GrpcPort/keepalive config, wire SiteStreamManager as ISiteStreamSubscriber, expose gRPC ports in docker-compose, add site seed script, update all 10 requirement docs + CLAUDE.md + README.md for the new dual-transport architecture.
This commit is contained in:
@@ -43,7 +43,7 @@ This project contains design documentation for a distributed SCADA system built
|
||||
2. Deployment Manager — Central-side deployment pipeline, system-wide artifact deployment, instance lifecycle.
|
||||
3. Site Runtime — Site-side actor hierarchy (Deployment Manager singleton, Instance/Script/Alarm Actors), script compilation, Akka stream.
|
||||
4. Data Connection Layer — Protocol abstraction (OPC UA, custom), subscription management, clean data pipe.
|
||||
5. Central–Site Communication — Akka.NET ClusterClient/ClusterClientReceptionist, message patterns, debug streaming.
|
||||
5. Central–Site Communication — Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data), message patterns, debug streaming.
|
||||
6. Store-and-Forward Engine — Buffering, fixed-interval retry, parking, SQLite persistence, replication.
|
||||
7. External System Gateway — External system definitions, API method invocation, database connections.
|
||||
8. Notification Service — Notification lists, email delivery, store-and-forward integration.
|
||||
@@ -81,7 +81,8 @@ This project contains design documentation for a distributed SCADA system built
|
||||
- Tag path resolution retried periodically for devices still booting.
|
||||
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses ClusterClient/ClusterClientReceptionist. Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
|
||||
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
|
||||
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
|
||||
|
||||
### External Integrations
|
||||
- External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
|
||||
@@ -126,7 +127,7 @@ This project contains design documentation for a distributed SCADA system built
|
||||
### UI & Monitoring
|
||||
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
|
||||
- UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components.
|
||||
- Debug view: real-time streaming via DebugStreamBridgeActor. Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
|
||||
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via SiteStreamGrpcClient, snapshot via ClusterClient). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
|
||||
- Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
|
||||
- Dead letter monitoring as a health metric.
|
||||
- Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search.
|
||||
|
||||
10
README.md
10
README.md
@@ -38,7 +38,7 @@ This document serves as the master index for the SCADA system design. The system
|
||||
| 2 | Deployment Manager | [docs/requirements/Component-DeploymentManager.md](docs/requirements/Component-DeploymentManager.md) | Central-side deployment pipeline with deployment ID/idempotency, per-instance operation lock, state transition matrix, all-or-nothing site apply, system-wide artifact deployment with per-site status. |
|
||||
| 3 | Site Runtime | [docs/requirements/Component-SiteRuntime.md](docs/requirements/Component-SiteRuntime.md) | Site-side actor hierarchy with explicit supervision strategies, staggered startup, script trust model (constrained APIs), Tell/Ask conventions, concurrency serialization, and site-wide Akka stream with per-subscriber backpressure. |
|
||||
| 4 | Data Connection Layer | [docs/requirements/Component-DataConnectionLayer.md](docs/requirements/Component-DataConnectionLayer.md) | Common data connection interface (OPC UA, custom), Become/Stash connection actor model, auto-reconnect, immediate bad quality on disconnect, transparent re-subscribe, synchronous write failures, tag path resolution retry. |
|
||||
| 5 | Central–Site Communication | [docs/requirements/Component-Communication.md](docs/requirements/Component-Communication.md) | Akka.NET remoting/cluster topology, 8 message patterns with per-pattern timeouts, application-level correlation IDs, transport heartbeat config, message ordering, connection failure behavior. |
|
||||
| 5 | Central–Site Communication | [docs/requirements/Component-Communication.md](docs/requirements/Component-Communication.md) | Dual transport: Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data). 8 message patterns with per-pattern timeouts, SiteStreamGrpcServer/Client, application-level correlation IDs, transport heartbeat config, gRPC keepalive, message ordering, connection failure behavior. |
|
||||
| 6 | Store-and-Forward Engine | [docs/requirements/Component-StoreAndForward.md](docs/requirements/Component-StoreAndForward.md) | Buffering (transient failures only), fixed-interval retry, parking, async best-effort replication, SQLite persistence at sites. |
|
||||
| 7 | External System Gateway | [docs/requirements/Component-ExternalSystemGateway.md](docs/requirements/Component-ExternalSystemGateway.md) | HTTP/REST + JSON, API key/Basic Auth, per-system timeout, dual call modes (Call/CachedCall), transient/permanent error classification, dedicated blocking I/O dispatcher, ADO.NET connection pooling. |
|
||||
| 8 | Notification Service | [docs/requirements/Component-NotificationService.md](docs/requirements/Component-NotificationService.md) | SMTP with OAuth2 (M365) or Basic Auth, BCC delivery, plain text, transient/permanent SMTP error classification, store-and-forward integration. |
|
||||
@@ -90,6 +90,8 @@ This document serves as the master index for the SCADA system design. The system
|
||||
│ └──────────┘ │
|
||||
│ ┌───────────────────────────────────┐ │
|
||||
│ │ Akka.NET Communication Layer │ │
|
||||
│ │ ClusterClient: command/control │ │
|
||||
│ │ gRPC Client: real-time streams │ │
|
||||
│ │ (correlation IDs, per-pattern │ │
|
||||
│ │ timeouts, message ordering) │ │
|
||||
│ └──────────────┬────────────────────┘ │
|
||||
@@ -98,7 +100,8 @@ This document serves as the master index for the SCADA system design. The system
|
||||
│ └───────────────────────────────────┘ (Config DB)│
|
||||
│ │ Machine Data DB│
|
||||
└─────────────────┼───────────────────────────────────┘
|
||||
│ Akka.NET Remoting
|
||||
│ Akka.NET Remoting (command/control)
|
||||
│ gRPC HTTP/2 (real-time data, port 8083)
|
||||
┌────────────┼────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
@@ -112,6 +115,9 @@ This document serves as the master index for the SCADA system design. The system
|
||||
│ │Site │ │ │ │Site │ │ │ │Site │ │
|
||||
│ │Runtm│ │ │ │Runtm│ │ │ │Runtm│ │
|
||||
│ ├─────┤ │ │ ├─────┤ │ │ ├─────┤ │
|
||||
│ │gRPC │ │ │ │gRPC │ │ │ │gRPC │ │
|
||||
│ │Srvr │ │ │ │Srvr │ │ │ │Srvr │ │
|
||||
│ ├─────┤ │ │ ├─────┤ │ │ ├─────┤ │
|
||||
│ │S&F │ │ │ │S&F │ │ │ │S&F │ │
|
||||
│ │Engine│ │ │ │Engine│ │ │ │Engine│ │
|
||||
│ ├─────┤ │ │ ├─────┤ │ │ ├─────┤ │
|
||||
|
||||
@@ -29,7 +29,8 @@ Local Docker deployment of the full ScadaLink cluster topology: a 2-node central
|
||||
│ (Test Plant A) │ │ (Test Plant B) │ │ (Test Plant C) │
|
||||
│ │ │ │ │ │
|
||||
│ node-a ◄──► node-b│ │ node-a ◄──► node-b│ │ node-a ◄──► node-b│
|
||||
│ :9021 :9022 │ │ :9031 :9032 │ │ :9041 :9042 │
|
||||
│ Akka :9021 :9022 │ │ Akka :9031 :9032 │ │ Akka :9041 :9042 │
|
||||
│ gRPC :9023 :9024 │ │ gRPC :9033 :9034 │ │ gRPC :9043 :9044 │
|
||||
└────────────────────┘ └────────────────────┘ └────────────────────┘
|
||||
```
|
||||
|
||||
@@ -39,7 +40,7 @@ Runs the web UI (Blazor Server), Template Engine, Deployment Manager, Security,
|
||||
|
||||
### Site Clusters (active/standby each)
|
||||
|
||||
Each site cluster runs Site Runtime, Data Connection Layer, Store-and-Forward, and Site Event Logging. Sites connect to OPC UA for device data and to the central cluster via Akka.NET remoting. Deployed configurations and S&F buffers are stored in local SQLite databases per node.
|
||||
Each site cluster runs Site Runtime, Data Connection Layer, Store-and-Forward, and Site Event Logging. Sites connect to OPC UA for device data and to the central cluster via Akka.NET remoting. Each site node also hosts a gRPC streaming server (port 8083) that central nodes connect to for real-time attribute value and alarm state streams. Deployed configurations and S&F buffers are stored in local SQLite databases per node.
|
||||
|
||||
| Site Cluster | Site Identifier | Central UI Name |
|
||||
|-------------|-----------------|-----------------|
|
||||
@@ -51,19 +52,19 @@ Each site cluster runs Site Runtime, Data Connection Layer, Store-and-Forward, a
|
||||
|
||||
### Application Nodes
|
||||
|
||||
| Node | Container Name | Host Web Port | Host Akka Port | Internal Ports |
|
||||
|------|---------------|---------------|----------------|----------------|
|
||||
| Traefik LB | `scadalink-traefik` | 9000 | — | 80 (proxy), 8080 (dashboard) |
|
||||
| Central A | `scadalink-central-a` | 9001 | 9011 | 5000 (web), 8081 (Akka) |
|
||||
| Central B | `scadalink-central-b` | 9002 | 9012 | 5000 (web), 8081 (Akka) |
|
||||
| Site-A A | `scadalink-site-a-a` | — | 9021 | 8082 (Akka) |
|
||||
| Site-A B | `scadalink-site-a-b` | — | 9022 | 8082 (Akka) |
|
||||
| Site-B A | `scadalink-site-b-a` | — | 9031 | 8082 (Akka) |
|
||||
| Site-B B | `scadalink-site-b-b` | — | 9032 | 8082 (Akka) |
|
||||
| Site-C A | `scadalink-site-c-a` | — | 9041 | 8082 (Akka) |
|
||||
| Site-C B | `scadalink-site-c-b` | — | 9042 | 8082 (Akka) |
|
||||
| Node | Container Name | Host Web Port | Host Akka Port | Host gRPC Port | Internal Ports |
|
||||
|------|---------------|---------------|----------------|----------------|----------------|
|
||||
| Traefik LB | `scadalink-traefik` | 9000 | — | — | 80 (proxy), 8080 (dashboard) |
|
||||
| Central A | `scadalink-central-a` | 9001 | 9011 | — | 5000 (web), 8081 (Akka) |
|
||||
| Central B | `scadalink-central-b` | 9002 | 9012 | — | 5000 (web), 8081 (Akka) |
|
||||
| Site-A A | `scadalink-site-a-a` | — | 9021 | 9023 | 8082 (Akka), 8083 (gRPC) |
|
||||
| Site-A B | `scadalink-site-a-b` | — | 9022 | 9024 | 8082 (Akka), 8083 (gRPC) |
|
||||
| Site-B A | `scadalink-site-b-a` | — | 9031 | 9033 | 8082 (Akka), 8083 (gRPC) |
|
||||
| Site-B B | `scadalink-site-b-b` | — | 9032 | 9034 | 8082 (Akka), 8083 (gRPC) |
|
||||
| Site-C A | `scadalink-site-c-a` | — | 9041 | 9043 | 8082 (Akka), 8083 (gRPC) |
|
||||
| Site-C B | `scadalink-site-c-b` | — | 9042 | 9044 | 8082 (Akka), 8083 (gRPC) |
|
||||
|
||||
Port block pattern: `90X1`/`90X2` where X = 0 (central), 1 (web), 2 (site-a), 3 (site-b), 4 (site-c).
|
||||
Port block pattern: `90X1`/`90X2` (Akka), `90X3`/`90X4` (gRPC) where X = 0 (central), 2 (site-a), 3 (site-b), 4 (site-c). gRPC streaming ports are used by central nodes to subscribe to real-time site data streams.
|
||||
|
||||
### Infrastructure Services (from `infra/docker-compose.yml`)
|
||||
|
||||
@@ -85,6 +86,7 @@ docker/
|
||||
├── docker-compose.yml # 8-node application stack
|
||||
├── build.sh # Build Docker image
|
||||
├── deploy.sh # Build + deploy all containers
|
||||
├── seed-sites.sh # Create test sites with Akka + gRPC addresses
|
||||
├── teardown.sh # Stop and remove containers
|
||||
├── central-node-a/
|
||||
│ ├── appsettings.Central.json # Central node A configuration
|
||||
@@ -130,6 +132,9 @@ cd infra && docker compose up -d && cd ..
|
||||
|
||||
# 2. Build and deploy all 8 ScadaLink nodes
|
||||
docker/deploy.sh
|
||||
|
||||
# 3. Seed test sites (first-time only, after cluster is healthy)
|
||||
docker/seed-sites.sh
|
||||
```
|
||||
|
||||
### After Code Changes
|
||||
|
||||
@@ -26,4 +26,7 @@ echo " Active node check: http://localhost:9001/health/active"
|
||||
echo " Traefik dashboard: http://localhost:8180"
|
||||
echo " Management API: http://localhost:9000/management"
|
||||
echo ""
|
||||
echo "To seed test sites (first-time setup):"
|
||||
echo " docker/seed-sites.sh"
|
||||
echo ""
|
||||
echo "Logs: docker compose -f $SCRIPT_DIR/docker-compose.yml logs -f"
|
||||
|
||||
@@ -40,6 +40,7 @@ services:
|
||||
SCADALINK_CONFIG: Site
|
||||
ports:
|
||||
- "9021:8082" # Akka remoting (host access for debugging)
|
||||
- "9023:8083" # gRPC streaming
|
||||
volumes:
|
||||
- ./site-a-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
|
||||
- ./site-a-node-a/data:/app/data
|
||||
@@ -55,6 +56,7 @@ services:
|
||||
SCADALINK_CONFIG: Site
|
||||
ports:
|
||||
- "9022:8082" # Akka remoting
|
||||
- "9024:8083" # gRPC streaming
|
||||
volumes:
|
||||
- ./site-a-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
|
||||
- ./site-a-node-b/data:/app/data
|
||||
@@ -70,6 +72,7 @@ services:
|
||||
SCADALINK_CONFIG: Site
|
||||
ports:
|
||||
- "9031:8082" # Akka remoting
|
||||
- "9033:8083" # gRPC streaming
|
||||
volumes:
|
||||
- ./site-b-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
|
||||
- ./site-b-node-a/data:/app/data
|
||||
@@ -85,6 +88,7 @@ services:
|
||||
SCADALINK_CONFIG: Site
|
||||
ports:
|
||||
- "9032:8082" # Akka remoting
|
||||
- "9034:8083" # gRPC streaming
|
||||
volumes:
|
||||
- ./site-b-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
|
||||
- ./site-b-node-b/data:/app/data
|
||||
@@ -100,6 +104,7 @@ services:
|
||||
SCADALINK_CONFIG: Site
|
||||
ports:
|
||||
- "9041:8082" # Akka remoting
|
||||
- "9043:8083" # gRPC streaming
|
||||
volumes:
|
||||
- ./site-c-node-a/appsettings.Site.json:/app/appsettings.Site.json:ro
|
||||
- ./site-c-node-a/data:/app/data
|
||||
@@ -115,6 +120,7 @@ services:
|
||||
SCADALINK_CONFIG: Site
|
||||
ports:
|
||||
- "9042:8082" # Akka remoting
|
||||
- "9044:8083" # gRPC streaming
|
||||
volumes:
|
||||
- ./site-c-node-b/appsettings.Site.json:/app/appsettings.Site.json:ro
|
||||
- ./site-c-node-b/data:/app/data
|
||||
|
||||
62
docker/seed-sites.sh
Executable file
62
docker/seed-sites.sh
Executable file
@@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# Seed the three test sites with Akka and gRPC addresses.
|
||||
# Run after deploy.sh once the central cluster is healthy.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - Infrastructure services running (infra/docker-compose up -d)
|
||||
# - Application containers running (docker/deploy.sh)
|
||||
# - Central cluster healthy (curl http://localhost:9000/health/ready)
|
||||
#
|
||||
# Usage:
|
||||
# docker/seed-sites.sh
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
CLI="dotnet run --project $PROJECT_ROOT/src/ScadaLink.CLI --"
|
||||
AUTH="--username multi-role --password password"
|
||||
URL="--url http://localhost:9000"
|
||||
|
||||
echo "=== Seeding ScadaLink Sites ==="
|
||||
|
||||
echo ""
|
||||
echo "Creating Site-A (Test Plant A)..."
|
||||
$CLI $URL $AUTH site create \
|
||||
--name "Test Plant A" \
|
||||
--identifier "site-a" \
|
||||
--description "Test site A - two-node cluster" \
|
||||
--node-a-address "akka.tcp://scadalink@scadalink-site-a-a:8082" \
|
||||
--node-b-address "akka.tcp://scadalink@scadalink-site-a-b:8082" \
|
||||
--grpc-node-a-address "http://scadalink-site-a-a:8083" \
|
||||
--grpc-node-b-address "http://scadalink-site-a-b:8083" \
|
||||
|| echo " (Site-A may already exist)"
|
||||
|
||||
echo ""
|
||||
echo "Creating Site-B (Test Plant B)..."
|
||||
$CLI $URL $AUTH site create \
|
||||
--name "Test Plant B" \
|
||||
--identifier "site-b" \
|
||||
--description "Test site B - two-node cluster" \
|
||||
--node-a-address "akka.tcp://scadalink@scadalink-site-b-a:8082" \
|
||||
--node-b-address "akka.tcp://scadalink@scadalink-site-b-b:8082" \
|
||||
--grpc-node-a-address "http://scadalink-site-b-a:8083" \
|
||||
--grpc-node-b-address "http://scadalink-site-b-b:8083" \
|
||||
|| echo " (Site-B may already exist)"
|
||||
|
||||
echo ""
|
||||
echo "Creating Site-C (Test Plant C)..."
|
||||
$CLI $URL $AUTH site create \
|
||||
--name "Test Plant C" \
|
||||
--identifier "site-c" \
|
||||
--description "Test site C - two-node cluster" \
|
||||
--node-a-address "akka.tcp://scadalink@scadalink-site-c-a:8082" \
|
||||
--node-b-address "akka.tcp://scadalink@scadalink-site-c-b:8082" \
|
||||
--grpc-node-a-address "http://scadalink-site-c-a:8083" \
|
||||
--grpc-node-b-address "http://scadalink-site-c-b:8083" \
|
||||
|| echo " (Site-C may already exist)"
|
||||
|
||||
echo ""
|
||||
echo "=== Site seeding complete ==="
|
||||
echo ""
|
||||
echo "Verify with: $CLI $URL $AUTH site list"
|
||||
@@ -4,7 +4,8 @@
|
||||
"Role": "Site",
|
||||
"NodeHostname": "scadalink-site-a-a",
|
||||
"SiteId": "site-a",
|
||||
"RemotingPort": 8082
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
"Role": "Site",
|
||||
"NodeHostname": "scadalink-site-a-b",
|
||||
"SiteId": "site-a",
|
||||
"RemotingPort": 8082
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
"Role": "Site",
|
||||
"NodeHostname": "scadalink-site-b-a",
|
||||
"SiteId": "site-b",
|
||||
"RemotingPort": 8082
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
"Role": "Site",
|
||||
"NodeHostname": "scadalink-site-b-b",
|
||||
"SiteId": "site-b",
|
||||
"RemotingPort": 8082
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
"Role": "Site",
|
||||
"NodeHostname": "scadalink-site-c-a",
|
||||
"SiteId": "site-c",
|
||||
"RemotingPort": 8082
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
"Role": "Site",
|
||||
"NodeHostname": "scadalink-site-c-b",
|
||||
"SiteId": "site-c",
|
||||
"RemotingPort": 8082
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
|
||||
@@ -88,7 +88,7 @@ scadalink instance delete <code>
|
||||
```
|
||||
scadalink site list [--format json|table]
|
||||
scadalink site get <site-id> [--format json|table]
|
||||
scadalink site create --name <name> --id <site-id>
|
||||
scadalink site create --name <name> --id <site-id> [--node-a-address <addr>] [--node-b-address <addr>] [--grpc-node-a-address <addr>] [--grpc-node-b-address <addr>]
|
||||
scadalink site update <site-id> --file <path>
|
||||
scadalink site delete <site-id>
|
||||
scadalink site area list <site-id>
|
||||
|
||||
@@ -24,7 +24,7 @@ Central cluster only. Sites have no user interface.
|
||||
|
||||
## Real-Time Updates
|
||||
|
||||
- **Debug view**: Real-time display of attribute values and alarm states via **streaming**. When the user opens a debug view, a `DebugStreamBridgeActor` on the central side subscribes to the site's Akka stream for the selected instance. The bridge actor delivers an initial `DebugViewSnapshot` followed by ongoing `AttributeValueChanged` and `AlarmStateChanged` events to the Blazor component via callbacks, which call `InvokeAsync(StateHasChanged)` to push UI updates through the built-in SignalR circuit.
|
||||
- **Debug view**: Real-time display of attribute values and alarm states via **gRPC streaming**. When the user opens a debug view, a `DebugStreamBridgeActor` on the central side opens a gRPC server-streaming subscription to the site's `SiteStreamGrpcServer` for the selected instance, then requests an initial `DebugViewSnapshot` via ClusterClient. Ongoing `AttributeValueChanged` and `AlarmStateChanged` events flow via the gRPC stream (not through ClusterClient) to the bridge actor, which delivers them to the Blazor component via callbacks that call `InvokeAsync(StateHasChanged)` to push UI updates through the built-in SignalR circuit.
|
||||
- **Health dashboard**: Site status, connection health, error rates, and buffer depths update via a **10-second auto-refresh timer**. Since health reports arrive from sites every 30 seconds, a 10s poll interval catches updates within one reporting cycle without unnecessary overhead.
|
||||
- **Deployment status**: Pending/in-progress/success/failed transitions **push to the UI immediately** via SignalR (built into Blazor Server). No polling required for deployment tracking.
|
||||
|
||||
@@ -66,7 +66,7 @@ Central cluster only. Sites have no user interface.
|
||||
- Configure SMTP settings.
|
||||
|
||||
### Site & Data Connection Management (Admin Role)
|
||||
- Create, edit, and delete site definitions.
|
||||
- Create, edit, and delete site definitions, including Akka node addresses (NodeA/NodeB) and gRPC node addresses (GrpcNodeA/GrpcNodeB).
|
||||
- Define data connections and assign them to sites (name, protocol type, connection details).
|
||||
|
||||
### Area Management (Admin Role)
|
||||
@@ -101,8 +101,8 @@ Central cluster only. Sites have no user interface.
|
||||
### Debug View (Deployment Role)
|
||||
- Select a deployed instance and open a live debug view.
|
||||
- Real-time streaming of all attribute values (with quality and timestamp) and alarm states for that instance.
|
||||
- The `DebugStreamService` creates a `DebugStreamBridgeActor` on the central side that subscribes to the site's Akka stream for the selected instance.
|
||||
- The bridge actor receives an initial `DebugViewSnapshot` followed by ongoing `AttributeValueChanged` and `AlarmStateChanged` events from the site.
|
||||
- The `DebugStreamService` creates a `DebugStreamBridgeActor` on the central side. The bridge actor opens a **gRPC server-streaming subscription** to the site's `SiteStreamGrpcServer` for the selected instance, then requests an initial `DebugViewSnapshot` via ClusterClient.
|
||||
- Ongoing events (`AttributeValueChanged`, `AlarmStateChanged`) flow via the gRPC stream directly to the bridge actor — they do not pass through ClusterClient.
|
||||
- Events are delivered to the Blazor component via callbacks, which call `InvokeAsync(StateHasChanged)` to push UI updates through the built-in SignalR circuit.
|
||||
- A pulsing "Live" indicator replaces the static "Connected" badge when streaming is active.
|
||||
- Stream includes attribute values formatted as `[InstanceUniqueName].[AttributePath].[AttributeName]` and alarm states formatted as `[InstanceUniqueName].[AlarmName]`.
|
||||
|
||||
@@ -106,7 +106,8 @@ The Host component wires CoordinatedShutdown into the Windows Service lifecycle
|
||||
Each node is configured with:
|
||||
- **Cluster seed nodes**: **Both nodes** are seed nodes — each node lists both itself and its partner. Either node can start first and form the cluster; the other joins when it starts. No startup ordering dependency.
|
||||
- **Cluster role**: Central or Site (plus site identifier for site clusters).
|
||||
- **Akka.NET remoting**: Hostname/port for inter-node and inter-cluster communication.
|
||||
- **Akka.NET remoting**: Hostname/port for inter-node and inter-cluster communication (default 8081 central, 8082 site).
|
||||
- **gRPC port** (site nodes only): Dedicated HTTP/2 port for the SiteStreamGrpcServer (default 8083). Separate from the Akka remoting port — gRPC uses Kestrel, Akka uses its own TCP transport.
|
||||
- **Local storage paths**: SQLite database locations (site nodes only).
|
||||
|
||||
## Windows Service
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
The Communication component manages all messaging between the central cluster and site clusters using Akka.NET. It provides the transport layer for deployments, instance lifecycle commands, integration routing, debug streaming, health reporting, and remote queries (parked messages, event logs).
|
||||
The Communication component manages all messaging between the central cluster and site clusters. It provides the transport layer for deployments, instance lifecycle commands, integration routing, debug streaming, health reporting, and remote queries (parked messages, event logs). Two transports are used: **Akka.NET ClusterClient** for command/control messaging and **gRPC server-streaming** for real-time data (attribute values, alarm states).
|
||||
|
||||
## Location
|
||||
|
||||
@@ -10,12 +10,15 @@ Both central and site clusters. Each side has communication actors that handle m
|
||||
|
||||
## Responsibilities
|
||||
|
||||
- Resolve site addresses from the configuration database and maintain a cached address map.
|
||||
- Establish and maintain cross-cluster connections using Akka.NET ClusterClient/ClusterClientReceptionist.
|
||||
- Resolve site addresses (Akka remoting and gRPC) from the configuration database and maintain a cached address map.
|
||||
- Establish and maintain cross-cluster connections using Akka.NET ClusterClient/ClusterClientReceptionist for command/control.
|
||||
- Establish and maintain per-site gRPC streaming connections for real-time data delivery (site→central).
|
||||
- Route messages between central and site clusters in a hub-and-spoke topology.
|
||||
- Broker requests from external systems (via central) to sites and return responses.
|
||||
- Support multiple concurrent message patterns (request/response, fire-and-forget, streaming).
|
||||
- Detect site connectivity status for health monitoring.
|
||||
- Host the **SiteStreamGrpcServer** on site nodes (Kestrel HTTP/2) to serve real-time event streams.
|
||||
- Manage per-site **SiteStreamGrpcClient** instances on central nodes via **SiteStreamGrpcClientFactory**.
|
||||
|
||||
## Communication Patterns
|
||||
|
||||
@@ -50,22 +53,55 @@ Both central and site clusters. Each side has communication actors that handle m
|
||||
- Site applies and acknowledges.
|
||||
|
||||
### 6. Debug Streaming (Site → Central)
|
||||
- **Pattern**: Subscribe/push with initial snapshot (no polling).
|
||||
- A **DebugStreamBridgeActor** (one per active debug session) is created on the central cluster by the **DebugStreamService**. The bridge actor sends a `SubscribeDebugViewRequest` to the site via `CentralCommunicationActor`. The site's `InstanceActor` stores the subscription's correlation ID and replies with an initial snapshot via the ClusterClient reply path.
|
||||
- Site requests a **snapshot** of all current attribute values and alarm states from the Instance Actor and sends it back to the bridge actor (via the ClusterClient reply path, which works for immediate responses).
|
||||
- For ongoing events, the InstanceActor wraps `AttributeValueChanged` and `AlarmStateChanged` in a `DebugStreamEvent(correlationId, event)` message and sends it to the local `SiteCommunicationActor`. The SiteCommunicationActor forwards it to central via its own ClusterClient (`ClusterClient.Send("/user/central-communication", event)`). The `CentralCommunicationActor` looks up the bridge actor by correlation ID and delivers the event. This follows the same site→central pattern as health reports.
|
||||
- **Pattern**: Subscribe/push with initial snapshot. Two transports: **ClusterClient** for the subscribe/unsubscribe handshake and initial snapshot, **gRPC server-streaming** for ongoing real-time events.
|
||||
- A **DebugStreamBridgeActor** (one per active debug session) is created on the central cluster by the **DebugStreamService**. The bridge actor first opens a **gRPC server-streaming subscription** to the site via `SiteStreamGrpcClient`, then sends a `SubscribeDebugViewRequest` to the site via `CentralCommunicationActor` (ClusterClient). The site's `InstanceActor` replies with an initial snapshot via the ClusterClient reply path.
|
||||
- **gRPC stream (real-time events)**: The site's **SiteStreamGrpcServer** receives the gRPC `SubscribeInstance` call and creates a **StreamRelayActor** that subscribes to **SiteStreamManager** for the requested instance. Events (`AttributeValueChanged`, `AlarmStateChanged`) flow from `SiteStreamManager` → `StreamRelayActor` → `Channel<SiteStreamEvent>` (bounded, 1000, DropOldest) → gRPC response stream → `SiteStreamGrpcClient` on central → `DebugStreamBridgeActor`.
|
||||
- The `DebugStreamEvent` message type no longer exists — events are not routed through ClusterClient. `SiteCommunicationActor` and `CentralCommunicationActor` have no role in streaming event delivery.
|
||||
- The bridge actor forwards received events to the consumer via callbacks (Blazor component or SignalR hub).
|
||||
- **Snapshot-to-stream handoff**: The gRPC stream is opened **before** the snapshot request to avoid missing events. The consumer applies the snapshot as baseline, then replays buffered gRPC events with timestamps newer than the snapshot (timestamp-based dedup).
|
||||
- Attribute value stream messages: `[InstanceUniqueName].[AttributePath].[AttributeName]`, value, quality, timestamp.
|
||||
- Alarm state stream messages: `[InstanceUniqueName].[AlarmName]`, state (active/normal), priority, timestamp.
|
||||
- Central sends an unsubscribe request when the debug session ends. The site removes its stream subscription and the bridge actor is stopped.
|
||||
- Central sends an unsubscribe request via ClusterClient when the debug session ends. The gRPC stream is cancelled. The site's `StreamRelayActor` is stopped and the SiteStreamManager subscription is removed.
|
||||
- The stream is session-based and temporary.
|
||||
|
||||
#### Site-Side gRPC Streaming Components
|
||||
|
||||
- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel<SiteStreamEvent>` to the gRPC response stream. Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready.
|
||||
- **StreamRelayActor**: Short-lived actor created per gRPC subscription. Receives domain events (`AttributeValueChanged`, `AlarmStateChanged`) from `SiteStreamManager`, converts them to protobuf `SiteStreamEvent` messages, and writes to the `Channel<SiteStreamEvent>` writer. Stopped when the gRPC stream is cancelled or the client disconnects.
|
||||
|
||||
#### Central-Side Debug Stream Components
|
||||
|
||||
- **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub.
|
||||
- **DebugStreamBridgeActor**: One per active debug session. Acts as the Akka-level subscriber registered with the site's `InstanceActor`. Receives real-time `AttributeValueChanged` and `AlarmStateChanged` events from the site and forwards them to the consumer via callbacks.
|
||||
- **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub. Injects `SiteStreamGrpcClientFactory` for gRPC stream creation.
|
||||
- **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` via ClusterClient. Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
|
||||
- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, converts protobuf messages to domain events, and invokes the `onEvent` callback.
|
||||
- **SiteStreamGrpcClientFactory**: Caches per-site `SiteStreamGrpcClient` instances. Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor`). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change.
|
||||
- **DebugStreamHub**: SignalR hub at `/hubs/debug-stream` for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the **Deployment** role. Server-to-client methods: `OnSnapshot`, `OnAttributeChanged`, `OnAlarmChanged`, `OnStreamTerminated`.
|
||||
|
||||
#### gRPC Proto Definition
|
||||
|
||||
The streaming protocol is defined in `sitestream.proto` (`src/ScadaLink.Communication/Protos/sitestream.proto`):
|
||||
|
||||
- **Service**: `SiteStreamService` with a single RPC `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)`.
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`).
|
||||
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
|
||||
- Proto field numbers are never reused. Old clients ignore unknown `oneof` variants.
|
||||
|
||||
#### gRPC Connection Keepalive
|
||||
|
||||
Three layers of dead-client detection prevent orphan streams on site nodes:
|
||||
|
||||
| Layer | Detects | Timeline | Mechanism |
|
||||
|-------|---------|----------|-----------|
|
||||
| TCP RST | Clean process death, connection close | 1–5s | OS-level TCP, `WriteAsync` throws |
|
||||
| gRPC keepalive PING | Network partition, silent crash, firewall drop | ~25s | HTTP/2 PING frames, `CancellationToken` fires |
|
||||
| Session timeout | Misconfigured keepalive, long-lived zombie streams | 4 hours | `CancellationTokenSource.CancelAfter` |
|
||||
|
||||
Keepalive settings are configurable via `CommunicationOptions`:
|
||||
- `GrpcKeepAlivePingDelay`: 15 seconds (default)
|
||||
- `GrpcKeepAlivePingTimeout`: 10 seconds (default)
|
||||
- `GrpcMaxStreamLifetime`: 4 hours (default)
|
||||
- `GrpcMaxConcurrentStreams`: 100 (default)
|
||||
|
||||
### 6a. Debug Snapshot (Central → Site)
|
||||
- **Pattern**: Request/Response (one-shot, no subscription).
|
||||
- Central sends a `DebugSnapshotRequest` (identified by instance unique name) to the site.
|
||||
@@ -91,12 +127,17 @@ Both central and site clusters. Each side has communication actors that handle m
|
||||
|
||||
```
|
||||
Central Cluster
|
||||
├── ClusterClient → Site A Cluster (SiteCommunicationActor via Receptionist)
|
||||
├── ClusterClient → Site B Cluster (SiteCommunicationActor via Receptionist)
|
||||
└── ClusterClient → Site N Cluster (SiteCommunicationActor via Receptionist)
|
||||
├── ClusterClient → Site A Cluster (SiteCommunicationActor via Receptionist) [command/control]
|
||||
├── ClusterClient → Site B Cluster (SiteCommunicationActor via Receptionist) [command/control]
|
||||
└── ClusterClient → Site N Cluster (SiteCommunicationActor via Receptionist) [command/control]
|
||||
│
|
||||
├── SiteStreamGrpcClient ◄── gRPC stream ── Site A (SiteStreamGrpcServer) [real-time data]
|
||||
├── SiteStreamGrpcClient ◄── gRPC stream ── Site B (SiteStreamGrpcServer) [real-time data]
|
||||
└── SiteStreamGrpcClient ◄── gRPC stream ── Site N (SiteStreamGrpcServer) [real-time data]
|
||||
|
||||
Site Clusters
|
||||
└── ClusterClient → Central Cluster (CentralCommunicationActor via Receptionist)
|
||||
└── ClusterClient → Central Cluster (CentralCommunicationActor via Receptionist) [command/control]
|
||||
└── SiteStreamGrpcServer (Kestrel HTTP/2, port 8083) → serves gRPC streams [real-time data]
|
||||
```
|
||||
|
||||
- Sites do **not** communicate with each other.
|
||||
@@ -107,8 +148,8 @@ Site Clusters
|
||||
|
||||
Central discovers site addresses through the **configuration database**, not runtime registration:
|
||||
|
||||
- Each site record in the Sites table includes optional **NodeAAddress** and **NodeBAddress** fields containing base Akka addresses of the site's cluster nodes (e.g., `akka.tcp://scadalink@host:port`).
|
||||
- The **CentralCommunicationActor** loads all site addresses from the database at startup and creates one **ClusterClient per site**, configured with both NodeA and NodeB as contact points.
|
||||
- Each site record in the Sites table includes optional **NodeAAddress** and **NodeBAddress** fields containing base Akka addresses of the site's cluster nodes (e.g., `akka.tcp://scadalink@host:port`), and optional **GrpcNodeAAddress** and **GrpcNodeBAddress** fields containing gRPC endpoints (e.g., `http://host:8083`).
|
||||
- The **CentralCommunicationActor** loads all site addresses from the database at startup and creates one **ClusterClient per site**, configured with both NodeA and NodeB as contact points. The **SiteStreamGrpcClientFactory** uses `GrpcNodeAAddress` / `GrpcNodeBAddress` to create per-site gRPC channels for streaming.
|
||||
- The address cache is **refreshed every 60 seconds** and **on-demand** when site records are added, edited, or deleted via the Central UI or CLI. ClusterClient instances are recreated when contact points change.
|
||||
- When routing a message to a site, central sends via `ClusterClient.Send("/user/site-communication", msg)`. **ClusterClient handles failover between NodeA and NodeB internally** — there is no application-level NodeA preference/NodeB fallback logic.
|
||||
- **Heartbeats** from sites serve **health monitoring only** — they do not serve as a registration or address discovery mechanism.
|
||||
@@ -166,7 +207,7 @@ The ManagementActor is registered at the well-known path `/user/management` on c
|
||||
## Connection Failure Behavior
|
||||
|
||||
- **In-flight messages**: When a connection drops while a request is in flight (e.g., deployment sent but no response received), the Akka ask pattern times out and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages.
|
||||
- **Debug streams**: Any connection interruption (failover or network blip) kills the debug stream. The `DebugStreamBridgeActor` is stopped and the consumer is notified via `OnStreamTerminated`. The engineer must reopen the debug view to re-establish the subscription with a fresh snapshot. There is no auto-resume.
|
||||
- **Debug streams**: Any gRPC stream interruption triggers reconnection logic in the `DebugStreamBridgeActor`. The bridge actor attempts to reconnect to the other site node endpoint (NodeB if NodeA failed, or vice versa), with up to 3 retries and 5-second backoff. If all retries fail, the consumer is notified via `OnStreamTerminated` and the bridge actor is stopped. Events during the reconnection gap are lost (acceptable for real-time debug view). On successful reconnection, the consumer can request a fresh snapshot to re-sync state.
|
||||
|
||||
## Failover Behavior
|
||||
|
||||
@@ -175,9 +216,11 @@ The ManagementActor is registered at the well-known path `/user/management` on c
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **Akka.NET Remoting + ClusterClient**: Provides the transport layer. ClusterClient/ClusterClientReceptionist used for all cross-cluster messaging.
|
||||
- **Akka.NET Remoting + ClusterClient**: Provides the command/control transport layer. ClusterClient/ClusterClientReceptionist used for cross-cluster command/control messaging (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots).
|
||||
- **gRPC (Grpc.AspNetCore + Grpc.Net.Client)**: Provides the real-time data streaming transport. Site nodes host a gRPC server (SiteStreamGrpcServer); central nodes create per-site gRPC clients (SiteStreamGrpcClient).
|
||||
- **Cluster Infrastructure**: Manages node roles and failover detection.
|
||||
- **Configuration Database**: Provides site node addresses (NodeAAddress, NodeBAddress) for address resolution.
|
||||
- **Configuration Database**: Provides site node addresses (NodeAAddress, NodeBAddress for Akka remoting; GrpcNodeAAddress, GrpcNodeBAddress for gRPC streaming) for address resolution.
|
||||
- **Site Runtime (SiteStreamManager)**: The SiteStreamGrpcServer subscribes to SiteStreamManager to receive real-time events for gRPC delivery.
|
||||
|
||||
## Interactions
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ The configuration database stores all central system data, organized by domain a
|
||||
- **Shared Scripts**: System-wide reusable script definitions (name, C# source code, parameter definitions, return value definitions).
|
||||
|
||||
### Sites & Data Connections
|
||||
- **Sites**: Site definitions (name, identifier, description).
|
||||
- **Sites**: Site definitions (name, identifier, description, NodeAAddress, NodeBAddress, GrpcNodeAAddress, GrpcNodeBAddress).
|
||||
- **Data Connections**: Data connection definitions (name, protocol type, connection details) with site assignments.
|
||||
|
||||
### External Systems & Database Connections
|
||||
|
||||
@@ -45,7 +45,7 @@ The Host must bind configuration sections from `appsettings.json` to strongly-ty
|
||||
|
||||
| Section | Options Class | Owner | Contents |
|
||||
|---------|--------------|-------|----------|
|
||||
| `ScadaLink:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort |
|
||||
| `ScadaLink:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort, GrpcPort (site only, default 8083) |
|
||||
| `ScadaLink:Cluster` | `ClusterOptions` | ClusterInfrastructure | SeedNodes, SplitBrainResolverStrategy, StableAfter, HeartbeatInterval, FailureDetectionThreshold, MinNrOfMembers |
|
||||
| `ScadaLink:Database` | `DatabaseOptions` | Host | Central: ConfigurationDb, MachineDataDb connection strings; Site: SQLite paths |
|
||||
|
||||
@@ -79,6 +79,7 @@ Before the Akka.NET actor system is created, the Host must validate all required
|
||||
- `NodeConfiguration.Role` must be a valid `NodeRole` value.
|
||||
- `NodeConfiguration.NodeHostname` must not be null or empty.
|
||||
- `NodeConfiguration.RemotingPort` must be in valid port range (1–65535).
|
||||
- Site nodes must have `GrpcPort` in valid port range (1–65535) and different from `RemotingPort`.
|
||||
- Site nodes must have a non-empty `SiteId`.
|
||||
- Central nodes must have non-empty `ConfigurationDb` and `MachineDataDb` connection strings.
|
||||
- Site nodes must have non-empty SQLite path values. Site nodes do **not** require a `ConfigurationDb` connection string — all configuration is received via artifact deployment and read from local SQLite.
|
||||
@@ -112,14 +113,24 @@ The Host must configure the Akka.NET actor system using Akka.Hosting with:
|
||||
|
||||
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the ManagementActor with it. This allows external processes (e.g., the CLI) to discover and communicate with the ManagementActor via ClusterClient without joining the cluster as full members. The receptionist is started as part of the Akka.NET bootstrap (REQ-HOST-6) on central nodes only.
|
||||
|
||||
### REQ-HOST-7: ASP.NET Web Endpoints (Central Only)
|
||||
### REQ-HOST-7: ASP.NET Web Endpoints
|
||||
|
||||
On central nodes, the Host must use `WebApplication.CreateBuilder` to produce a full ASP.NET Core host with Kestrel, and must map web endpoints for:
|
||||
|
||||
- Central UI (via `MapCentralUI()` extension method).
|
||||
- Inbound API (via `MapInboundAPI()` extension method).
|
||||
|
||||
On site nodes, the Host must use `Host.CreateDefaultBuilder` to produce a generic `IHost` — **not** a `WebApplication`. This ensures no Kestrel server is started, no HTTP port is opened, and no web endpoint or middleware pipeline is configured. Site nodes are headless and must never accept inbound HTTP connections.
|
||||
On site nodes, the Host must also use `WebApplication.CreateBuilder` (not `Host.CreateDefaultBuilder`) to host the **SiteStreamGrpcServer** via Kestrel HTTP/2 on the configured `GrpcPort` (default 8083). Kestrel is configured with `HttpProtocols.Http2` on the gRPC port only — no HTTP/1.1 web endpoints are exposed. The gRPC service is mapped via `MapGrpcService<SiteStreamGrpcServer>()`.
|
||||
|
||||
**Startup ordering (site nodes)**:
|
||||
1. Actor system and SiteStreamManager must be initialized before gRPC begins accepting connections.
|
||||
2. The gRPC server rejects streams with `StatusCode.Unavailable` until the actor system is ready.
|
||||
|
||||
**Shutdown ordering (site nodes)**:
|
||||
1. On `CoordinatedShutdown`, stop accepting new gRPC streams first.
|
||||
2. Cancel all active gRPC streams (triggering client-side reconnect).
|
||||
3. Tear down actors.
|
||||
4. Use `IHostApplicationLifetime.ApplicationStopping` to signal the gRPC server.
|
||||
|
||||
### REQ-HOST-8: Structured Logging
|
||||
|
||||
|
||||
@@ -113,7 +113,7 @@ Deployment Manager Singleton (Cluster Singleton)
|
||||
|
||||
### Debug View Support
|
||||
- On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states.
|
||||
- Subsequent changes are delivered via the site-wide Akka stream, filtered by instance unique name.
|
||||
- Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
|
||||
- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established.
|
||||
|
||||
### Supervision Strategy
|
||||
@@ -280,10 +280,16 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
- Script Execution Actors may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
|
||||
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.
|
||||
|
||||
## SiteStreamManager and gRPC Integration
|
||||
|
||||
- The `SiteStreamManager` implements the `ISiteStreamSubscriber` interface, allowing the Communication Layer's `SiteStreamGrpcServer` to subscribe to the stream for cross-cluster delivery via gRPC.
|
||||
- When a gRPC `SubscribeInstance` call arrives, the `SiteStreamGrpcServer` creates a `StreamRelayActor` and subscribes it to `SiteStreamManager` for the requested instance. Events flow from `SiteStreamManager` → `StreamRelayActor` → `Channel<SiteStreamEvent>` → gRPC response stream to central.
|
||||
- The `SiteStreamManager` filters events by instance unique name and forwards matching events to all registered subscribers (both local debug consumers and gRPC relay actors).
|
||||
|
||||
## Site-Wide Stream Backpressure
|
||||
|
||||
- The site-wide Akka stream uses **per-subscriber buffering** with bounded buffers. Each subscriber (debug view, future consumers) gets an independent buffer.
|
||||
- If a subscriber falls behind (e.g., slow network on debug view), its buffer fills and oldest events are dropped. This does not affect other subscribers or the publishing Instance Actors.
|
||||
- The site-wide Akka stream uses **per-subscriber buffering** with bounded buffers. Each subscriber (gRPC relay actors, future consumers) gets an independent buffer.
|
||||
- If a subscriber falls behind (e.g., slow network on gRPC stream), its buffer fills and oldest events are dropped. This does not affect other subscribers or the publishing Instance Actors.
|
||||
- Instance Actors publish to the stream with **fire-and-forget** semantics — publishing never blocks the actor.
|
||||
|
||||
## Error Handling
|
||||
|
||||
@@ -45,11 +45,13 @@
|
||||
- **Machine Data Database**: A separate database for collected machine data (e.g., telemetry, measurements, events).
|
||||
|
||||
### 2.2 Communication: Central ↔ Site
|
||||
- Central-to-site and site-to-central communication uses **Akka.NET ClusterClient/ClusterClientReceptionist** for cross-cluster messaging with automatic failover.
|
||||
- **Site addressing**: Site Akka base addresses (NodeA and NodeB) are stored in the **Sites database table** and configured via the Central UI. Central creates a ClusterClient per site using both addresses as contact points (cached in memory, refreshed periodically and on admin changes) rather than relying on runtime registration messages from sites.
|
||||
- Two transport layers are used for central-site communication:
|
||||
- **Akka.NET ClusterClient/ClusterClientReceptionist**: Handles **command/control** messaging — deployments, instance lifecycle commands, subscribe/unsubscribe handshake, debug snapshots, health reports, remote queries, and integration routing. Provides automatic failover between contact points.
|
||||
- **gRPC server-streaming (site→central)**: Handles **real-time data streaming** — attribute value updates and alarm state changes. Each site node hosts a **SiteStreamGrpcServer** on a dedicated HTTP/2 port (Kestrel, default port 8083). Central creates per-site **SiteStreamGrpcClient** instances to subscribe to site streams. gRPC provides HTTP/2 flow control and per-stream backpressure that ClusterClient lacks.
|
||||
- **Site addressing**: Site Akka base addresses (NodeA and NodeB) and gRPC endpoints (GrpcNodeAAddress and GrpcNodeBAddress) are stored in the **Sites database table** and configured via the Central UI or CLI. Central creates a ClusterClient per site using both Akka addresses as contact points, and per-site gRPC clients using the gRPC addresses.
|
||||
- **Central contact points**: Sites configure **multiple central contact points** (both central node addresses) for redundancy. ClusterClient handles failover between central nodes automatically.
|
||||
- **Central as integration hub**: Central brokers requests between external systems and sites. For example, a recipe manager sends a recipe to central, which routes it to the appropriate site. MES requests machine values from central, which routes the request to the site and returns the response.
|
||||
- **Real-time data streaming** is not continuous for all machine data. The only real-time stream is an **on-demand debug view** — an engineer in the central UI can open a live view of a specific instance's tag values and alarm states for troubleshooting purposes. This is session-based and temporary. The debug view subscribes to the site-wide Akka stream filtered by instance (see Section 8.1).
|
||||
- **Real-time data streaming** is not continuous for all machine data. The only real-time stream is an **on-demand debug view** — an engineer in the central UI can open a live view of a specific instance's tag values and alarm states for troubleshooting purposes. This is session-based and temporary. The debug view subscribes via gRPC to the site's SiteStreamManager filtered by instance (see Section 8.1).
|
||||
|
||||
### 2.3 Site-Level Storage & Interface
|
||||
- Sites have **no user interface** — they are headless collectors, forwarders, and script executors.
|
||||
@@ -362,7 +364,7 @@ The central cluster hosts a **configuration and management UI** (no live machine
|
||||
- **Database Connection Management**: Define named database connections for script use.
|
||||
- **Inbound API Management**: Manage API keys (create, enable/disable, delete). Define API methods (name, parameters, return values, approved keys, implementation script). *(Admin role for keys, Design role for methods.)*
|
||||
- **Instance Management**: Create instances from templates, bind data connections (per-attribute, with **bulk assignment** UI for selecting multiple attributes and assigning a data connection at once), set instance-level attribute overrides, assign instances to areas. **Disable** or **delete** instances.
|
||||
- **Site & Data Connection Management**: Define sites (including optional NodeAAddress and NodeBAddress fields for Akka remoting paths), manage data connections and assign them to sites.
|
||||
- **Site & Data Connection Management**: Define sites (including optional NodeAAddress and NodeBAddress fields for Akka remoting paths, and optional GrpcNodeAAddress and GrpcNodeBAddress fields for gRPC streaming endpoints), manage data connections and assign them to sites.
|
||||
- **Area Management**: Define hierarchical area structures per site for organizing instances.
|
||||
- **Deployment**: View diffs between deployed and current template-derived configurations, deploy updates to individual instances. Filter instances by area. Pre-deployment validation runs automatically before any deployment is sent.
|
||||
- **System-Wide Artifact Deployment**: Explicitly deploy shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration to all sites or to an individual site (requires Deployment role). Per-site deployment is available via the Sites admin page.
|
||||
@@ -373,7 +375,7 @@ The central cluster hosts a **configuration and management UI** (no live machine
|
||||
- **Site Event Log Viewer**: Query and view operational event logs from site clusters (see Section 12).
|
||||
|
||||
### 8.1 Debug View
|
||||
- **Subscribe-on-demand**: When an engineer opens a debug view for an instance, central subscribes to the **site-wide Akka stream** filtered by instance unique name. The site first provides a **snapshot** of all current attribute values and alarm states from the Instance Actor, then streams subsequent changes from the Akka stream.
|
||||
- **Subscribe-on-demand**: When an engineer opens a debug view for an instance, central opens a **gRPC server-streaming subscription** to the site's `SiteStreamGrpcServer` for the instance, then requests a **snapshot** of all current attribute values and alarm states via ClusterClient. The gRPC stream delivers subsequent attribute value and alarm state changes directly from the site's `SiteStreamManager`.
|
||||
- Attribute value stream messages are structured as: `[InstanceUniqueName].[AttributePath].[AttributeName]`, attribute value, attribute quality, attribute change timestamp.
|
||||
- Alarm state stream messages are structured as: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.
|
||||
- The stream continues until the engineer **closes the debug view**, at which point central unsubscribes and the site stops streaming.
|
||||
|
||||
@@ -33,6 +33,18 @@ public class CommunicationOptions
|
||||
/// </summary>
|
||||
public List<string> CentralContactPoints { get; set; } = new();
|
||||
|
||||
/// <summary>gRPC keepalive ping interval for streaming connections.</summary>
|
||||
public TimeSpan GrpcKeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(15);
|
||||
|
||||
/// <summary>gRPC keepalive ping timeout — stream is considered dead if no response within this period.</summary>
|
||||
public TimeSpan GrpcKeepAlivePingTimeout { get; set; } = TimeSpan.FromSeconds(10);
|
||||
|
||||
/// <summary>Maximum lifetime for a single gRPC stream before the server forces re-establishment.</summary>
|
||||
public TimeSpan GrpcMaxStreamLifetime { get; set; } = TimeSpan.FromHours(4);
|
||||
|
||||
/// <summary>Maximum number of concurrent gRPC streaming subscriptions per site node.</summary>
|
||||
public int GrpcMaxConcurrentStreams { get; set; } = 100;
|
||||
|
||||
/// <summary>Akka.Remote transport heartbeat interval.</summary>
|
||||
public TimeSpan TransportHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ namespace ScadaLink.Communication.Grpc;
|
||||
public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
{
|
||||
private readonly ISiteStreamSubscriber _streamSubscriber;
|
||||
private readonly ActorSystem _actorSystem;
|
||||
private ActorSystem? _actorSystem;
|
||||
private readonly ILogger<SiteStreamGrpcServer> _logger;
|
||||
private readonly ConcurrentDictionary<string, StreamEntry> _activeStreams = new();
|
||||
private readonly int _maxConcurrentStreams;
|
||||
@@ -24,21 +24,25 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
|
||||
public SiteStreamGrpcServer(
|
||||
ISiteStreamSubscriber streamSubscriber,
|
||||
ActorSystem actorSystem,
|
||||
ILogger<SiteStreamGrpcServer> logger,
|
||||
int maxConcurrentStreams = 100)
|
||||
{
|
||||
_streamSubscriber = streamSubscriber;
|
||||
_actorSystem = actorSystem;
|
||||
_logger = logger;
|
||||
_maxConcurrentStreams = maxConcurrentStreams;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Marks the server as ready to accept subscriptions.
|
||||
/// Called after the site runtime is fully initialized.
|
||||
/// Marks the server as ready to accept subscriptions and injects the ActorSystem.
|
||||
/// Called after the site runtime actor system is fully initialized.
|
||||
/// The ActorSystem is set here rather than via the constructor so that
|
||||
/// the gRPC server can be created by DI before the actor system exists.
|
||||
/// </summary>
|
||||
public void SetReady() => _ready = true;
|
||||
public void SetReady(ActorSystem actorSystem)
|
||||
{
|
||||
_actorSystem = actorSystem;
|
||||
_ready = true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Number of currently active streaming subscriptions. Exposed for diagnostics.
|
||||
@@ -72,7 +76,7 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
new BoundedChannelOptions(1000) { FullMode = BoundedChannelFullMode.DropOldest });
|
||||
|
||||
var actorSeq = Interlocked.Increment(ref _actorCounter);
|
||||
var relayActor = _actorSystem.ActorOf(
|
||||
var relayActor = _actorSystem!.ActorOf(
|
||||
Props.Create(typeof(Actors.StreamRelayActor), request.CorrelationId, channel.Writer),
|
||||
$"stream-relay-{request.CorrelationId}-{actorSeq}");
|
||||
|
||||
@@ -96,7 +100,7 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
finally
|
||||
{
|
||||
_streamSubscriber.RemoveSubscriber(relayActor);
|
||||
_actorSystem.Stop(relayActor);
|
||||
_actorSystem!.Stop(relayActor);
|
||||
channel.Writer.TryComplete();
|
||||
|
||||
// Only remove our own entry -- a replacement stream may have already taken the slot
|
||||
|
||||
@@ -216,7 +216,8 @@ akka {{
|
||||
var storage = _serviceProvider.GetRequiredService<SiteStorageService>();
|
||||
var compilationService = _serviceProvider.GetRequiredService<ScriptCompilationService>();
|
||||
var sharedScriptLibrary = _serviceProvider.GetRequiredService<SharedScriptLibrary>();
|
||||
var streamManager = _serviceProvider.GetService<SiteStreamManager>();
|
||||
var streamManager = _serviceProvider.GetRequiredService<SiteStreamManager>();
|
||||
streamManager.Initialize(_actorSystem!);
|
||||
var siteRuntimeOptionsValue = _serviceProvider.GetService<IOptions<SiteRuntimeOptions>>()?.Value
|
||||
?? new SiteRuntimeOptions();
|
||||
var dmLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
|
||||
@@ -325,5 +326,9 @@ akka {{
|
||||
"Created ClusterClient to central with {Count} contact point(s) for site {SiteId}",
|
||||
contacts.Count, _nodeOptions.SiteId);
|
||||
}
|
||||
|
||||
// Gate gRPC subscriptions until the actor system and SiteStreamManager are initialized
|
||||
var grpcServer = _serviceProvider.GetService<ScadaLink.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
grpcServer?.SetReady(_actorSystem!);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -156,22 +156,40 @@ try
|
||||
}
|
||||
else if (nodeRole.Equals("Site", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
var builder = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder(args);
|
||||
builder.ConfigureAppConfiguration(config => config.AddConfiguration(configuration));
|
||||
var builder = WebApplication.CreateBuilder(args);
|
||||
builder.Configuration.AddConfiguration(configuration);
|
||||
|
||||
// WP-14: Serilog
|
||||
builder.UseSerilog();
|
||||
builder.Host.UseSerilog();
|
||||
|
||||
// WP-17: Windows Service support (no-op when not running as a Windows Service)
|
||||
builder.UseWindowsService();
|
||||
builder.Host.UseWindowsService();
|
||||
|
||||
builder.ConfigureServices((context, services) =>
|
||||
// Read GrpcPort from config (NodeOptions already has default 8083)
|
||||
var grpcPort = configuration.GetValue<int>("ScadaLink:Node:GrpcPort", 8083);
|
||||
|
||||
// Configure Kestrel for HTTP/2 only on the gRPC port
|
||||
builder.WebHost.ConfigureKestrel(options =>
|
||||
{
|
||||
SiteServiceRegistration.Configure(services, context.Configuration);
|
||||
options.ListenAnyIP(grpcPort, listenOptions =>
|
||||
{
|
||||
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2;
|
||||
});
|
||||
});
|
||||
|
||||
var host = builder.Build();
|
||||
await host.RunAsync();
|
||||
// gRPC server registration
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddSingleton<ScadaLink.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
|
||||
// Existing site service registrations
|
||||
SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
|
||||
|
||||
var app = builder.Build();
|
||||
|
||||
// Map gRPC service — resolves the singleton SiteStreamGrpcServer from DI
|
||||
app.MapGrpcService<ScadaLink.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
|
||||
await app.RunAsync();
|
||||
}
|
||||
else
|
||||
{
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
</PackageReference>
|
||||
<PackageReference Include="Grpc.AspNetCore" Version="2.71.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Hosting.WindowsServices" Version="10.0.5" />
|
||||
<PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
|
||||
<PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../ScadaLink.Commons/ScadaLink.Commons.csproj" />
|
||||
<ProjectReference Include="../ScadaLink.Communication/ScadaLink.Communication.csproj" />
|
||||
<ProjectReference Include="../ScadaLink.HealthMonitoring/ScadaLink.HealthMonitoring.csproj" />
|
||||
<ProjectReference Include="../ScadaLink.StoreAndForward/ScadaLink.StoreAndForward.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ScadaLink.Communication.Grpc;
|
||||
using ScadaLink.Commons.Interfaces.Repositories;
|
||||
using ScadaLink.SiteRuntime.Persistence;
|
||||
using ScadaLink.SiteRuntime.Repositories;
|
||||
using ScadaLink.SiteRuntime.Scripts;
|
||||
using ScadaLink.SiteRuntime.Streaming;
|
||||
|
||||
namespace ScadaLink.SiteRuntime;
|
||||
|
||||
@@ -40,6 +43,17 @@ public static class ServiceCollectionExtensions
|
||||
// WP-17: Shared script library
|
||||
services.AddSingleton<SharedScriptLibrary>();
|
||||
|
||||
// WP-23: Site stream manager — registered as singleton and exposed as ISiteStreamSubscriber
|
||||
// so the gRPC server can subscribe relay actors to instance events.
|
||||
// ActorSystem is injected later via Initialize() after AkkaHostedService starts.
|
||||
services.AddSingleton(sp =>
|
||||
{
|
||||
var options = sp.GetRequiredService<IOptions<SiteRuntimeOptions>>().Value;
|
||||
var logger = sp.GetRequiredService<ILogger<SiteStreamManager>>();
|
||||
return new SiteStreamManager(options, logger);
|
||||
});
|
||||
services.AddSingleton<ISiteStreamSubscriber>(sp => sp.GetRequiredService<SiteStreamManager>());
|
||||
|
||||
// Site-local repository implementations backed by SQLite
|
||||
services.AddScoped<IExternalSystemRepository, SiteExternalSystemRepository>();
|
||||
services.AddScoped<INotificationRepository, SiteNotificationRepository>();
|
||||
|
||||
@@ -3,6 +3,7 @@ using Akka.Actor;
|
||||
using Akka.Streams;
|
||||
using Akka.Streams.Dsl;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ScadaLink.Communication.Grpc;
|
||||
using ScadaLink.Commons.Messages.Streaming;
|
||||
|
||||
namespace ScadaLink.SiteRuntime.Streaming;
|
||||
@@ -13,10 +14,12 @@ namespace ScadaLink.SiteRuntime.Streaming;
|
||||
/// Subscribers get per-subscriber bounded buffers with drop-oldest overflow.
|
||||
///
|
||||
/// Filterable by instance name for debug view (WP-25).
|
||||
/// Implements ISiteStreamSubscriber so the gRPC server can subscribe actors
|
||||
/// to instance events without referencing SiteRuntime directly.
|
||||
/// </summary>
|
||||
public class SiteStreamManager
|
||||
public class SiteStreamManager : ISiteStreamSubscriber
|
||||
{
|
||||
private readonly ActorSystem _system;
|
||||
private ActorSystem? _system;
|
||||
private readonly int _bufferSize;
|
||||
private readonly ILogger<SiteStreamManager> _logger;
|
||||
private readonly object _lock = new();
|
||||
@@ -25,20 +28,21 @@ public class SiteStreamManager
|
||||
private readonly Dictionary<string, SubscriptionInfo> _subscriptions = new();
|
||||
|
||||
public SiteStreamManager(
|
||||
ActorSystem system,
|
||||
SiteRuntimeOptions options,
|
||||
ILogger<SiteStreamManager> logger)
|
||||
{
|
||||
_system = system;
|
||||
_bufferSize = options.StreamBufferSize;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes the stream source. Must be called after ActorSystem is ready.
|
||||
/// The ActorSystem is passed here rather than via the constructor so that
|
||||
/// SiteStreamManager can be created by DI before the actor system exists.
|
||||
/// </summary>
|
||||
public void Initialize()
|
||||
public void Initialize(ActorSystem system)
|
||||
{
|
||||
_system = system;
|
||||
var materializer = _system.Materializer();
|
||||
|
||||
var source = Source.ActorRef<ISiteStreamEvent>(
|
||||
|
||||
@@ -24,7 +24,7 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
|
||||
private SiteStreamGrpcServer CreateServer(int maxStreams = 100)
|
||||
{
|
||||
return new SiteStreamGrpcServer(_subscriber, Sys, _logger, maxStreams);
|
||||
return new SiteStreamGrpcServer(_subscriber, _logger, maxStreams);
|
||||
}
|
||||
|
||||
private static InstanceStreamRequest MakeRequest(string correlationId = "corr-1", string instance = "Site1.Pump01")
|
||||
@@ -55,7 +55,7 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
public async Task RejectsWhenMaxStreamsReached()
|
||||
{
|
||||
var server = CreateServer(maxStreams: 1);
|
||||
server.SetReady();
|
||||
server.SetReady(Sys);
|
||||
|
||||
// Start one stream that blocks
|
||||
var cts1 = new CancellationTokenSource();
|
||||
@@ -86,7 +86,7 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
public async Task CancelsDuplicateCorrelationId()
|
||||
{
|
||||
var server = CreateServer();
|
||||
server.SetReady();
|
||||
server.SetReady(Sys);
|
||||
|
||||
var cts1 = new CancellationTokenSource();
|
||||
var context1 = CreateMockContext(cts1.Token);
|
||||
@@ -121,7 +121,7 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
public async Task CleansUpOnCancellation()
|
||||
{
|
||||
var server = CreateServer();
|
||||
server.SetReady();
|
||||
server.SetReady(Sys);
|
||||
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
@@ -142,7 +142,7 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
public async Task SubscribesAndRemovesFromStreamManager()
|
||||
{
|
||||
var server = CreateServer();
|
||||
server.SetReady();
|
||||
server.SetReady(Sys);
|
||||
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
@@ -167,7 +167,7 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
public async Task WritesEventsToResponseStream()
|
||||
{
|
||||
var server = CreateServer();
|
||||
server.SetReady();
|
||||
server.SetReady(Sys);
|
||||
|
||||
// Capture the relay actor so we can send it events
|
||||
IActorRef? capturedActor = null;
|
||||
@@ -212,7 +212,7 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
{
|
||||
var server = CreateServer();
|
||||
// Initially not ready -- just verify the property works
|
||||
server.SetReady();
|
||||
server.SetReady(Sys);
|
||||
// No assertion needed -- the other tests verify that SetReady enables streaming
|
||||
Assert.Equal(0, server.ActiveStreamCount);
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Components.Authorization;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
@@ -21,10 +22,12 @@ using ScadaLink.ManagementService;
|
||||
using ScadaLink.NotificationService;
|
||||
using ScadaLink.Security;
|
||||
using ScadaLink.SiteEventLogging;
|
||||
using ScadaLink.Communication.Grpc;
|
||||
using ScadaLink.SiteRuntime;
|
||||
using ScadaLink.SiteRuntime.Persistence;
|
||||
using ScadaLink.SiteRuntime.Repositories;
|
||||
using ScadaLink.SiteRuntime.Scripts;
|
||||
using ScadaLink.SiteRuntime.Streaming;
|
||||
using ScadaLink.StoreAndForward;
|
||||
using ScadaLink.TemplateEngine;
|
||||
using ScadaLink.TemplateEngine.Flattening;
|
||||
@@ -274,45 +277,45 @@ public class CentralCompositionRootTests : IDisposable
|
||||
/// <summary>
|
||||
/// Verifies every expected DI service resolves from the Site composition root.
|
||||
/// Uses the extracted SiteServiceRegistration.Configure() so the test always
|
||||
/// matches the real Program.cs registration.
|
||||
/// matches the real Program.cs registration (WebApplicationBuilder + gRPC).
|
||||
/// </summary>
|
||||
public class SiteCompositionRootTests : IDisposable
|
||||
{
|
||||
private readonly IHost _host;
|
||||
private readonly WebApplication _host;
|
||||
private readonly string _tempDbPath;
|
||||
|
||||
public SiteCompositionRootTests()
|
||||
{
|
||||
_tempDbPath = Path.Combine(Path.GetTempPath(), $"scadalink_test_{Guid.NewGuid()}.db");
|
||||
|
||||
var builder = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder();
|
||||
builder.ConfigureAppConfiguration(config =>
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.Sources.Clear();
|
||||
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
config.Sources.Clear();
|
||||
config.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["ScadaLink:Node:Role"] = "Site",
|
||||
["ScadaLink:Node:NodeHostname"] = "test-site",
|
||||
["ScadaLink:Node:SiteId"] = "TestSite",
|
||||
["ScadaLink:Node:RemotingPort"] = "0",
|
||||
["ScadaLink:Database:SiteDbPath"] = _tempDbPath,
|
||||
});
|
||||
["ScadaLink:Node:Role"] = "Site",
|
||||
["ScadaLink:Node:NodeHostname"] = "test-site",
|
||||
["ScadaLink:Node:SiteId"] = "TestSite",
|
||||
["ScadaLink:Node:RemotingPort"] = "0",
|
||||
["ScadaLink:Node:GrpcPort"] = "0",
|
||||
["ScadaLink:Database:SiteDbPath"] = _tempDbPath,
|
||||
});
|
||||
builder.ConfigureServices((context, services) =>
|
||||
{
|
||||
SiteServiceRegistration.Configure(services, context.Configuration);
|
||||
|
||||
// Keep AkkaHostedService in DI (other services depend on it)
|
||||
// but prevent it from starting by removing only its IHostedService registration.
|
||||
AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(services);
|
||||
});
|
||||
// gRPC server registration (mirrors Program.cs site section)
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddSingleton<ScadaLink.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
|
||||
SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
|
||||
|
||||
// Keep AkkaHostedService in DI (other services depend on it)
|
||||
// but prevent it from starting by removing only its IHostedService registration.
|
||||
AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services);
|
||||
|
||||
_host = builder.Build();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_host.Dispose();
|
||||
(_host as IDisposable)?.Dispose();
|
||||
try { File.Delete(_tempDbPath); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
@@ -333,6 +336,9 @@ public class SiteCompositionRootTests : IDisposable
|
||||
new object[] { typeof(SiteStorageService) },
|
||||
new object[] { typeof(ScriptCompilationService) },
|
||||
new object[] { typeof(SharedScriptLibrary) },
|
||||
new object[] { typeof(SiteStreamManager) },
|
||||
new object[] { typeof(ISiteStreamSubscriber) },
|
||||
new object[] { typeof(SiteStreamGrpcServer) },
|
||||
new object[] { typeof(IDataConnectionFactory) },
|
||||
new object[] { typeof(StoreAndForwardStorage) },
|
||||
new object[] { typeof(StoreAndForwardService) },
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using System.Reflection;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
@@ -65,70 +67,74 @@ public class HostStartupTests : IDisposable
|
||||
[Fact]
|
||||
public void SiteRole_StartsWithoutError()
|
||||
{
|
||||
var builder = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder();
|
||||
builder.ConfigureAppConfiguration(config =>
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.Sources.Clear();
|
||||
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
config.Sources.Clear();
|
||||
config.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["ScadaLink:Node:Role"] = "Site",
|
||||
["ScadaLink:Node:NodeHostname"] = "test-site",
|
||||
["ScadaLink:Node:SiteId"] = "TestSite",
|
||||
["ScadaLink:Node:RemotingPort"] = "0",
|
||||
});
|
||||
});
|
||||
builder.ConfigureServices((context, services) =>
|
||||
{
|
||||
SiteServiceRegistration.Configure(services, context.Configuration);
|
||||
["ScadaLink:Node:Role"] = "Site",
|
||||
["ScadaLink:Node:NodeHostname"] = "test-site",
|
||||
["ScadaLink:Node:SiteId"] = "TestSite",
|
||||
["ScadaLink:Node:RemotingPort"] = "0",
|
||||
["ScadaLink:Node:GrpcPort"] = "0",
|
||||
});
|
||||
|
||||
var host = builder.Build();
|
||||
_disposables.Add(host);
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddSingleton<ScadaLink.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
|
||||
|
||||
// Remove AkkaHostedService from running
|
||||
AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services);
|
||||
|
||||
var app = builder.Build();
|
||||
_disposables.Add(app);
|
||||
|
||||
// Build succeeds = DI container is valid and all services resolve
|
||||
Assert.NotNull(host);
|
||||
Assert.NotNull(host.Services);
|
||||
Assert.NotNull(app);
|
||||
Assert.NotNull(app.Services);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SiteRole_DoesNotConfigureKestrel()
|
||||
public void SiteRole_ConfiguresKestrelForGrpc()
|
||||
{
|
||||
var builder = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder();
|
||||
builder.ConfigureAppConfiguration(config =>
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.Sources.Clear();
|
||||
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
config.Sources.Clear();
|
||||
config.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
["ScadaLink:Node:Role"] = "Site",
|
||||
["ScadaLink:Node:NodeHostname"] = "test-site",
|
||||
["ScadaLink:Node:SiteId"] = "TestSite",
|
||||
["ScadaLink:Node:RemotingPort"] = "0",
|
||||
["ScadaLink:Node:GrpcPort"] = "0",
|
||||
});
|
||||
|
||||
builder.WebHost.ConfigureKestrel(options =>
|
||||
{
|
||||
options.ListenAnyIP(0, listenOptions =>
|
||||
{
|
||||
["ScadaLink:Node:Role"] = "Site",
|
||||
["ScadaLink:Node:NodeHostname"] = "test-site",
|
||||
["ScadaLink:Node:SiteId"] = "TestSite",
|
||||
["ScadaLink:Node:RemotingPort"] = "0",
|
||||
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2;
|
||||
});
|
||||
});
|
||||
builder.ConfigureServices((context, services) =>
|
||||
{
|
||||
SiteServiceRegistration.Configure(services, context.Configuration);
|
||||
});
|
||||
|
||||
var host = builder.Build();
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddSingleton<ScadaLink.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
|
||||
|
||||
// Verify no Kestrel server or web host is registered.
|
||||
// Host.CreateDefaultBuilder does not add Kestrel, so there should be no IServer.
|
||||
// Remove AkkaHostedService from running
|
||||
AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services);
|
||||
|
||||
var app = builder.Build();
|
||||
|
||||
// Verify Kestrel IS configured (site now hosts gRPC via WebApplicationBuilder)
|
||||
var serverType = Type.GetType(
|
||||
"Microsoft.AspNetCore.Hosting.Server.IServer, Microsoft.AspNetCore.Hosting.Server.Abstractions");
|
||||
|
||||
if (serverType != null)
|
||||
{
|
||||
var server = host.Services.GetService(serverType);
|
||||
Assert.Null(server);
|
||||
var server = app.Services.GetService(serverType);
|
||||
Assert.NotNull(server);
|
||||
}
|
||||
|
||||
// Additionally verify no HTTP URLs are configured
|
||||
var config = host.Services.GetRequiredService<IConfiguration>();
|
||||
var urls = config["urls"] ?? config["ASPNETCORE_URLS"];
|
||||
Assert.Null(urls);
|
||||
|
||||
host.Dispose();
|
||||
(app as IDisposable)?.Dispose();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
using System.Threading.Channels;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using NSubstitute;
|
||||
using ScadaLink.Commons.Messages.Streaming;
|
||||
using ScadaLink.Communication.Actors;
|
||||
using ScadaLink.Communication.Grpc;
|
||||
|
||||
namespace ScadaLink.IntegrationTests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Integration tests for the gRPC streaming pipeline.
|
||||
/// Tests the full in-process flow: subscribe request -> StreamRelayActor creation ->
|
||||
/// domain events via Akka Tell -> Channel relay -> gRPC response stream writes.
|
||||
///
|
||||
/// These tests exercise the real SiteStreamGrpcServer, StreamRelayActor, and Channel
|
||||
/// wiring together with a real Akka actor system, using only mocked gRPC transport
|
||||
/// (IServerStreamWriter + ServerCallContext).
|
||||
///
|
||||
/// Full end-to-end gRPC-over-HTTP/2 tests are performed manually against the Docker
|
||||
/// cluster (docker/deploy.sh + docker/seed-sites.sh + CLI debug-stream).
|
||||
/// </summary>
|
||||
public class GrpcStreamIntegrationTests : TestKit
|
||||
{
|
||||
/// <summary>
|
||||
/// End-to-end pipeline test: subscribe -> relay actor receives domain events ->
|
||||
/// events flow through Channel to gRPC response stream.
|
||||
/// Validates attribute value changes arrive with correct protobuf mapping.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Pipeline_AttributeValueChanged_FlowsToResponseStream()
|
||||
{
|
||||
// Arrange: capture the relay actor created by the gRPC server
|
||||
IActorRef? relayActor = null;
|
||||
var subscriber = Substitute.For<ISiteStreamSubscriber>();
|
||||
subscriber.Subscribe(Arg.Any<string>(), Arg.Any<IActorRef>())
|
||||
.Returns(ci =>
|
||||
{
|
||||
relayActor = ci.Arg<IActorRef>();
|
||||
return "sub-integ-1";
|
||||
});
|
||||
|
||||
var server = new SiteStreamGrpcServer(
|
||||
subscriber,
|
||||
NullLogger<SiteStreamGrpcServer>.Instance);
|
||||
server.SetReady(Sys);
|
||||
|
||||
var writtenEvents = new List<SiteStreamEvent>();
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
writer.WriteAsync(Arg.Any<SiteStreamEvent>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask)
|
||||
.AndDoes(ci => writtenEvents.Add(ci.Arg<SiteStreamEvent>()));
|
||||
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
|
||||
var request = new InstanceStreamRequest
|
||||
{
|
||||
CorrelationId = "integ-attr-1",
|
||||
InstanceUniqueName = "SiteA.Pump01"
|
||||
};
|
||||
|
||||
// Act: start the subscription stream
|
||||
var streamTask = Task.Run(() => server.SubscribeInstance(request, writer, context));
|
||||
|
||||
await WaitForConditionAsync(() => relayActor != null);
|
||||
|
||||
// Send domain events to the relay actor (simulating what SiteStreamManager does)
|
||||
var ts1 = new DateTimeOffset(2026, 3, 21, 14, 0, 0, TimeSpan.Zero);
|
||||
var ts2 = new DateTimeOffset(2026, 3, 21, 14, 0, 1, TimeSpan.Zero);
|
||||
|
||||
relayActor!.Tell(new AttributeValueChanged(
|
||||
"SiteA.Pump01", "Modules.Flow", "CurrentGPM", 125.3, "Good", ts1));
|
||||
relayActor.Tell(new AttributeValueChanged(
|
||||
"SiteA.Pump01", "Modules.Pressure", "CurrentPSI", 48.7, "Uncertain", ts2));
|
||||
|
||||
await WaitForConditionAsync(() => writtenEvents.Count >= 2);
|
||||
|
||||
// Cleanup
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
|
||||
// Assert: both events arrived in order with correct protobuf mapping
|
||||
Assert.Equal(2, writtenEvents.Count);
|
||||
|
||||
var evt1 = writtenEvents[0];
|
||||
Assert.Equal("integ-attr-1", evt1.CorrelationId);
|
||||
Assert.Equal(SiteStreamEvent.EventOneofCase.AttributeChanged, evt1.EventCase);
|
||||
Assert.Equal("SiteA.Pump01", evt1.AttributeChanged.InstanceUniqueName);
|
||||
Assert.Equal("Modules.Flow", evt1.AttributeChanged.AttributePath);
|
||||
Assert.Equal("CurrentGPM", evt1.AttributeChanged.AttributeName);
|
||||
Assert.Equal("125.3", evt1.AttributeChanged.Value);
|
||||
Assert.Equal(Quality.Good, evt1.AttributeChanged.Quality);
|
||||
Assert.Equal(Timestamp.FromDateTimeOffset(ts1), evt1.AttributeChanged.Timestamp);
|
||||
|
||||
var evt2 = writtenEvents[1];
|
||||
Assert.Equal("integ-attr-1", evt2.CorrelationId);
|
||||
Assert.Equal("Modules.Pressure", evt2.AttributeChanged.AttributePath);
|
||||
Assert.Equal(Quality.Uncertain, evt2.AttributeChanged.Quality);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// End-to-end pipeline test for alarm state changes.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Pipeline_AlarmStateChanged_FlowsToResponseStream()
|
||||
{
|
||||
IActorRef? relayActor = null;
|
||||
var subscriber = Substitute.For<ISiteStreamSubscriber>();
|
||||
subscriber.Subscribe(Arg.Any<string>(), Arg.Any<IActorRef>())
|
||||
.Returns(ci =>
|
||||
{
|
||||
relayActor = ci.Arg<IActorRef>();
|
||||
return "sub-integ-2";
|
||||
});
|
||||
|
||||
var server = new SiteStreamGrpcServer(
|
||||
subscriber,
|
||||
NullLogger<SiteStreamGrpcServer>.Instance);
|
||||
server.SetReady(Sys);
|
||||
|
||||
var writtenEvents = new List<SiteStreamEvent>();
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
writer.WriteAsync(Arg.Any<SiteStreamEvent>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask)
|
||||
.AndDoes(ci => writtenEvents.Add(ci.Arg<SiteStreamEvent>()));
|
||||
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
|
||||
var request = new InstanceStreamRequest
|
||||
{
|
||||
CorrelationId = "integ-alarm-1",
|
||||
InstanceUniqueName = "SiteA.Pump01"
|
||||
};
|
||||
|
||||
var streamTask = Task.Run(() => server.SubscribeInstance(request, writer, context));
|
||||
await WaitForConditionAsync(() => relayActor != null);
|
||||
|
||||
var ts = new DateTimeOffset(2026, 3, 21, 14, 5, 0, TimeSpan.Zero);
|
||||
relayActor!.Tell(new AlarmStateChanged(
|
||||
"SiteA.Pump01", "HighPressure",
|
||||
Commons.Types.Enums.AlarmState.Active, 3, ts));
|
||||
|
||||
await WaitForConditionAsync(() => writtenEvents.Count >= 1);
|
||||
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
|
||||
Assert.Single(writtenEvents);
|
||||
var evt = writtenEvents[0];
|
||||
Assert.Equal("integ-alarm-1", evt.CorrelationId);
|
||||
Assert.Equal(SiteStreamEvent.EventOneofCase.AlarmChanged, evt.EventCase);
|
||||
Assert.Equal("SiteA.Pump01", evt.AlarmChanged.InstanceUniqueName);
|
||||
Assert.Equal("HighPressure", evt.AlarmChanged.AlarmName);
|
||||
Assert.Equal(AlarmStateEnum.AlarmStateActive, evt.AlarmChanged.State);
|
||||
Assert.Equal(3, evt.AlarmChanged.Priority);
|
||||
Assert.Equal(Timestamp.FromDateTimeOffset(ts), evt.AlarmChanged.Timestamp);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that mixed event types (attribute + alarm) flow through the same stream.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Pipeline_MixedEvents_FlowInOrder()
|
||||
{
|
||||
IActorRef? relayActor = null;
|
||||
var subscriber = Substitute.For<ISiteStreamSubscriber>();
|
||||
subscriber.Subscribe(Arg.Any<string>(), Arg.Any<IActorRef>())
|
||||
.Returns(ci =>
|
||||
{
|
||||
relayActor = ci.Arg<IActorRef>();
|
||||
return "sub-integ-3";
|
||||
});
|
||||
|
||||
var server = new SiteStreamGrpcServer(
|
||||
subscriber,
|
||||
NullLogger<SiteStreamGrpcServer>.Instance);
|
||||
server.SetReady(Sys);
|
||||
|
||||
var writtenEvents = new List<SiteStreamEvent>();
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
writer.WriteAsync(Arg.Any<SiteStreamEvent>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask)
|
||||
.AndDoes(ci => writtenEvents.Add(ci.Arg<SiteStreamEvent>()));
|
||||
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
|
||||
var request = new InstanceStreamRequest
|
||||
{
|
||||
CorrelationId = "integ-mixed-1",
|
||||
InstanceUniqueName = "SiteB.Motor01"
|
||||
};
|
||||
|
||||
var streamTask = Task.Run(() => server.SubscribeInstance(request, writer, context));
|
||||
await WaitForConditionAsync(() => relayActor != null);
|
||||
|
||||
var ts = DateTimeOffset.UtcNow;
|
||||
|
||||
// Send interleaved attribute and alarm events
|
||||
relayActor!.Tell(new AttributeValueChanged(
|
||||
"SiteB.Motor01", "Speed", "RPM", 1750.0, "Good", ts));
|
||||
relayActor.Tell(new AlarmStateChanged(
|
||||
"SiteB.Motor01", "OverSpeed",
|
||||
Commons.Types.Enums.AlarmState.Active, 4, ts));
|
||||
relayActor.Tell(new AttributeValueChanged(
|
||||
"SiteB.Motor01", "Temperature", "BearingTemp", 85.2, "Good", ts));
|
||||
|
||||
await WaitForConditionAsync(() => writtenEvents.Count >= 3);
|
||||
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
|
||||
Assert.Equal(3, writtenEvents.Count);
|
||||
Assert.Equal(SiteStreamEvent.EventOneofCase.AttributeChanged, writtenEvents[0].EventCase);
|
||||
Assert.Equal(SiteStreamEvent.EventOneofCase.AlarmChanged, writtenEvents[1].EventCase);
|
||||
Assert.Equal(SiteStreamEvent.EventOneofCase.AttributeChanged, writtenEvents[2].EventCase);
|
||||
Assert.All(writtenEvents, e => Assert.Equal("integ-mixed-1", e.CorrelationId));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that when a stream is cancelled, the subscriber is cleaned up
|
||||
/// and the active stream count returns to zero.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Pipeline_Cancellation_CleansUpRelayActorAndSubscription()
|
||||
{
|
||||
var subscriber = Substitute.For<ISiteStreamSubscriber>();
|
||||
subscriber.Subscribe(Arg.Any<string>(), Arg.Any<IActorRef>())
|
||||
.Returns("sub-integ-4");
|
||||
|
||||
var server = new SiteStreamGrpcServer(
|
||||
subscriber,
|
||||
NullLogger<SiteStreamGrpcServer>.Instance);
|
||||
server.SetReady(Sys);
|
||||
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
|
||||
var request = new InstanceStreamRequest
|
||||
{
|
||||
CorrelationId = "integ-cleanup-1",
|
||||
InstanceUniqueName = "SiteC.Valve01"
|
||||
};
|
||||
|
||||
var streamTask = Task.Run(() => server.SubscribeInstance(request, writer, context));
|
||||
await WaitForConditionAsync(() => server.ActiveStreamCount == 1);
|
||||
|
||||
// Cancel the stream
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
|
||||
// Verify cleanup
|
||||
Assert.Equal(0, server.ActiveStreamCount);
|
||||
subscriber.Received(1).Subscribe("SiteC.Valve01", Arg.Any<IActorRef>());
|
||||
subscriber.Received(1).RemoveSubscriber(Arg.Any<IActorRef>());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that a duplicate correlation ID cancels the first stream and
|
||||
/// the second stream continues to receive events.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Pipeline_DuplicateCorrelationId_ReplacesStream()
|
||||
{
|
||||
IActorRef? relayActor2 = null;
|
||||
var callCount = 0;
|
||||
var subscriber = Substitute.For<ISiteStreamSubscriber>();
|
||||
subscriber.Subscribe(Arg.Any<string>(), Arg.Any<IActorRef>())
|
||||
.Returns(ci =>
|
||||
{
|
||||
callCount++;
|
||||
if (callCount == 2)
|
||||
relayActor2 = ci.Arg<IActorRef>();
|
||||
return $"sub-dup-{callCount}";
|
||||
});
|
||||
|
||||
var server = new SiteStreamGrpcServer(
|
||||
subscriber,
|
||||
NullLogger<SiteStreamGrpcServer>.Instance);
|
||||
server.SetReady(Sys);
|
||||
|
||||
// First stream
|
||||
var writer1 = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
var cts1 = new CancellationTokenSource();
|
||||
var context1 = CreateMockContext(cts1.Token);
|
||||
|
||||
var stream1Task = Task.Run(() => server.SubscribeInstance(
|
||||
new InstanceStreamRequest { CorrelationId = "integ-dup", InstanceUniqueName = "SiteA.Pump01" },
|
||||
writer1, context1));
|
||||
|
||||
await WaitForConditionAsync(() => server.ActiveStreamCount == 1);
|
||||
|
||||
// Second stream with same correlation ID -- should cancel first
|
||||
var writtenEvents2 = new List<SiteStreamEvent>();
|
||||
var writer2 = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
writer2.WriteAsync(Arg.Any<SiteStreamEvent>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask)
|
||||
.AndDoes(ci => writtenEvents2.Add(ci.Arg<SiteStreamEvent>()));
|
||||
|
||||
var cts2 = new CancellationTokenSource();
|
||||
var context2 = CreateMockContext(cts2.Token);
|
||||
|
||||
var stream2Task = Task.Run(() => server.SubscribeInstance(
|
||||
new InstanceStreamRequest { CorrelationId = "integ-dup", InstanceUniqueName = "SiteA.Pump01" },
|
||||
writer2, context2));
|
||||
|
||||
// First stream should complete
|
||||
await stream1Task;
|
||||
await WaitForConditionAsync(() => relayActor2 != null);
|
||||
|
||||
// Send event to second relay
|
||||
var ts = DateTimeOffset.UtcNow;
|
||||
relayActor2!.Tell(new AttributeValueChanged(
|
||||
"SiteA.Pump01", "Flow", "GPM", 100.0, "Good", ts));
|
||||
|
||||
await WaitForConditionAsync(() => writtenEvents2.Count >= 1);
|
||||
|
||||
cts2.Cancel();
|
||||
await stream2Task;
|
||||
|
||||
Assert.Single(writtenEvents2);
|
||||
Assert.Equal("integ-dup", writtenEvents2[0].CorrelationId);
|
||||
}
|
||||
|
||||
private static ServerCallContext CreateMockContext(CancellationToken cancellationToken = default)
|
||||
{
|
||||
var context = Substitute.For<ServerCallContext>();
|
||||
context.CancellationToken.Returns(cancellationToken);
|
||||
return context;
|
||||
}
|
||||
|
||||
private static async Task WaitForConditionAsync(Func<bool> condition, int timeoutMs = 5000)
|
||||
{
|
||||
var deadline = DateTime.UtcNow.AddMilliseconds(timeoutMs);
|
||||
while (!condition() && DateTime.UtcNow < deadline)
|
||||
{
|
||||
await Task.Delay(25);
|
||||
}
|
||||
|
||||
Assert.True(condition(), $"Condition not met within {timeoutMs}ms");
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,7 @@
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Akka.TestKit.Xunit2" Version="1.5.62" />
|
||||
<PackageReference Include="coverlet.collector" Version="6.0.4" />
|
||||
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.5" />
|
||||
<PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.5" />
|
||||
|
||||
@@ -19,8 +19,8 @@ public class SiteStreamManagerTests : TestKit, IDisposable
|
||||
{
|
||||
var options = new SiteRuntimeOptions { StreamBufferSize = 100 };
|
||||
_streamManager = new SiteStreamManager(
|
||||
Sys, options, NullLogger<SiteStreamManager>.Instance);
|
||||
_streamManager.Initialize();
|
||||
options, NullLogger<SiteStreamManager>.Instance);
|
||||
_streamManager.Initialize(Sys);
|
||||
}
|
||||
|
||||
void IDisposable.Dispose()
|
||||
|
||||
Reference in New Issue
Block a user