4 Commits

Author SHA1 Message Date
Joseph Doherty f66dc031a4 fix(health): route site heartbeats into the aggregator
CentralCommunicationActor.HandleHeartbeat was forwarding each incoming
HeartbeatMessage to Context.Parent, which resolves to the /user
guardian — a non-actor. Every site heartbeat went straight to dead
letters (~1026 per central node per 30 minutes at the default ~2s
interval across three sites).

The aggregator now exposes MarkHeartbeat(siteId, receivedAt) which
bumps LastReportReceivedAt on already-known sites (and clears IsOnline
if it had flipped) without touching LatestReport. Heartbeats from
unregistered sites are dropped — first registration still happens on
the first full report. CentralCommunicationActor calls this in place
of the no-op Tell.

The result: heartbeats now serve their stated health-monitoring
purpose (per CLAUDE.md) by keeping a site marked online between the
30s full reports if a single report is briefly delayed, and the dead
letter noise disappears entirely.
2026-05-13 08:11:43 -04:00
Joseph Doherty 7bba48a14a feat(ui/monitoring): redesign Parked Messages page with filters, drawer, and bulk actions
Triage was painful on the old layout: a lone Site dropdown sat on a sparse
row, errors were truncated mid-sentence with a per-row View/Hide toggle
that on expand pushed an unwrapped <pre> through the table and shoved the
Actions column off-screen, all rows looked the same regardless of age or
attempt count, and OriginInstance — which tells you which instance
produced the failure — wasn't displayed at all even though the data was
on the entity.

This pass:

- Adds a real filter bar: Site, Category, Target system, Origin instance,
  Age window, free-text search. Category/Target/Origin/Age/Search filter
  the loaded page client-side; Site still drives the server query (and
  changing site now auto-queries — one fewer click).
- Replaces the in-table expansion with an Offcanvas detail drawer.
  Clicking a row slides in a side panel with full message ID + copy,
  category label, origin, attempts, both timestamps in relative + absolute
  form, the complete error (pre-wrap, scrollable), and big Retry / Discard
  buttons. The table never overflows.
- Stacks Target + Method into one column (target in semibold, method
  small/muted below) and surfaces Origin as a code-styled chip in a new
  column ("—" muted when null).
- Severity left-border on each row, derived client-side from
  AttemptCount/MaxAttempts and age of the last attempt: red when retries
  are exhausted and last attempt was in the past hour, amber when
  exhausted but stale, muted grey otherwise.
- Mini attempt progress bar under the n/max count, red when fully
  exhausted and amber while partial.
- Relative timestamps ("5m ago", "1h ago", "2d ago") with absolute UTC on
  hover via the title attribute — applies in both the table and the drawer.
- Bulk select: header checkbox selects the filtered set, per-row
  checkboxes. When ≥1 selected, a sticky action strip slides in below the
  filter bar offering Retry selected / Discard selected with the usual
  confirm dialog. Toast reports per-item success/failure counts.
- Summary line next to the title: "N parked · K target systems · oldest
  Xh ago" (and "(showing M of N)" when filters are active).
- ParkedMessageEntry contract extended additively with MaxAttempts,
  Category, and OriginInstance so the UI has the data it needs for
  severity, the category filter, and the new column.
- Bumped page size from 25 to 50 to better match the dense layout.
2026-05-13 08:05:22 -04:00
Joseph Doherty 1c2dc45803 feat(ui/api-methods): pick approved API keys when editing a method
The ApiMethod entity had an ApprovedApiKeyIds column and ApiKeyValidator
read it, but no UI/CLI/seed code ever wrote to it. Result: any inbound
POST /api/{method} was rejected with 403 "API key not approved for this
method" regardless of which key was sent.

Add an "Approved API Keys" subsection to the method form, between
Timeout and Parameters: vertical list of checkboxes, one per ApiKey
row (with a "Disabled" badge for disabled keys, and a link to
/admin/api-keys when none exist). OnInitializedAsync loads all keys and
parses the existing comma-separated IDs; Save() serializes the selected
set back to the entity on both create and edit paths.

Re-uses IInboundApiRepository.GetAllApiKeysAsync — no repo or migration
changes needed.
2026-05-13 07:12:44 -04:00
Joseph Doherty 1822e3c76f fix(store-and-forward): wire up parked-message handler and start S&F service on sites
The Parked Messages page returned "Parked message handler not available"
because no actor was ever registered for ParkedMessages, and Retry/Discard
requests had no Receive at all (would have hit deadletters). On top of
that, StoreAndForwardService.StartAsync() was never called anywhere, so
the sf_messages SQLite table was never created and the retry timer never
ran — silently breaking all of S&F.

- New ParkedMessageHandlerActor bridges StoreAndForwardService.{Get,Retry,Discard}
  using the Sender→Task→PipeTo pattern already used in DeploymentManagerActor.
- SiteCommunicationActor now routes ParkedMessageRetryRequest and
  ParkedMessageDiscardRequest the same way as the existing Query handler.
- AkkaHostedService.RegisterSiteActors() resolves StoreAndForwardService,
  calls StartAsync() to create the schema and start the timer, then
  creates and registers the handler actor.
2026-05-13 07:12:37 -04:00
12 changed files with 879 additions and 191 deletions
@@ -28,6 +28,40 @@
<label class="form-label">Timeout (seconds)</label>
<input type="number" class="form-control" @bind="_timeoutSeconds" min="1" />
</div>
<div class="mb-3">
<label class="form-label">Approved API Keys</label>
@if (_allKeys.Count == 0)
{
<div class="form-text">
No API keys configured.
<a href="/admin/api-keys">Create one</a> to authorize callers for this method.
</div>
}
else
{
<div class="border rounded p-2" style="max-height: 220px; overflow-y: auto;">
@foreach (var key in _allKeys)
{
var checkboxId = $"approved-key-{key.Id}";
<div class="form-check">
<input class="form-check-input" type="checkbox" id="@checkboxId"
checked="@_selectedKeyIds.Contains(key.Id)"
@onchange="e => ToggleKey(key.Id, (bool)e.Value!)" />
<label class="form-check-label" for="@checkboxId">
@key.Name
@if (!key.IsEnabled)
{
<span class="badge bg-secondary ms-1">Disabled</span>
}
</label>
</div>
}
</div>
<div class="form-text">
Callers must present a checked key in the <code>X-API-Key</code> header to invoke this method.
</div>
}
</div>
<div class="mb-3">
<label class="form-label">Parameters</label>
<SchemaBuilder Mode="object"
@@ -77,9 +111,17 @@
= Array.Empty<ScadaLink.CentralUI.ScriptAnalysis.DiagnosticMarker>();
private ApiMethod? _existing;
private List<ApiKey> _allKeys = new();
private HashSet<int> _selectedKeyIds = new();
protected override async Task OnInitializedAsync()
{
try
{
_allKeys = (await InboundApiRepository.GetAllApiKeysAsync()).ToList();
}
catch (Exception ex) { _formError = ex.Message; }
if (Id.HasValue)
{
try
@@ -92,6 +134,7 @@
_timeoutSeconds = _existing.TimeoutSeconds;
_params = _existing.ParameterDefinitions;
_returns = _existing.ReturnDefinition;
_selectedKeyIds = ParseApprovedKeyIds(_existing.ApprovedApiKeyIds);
}
}
catch (Exception ex) { _formError = ex.Message; }
@@ -99,6 +142,25 @@
_loading = false;
}
private static HashSet<int> ParseApprovedKeyIds(string? value)
{
if (string.IsNullOrWhiteSpace(value))
return new HashSet<int>();
return value.Split(',', StringSplitOptions.RemoveEmptyEntries)
.Select(s => int.TryParse(s.Trim(), out var id) ? id : -1)
.Where(id => id > 0)
.ToHashSet();
}
private void ToggleKey(int keyId, bool isChecked)
{
if (isChecked) _selectedKeyIds.Add(keyId);
else _selectedKeyIds.Remove(keyId);
}
private string? SerializeApprovedKeyIds() =>
_selectedKeyIds.Count == 0 ? null : string.Join(",", _selectedKeyIds.OrderBy(id => id));
private async Task Save()
{
_formError = null;
@@ -110,12 +172,14 @@
try
{
var approvedKeyIds = SerializeApprovedKeyIds();
if (_existing != null)
{
_existing.Script = _script;
_existing.TimeoutSeconds = _timeoutSeconds;
_existing.ParameterDefinitions = _params?.Trim();
_existing.ReturnDefinition = _returns?.Trim();
_existing.ApprovedApiKeyIds = approvedKeyIds;
await InboundApiRepository.UpdateApiMethodAsync(_existing);
}
else
@@ -124,7 +188,8 @@
{
TimeoutSeconds = _timeoutSeconds,
ParameterDefinitions = _params?.Trim(),
ReturnDefinition = _returns?.Trim()
ReturnDefinition = _returns?.Trim(),
ApprovedApiKeyIds = approvedKeyIds
};
await InboundApiRepository.AddApiMethodAsync(m);
}
@@ -3,6 +3,7 @@
@using ScadaLink.Commons.Entities.Sites
@using ScadaLink.Commons.Interfaces.Repositories
@using ScadaLink.Commons.Messages.RemoteQuery
@using ScadaLink.Commons.Types.Enums
@using ScadaLink.Communication
@inject ISiteRepository SiteRepository
@inject CommunicationService CommunicationService
@@ -10,22 +11,94 @@
@inject IDialogService Dialog
<div class="container-fluid mt-3">
<h4 class="mb-3">Parked Messages</h4>
<ToastNotification @ref="_toast" />
<div class="row mb-3 g-2 align-items-end">
<div class="col-md-3">
<label class="form-label small" for="pm-filter-site">Site</label>
<select id="pm-filter-site" class="form-select form-select-sm" aria-label="Site" @bind="_selectedSiteId">
<option value="">Select site...</option>
<div class="d-flex align-items-baseline flex-wrap mb-3">
<h4 class="mb-0 me-3">Parked Messages</h4>
@if (_messages != null && _messages.Count > 0)
{
<span class="text-muted small">
@_totalCount parked · @DistinctTargets target system@(DistinctTargets == 1 ? "" : "s")
@if (OldestMessage != null)
{
<span> · oldest @Relative(OldestMessage.LastAttemptTimestamp)</span>
}
@if (FilteredCount != _messages.Count)
{
<span class="ms-2">(showing @FilteredCount of @_messages.Count)</span>
}
</span>
}
</div>
<div class="card mb-3">
<div class="card-body py-2">
<div class="row g-2 align-items-end">
<div class="col-auto">
<label class="form-label small mb-1" for="pm-filter-site">Site</label>
<select id="pm-filter-site" class="form-select form-select-sm" style="min-width: 180px;"
value="@_selectedSiteId" @onchange="OnSiteChanged">
<option value="">Select site…</option>
@foreach (var site in _sites)
{
<option value="@site.SiteIdentifier">@site.Name</option>
}
</select>
</div>
<div class="col-md-2">
<div class="col-auto">
<label class="form-label small mb-1" for="pm-filter-cat">Category</label>
<select id="pm-filter-cat" class="form-select form-select-sm" style="min-width: 150px;"
@bind="_categoryFilter">
<option value="">All</option>
<option value="ExternalSystem">External system</option>
<option value="Notification">Notification</option>
<option value="CachedDbWrite">DB write</option>
</select>
</div>
<div class="col-auto">
<label class="form-label small mb-1" for="pm-filter-target">Target</label>
<select id="pm-filter-target" class="form-select form-select-sm" style="min-width: 160px;"
@bind="_targetFilter">
<option value="">Any</option>
@foreach (var t in DistinctTargetsList)
{
<option value="@t">@t</option>
}
</select>
</div>
<div class="col-auto">
<label class="form-label small mb-1" for="pm-filter-origin">Origin</label>
<select id="pm-filter-origin" class="form-select form-select-sm" style="min-width: 160px;"
@bind="_originFilter">
<option value="">Any</option>
<option value="__none__">(none)</option>
@foreach (var o in DistinctOriginsList)
{
<option value="@o">@o</option>
}
</select>
</div>
<div class="col-auto">
<label class="form-label small mb-1" for="pm-filter-age">Age</label>
<select id="pm-filter-age" class="form-select form-select-sm" style="min-width: 130px;"
@bind="_ageFilter">
<option value="All">All</option>
<option value="LastHour">Last hour</option>
<option value="LastDay">Last 24h</option>
<option value="LastWeek">Last 7d</option>
</select>
</div>
<div class="col">
<label class="form-label small mb-1" for="pm-filter-search">Search</label>
<input id="pm-filter-search" type="search" class="form-control form-control-sm"
placeholder="ID, target, method, error…"
@bind="_searchFilter" @bind:event="oninput" />
</div>
<div class="col-auto">
<button class="btn btn-outline-secondary btn-sm" @onclick="ClearFilters"
disabled="@(!HasActiveFilters)">Clear</button>
</div>
<div class="col-auto">
<button class="btn btn-primary btn-sm" @onclick="Search"
disabled="@(string.IsNullOrEmpty(_selectedSiteId) || _searching)">
@if (_searching) { <span class="spinner-border spinner-border-sm me-1" role="status"></span> }
@@ -33,163 +106,288 @@
</button>
</div>
</div>
</div>
</div>
@if (_selectedIds.Count > 0)
{
<div class="alert alert-secondary py-2 d-flex align-items-center mb-3">
<strong class="me-3">@_selectedIds.Count selected</strong>
<button class="btn btn-outline-success btn-sm me-2"
@onclick="BulkRetry" disabled="@_bulkInProgress">
@if (_bulkInProgress && _bulkAction == "Retry") { <span class="spinner-border spinner-border-sm me-1" role="status"></span> }
Retry selected
</button>
<button class="btn btn-outline-danger btn-sm me-2"
@onclick="BulkDiscard" disabled="@_bulkInProgress">
@if (_bulkInProgress && _bulkAction == "Discard") { <span class="spinner-border spinner-border-sm me-1" role="status"></span> }
Discard selected
</button>
<button type="button" class="btn-close ms-auto"
aria-label="Clear selection" @onclick="ClearSelection"></button>
</div>
}
@if (_errorMessage != null)
{
<div class="alert alert-danger">@_errorMessage</div>
}
@if (_messages != null)
@if (_messages == null)
{
<table class="table table-sm table-striped table-hover">
<thead class="table-dark">
@if (!string.IsNullOrEmpty(_selectedSiteId) && _searching)
{
<div class="text-muted small">Loading…</div>
}
}
else if (_messages.Count == 0)
{
<div class="card">
<div class="card-body text-center text-muted py-5">
<div class="fs-5 mb-1">No parked messages</div>
<div class="small">Nothing has failed enough to give up on at this site.</div>
</div>
</div>
}
else
{
var filtered = FilteredMessages;
<div class="table-responsive">
<table class="table table-sm table-hover mb-2 align-middle parked-table">
<thead class="table-light">
<tr>
<th style="width: 1%;"></th>
<th>Message ID</th>
<th>Target System</th>
<th>Method</th>
<th style="width: 36px;">
<input class="form-check-input" type="checkbox"
checked="@AllFilteredSelected"
@onchange="ToggleSelectAll"
aria-label="Select all" />
</th>
<th>Target / Method</th>
<th>Origin</th>
<th>Error</th>
<th>Attempts</th>
<th>Original</th>
<th>Last Attempt</th>
<th style="width: 120px;">Actions</th>
<th style="width: 110px;">Attempts</th>
<th style="width: 160px;">Last attempt</th>
</tr>
</thead>
<tbody>
@if (_messages.Count == 0)
@if (filtered.Count == 0)
{
<tr><td colspan="9" class="text-muted text-center">No parked messages.</td></tr>
<tr><td colspan="6" class="text-muted text-center py-3">No messages match the current filters.</td></tr>
}
@for (int i = 0; i < _messages.Count; i++)
@foreach (var msg in filtered)
{
var idx = i;
var msg = _messages[idx];
var idShort = msg.MessageId[..Math.Min(12, msg.MessageId.Length)];
var expanded = _expandedRows.Contains(idx);
var retryActive = _actionInProgress && _activeMessageId == msg.MessageId && _activeAction == "Retry";
var discardActive = _actionInProgress && _activeMessageId == msg.MessageId && _activeAction == "Discard";
<tr>
<td>
<button class="btn btn-link btn-sm p-0"
@onclick="() => ToggleRow(idx)"
aria-label="@(expanded ? "Hide error details" : "View error details")">
@(expanded ? "Hide" : "View")
</button>
var isSelected = _selectedIds.Contains(msg.MessageId);
<tr @key="msg.MessageId"
class="parked-row @SeverityClass(msg) @(isSelected ? "table-active" : "")"
@onclick="() => OpenDrawer(msg)"
style="cursor: pointer;">
<td @onclick:stopPropagation="true">
<input class="form-check-input" type="checkbox"
checked="@isSelected"
@onchange="e => ToggleSelect(msg.MessageId, (bool)e.Value!)"
aria-label="@($"Select {msg.MessageId[..Math.Min(8, msg.MessageId.Length)]}")" />
</td>
<td class="small">
<code class="small">@idShort…</code>
<button class="btn btn-link btn-sm p-0 ms-1"
@onclick="() => CopyAsync(msg.MessageId)"
title="Copy message ID"
aria-label="Copy message ID @msg.MessageId">📋</button>
</td>
<td class="small">@msg.TargetSystem</td>
<td class="small">@msg.MethodName</td>
<td class="small text-danger text-truncate" style="max-width: 320px;">@msg.ErrorMessage</td>
<td class="small text-center">@msg.AttemptCount</td>
<td class="small"><TimestampDisplay Value="@msg.OriginalTimestamp" /></td>
<td class="small"><TimestampDisplay Value="@msg.LastAttemptTimestamp" /></td>
<td>
<button class="btn btn-outline-success btn-sm py-0 px-1 me-1"
@onclick="() => RetryMessage(msg)"
disabled="@_actionInProgress"
title="Retry message (move back to pending)"
aria-label="Retry message @idShort">
@if (retryActive)
<div class="fw-semibold">@msg.TargetSystem</div>
<div class="small text-muted">@msg.MethodName</div>
</td>
<td>
@if (!string.IsNullOrEmpty(msg.OriginInstance))
{
<span class="spinner-border spinner-border-sm me-1" role="status"></span>
<code class="small">@msg.OriginInstance</code>
}
Retry
</button>
<button class="btn btn-outline-danger btn-sm py-0 px-1"
@onclick="() => DiscardMessage(msg)"
disabled="@_actionInProgress"
title="Permanently discard message"
aria-label="Discard message @idShort">
@if (discardActive)
else
{
<span class="spinner-border spinner-border-sm me-1" role="status"></span>
<span class="text-muted small"></span>
}
Discard
</button>
</td>
<td>
<div class="text-danger small parked-error-clamp">@msg.ErrorMessage</div>
</td>
<td>
<div class="small font-monospace">
@msg.AttemptCount<span class="text-muted">/@msg.MaxAttempts</span>
</div>
<div class="progress mt-1" style="height: 3px;">
<div class="progress-bar @AttemptBarClass(msg)"
role="progressbar"
style="width: @AttemptPercent(msg)%;"
aria-valuenow="@msg.AttemptCount"
aria-valuemin="0"
aria-valuemax="@Math.Max(1, msg.MaxAttempts)"></div>
</div>
</td>
<td>
<div class="small" title="@AbsoluteUtc(msg.LastAttemptTimestamp)">
@Relative(msg.LastAttemptTimestamp)
</div>
</td>
</tr>
@if (expanded)
{
<tr>
<td colspan="9">
<pre class="small mb-0">@msg.ErrorMessage</pre>
</td>
</tr>
}
}
</tbody>
</table>
</div>
@if (_totalCount > 0)
@if (_totalCount > _pageSize)
{
<div class="d-flex justify-content-between align-items-center">
<span class="text-muted small">Page @_pageNumber of @((_totalCount + _pageSize - 1) / _pageSize) (@_totalCount total)</span>
<span class="text-muted small">
Page @_pageNumber of @((_totalCount + _pageSize - 1) / _pageSize) · @_totalCount total
</span>
<div>
<button class="btn btn-outline-secondary btn-sm me-1" @onclick="PrevPage" disabled="@(_pageNumber <= 1)">Previous</button>
<button class="btn btn-outline-secondary btn-sm" @onclick="NextPage" disabled="@(_messages.Count < _pageSize)">Next</button>
<button class="btn btn-outline-secondary btn-sm me-1"
@onclick="PrevPage" disabled="@(_pageNumber <= 1)">Previous</button>
<button class="btn btn-outline-secondary btn-sm"
@onclick="NextPage" disabled="@(_messages.Count < _pageSize)">Next</button>
</div>
</div>
}
}
</div>
@if (_drawerMessage != null)
{
<div class="offcanvas-backdrop fade show" @onclick="CloseDrawer"></div>
<div class="offcanvas offcanvas-end show parked-drawer" tabindex="-1" style="visibility: visible;">
<div class="offcanvas-header border-bottom">
<div>
<div class="text-muted small text-uppercase">Parked message</div>
<h5 class="offcanvas-title mb-0">@_drawerMessage.TargetSystem</h5>
<div class="small text-muted">@_drawerMessage.MethodName</div>
</div>
<button type="button" class="btn-close" aria-label="Close" @onclick="CloseDrawer"></button>
</div>
<div class="offcanvas-body small">
<dl class="row mb-3">
<dt class="col-4 text-muted fw-normal">Message ID</dt>
<dd class="col-8 d-flex align-items-center gap-2">
<code class="text-truncate" style="min-width: 0;">@_drawerMessage.MessageId</code>
<button class="btn btn-link btn-sm p-0" title="Copy message ID"
@onclick="() => CopyAsync(_drawerMessage.MessageId)">📋</button>
</dd>
<dt class="col-4 text-muted fw-normal">Category</dt>
<dd class="col-8">@CategoryLabel(_drawerMessage.Category)</dd>
<dt class="col-4 text-muted fw-normal">Origin instance</dt>
<dd class="col-8">
@if (!string.IsNullOrEmpty(_drawerMessage.OriginInstance))
{
<code>@_drawerMessage.OriginInstance</code>
}
else
{
<span class="text-muted">—</span>
}
</dd>
<dt class="col-4 text-muted fw-normal">Attempts</dt>
<dd class="col-8 font-monospace">@_drawerMessage.AttemptCount / @_drawerMessage.MaxAttempts</dd>
<dt class="col-4 text-muted fw-normal">Originally enqueued</dt>
<dd class="col-8">
@Relative(_drawerMessage.OriginalTimestamp)
<span class="text-muted">· @AbsoluteUtc(_drawerMessage.OriginalTimestamp)</span>
</dd>
<dt class="col-4 text-muted fw-normal">Last attempt</dt>
<dd class="col-8">
@Relative(_drawerMessage.LastAttemptTimestamp)
<span class="text-muted">· @AbsoluteUtc(_drawerMessage.LastAttemptTimestamp)</span>
</dd>
</dl>
<div class="text-muted text-uppercase small fw-semibold mb-1">Error</div>
<pre class="bg-light border rounded p-2 small mb-0 parked-error-pre">@_drawerMessage.ErrorMessage</pre>
</div>
<div class="border-top p-3 d-flex gap-2">
<button class="btn btn-outline-success btn-sm flex-grow-1"
@onclick="RetryFromDrawer" disabled="@_actionInProgress">
@if (_actionInProgress && _activeAction == "Retry") { <span class="spinner-border spinner-border-sm me-1" role="status"></span> }
Retry
</button>
<button class="btn btn-outline-danger btn-sm flex-grow-1"
@onclick="DiscardFromDrawer" disabled="@_actionInProgress">
@if (_actionInProgress && _activeAction == "Discard") { <span class="spinner-border spinner-border-sm me-1" role="status"></span> }
Discard
</button>
</div>
</div>
}
<style>
.parked-row { border-left: 3px solid transparent; }
.parked-row.sev-danger { border-left-color: var(--bs-danger); }
.parked-row.sev-warning { border-left-color: var(--bs-warning); }
.parked-row.sev-secondary { border-left-color: var(--bs-secondary-bg-subtle); }
.parked-error-clamp {
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
max-width: 520px;
}
.parked-drawer { width: min(560px, 95vw); }
.parked-error-pre { white-space: pre-wrap; word-break: break-word; max-height: 300px; overflow-y: auto; }
.parked-table tbody tr { transition: background-color 0.1s ease; }
</style>
@code {
private List<Site> _sites = new();
private string _selectedSiteId = string.Empty;
private List<ParkedMessageEntry>? _messages;
private int _totalCount;
private int _pageNumber = 1;
private int _pageSize = 25;
private int _pageSize = 50;
private bool _searching;
private string? _errorMessage;
// Filters
private string _categoryFilter = string.Empty;
private string _targetFilter = string.Empty;
private string _originFilter = string.Empty;
private string _ageFilter = "All";
private string _searchFilter = string.Empty;
// Selection
private readonly HashSet<string> _selectedIds = new();
private bool _bulkInProgress;
private string? _bulkAction;
// Per-row action state
private bool _actionInProgress;
private string? _activeMessageId;
private string? _activeAction;
// Drawer
private ParkedMessageEntry? _drawerMessage;
private ToastNotification _toast = default!;
private readonly HashSet<int> _expandedRows = new();
protected override async Task OnInitializedAsync()
{
_sites = (await SiteRepository.GetAllSitesAsync()).ToList();
}
private async Task OnSiteChanged(ChangeEventArgs e)
{
_selectedSiteId = e.Value?.ToString() ?? string.Empty;
if (!string.IsNullOrEmpty(_selectedSiteId))
{
await Search();
}
else
{
_messages = null;
_selectedIds.Clear();
_drawerMessage = null;
}
}
private async Task Search()
{
_pageNumber = 1;
_expandedRows.Clear();
_selectedIds.Clear();
_drawerMessage = null;
await FetchPage();
}
private async Task PrevPage() { _pageNumber--; await FetchPage(); }
private async Task NextPage() { _pageNumber++; await FetchPage(); }
private void ToggleRow(int idx)
{
if (!_expandedRows.Add(idx))
{
_expandedRows.Remove(idx);
}
}
private async Task CopyAsync(string text)
{
try
{
await JS.InvokeVoidAsync("navigator.clipboard.writeText", text);
_toast.ShowSuccess("Copied to clipboard.");
}
catch
{
_toast.ShowError("Copy failed.");
}
}
private async Task PrevPage() { _pageNumber--; _selectedIds.Clear(); await FetchPage(); }
private async Task NextPage() { _pageNumber++; _selectedIds.Clear(); await FetchPage(); }
private async Task FetchPage()
{
@@ -205,12 +403,10 @@
Timestamp: DateTimeOffset.UtcNow);
var response = await CommunicationService.QueryParkedMessagesAsync(_selectedSiteId, request);
if (response.Success)
{
_messages = response.Messages.ToList();
_totalCount = response.TotalCount;
_expandedRows.Clear();
}
else
{
@@ -224,73 +420,302 @@
_searching = false;
}
private async Task RetryMessage(ParkedMessageEntry msg)
private void ClearFilters()
{
_actionInProgress = true;
_activeMessageId = msg.MessageId;
_activeAction = "Retry";
try
_categoryFilter = string.Empty;
_targetFilter = string.Empty;
_originFilter = string.Empty;
_ageFilter = "All";
_searchFilter = string.Empty;
}
private bool HasActiveFilters =>
!string.IsNullOrEmpty(_categoryFilter) ||
!string.IsNullOrEmpty(_targetFilter) ||
!string.IsNullOrEmpty(_originFilter) ||
_ageFilter != "All" ||
!string.IsNullOrEmpty(_searchFilter);
private List<ParkedMessageEntry> FilteredMessages
{
var request = new ParkedMessageRetryRequest(
CorrelationId: Guid.NewGuid().ToString("N"),
SiteId: _selectedSiteId,
MessageId: msg.MessageId,
Timestamp: DateTimeOffset.UtcNow);
var response = await CommunicationService.RetryParkedMessageAsync(_selectedSiteId, request);
if (response.Success)
get
{
_toast.ShowSuccess($"Message {msg.MessageId[..Math.Min(12, msg.MessageId.Length)]} queued for retry.");
await FetchPage();
if (_messages == null) return new();
IEnumerable<ParkedMessageEntry> q = _messages;
if (!string.IsNullOrEmpty(_categoryFilter) &&
Enum.TryParse<StoreAndForwardCategory>(_categoryFilter, out var cat))
q = q.Where(m => m.Category == cat);
if (!string.IsNullOrEmpty(_targetFilter))
q = q.Where(m => m.TargetSystem == _targetFilter);
if (_originFilter == "__none__")
q = q.Where(m => string.IsNullOrEmpty(m.OriginInstance));
else if (!string.IsNullOrEmpty(_originFilter))
q = q.Where(m => m.OriginInstance == _originFilter);
if (_ageFilter != "All")
{
var cutoff = _ageFilter switch
{
"LastHour" => DateTimeOffset.UtcNow.AddHours(-1),
"LastDay" => DateTimeOffset.UtcNow.AddDays(-1),
"LastWeek" => DateTimeOffset.UtcNow.AddDays(-7),
_ => DateTimeOffset.MinValue
};
q = q.Where(m => m.LastAttemptTimestamp >= cutoff);
}
if (!string.IsNullOrEmpty(_searchFilter))
{
var s = _searchFilter.Trim();
q = q.Where(m =>
m.MessageId.Contains(s, StringComparison.OrdinalIgnoreCase) ||
m.TargetSystem.Contains(s, StringComparison.OrdinalIgnoreCase) ||
m.MethodName.Contains(s, StringComparison.OrdinalIgnoreCase) ||
m.ErrorMessage.Contains(s, StringComparison.OrdinalIgnoreCase) ||
(m.OriginInstance ?? string.Empty).Contains(s, StringComparison.OrdinalIgnoreCase));
}
return q.ToList();
}
}
private int FilteredCount => FilteredMessages.Count;
private int DistinctTargets => _messages?.Select(m => m.TargetSystem).Distinct().Count() ?? 0;
private IEnumerable<string> DistinctTargetsList =>
_messages?.Select(m => m.TargetSystem).Distinct().OrderBy(s => s).ToList()
?? (IEnumerable<string>)Array.Empty<string>();
private IEnumerable<string> DistinctOriginsList =>
_messages?.Where(m => !string.IsNullOrEmpty(m.OriginInstance))
.Select(m => m.OriginInstance!).Distinct().OrderBy(s => s).ToList()
?? (IEnumerable<string>)Array.Empty<string>();
private ParkedMessageEntry? OldestMessage =>
_messages?.OrderBy(m => m.LastAttemptTimestamp).FirstOrDefault();
// ── Selection ──
private bool AllFilteredSelected
{
get
{
var filtered = FilteredMessages;
return filtered.Count > 0 && filtered.All(m => _selectedIds.Contains(m.MessageId));
}
}
private void ToggleSelect(string id, bool isChecked)
{
if (isChecked) _selectedIds.Add(id);
else _selectedIds.Remove(id);
}
private void ToggleSelectAll(ChangeEventArgs e)
{
var on = (bool)e.Value!;
var filtered = FilteredMessages;
if (on)
{
foreach (var m in filtered) _selectedIds.Add(m.MessageId);
}
else
{
_toast.ShowError(response.ErrorMessage ?? "Retry failed.");
foreach (var m in filtered) _selectedIds.Remove(m.MessageId);
}
}
catch (Exception ex)
{
_toast.ShowError($"Retry failed: {ex.Message}");
}
_activeMessageId = null;
_activeAction = null;
_actionInProgress = false;
}
private async Task DiscardMessage(ParkedMessageEntry msg)
private void ClearSelection() => _selectedIds.Clear();
// ── Drawer ──
private void OpenDrawer(ParkedMessageEntry msg) => _drawerMessage = msg;
private void CloseDrawer() => _drawerMessage = null;
private async Task RetryFromDrawer()
{
if (_drawerMessage == null) return;
var msg = _drawerMessage;
await RetrySingle(msg);
CloseDrawer();
}
private async Task DiscardFromDrawer()
{
if (_drawerMessage == null) return;
var msg = _drawerMessage;
var ok = await DiscardSingle(msg);
if (ok) CloseDrawer();
}
// ── Bulk ──
private async Task BulkRetry()
{
var ids = _selectedIds.ToList();
if (ids.Count == 0) return;
var confirmed = await Dialog.ConfirmAsync(
"Discard Parked Message",
$"Permanently discard message {msg.MessageId[..Math.Min(12, msg.MessageId.Length)]}? This cannot be undone.",
"Retry parked messages",
$"Move {ids.Count} message{(ids.Count == 1 ? "" : "s")} back to the pending queue?");
if (!confirmed) return;
_bulkInProgress = true;
_bulkAction = "Retry";
int success = 0, failed = 0;
foreach (var id in ids)
{
try
{
var req = new ParkedMessageRetryRequest(Guid.NewGuid().ToString("N"), _selectedSiteId, id, DateTimeOffset.UtcNow);
var resp = await CommunicationService.RetryParkedMessageAsync(_selectedSiteId, req);
if (resp.Success) success++; else failed++;
}
catch { failed++; }
}
_toast.ShowSuccess($"{success} queued for retry" + (failed > 0 ? $", {failed} failed" : "."));
_selectedIds.Clear();
_bulkInProgress = false;
_bulkAction = null;
await FetchPage();
}
private async Task BulkDiscard()
{
var ids = _selectedIds.ToList();
if (ids.Count == 0) return;
var confirmed = await Dialog.ConfirmAsync(
"Discard parked messages",
$"Permanently discard {ids.Count} message{(ids.Count == 1 ? "" : "s")}? This cannot be undone.",
danger: true);
if (!confirmed) return;
_actionInProgress = true;
_activeMessageId = msg.MessageId;
_activeAction = "Discard";
_bulkInProgress = true;
_bulkAction = "Discard";
int success = 0, failed = 0;
foreach (var id in ids)
{
try
{
var request = new ParkedMessageDiscardRequest(
CorrelationId: Guid.NewGuid().ToString("N"),
SiteId: _selectedSiteId,
MessageId: msg.MessageId,
Timestamp: DateTimeOffset.UtcNow);
var response = await CommunicationService.DiscardParkedMessageAsync(_selectedSiteId, request);
if (response.Success)
{
_toast.ShowSuccess($"Message {msg.MessageId[..Math.Min(12, msg.MessageId.Length)]} discarded.");
var req = new ParkedMessageDiscardRequest(Guid.NewGuid().ToString("N"), _selectedSiteId, id, DateTimeOffset.UtcNow);
var resp = await CommunicationService.DiscardParkedMessageAsync(_selectedSiteId, req);
if (resp.Success) success++; else failed++;
}
catch { failed++; }
}
_toast.ShowSuccess($"{success} discarded" + (failed > 0 ? $", {failed} failed" : "."));
_selectedIds.Clear();
_bulkInProgress = false;
_bulkAction = null;
await FetchPage();
}
else
// ── Single actions ──
private async Task RetrySingle(ParkedMessageEntry msg)
{
_toast.ShowError(response.ErrorMessage ?? "Discard failed.");
}
}
catch (Exception ex)
_actionInProgress = true;
_activeAction = "Retry";
try
{
_toast.ShowError($"Discard failed: {ex.Message}");
var req = new ParkedMessageRetryRequest(Guid.NewGuid().ToString("N"), _selectedSiteId, msg.MessageId, DateTimeOffset.UtcNow);
var resp = await CommunicationService.RetryParkedMessageAsync(_selectedSiteId, req);
if (resp.Success)
{
_toast.ShowSuccess($"Message {ShortId(msg.MessageId)} queued for retry.");
await FetchPage();
}
_activeMessageId = null;
_activeAction = null;
else _toast.ShowError(resp.ErrorMessage ?? "Retry failed.");
}
catch (Exception ex) { _toast.ShowError($"Retry failed: {ex.Message}"); }
_actionInProgress = false;
_activeAction = null;
}
private async Task<bool> DiscardSingle(ParkedMessageEntry msg)
{
var confirmed = await Dialog.ConfirmAsync(
"Discard parked message",
$"Permanently discard message {ShortId(msg.MessageId)}? This cannot be undone.",
danger: true);
if (!confirmed) return false;
_actionInProgress = true;
_activeAction = "Discard";
bool ok = false;
try
{
var req = new ParkedMessageDiscardRequest(Guid.NewGuid().ToString("N"), _selectedSiteId, msg.MessageId, DateTimeOffset.UtcNow);
var resp = await CommunicationService.DiscardParkedMessageAsync(_selectedSiteId, req);
if (resp.Success)
{
_toast.ShowSuccess($"Message {ShortId(msg.MessageId)} discarded.");
ok = true;
await FetchPage();
}
else _toast.ShowError(resp.ErrorMessage ?? "Discard failed.");
}
catch (Exception ex) { _toast.ShowError($"Discard failed: {ex.Message}"); }
_actionInProgress = false;
_activeAction = null;
return ok;
}
private async Task CopyAsync(string text)
{
try
{
await JS.InvokeVoidAsync("navigator.clipboard.writeText", text);
_toast.ShowSuccess("Copied to clipboard.");
}
catch { _toast.ShowError("Copy failed."); }
}
// ── Helpers ──
private static string ShortId(string id) => id[..Math.Min(12, id.Length)];
private static string Relative(DateTimeOffset t)
{
var diff = DateTimeOffset.UtcNow - t;
if (diff.TotalSeconds < 0) return "just now";
if (diff.TotalSeconds < 60) return "just now";
if (diff.TotalMinutes < 60) return $"{(int)diff.TotalMinutes}m ago";
if (diff.TotalHours < 24) return $"{(int)diff.TotalHours}h ago";
if (diff.TotalDays < 30) return $"{(int)diff.TotalDays}d ago";
return t.UtcDateTime.ToString("yyyy-MM-dd");
}
private static string AbsoluteUtc(DateTimeOffset t) =>
$"{t.UtcDateTime:yyyy-MM-dd HH:mm:ss} UTC";
private static string SeverityClass(ParkedMessageEntry msg)
{
var exhausted = msg.MaxAttempts > 0 && msg.AttemptCount >= msg.MaxAttempts;
if (!exhausted) return "sev-secondary";
var age = DateTimeOffset.UtcNow - msg.LastAttemptTimestamp;
return age < TimeSpan.FromHours(1) ? "sev-danger" : "sev-warning";
}
private static int AttemptPercent(ParkedMessageEntry msg)
{
if (msg.MaxAttempts <= 0) return 100;
var pct = (int)Math.Round(msg.AttemptCount * 100.0 / msg.MaxAttempts);
return Math.Clamp(pct, 0, 100);
}
private static string AttemptBarClass(ParkedMessageEntry msg) =>
msg.AttemptCount >= msg.MaxAttempts ? "bg-danger" : "bg-warning";
private static string CategoryLabel(StoreAndForwardCategory c) => c switch
{
StoreAndForwardCategory.ExternalSystem => "External system",
StoreAndForwardCategory.Notification => "Notification",
StoreAndForwardCategory.CachedDbWrite => "Cached DB write",
_ => c.ToString()
};
}
@@ -1,3 +1,5 @@
using ScadaLink.Commons.Types.Enums;
namespace ScadaLink.Commons.Messages.RemoteQuery;
/// <summary>
@@ -10,7 +12,10 @@ public record ParkedMessageEntry(
string ErrorMessage,
int AttemptCount,
DateTimeOffset OriginalTimestamp,
DateTimeOffset LastAttemptTimestamp);
DateTimeOffset LastAttemptTimestamp,
int MaxAttempts = 0,
StoreAndForwardCategory Category = StoreAndForwardCategory.ExternalSystem,
string? OriginInstance = null);
public record ParkedMessageQueryResponse(
string CorrelationId,
@@ -100,8 +100,8 @@ public class CentralCommunicationActor : ReceiveActor
private void HandleHeartbeat(HeartbeatMessage heartbeat)
{
// Forward heartbeat to parent for any interested central actors
Context.Parent.Tell(heartbeat);
var aggregator = _serviceProvider.GetService<ICentralHealthAggregator>();
aggregator?.MarkHeartbeat(heartbeat.SiteId, heartbeat.Timestamp);
}
/// <summary>
@@ -136,6 +136,28 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
}
});
Receive<ParkedMessageRetryRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageRetryResponse(
msg.CorrelationId, false, "Parked message handler not available"));
}
});
Receive<ParkedMessageDiscardRequest>(msg =>
{
if (_parkedMessageHandler != null)
_parkedMessageHandler.Forward(msg);
else
{
Sender.Tell(new ParkedMessageDiscardResponse(
msg.CorrelationId, false, "Parked message handler not available"));
}
});
// Internal: send heartbeat tick
Receive<SendHeartbeat>(_ => SendHeartbeatToCentral());
@@ -76,6 +76,26 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
});
}
/// <summary>
/// Bumps the last-seen timestamp for a site already known via a prior
/// SiteHealthReport. Heartbeats from sites we have not yet received a
/// full report from are ignored — registration only happens on report.
/// </summary>
public void MarkHeartbeat(string siteId, DateTimeOffset receivedAt)
{
if (!_siteStates.TryGetValue(siteId, out var state))
return;
if (receivedAt > state.LastReportReceivedAt)
state.LastReportReceivedAt = receivedAt;
if (!state.IsOnline)
{
state.IsOnline = true;
_logger.LogInformation("Site {SiteId} is back online (heartbeat)", siteId);
}
}
/// <summary>
/// Get the current health state for all known sites.
/// </summary>
@@ -9,6 +9,15 @@ namespace ScadaLink.HealthMonitoring;
public interface ICentralHealthAggregator
{
void ProcessReport(SiteHealthReport report);
/// <summary>
/// Bumps the last-seen timestamp for a site already known via a prior
/// SiteHealthReport. Used to keep a site marked online between full
/// 30s reports when ~2s heartbeats are arriving — protects against the
/// 60s offline threshold firing on a transiently delayed report.
/// </summary>
void MarkHeartbeat(string siteId, DateTimeOffset receivedAt);
IReadOnlyDictionary<string, SiteHealthState> GetAllSiteStates();
SiteHealthState? GetSiteState(string siteId);
}
@@ -316,6 +316,21 @@ akka {{
siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.EventLog, eventLogHandler));
}
// Parked message handler — bridges Akka to StoreAndForwardService
var storeAndForwardService = _serviceProvider.GetService<StoreAndForwardService>();
if (storeAndForwardService != null)
{
// Initialize SQLite schema and start the retry timer. Must complete before
// any actor or HTTP handler touches the service.
storeAndForwardService.StartAsync().GetAwaiter().GetResult();
var parkedMessageHandler = _actorSystem.ActorOf(
Props.Create(() => new ParkedMessageHandlerActor(
storeAndForwardService, _nodeOptions.SiteId!)),
"parked-message-handler");
siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.ParkedMessages, parkedMessageHandler));
}
// Register SiteCommunicationActor with ClusterClientReceptionist so central ClusterClients can reach it
ClusterClientReceptionist.Get(_actorSystem).RegisterService(siteCommActor);
@@ -0,0 +1,122 @@
using System.Text.Json;
using Akka.Actor;
using Akka.Event;
using ScadaLink.Commons.Messages.RemoteQuery;
namespace ScadaLink.StoreAndForward;
/// <summary>
/// Akka actor bridge for <see cref="StoreAndForwardService"/> parked-message operations.
/// Receives Query/Retry/Discard requests from the SiteCommunicationActor and replies
/// with the matching response records.
/// </summary>
public class ParkedMessageHandlerActor : ReceiveActor
{
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly StoreAndForwardService _service;
private readonly string _siteId;
public ParkedMessageHandlerActor(StoreAndForwardService service, string siteId)
{
_service = service;
_siteId = siteId;
Receive<ParkedMessageQueryRequest>(HandleQuery);
Receive<ParkedMessageRetryRequest>(HandleRetry);
Receive<ParkedMessageDiscardRequest>(HandleDiscard);
}
private void HandleQuery(ParkedMessageQueryRequest msg)
{
var sender = Sender;
var siteId = _siteId;
_service.GetParkedMessagesAsync(category: null, msg.PageNumber, msg.PageSize)
.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
var entries = t.Result.Messages
.Select(m => new ParkedMessageEntry(
MessageId: m.Id,
TargetSystem: m.Target,
MethodName: ExtractMethodName(m.PayloadJson, m.Category),
ErrorMessage: m.LastError ?? string.Empty,
AttemptCount: m.RetryCount,
OriginalTimestamp: m.CreatedAt,
LastAttemptTimestamp: m.LastAttemptAt ?? m.CreatedAt,
MaxAttempts: m.MaxRetries,
Category: m.Category,
OriginInstance: m.OriginInstanceName))
.ToList();
return new ParkedMessageQueryResponse(
msg.CorrelationId, siteId, entries, t.Result.TotalCount,
msg.PageNumber, msg.PageSize, true, null, DateTimeOffset.UtcNow);
}
return new ParkedMessageQueryResponse(
msg.CorrelationId, siteId, [], 0, msg.PageNumber, msg.PageSize,
false, t.Exception?.GetBaseException().Message, DateTimeOffset.UtcNow);
}).PipeTo(sender);
}
private void HandleRetry(ParkedMessageRetryRequest msg)
{
var sender = Sender;
_service.RetryParkedMessageAsync(msg.MessageId)
.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
return new ParkedMessageRetryResponse(
msg.CorrelationId, t.Result,
t.Result ? null : "Message not found or no longer parked.");
}
return new ParkedMessageRetryResponse(
msg.CorrelationId, false, t.Exception?.GetBaseException().Message);
}).PipeTo(sender);
}
private void HandleDiscard(ParkedMessageDiscardRequest msg)
{
var sender = Sender;
_service.DiscardParkedMessageAsync(msg.MessageId)
.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
return new ParkedMessageDiscardResponse(
msg.CorrelationId, t.Result,
t.Result ? null : "Message not found or no longer parked.");
}
return new ParkedMessageDiscardResponse(
msg.CorrelationId, false, t.Exception?.GetBaseException().Message);
}).PipeTo(sender);
}
private static string ExtractMethodName(string payloadJson, Commons.Types.Enums.StoreAndForwardCategory category)
{
if (string.IsNullOrEmpty(payloadJson))
return category.ToString();
try
{
using var doc = JsonDocument.Parse(payloadJson);
var root = doc.RootElement;
if (root.TryGetProperty("MethodName", out var method) && method.ValueKind == JsonValueKind.String)
return method.GetString() ?? category.ToString();
if (root.TryGetProperty("Subject", out var subject) && subject.ValueKind == JsonValueKind.String)
return subject.GetString() ?? category.ToString();
}
catch (JsonException)
{
}
return category.ToString();
}
}
@@ -8,6 +8,7 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Akka" Version="1.5.62" />
<PackageReference Include="Microsoft.Data.Sqlite" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.7" />
@@ -11,6 +11,7 @@ using ScadaLink.Commons.Messages.Deployment;
using ScadaLink.Commons.Messages.DebugView;
using ScadaLink.Commons.Messages.Health;
using ScadaLink.Communication.Actors;
using ScadaLink.HealthMonitoring;
using Akka.TestKit;
namespace ScadaLink.Communication.Tests;
@@ -140,25 +141,27 @@ public class CentralCommunicationActorTests : TestKit
}
[Fact]
public void Heartbeat_ForwardedToParent()
public void Heartbeat_BumpsAggregatorTimestamp()
{
var mockRepo = Substitute.For<ISiteRepository>();
mockRepo.GetAllSitesAsync(Arg.Any<CancellationToken>())
.Returns(new List<Site>());
var aggregator = Substitute.For<ICentralHealthAggregator>();
var services = new ServiceCollection();
services.AddScoped(_ => mockRepo);
services.AddSingleton(aggregator);
var sp = services.BuildServiceProvider();
var siteClientFactory = Substitute.For<ISiteClientFactory>();
var parentProbe = CreateTestProbe();
var centralActor = parentProbe.ChildActorOf(
var centralActor = Sys.ActorOf(
Props.Create(() => new CentralCommunicationActor(sp, siteClientFactory)));
var heartbeat = new HeartbeatMessage("site1", "host1", true, DateTimeOffset.UtcNow);
centralActor.Tell(heartbeat);
var timestamp = DateTimeOffset.UtcNow;
centralActor.Tell(new HeartbeatMessage("site1", "host1", true, timestamp));
parentProbe.ExpectMsg<HeartbeatMessage>(msg => msg.SiteId == "site1");
AwaitAssert(() => aggregator.Received(1).MarkHeartbeat("site1", timestamp));
}
[Fact]
@@ -25,6 +25,7 @@
<ItemGroup>
<ProjectReference Include="../../src/ScadaLink.Communication/ScadaLink.Communication.csproj" />
<ProjectReference Include="../../src/ScadaLink.Commons/ScadaLink.Commons.csproj" />
<ProjectReference Include="../../src/ScadaLink.HealthMonitoring/ScadaLink.HealthMonitoring.csproj" />
</ItemGroup>
</Project>