- Adds SourceNode varchar(64) NULL to AuditLog, Notifications, and SiteCalls
tables with role-name semantics: node-a/node-b for site rows (qualified by
SourceSiteId), central-a/central-b for central direct-write rows.
- New IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc) index.
- Reframes CLAUDE.md from documentation-only to implementation project.
- Adds docs/plans/2026-05-23-audit-source-node.md + tasks.json companion.
This project contains design documentation for a distributed SCADA system built on Akka.NET. The documents describe a hub-and-spoke architecture with a central cluster and multiple site clusters.
This is the full **implementation** project for ScadaLink — a distributed SCADA system built on Akka.NET in a hub-and-spoke architecture (one central cluster + multiple site clusters). It contains source code, tests, deployable docker topology, and the design documentation that the code implements. The design docs are the spec; `src/` is the binary.
When a change is requested, the default assumption is: update the design doc *and* the code *and* the tests *and* (if it ships) the docker deploy — together, in one session, with `git diff` review before committing.
## Project Structure
-`src/` — C#/.NET implementation, one project per component (e.g. `ScadaLink.AuditLog`, `ScadaLink.NotificationOutbox`, `ScadaLink.SiteCallAudit`, `ScadaLink.CentralUI`, `ScadaLink.Host`, …). Solution file: `ScadaLink.slnx`.
-`tests/` — Test projects (unit + integration).
-`docker/` — 8-node cluster topology (2 central + 3 sites), `deploy.sh`, per-node `appsettings.*.json`. See [`docker/README.md`](docker/README.md) for setup, ports, and management commands. Rebuild + redeploy with `bash docker/deploy.sh`.
-`infra/` — Docker Compose for local test services (LDAP, MS SQL, OPC UA, SMTP, REST API, Traefik).
-`README.md` — Master index with component table and architecture diagrams.
-`docs/requirements/HighLevelReqs.md` — Complete high-level requirements covering all functional areas.
-`docs/requirements/Component-*.md` — Individual component design documents (one per component).
-`docs/requirements/Component-*.md` — Individual component design documents (one per component) — the spec the code implements.
-`docs/test_infra/test_infra.md` — Master test infrastructure doc (OPC UA, LDAP, MS SQL, SMTP, REST API, Traefik).
-`docs/plans/` — Design decision documents from refinement sessions.
-`docs/plans/` — Design decision and implementation-plan documents from refinement sessions.
-`AkkaDotNet/` — Akka.NET reference documentation and best practices notes.
-`infra/` — Docker Compose and config files for local test services.
-`docker/` — Docker infrastructure for the 8-node cluster topology (2 central + 3 sites). See [`docker/README.md`](docker/README.md) for cluster setup, port allocation, and management commands.
## Document Conventions
@@ -31,10 +35,11 @@ This project contains design documentation for a distributed SCADA system built
## Editing Rules
- Edit documents in place. Do not create copies or backup files.
- When a change affects multiple documents, update all affected documents in the same session.
- Edit documents and code in place. Do not create copies or backup files.
- When a change affects multiple documents or projects, update them all in the same session — design doc, entities/repos, actors/services, UI, tests, migrations, and deploy config travel together.
- Use `git diff` to review changes before committing.
- Commit related changes together with a descriptive message summarizing the design decision.
- Commit related changes together with a descriptive message summarizing the design decision and the implementation slice.
- After non-trivial code changes, build (`dotnet build ScadaLink.slnx`) and run relevant tests before declaring done; for cluster-runtime changes, rebuild the image with `bash docker/deploy.sh`.
## Current Component List (23 components)
@@ -140,6 +145,7 @@ This project contains design documentation for a distributed SCADA system built
- Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
- 365-day central retention with monthly partition-switch purge; 7-day site SQLite retention with a hard `ForwardState` invariant (no row purged until forwarded or reconciled).
- Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE); hash-chain tamper evidence and Parquet archival are deferred to v1.x.
- Node-of-origin is captured alongside site-of-origin: `SourceNode` (`varchar(64)` NULL) on `AuditLog`, `Notifications`, and `SiteCalls` — `node-a`/`node-b` for site rows (qualified by `SourceSiteId`/`SourceSite`), `central-a`/`central-b` for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` on `AuditLog`.
- Central UI: new top-level **Audit** nav group + Audit Log page, with drill-ins from Notifications, Site Calls, External Systems, Inbound API Keys, Sites, and Instances.
# Audit `SourceNode` Stamping — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Capture the cluster node of origin (`node-a` / `node-b` for site rows, `central-a` / `central-b` for central direct-write rows) on every `AuditLog`, `Notifications`, and `SiteCalls` row, end-to-end from the writing node through telemetry / reconciliation to the Central UI.
**Architecture:** Introduce a single `INodeIdentityProvider` exposing the local node name from `NodeOptions.NodeName` (new config key, bound from `ScadaLink:Node:NodeName`). Stamp `SourceNode` at the *writing node* — site `SqliteAuditWriter` for site rows, `CentralAuditWriter` for central direct-writes — and carry it verbatim through the existing gRPC `AuditEventDto` / `SiteCallOperationalDto` envelopes (additive new field), the `NotificationSubmit` S&F payload, and the `CachedCallTelemetry` packet. EF Core migrations add the column to all three central tables; site-side SQLite schemas use the existing idempotent `PRAGMA table_info` + `ALTER TABLE ADD COLUMN` pattern. UI gets a new "Node" column + filter on the three grids.
- Cross-table `SourceNode`-aware KPIs (e.g., "per-node stuck count"). The data is captured; dashboards stay site-keyed until a real ask lands.
- Backfilling existing rows with a best-guess `SourceNode`. New rows get the column; legacy rows stay `NULL`.
- Renaming `NodeHostname` (the Docker container hostname) — it stays as the diagnostic hostname; `NodeName` is the new semantic role-within-cluster name.
---
## Conventions
- All file paths are relative to repo root (`/Users/dohertj2/Desktop/scadalink-design`).
- TDD: red → green → commit. Failing test first, then implementation, then verify, then commit.
- One coherent change per commit. Commit messages prefix with the affected slice: `feat(audit):`, `feat(notif-outbox):`, `feat(sitecall-audit):`, `chore(docker):`, etc.
- After each task, run the targeted test set (`dotnet test <project> --filter <name>`) and the full solution build (`dotnet build ScadaLink.slnx`) before the commit.
- For tasks whose migration name needs a timestamp, use `date -u +%Y%m%d%H%M%S` at the moment of execution. The exact name doesn't matter for correctness; just keep them monotonically ordered.
---
## Task 0: Branch + Snapshot
**Files:** none (git only)
**Step 1: Create feature branch**
```bash
git checkout -b feature/audit-source-node
git status
```
Expected: branch created, working tree shows only the prior design-doc edits + the modified `appsettings.*.json` files already in `git status`.
**Step 2: Stash unrelated dirty files**
`docker/central-node-{a,b}/appsettings.Central.json` and `src/ScadaLink.CentralUI/Components/Pages/Login.razor` are dirty from prior unrelated work — do **NOT** include them in this feature's commits. Verify with `git diff --stat` what's already modified, and either revert or stash:
In `AuditEvent.cs`, add between `SourceSiteId` and `SourceInstanceId` (mirroring the design doc):
```csharp
publicstring?SourceNode{get;init;}
```
**Step 3: Resolve compile errors**
The record is `sealed` and used widely. Most usages will be init-only and won't break; positional constructors (if any) will. Fix them by adding `SourceNode = …` initializers OR by leaving `SourceNode = null` where the caller doesn't know the node yet (writer-level stamping happens in Task 9 + 10).
**Step 4: Run + commit**
```bash
dotnet test tests/ScadaLink.AuditLog.Tests --filter SourceNode -v n
**Step 2: Add `SourceNode` to `SiteCallOperational`**
Insert `string? SourceNode` between `SourceSite` and `Status`. **Update all callers** — the C# compiler will list every site that constructs the record. Most are in:
- Modify: tests in `tests/ScadaLink.NotificationOutbox.Tests/`
**Step 1: Failing test**
```csharp
[Fact]
publicvoidNotificationSubmit_carries_SourceNode()
{
varsubmit=newNotificationSubmit(
NotificationId:Guid.NewGuid().ToString("D"),
ListName:"ops-team",
Subject:"x",
Body:"y",
SourceSiteId:"site-a",
SourceInstanceId:"instance-1",
SourceScript:"OnAlarm",
SourceNode:"node-a",// new
SiteEnqueuedAt:DateTimeOffset.UtcNow);
Assert.Equal("node-a",submit.SourceNode);
}
```
Run: expected FAIL.
**Step 2: Add `SourceNode` to both**
In `Notification.cs`: `public string? SourceNode { get; set; }`.
In `NotificationMessages.cs`, extend the record additively (defaulted optional positional arg so existing callers compile — keep behind the optional `OriginExecutionId`/`OriginParentExecutionId` to preserve the existing tail):
```csharp
publicrecordNotificationSubmit(
stringNotificationId,
stringListName,
stringSubject,
stringBody,
stringSourceSiteId,
string?SourceInstanceId,
string?SourceScript,
DateTimeOffsetSiteEnqueuedAt,
Guid?OriginExecutionId=null,
Guid?OriginParentExecutionId=null,
string?SourceNode=null);// new, tail
```
**Step 3: Run + commit**
```bash
dotnet build ScadaLink.slnx
dotnet test tests/ScadaLink.NotificationOutbox.Tests --filter SourceNode -v n
Pattern from existing `AddAuditLogTableMigrationTests.cs`. Apply migration against a fresh MS SQL test fixture; assert `INFORMATION_SCHEMA.COLUMNS` contains `SourceNode varchar(64) NULL` and `sys.indexes` contains `IX_AuditLog_Node_Occurred` with columns `(SourceNode, OccurredAtUtc)`.
Run: expected FAIL.
**Step 2: Add migration via EF CLI**
From repo root:
```bash
dotnet ef migrations add AddAuditLogSourceNode \
--project src/ScadaLink.ConfigurationDatabase \
--startup-project src/ScadaLink.Host \
--context ScadaLinkDbContext
```
Hand-edit the generated `Up()` / `Down()` to verify shape:
```csharp
migrationBuilder.AddColumn<string>(
name:"SourceNode",
table:"AuditLog",
type:"varchar(64)",
unicode:false,
maxLength:64,
nullable:true);
migrationBuilder.CreateIndex(
name:"IX_AuditLog_Node_Occurred",
table:"AuditLog",
columns:new[]{"SourceNode","OccurredAtUtc"});
```
`Down()` drops the index then the column.
**Step 3: Update EF configuration**
In `AuditLogEntityTypeConfiguration.cs`, mirror the design doc:
// 1. open a SQLite file and create the OLD schema (no SourceNode)
// 2. open SqliteAuditWriter against the same file
// 3. assert SourceNode column now exists via PRAGMA table_info
}
```
Run: expected FAIL.
**Step 2: Update `InitializeSchema`**
Add `SourceNode TEXT NULL` to the `CREATE TABLE IF NOT EXISTS AuditLog (...)` DDL. Add a second `PRAGMA table_info`-based upgrade block matching the existing `ExecutionId` / `ParentExecutionId` pattern:
- Modify: tests under `tests/ScadaLink.SiteRuntime.Tests/Tracking/` (or closest)
**Step 1: Failing test** — same pattern as Task 9, asserting the `OperationTracking` table grows a `SourceNode TEXT NULL` column on both fresh and pre-existing DBs.
**Step 1: Failing test** — mirror Task 11 but for the central writer (Inbound API / Notification Outbox dispatcher path).
**Step 2: Implementation** — inject `INodeIdentityProvider`, stamp before `repo.InsertIfNotExistsAsync`. Same "caller wins" semantics.
**Step 3: Update `AuditLogRepository.InsertIfNotExistsAsync` to persist `SourceNode`**
This is in `src/ScadaLink.ConfigurationDatabase/Repositories/AuditLogRepository.cs` — extend the parameterized INSERT.
**Step 4: Run + commit**
```bash
git commit -m "feat(audit): stamp SourceNode at CentralAuditWriter + persist via AuditLogRepository"
```
---
## Task 13: Site → central — carry `SourceNode` on `NotificationSubmit`
**Files:**
- Modify: site code that constructs `NotificationSubmit` (likely under `src/ScadaLink.SiteRuntime/Scripts/` or `src/ScadaLink.NotificationService/Site/...`)
- Modify: S&F buffer schema if NotificationSubmit is serialized to SQLite (check `src/ScadaLink.StoreAndForward/StoreAndForwardStorage.cs` for a notification-specific column — likely the whole DTO is serialized as a blob, no schema change needed)
- Modify: `src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs``HandleSubmit` to copy `SourceNode` into the `Notification` row
## Task 14: Site → central — carry `SourceNode` on `SiteCallOperational` + cached telemetry
**Files:**
- Modify: site emitters of `SiteCallOperational` — `src/ScadaLink.ExternalSystemGateway/ExternalSystemClient.cs`, `src/ScadaLink.DataConnectionLayer/...` (cached DB write), and the S&F retry path that emits `Attempted` packets
- Modify: `src/ScadaLink.SiteCallAudit/SiteCallAuditActor.cs` (and any `SiteCallAuditIngestActor` that maps `SiteCallOperational` → `SiteCall`)
- Modify: `src/ScadaLink.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` — extend monotonic upsert SQL to include `SourceNode`
- Modify: tests in `tests/ScadaLink.SiteCallAudit.Tests/SiteCallAuditActorTests.cs` and `tests/ScadaLink.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs`
**Step 1: Failing test — central persists `SourceNode` on upsert; subsequent upsert with same id keeps SourceNode set even if newer packet has it null**
- Site emitters: inject `INodeIdentityProvider`, pass `SourceNode = nodeIdentity.NodeName` on construction.
- Repository: include `SourceNode` in the INSERT branch. In the conditional monotonic UPDATE branch, use `SourceNode = COALESCE(@SourceNode, SourceNode)` so later packets with a null don't blank out a previously-stamped value.
- Extend `AuditLogQuery` with `IReadOnlyList<string>? SourceNodes` (multi-select like `Sites`).
- Extend the LINQ filter (`q.SourceNodes is { Count: > 0 } ? source.Where(r => q.SourceNodes.Contains(r.SourceNode)) : source`).
- Add a new column descriptor `("Node", row => row.SourceNode ?? "—")` between `Site` and `Channel` in `AllColumns`.
- Add a multi-select filter chip in `AuditFilterBar.razor`. Populate node options by `SELECT DISTINCT SourceNode FROM AuditLog WHERE SourceNode IS NOT NULL` (cached for 60s).
## Task 16: Central UI — add Node column + filter to Notifications grid
**Files:** mirror Task 15 for `src/ScadaLink.CentralUI/Components/Pages/Notifications/NotificationReport.razor` and its query service. No filter populate-from-DB required if scope is per-site already; a free-text Node filter is acceptable for v1.
Add `"NodeName": "<value>"` to the existing `"Node": { … }` object in each file. **Do not** disturb the two `central-*` files that show up dirty in pre-existing `git status` from before this branch — re-apply this change cleanly after stashing.
git commit -m "chore(docker): set NodeName on all 8 cluster nodes"
```
---
## Task 19: Full-solution build + targeted test sweep
**Step 1: Build**
```bash
dotnet build ScadaLink.slnx
```
Expected: 0 errors, 0 warnings.
**Step 2: Run the touched test projects**
```bash
dotnet test tests/ScadaLink.AuditLog.Tests/ScadaLink.AuditLog.Tests.csproj
dotnet test tests/ScadaLink.NotificationOutbox.Tests/ScadaLink.NotificationOutbox.Tests.csproj
dotnet test tests/ScadaLink.SiteCallAudit.Tests/ScadaLink.SiteCallAudit.Tests.csproj
dotnet test tests/ScadaLink.Communication.Tests/ScadaLink.Communication.Tests.csproj
dotnet test tests/ScadaLink.ConfigurationDatabase.Tests/ScadaLink.ConfigurationDatabase.Tests.csproj
dotnet test tests/ScadaLink.CentralUI.Tests/ScadaLink.CentralUI.Tests.csproj
```
Expected: all green. If anything fails, fix it before proceeding.
**Step 3: Full solution test (sanity)**
```bash
dotnet test ScadaLink.slnx --no-build
```
---
## Task 20: Docker redeploy + smoke verify
**Step 1: Rebuild + redeploy the cluster**
```bash
bash docker/deploy.sh
```
Expected: image rebuilt, all 8 containers up. Watch for migration apply on central startup.
**Step 2: CLI smoke — generate one inbound, one notification, one cached call; verify SourceNode populated**
```bash
# Trigger something that emits each kind. The exact CLI commands depend on
# scripts deployed at the time; at minimum:
# 1. POST to inbound API → produces an InboundRequest row
# 2. Run a script that calls Notify.Send → produces NotifySend + NotifyDeliver
# 3. Run a script that calls ExternalSystem.CachedCall → produces CachedSubmit/Forwarded/Attempted/Delivered
# Then check SourceNode is populated:
docker exec scadalink-mssql /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P 'YourStrong!Passw0rd' -Q "SELECT TOP 20 Channel, Kind, SourceSiteId, SourceNode, Status FROM scadalink.dbo.AuditLog ORDER BY OccurredAtUtc DESC"
```
Expected:
-`InboundRequest` rows: `SourceSiteId IS NULL`, `SourceNode IN ('central-a','central-b')`.
-`NotifyDeliver` rows: `SourceSiteId IS NULL`, `SourceNode IN ('central-a','central-b')`.
- Site-originated rows: `SourceSiteId = 'site-a'`, `SourceNode IN ('node-a','node-b')`.
**Step 3: UI smoke**
Open `http://localhost:9000`, log in as `multi-role` / `password`, hit each of:
**Step 4: Final commit (if anything had to be tweaked during smoke)**
```bash
git add -A
git status # verify nothing surprising
git commit -m "chore(audit): smoke-verify SourceNode end-to-end across cluster"
```
---
## Acceptance Criteria (the whole-plan checklist)
- [ ] Every audit row written from this commit forward carries `SourceNode` populated (`central-a/b` for central direct-write, `node-a/b` for site rows).
- [ ] Every new `Notifications` and `SiteCalls` row carries `SourceNode` from the site.
- [ ]`IX_AuditLog_Node_Occurred` exists in central MS SQL.
- [ ] Site SQLite `AuditLog` and `OperationTracking` tables both have `SourceNode TEXT NULL`; existing site DBs are upgraded idempotently on startup.
- [ ] Proto `AuditEventDto.source_node = 22` and `SiteCallOperationalDto.source_node = 12` exist; no field numbers reused.
- [ ] Central UI Audit Log, Notifications, and Site Calls pages all display a "Node" column and support filtering by it.
- [ ] All test projects green.
- [ ] Cluster comes up clean via `bash docker/deploy.sh`; CLI smoke confirms expected node names land in the central tables.
- [ ] Design docs (already committed in Task 0) match the implementation.
@@ -86,6 +86,7 @@ row per lifecycle event across all channels.
| `ExecutionId` | `uniqueidentifier` NULL | The originating script execution / inbound request — the universal per-run correlation value; distinct from `CorrelationId`, which is the per-operation lifecycle id. Stamped on *every* audit row emitted by one execution. |
| `ParentExecutionId` | `uniqueidentifier` NULL | The `ExecutionId` of the execution that *spawned* this run — the cross-execution correlation pointer. Set on every row of an inbound-API-routed site script run (= the inbound request's `ExecutionId`); NULL for a top-level run (inbound, tag-change / timer-triggered, un-bridged). |
| `SourceNode` | `varchar(64)` NULL | The cluster node on which the event was emitted — `node-a` / `node-b` for site rows (qualified by `SourceSiteId`), `central-a` / `central-b` for central-originated rows. Nullable so reconciled rows from a node that has since been retired don't block ingest. |
-`IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` — per-node filters ("everything `central-a` did in window X", or pinning a misbehaving site node).
-`IX_AuditLog_CorrelationId (CorrelationId)` — drilldown from a single operation.
-`IX_AuditLog_Execution (ExecutionId)` — drilldown to every action of one script execution / inbound request.
-`IX_AuditLog_ParentExecution (ParentExecutionId)` — cross-execution drilldown: the downward leg of the execution-tree walk seeks on it (`ParentExecutionId = ancestor.ExecutionId`), and it backs the `parentExecutionId` filter.
@@ -172,7 +174,9 @@ generalises to it with no schema change once that spawn point is threaded.
A SQLite database file on each site node, alongside the Store-and-Forward
buffer. Same schema as central minus `IngestedAtUtc` (irrelevant at the source),
plus a `ForwardState` column with values `Pending | Forwarded | Reconciled` that
drives the telemetry loop and reconciliation pull.
drives the telemetry loop and reconciliation pull.`SourceNode` is stamped by the
writing node itself (`node-a` / `node-b`) at append time and travels with the row
through telemetry and reconciliation unchanged.
**Site SQLite retention rule (hard invariant):**
@@ -233,7 +237,9 @@ instead. The Notification Outbox dispatcher writes
`Notification.NotifyDeliver` with `Status=Attempted` per delivery attempt and
`Notification.NotifyDeliver` with `Status=Delivered`/`Parked`/`Discarded` on
terminal status. Central direct-writes use the same insert-if-not-exists
semantics keyed on `EventId`.
semantics keyed on `EventId`.`SourceSiteId` is NULL on all central direct-write
rows; `SourceNode` is stamped to the local central node's role name
| `SourceNode` | The cluster node on which the notification was enqueued — `node-a` / `node-b` for site-originated rows (qualified by `SourceSiteId`). Nullable. Carried verbatim from the site through the S&F handoff. |
| `SiteEnqueuedAt` | When the script called `Send()` (carried from the site). |
- **SourceNode** — the cluster node on which the call was issued (`node-a` /
`node-b`, qualified by `SourceSite`). Nullable. Stamped site-side at submit
time and carried verbatim through the combined `CachedCallTelemetry` packet,
reconciliation pulls, and the central upsert.
- **Kind** — `TrackedOperationKind` enum: `ExternalCall` or `DatabaseWrite`.
- **TargetSummary** — external system + method name for an `ExternalCall`; for a
`DatabaseWrite`, just the database connection name — intentionally not the SQL
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Joseph Doherty