ScadaBridge/docs/deployment/topology-guide.md

# ScadaBridge Cluster Topology Guide

## Architecture Overview

ScadaBridge uses a hub-and-spoke architecture:
- **Central Cluster**: Two-node active/standby Akka.NET cluster for management, UI, and coordination.
- **Site Clusters**: Two-node active/standby Akka.NET clusters at each remote site for data collection and local processing.

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    USERS["Users<br/>(HTTPS / LB)"]

    subgraph CENTRAL["Central Cluster"]
        NA["Node A<br/>Active"]
        NB["Node B<br/>Standby"]
        NA <--> NB
    end

    USERS --> NA
    CENTRAL --> SITE01
    CENTRAL --> SITE02
    CENTRAL --> SITE03
    CENTRAL --> SITEN

    subgraph SITE01["Site 01"]
        S01A["A<br/>Active"]
        S01B["B<br/>Standby"]
    end
    subgraph SITE02["Site 02"]
        S02A["A<br/>Active"]
        S02B["B<br/>Standby"]
    end
    subgraph SITE03["Site 03"]
        S03A["A<br/>Active"]
        S03B["B<br/>Standby"]
    end
    subgraph SITEN["Site N"]
        SNA["A<br/>Active"]
        SNB["B<br/>Standby"]
    end

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
    class USERS dec
    class CENTRAL proc
    class NA,S01A,S02A,S03A,SNA start
    class NB,S01B,S02B,S03B,SNB muted
    class SITE01,SITE02,SITE03,SITEN warn
```

## Central Cluster Setup

### Cluster Configuration

Both central nodes must be configured as seed nodes for each other:

**Node A** (`central-01.example.com`):
```json
{
  "ScadaBridge": {
    "Node": {
      "Role": "Central",
      "NodeHostname": "central-01.example.com",
      "RemotingPort": 8081
    },
    "Cluster": {
      "SeedNodes": [
        "akka.tcp://scadabridge@central-01.example.com:8081",
        "akka.tcp://scadabridge@central-02.example.com:8081"
      ]
    }
  }
}
```

**Node B** (`central-02.example.com`):
```json
{
  "ScadaBridge": {
    "Node": {
      "Role": "Central",
      "NodeHostname": "central-02.example.com",
      "RemotingPort": 8081
    },
    "Cluster": {
      "SeedNodes": [
        "akka.tcp://scadabridge@central-02.example.com:8081",
        "akka.tcp://scadabridge@central-01.example.com:8081"
      ]
    }
  }
}
```

> **Seed order is load-bearing — each node lists ITSELF first** (decision 2026-07-22). Note Node B's list is the reverse of Node A's. Akka only lets `seed-nodes[0]` form a *new* cluster, so a node listing its partner first can never boot while that partner is down. `StartupValidator` rejects the boot if the ordering is wrong, comparing host **and** port; use the same spelling of the hostname in `NodeHostname` and in the seed URI, since Akka does no DNS canonicalisation (`central-02` and `central-02.example.com` are different seed identities). See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.

### Cluster Behavior

- **Split-brain resolver**: `auto-down` (`AutoDowning` provider, `auto-down-unreachable-after` = 15s) since the 2026-07-21 availability-over-partition-safety decision — the leader among the *reachable* members downs the unreachable peer, so a hard crash of **either** node fails over. Accepted trade: a real partition leaves both sides active until an operator restarts one. `keep-oldest` (with `down-if-alone = on`) remains a supported `SplitBrainResolverStrategy` value, but in a two-node cluster it cannot survive a crash of the oldest node. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- **Minimum members**: `min-nr-of-members = 1` — a single node can form a cluster.
- **Failure detection**: 2-second heartbeat interval, 10-second threshold.
- **Total failover time**: ~25 seconds from node failure to singleton migration.
- **Singleton handover**: Uses CoordinatedShutdown for graceful migration.

### Shared State

Both central nodes share state through:
- **SQL Server**: All configuration, deployment records, templates, and audit logs.
- **JWT signing key**: Same `JwtSigningKey` in both nodes' configuration.
- **Data Protection keys**: Shared key ring (stored in SQL Server or shared file path).
- **Inbound API keys**: shared **only** when `ScadaBridge:InboundApi:ApiKeyStore:Provider = SqlServer`
  (design 2026-09-02, register row 108) — both nodes then read/write the same `zbauth` schema in the
  configuration database, so create/revoke/rotate/scope changes are cluster-wide immediately. The
  library default, `Provider = Sqlite`, stores keys in a **per-node** file instead; running `Sqlite`
  on a central pair is a documented misconfiguration — `StartupValidator` logs a boot-time Warning
  because a failover then 401s every inbound client until keys are re-issued on the survivor. Both
  docker rigs run `SqlServer`; see `docs/operations/inbound-api-key-reissue.md` → *Central pairs* to
  migrate an existing pair.
- **Secrets store**: shared **only** when `Secrets:Store:Provider = SqlServer` (design 2026-09-28,
  `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`) — both nodes then read/write the
  same shared SQL Server secret store, so `/admin/secrets` create/rotate/delete changes are
  cluster-wide immediately and both nodes deploy the same secret set to sites regardless of which
  is active. The library default, `Provider = Sqlite`, stores secrets in a **per-node** file instead;
  running `Sqlite` on a central pair is a documented misconfiguration — `StartupValidator` logs a
  boot-time Warning. See *Secrets: shared store, per-site deploy, and the KEK* below for the KEK
  requirement this store does **not** by itself satisfy.

### Load Balancer

A load balancer sits in front of both central nodes for the Blazor Server UI:
- Health check: `GET /health/ready`
- Protocol: HTTPS (TLS termination at LB or pass-through)
- Sticky sessions: Not required (JWT + shared Data Protection keys)
- If the active node fails, the LB routes to the standby (which becomes active after singleton migration).

## Site Cluster Setup

### Cluster Configuration

Each site has its own two-node cluster:

**Site Node A** (`site-01-a.example.com`):
```json
{
  "ScadaBridge": {
    "Node": {
      "Role": "Site",
      "NodeHostname": "site-01-a.example.com",
      "SiteId": "plant-north",
      "RemotingPort": 8081
    },
    "Cluster": {
      "SeedNodes": [
        "akka.tcp://scadabridge@site-01-a.example.com:8081",
        "akka.tcp://scadabridge@site-01-b.example.com:8081"
      ]
    }
  }
}
```

> **Site Node B reverses this list** — `site-01-b` first, `site-01-a` second — per the self-first seed rule above. It applies to site pairs exactly as it does to the central pair: without it, `site-01-b` cannot boot while `site-01-a` is down.

### Site Cluster Behavior

- Same split-brain resolver as central (`auto-down`, per the 2026-07-21 decision — see the Central Cluster Behavior note above).
- Singleton actors: Site Deployment Manager migrates on failover.
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
  asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
  share a SQLite file.
- CDC capture triggers are installed **only on a node that has replication configured** —
  `LocalDb:Replication:PeerAddress` *or* `LocalDb:Replication:ApiKey`. Either key counts, because
  only the initiating half of a pair sets `PeerAddress` (one bidirectional stream, dialled by one
  side); the passive half carries the key alone. A deliberately unreplicated node — site-b and
  site-c on the rig — runs with no triggers at all and stops paying the per-write capture cost.
- **Stale-trigger cleanup is automatic** (LocalDb 0.2.0). A node with no replication configured does
  not merely skip registration — at boot it calls `DeregisterReplicated` on all ten tables, dropping
  any capture triggers an earlier build installed and pruning those tables' oplog and row-version
  rows. It is idempotent, so a file that was never registered reports nothing to clean; when
  something *was* cleaned the node logs it once at Information. Recreating the data volume is no
  longer required to stop an in-place-upgraded node from capturing.

#### Turning replication ON for a site that has been running without it

**Supported as of LocalDb 0.2.0.** Set the keys on **both** nodes and restart both (see the
stop-and-start-together rule below — this is a pair-wide change, not a rolling one). Existing rows
are carried across:

- **Pre-existing rows are baselined automatically.** ScadaBridge registers every replicated table
  with `baselineExistingRows: true`. Capture is change-data-capture, so rows written while the node
  had no triggers appear in neither the oplog nor `__localdb_row_version` — and LocalDb's snapshot
  resync streams from that ledger. Baselining seeds the ledger for those rows at the LWW floor
  (HLC `0`, stamped with the node's own id) and flags a snapshot resync, so the peer actually
  receives them. Copying one node's database onto the other beforehand is no longer necessary.
- **What the floor means for conflicts.** Every genuine HLC is a UTC millisecond shifted left 16
  bits, so it is strictly greater than `0`: a baselined row loses to any real remote write of the
  same key and wins only where the peer holds no version of that key at all. The one ambiguous case
  is **both** nodes baselining the same key (e.g. both were restored from the same legacy file) —
  both hold HLC `0` and the node-id tie-break decides. That is convergent but arbitrary as to which
  content survives, so if the two files may disagree on a key, start both nodes from one node's
  database.
- **Seeding is idempotent** (`ON CONFLICT DO NOTHING`), so a row that already has a genuine version
  keeps it and no snapshot is flagged. Booting with baselining on every start is free after the
  first.
- **The flag is per-table as of LocalDb 0.3.0.** Turning replication on for the first time still
  seeds all ten tables at once, so the flagged set is every registered table and the pair exchanges
  an ordinary full snapshot — that part is unchanged, so a site whose `site_events` table is at its
  1 GB cap should still be purged before the first enable, and one slow first sync expected. What
  changes is every *later* edit: adding an eleventh table to
  `SiteLocalDbSetup.ReplicatedTables` on a site that is already replicating now snapshots that one
  table, where through 0.2.x the single per-database flag re-streamed all eleven in full, in both
  directions at once. Upgrading a replicating pair to 0.3.0 in place is a no-op for this: the ten
  tables are already ledgered, so nothing seeds and nothing is flagged.
- **Mixed 0.2.x/0.3.0 versions still sync.** Scoped snapshots are negotiated per session by
  capability, and a peer that does not advertise it is sent a full snapshot — so the pin bump does
  not have to be simultaneous for replication's sake. (It still has to be simultaneous for the
  stop-and-start-together rule below, which is a separate constraint.)

Turning replication back **OFF** is likewise a both-nodes change: deregistration must be symmetric,
because the sync handshake compares the two nodes' registered-table digests fail-closed — a node
that drops a table its peer still replicates stops syncing with a schema-mismatch error rather than
diverging silently. Turning it on again later re-baselines, which is what makes the ledger prune on
deregistration safe.

#### Reading the replication backlog — the daily `site_events` purge burst

The site health report carries `LocalDbReplicationConnected` and `LocalDbOplogBacklog` (nullable —
**null means "no reading", not "disconnected with an empty backlog"**), and the same numbers export
as the `localdb_*` Prometheus series (`localdb_oplog_depth` is the backlog gauge). A healthy pair
sits at a backlog of roughly zero, so a sudden spike naturally reads as a replication problem.

**One expected spike is not a problem: the daily `site_events` retention purge.** `site_events` is
one of the ten replicated tables, and its retention DELETE is captured by CDC exactly like an
ordinary write — by design, since LocalDb Phase 2 there is deliberately no purge-exemption path
(the same property that makes a mass DELETE dangerous, which is why `ReplaceAllAsync` was deleted
rather than reinstated). One oplog row is therefore queued per deleted event, and the backlog jumps
by the size of the batch the tick deleted.

**The spike is bounded per tick** (register row 34, measured:
`docs/plans/2026-08-15-site-events-purge-burst-measurement.md`). One purge tick deletes at most
`ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick` rows — default **250 000**, across the retention
purge and the cap trim combined — so the height of the spike is a property of the *configuration*,
not of how much history had expired. That matters because the burst used to be unbounded: a full
1 GB event log holds ~2.73 million rows, and expiring all of them at once queued 2 731 947 oplog
rows, **2.7x LocalDb's `MaxOplogRows` (default 1 000 000)** — enough for the maintenance tick to
prune the oplog and flag a *full ten-table snapshot resync*. A healthy peer blunts that (the oplog
drains while the purge runs — measured peak was 391 000, not 2.73 M), which is precisely the problem:
the old safety margin was a function of how fast the peer happened to be, and vanished when the peer
was down. Set `MaxPurgedRowsPerTick` to `0` to restore the old unbounded behaviour.

- **When.** Every `ScadaBridge:SiteEventLog:PurgeInterval` (default **24 h**), plus once at
  startup — **except** after a tick that hit the ceiling, which schedules the next tick after
  `ScadaBridge:SiteEventLog:PurgeCatchUpInterval` (default **5 minutes**) instead. So a large
  backlog appears as a *run* of bounded spikes minutes apart rather than one huge one; a
  worst-case 2.73 M-row backlog converges in ~11 ticks ≈ 55 minutes. The cadence is anchored to the
  **end of the previous purge**, not to a wall-clock hour, so the burst lands at a different time of
  day after each failover or restart — do not expect it at a fixed hour. The storage-cap trim
  (default 1 GB) can produce the same shape off-schedule, and draws from the same per-tick budget.
- **Where it shows.** Only on a node with replication configured — the rig's site-a. site-b/site-c
  have no capture triggers at all and report no backlog for a purge.
- **What healthy looks like.** `LocalDbReplicationConnected` stays **true** across the spike, and
  the backlog drains back to ~0 as the peer acks the batch. Measured on a loopback pair at
  ~10 800 tombstones/second, so a full 250 000-row tick drains in well under a minute (delta
  messages are bounded by `LocalDb:Replication:MaxBatchBytes`, default 2 MB, and secondarily by
  `MaxBatchSize`). No dead letters, no schema-mismatch errors.
- **What is actually wrong.** `LocalDbReplicationConnected` **false** while the backlog climbs, a
  backlog that keeps rising across successive readings rather than draining, or a backlog that
  never returns near zero between bursts. Those point at the sync stream — an ApiKey mismatch
  (fail-closed: the pair simply stops converging), an unreachable peer, or an asymmetric
  registered-table set.

**Correlating it in the log.** When a purge on a replication-enabled node actually deletes rows it
logs an Information line next to the purge count:

```
Purged 41230 events older than 30 days
Purged 41230 site_events rows on a replication-enabled node — a transient LocalDb oplog backlog
of up to that many rows is expected while the deletes replicate to the peer. It drains on its own;
LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.
A tick deletes at most 250000 rows, so the spike is bounded by that number rather than by how much
history had expired.
```

An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
active node's log, the purge is *not* the explanation and the spike is worth investigating.

The second line is emitted **once per purge tick**, and its count covers the retention purge and
the storage-cap trim together. If you are comparing it against the `Purged N events older than…`
line above it, expect the two numbers to differ whenever the cap trim also ran — the note reports
the tick's whole contribution to the oplog, which is what a backlog spike is made of. (Through
2026-08-16 the note was emitted per phase, so a tick that ran both produced two near-identical
lines; register row 84.)

**A tick that hit the ceiling adds a Warning**, rate-limited to one line per 15 minutes so a
converging backlog does not flood the log:

```
Event log purge stopped at its per-tick ceiling of 250000 rows (250000 deleted); expired events
remain and the next purge runs in 00:05:00 instead of 1.00:00:00.
```

Every hit — throttled line or not — increments the counter
`scadabridge_site_event_log_purge_ceiling_hits`. **Read the counter's shape, not a single hit:** a
short run of hits that stops is a backlog converging (normal, e.g. the first purge after enabling
replication on a long-running site); hits that never stop mean the site writes events faster than
one paced tick can purge, i.e. `MaxPurgedRowsPerTick` or `PurgeCatchUpInterval` is mis-sized for
that site's event rate.

**Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
row 33):

- **The "Replication backlog" badge** on the site tile of the Health dashboard. It is deliberately
  a *sustained* signal, not a bare threshold: the badge appears only once the backlog has stayed
  above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
  `ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
  purge burst described above drains in seconds to a couple of minutes, so it never reaches the
  dwell and never badges the site — which is the entire reason the dwell exists. **The per-tick
  ceiling reinforces that rather than threatening it:** pacing lowers the spike (250 000 rows
  maximum instead of an unbounded 2.7 M) at the cost of more ticks, and a 250 000-row tick was
  measured draining in well under a minute — even allowing 10x for a real inter-node link that is
  ~4 minutes, against a 15-minute dwell and a 5-minute gap to the next tick. (The *unpaced* 2.73 M
  burst drained in 6.0 minutes on loopback: inside the dwell, but with only 2.5x of margin, so a
  materially slower link would have badged.) A bare threshold
  would fire once a day on every replicated site and be ignored within a week. The dwell is
  validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
  single sample and would collapse back into a bare threshold.
- **The "LocalDb Replication Backlog" trend chart** in the dashboard's per-site Site Health Trends
  panel, backed by the KPI history store (sampled every minute, retained 90 days raw / 365 days
  hourly). This is what answers the question a point-in-time reading cannot: *did that spike
  drain?* A past purge burst appears as a spike that returns to ~0; a real fault appears as a
  level that never comes back down.

**Both surfaces are silent on an unreplicated site, by design.** `LocalDbOplogBacklog` and
`LocalDbReplicationConnected` are nullable, and null means "no reading", never zero. The KPI
sampler therefore writes **no sample at all** rather than a fabricated 0, so site-b/site-c show an
empty chart rather than a flat zero line, and they never accumulate the dwell anchor the badge
keys off. A gap in those series means "not replicated / not reporting" — which is the truth. Note
that `false` and null differ for the connected flag: a configured link that is DOWN samples as 0,
while no link at all samples as nothing.

### Site Pair Upgrades — stop and start BOTH nodes together

**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby
could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler
along with the replicator, so a mixed-version pair has no common replication path: the two nodes
will run, but they will not converge, and the divergence is silent.

Stop both nodes of a site pair, upgrade both, then start both.

**A release that adds a column to a replicated table is the same constraint, enforced louder.**
`ReplicatedTable.ComputeDigest` hashes every column name and declared type, and the sync handshake
compares those digests fail-closed — so a mixed-version pair whose builds disagree about a table's
columns does not diverge silently, it refuses to sync with a `LocalDbSchemaMismatchException`.
Stopping and starting both nodes together, which is already the rule above, is the whole fix; no
manual trigger or oplog migration is needed, because `RegisterReplicated` re-reads
`PRAGMA table_info` on every boot and rebuilds the capture triggers (DROP + CREATE) over the
current column set. The site schema's own `AddColumnIfMissing` runs before registration on that
same boot, so a node picks up the new columns and the matching triggers in one start.
(Most recent instance: `external_systems` gained `max_retries` / `retry_delay_ms` for the
per-system store-and-forward retry policy.)

**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
nothing left to suppress the stale row the returning node still holds. Within the retention window
a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
database from its peer rather than letting it rejoin.

### Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM

SQLite honours `PRAGMA auto_vacuum` only on a table-less database, and until arch-review
2026-08-19 P4-6 the pragma ran AFTER LocalDb had created its bookkeeping tables — so every site
file created before that fix is `auto_vacuum=NONE`, and the event-log cap trim's
`PRAGMA incremental_vacuum` is a silent no-op on it: freed `site_events` pages are reused, but the
file never shrinks (deferred-work register row 104). Fresh files are now created
`auto_vacuum=INCREMENTAL`; existing files need a **full `VACUUM` with the pragma set**, which
rewrites the file and is therefore deliberately NOT done at boot (minutes of exclusive lock on a
~1 GB file).

Run it in a maintenance window with **both nodes of the pair stopped** (the rule above):

```bash
DRY_RUN=1 bash docker/vacuum-site-localdb.sh --all   # report: 0 = NONE, 2 = INCREMENTAL
bash docker/vacuum-site-localdb.sh a                 # one pair: stop both → VACUUM both → start both
```

The script integrity-checks each file before and after and leaves the pair stopped if the
post-check fails. It is safe for a replicated pair: `VACUUM` fires no triggers and changes no
row content, the LWW ledger keys by declared primary key, and the oplog `seq` watermark is an
`INTEGER PRIMARY KEY AUTOINCREMENT` — all preserved by `VACUUM`. Verified on the docker rig
2026-08-20 on all three pairs: site-a (replicated) came back `LocalDbReplicationConnected=true`
with backlog 0 and no schema or watermark change. For a non-docker deployment the recipe is the
same three statements against the stopped node's `LocalDb:Path` file:
`PRAGMA auto_vacuum = INCREMENTAL; VACUUM; PRAGMA integrity_check;`.

### Docker-rig data files on macOS/OrbStack — a virtiofs corruption + availability hazard

**Status: MITIGATED 2026-08-18 (register row 79) — the rigs' data dirs moved to named docker
volumes**, taking virtiofs out of the SQLite write path entirely. The hazard description and the
proven recovery runbook below are retained because they explain *why* the volumes exist, and
because failure mode 2 (the VM idle-sleep) is unaffected by the move.

**Scope: the local `docker/` and `docker-env2/` development rigs on macOS only.** This is a
host-level property of the filesystem the rig's data files sit on, **not a ScadaBridge defect** and
not something a production Linux deployment is exposed to. It is documented here because it has
already cost two databases and half a session, and because the recovery is non-obvious.

Until 2026-08-18, every ScadaBridge data file on the rig rode a **relative bind mount** out of the
repo working tree — `./site-a-node-a/data:/app/data` and its siblings, which is where
`site-localdb.db`, `auditlog.db` and the secrets store all live. On macOS that path crosses
**virtiofs** to reach the Linux VM, and **virtiofs does not give SQLite true `fsync` durability**.
SQLite's crash-safety contract assumes a successful `fsync` means the bytes are on stable storage;
when that is not true, an abrupt container kill mid-write can leave a torn file.
`docker compose up --force-recreate` — which `deploy.sh` uses — is exactly such an abrupt kill.
Today the same paths are **named volumes** (`scadabridge-<node>-data`,
`scadabridge-env2-<node>-data`) on native ext4 inside the OrbStack VM, where `fsync` is real.

Two distinct failure modes have been observed live (both 2026-08-15):

**1. Silent database corruption.** Both site-a LocalDb files were found malformed:

```
SQLite Error 11: 'database disk image is malformed'
```

Note how differently the two nodes presented, because this is the part that makes it hard to spot:

- **site-a-b** surfaced it as a *running* fault — every sync call threw, replication reported
  `connected: false` with a backlog of 892, and the faulted session retried about once a minute.
  The node stayed up and otherwise healthy-looking.
- **site-a-a** was corrupt too, but **latently** — nothing on the running build ever read the
  damaged pages. It only became fatal on the *next upgrade*, when a new build's boot-time
  re-registration read pages the old build never touched: `SqliteLocalDb` construction threw and
  the container **SIGABRT crash-looped** (×12).

The lesson to carry: a clean-looking node proves nothing about its file. Corruption can sit latent
until an unrelated code change happens to read further into the database.

**2. The OrbStack VM idle-sleeping mid-operation.** The VM can suspend under its idle policy while
work is in flight. `~/.orbstack/log/vmgr.log` records it plainly:

```
vmgr ... msg=sleep
```

On wake, the virtiofs layer logs repeated errors of the form:

```
refresh_nodeid: race with new lookup
```

and `docker` hangs — commands stop returning, `docker ps` comes back empty — for as long as it
takes to notice. **The fix is `orb restart docker`**; data survives and containers auto-restart.
Observed cost: roughly 20 minutes of a session lost to diagnosing what looked like a cluster
failure but was a host one. If the rig goes unresponsive as a whole, check `vmgr.log` **before**
investigating ScadaBridge.

#### Recovery runbook — LocalDb corruption on a site pair (proven)

This is the procedure that recovered site-a. It relies on the pair being able to reseed itself,
so it is a *replicated* pair's recovery; an unreplicated node has no peer to reseed from and must
be restored from a backup instead.

1. **Stop BOTH nodes of the site pair, together.** Not one at a time — this is the same
   stop-and-start-together rule as *Site Pair Upgrades* above, for the same reason.
2. **Quarantine the database and its sidecars — do not delete them.** Rename all three, keeping
   them for forensics. The data dir now lives in the node's named volume, so run the renames
   through a throwaway container (don't write through the host-side `~/OrbStack` view):

   ```sh
   # per affected node — volume name is scadabridge-<node>-data
   docker run --rm -v scadabridge-site-a-node-a-data:/d alpine sh -c '
     cd /d
     mv site-localdb.db     site-localdb.db.corrupt-$(date +%F)
     mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F)
     mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F)'
   ```

   The `-wal` and `-shm` files **must** move with the main file. Leaving a stale WAL beside a fresh
   database is its own corruption.
3. **Start both nodes together.** The fresh files reseed themselves from two independent paths:
   `SiteReconciliationActor`'s node-startup fetch pulls config back from central when central
   reports gaps, and LocalDb's **CDC snapshot resync** merges the peer's rows under LWW.
4. **Verify convergence**, do not assume it: `LocalDbReplicationConnected` should read `true` and
   `LocalDbOplogBacklog` should settle to ~0 on the site health report. (See *Reading the
   replication backlog* above for why a transient spike is normal and not a fault.)

On the live recovery, both quarantined files were large (394 MB and 416 MB), the pair converged to
`connected: true` / backlog 0, and a schema change in flight at the time additionally proved the
digest handshake green.

#### Mitigation applied (2026-08-18): named docker volumes

The "accept + runbook" position was retired without waiting for a second corruption. All ten
data dirs (8 `docker/` nodes + 2 `docker-env2/` site nodes) now ride named volumes
(`scadabridge-<node>-data` / `scadabridge-env2-<node>-data`), declared in each compose file with
explicit `name:`s and seeded from the old bind-mount dirs at migration time. What changed the
original trade-off calculus: the assumed cost — losing host-side inspectability — turned out to
be much smaller than documented, because OrbStack exposes every volume on the Mac at
`~/OrbStack/docker/volumes/<name>/`, so `site-localdb.db` can still be opened from the
host for **read-only inspection**. (Never write through that path: it reintroduces exactly the
virtiofs layer the volumes exist to avoid.) The appsettings and logs bind mounts deliberately
stayed: config is read-only, and Serilog's append-only log files have no fsync-durability
requirement worth losing `tail -f` from the repo tree for.

Residual exposure: **failure mode 2 is availability-only and still real** — the VM can still
idle-sleep mid-operation and hang `docker` until `orb restart docker`; data now survives that
on ext4. Disabling OrbStack's idle pause remains an un-version-controllable host setting and was
left alone. The old `docker/<node>/data/` dirs were left in place as a dormant pre-migration
backup; they are no longer mounted and can be deleted once the volumes have proven themselves.

### Central-Site Communication

Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed
in Phase 4 of the ClusterClient→gRPC migration (2026-07-23), and Akka remoting no longer crosses the
boundary at all:**

- **gRPC command/control** — both directions, on sticky-failover channel pairs, dialled directly (no
  receptionist, no "active central" to identify — each side dials both of the peer's node endpoints):
  - *Site → central* to the central-hosted **`CentralControlService`** (`GrpcCentralTransport`): the
    site lists the central nodes' gRPC endpoints in `ScadaBridge:Communication:CentralGrpcEndpoints`
    (e.g. `http://scadabridge-central-a:8083`, the central's `CentralGrpcPort`, default 8083 — direct
    h2c, **not** via Traefik, which is HTTP/1 only). A Site node must list at least one; central nodes
    leave it empty.
  - *Central → site* to the site-hosted **`SiteCommandService`** (`GrpcSiteTransport`): central dials
    the site's `GrpcNodeAAddress` / `GrpcNodeBAddress` (from the Site entity), NodeA→NodeB failover.
- **gRPC streaming + audit pull** — real-time data and audit/telemetry pull on the site-hosted
  **`SiteStreamService`**. Note the direction is inverted from the data flow: each **site node hosts
  the server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.

#### gRPC control-plane preshared key (required)

Every site node must set `ScadaBridge:Communication:GrpcPsk`, and central must hold the same
value for that site. **`StartupValidator` refuses to boot a site node without it**, deliberately:
the gate is fail-closed, so an unset key would leave the node joined, healthy-looking and
answering heartbeats while refusing every gRPC call — no live subscriptions, no audit pull, no
cached-telemetry ingest.

| Side | Where the key lives |
|---|---|
| Site node (both nodes of the pair, identical) | `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}` |
| Central | secret `SB-GRPC-PSK-<siteId>` in its store — **or** `ScadaBridge:Communication:SitePsks:<siteId>` |

The store is the source that matters in production, because sites are added at runtime and their
keys cannot be enumerated in configuration at boot; `SitePsks` covers a host running without a
master key (the docker rig) and one-off pins.

One key **per site**, never one for the fleet: a compromised site must not yield another site's
key. And never share it with `LocalDb:Replication:ApiKey` — that authenticates the *pair partner*
for database replication, a different trust relationship on the same listener.

**Rotation:** set the new value on both sides, then restart the pair (pairs restart together
anyway — see above). **Upgrading to a build that has this gate requires seeding the key first**,
including in the on-host `deploy/` overlays.

The key is a bearer token over plaintext h2c, so it is readable and replayable by anyone on the
path. That is the accepted posture today — the same trusted-network assumption the boundary
already made, now with authentication rather than none. TLS on these listeners is follow-on
hardening and needs no change to the key design.

#### Secrets: shared store, per-site deploy, and the KEK

Site secrets ride the artifact deploy (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`;
see that doc's *Implementation notes (as built)* for where the shipped code deviates from the
original design). This section is the operational summary; component detail lives in
Component-Host.md → *Secrets*, Component-DeploymentManager.md → *Site secrets*, and
Component-SiteRuntime.md → *Deployed Secrets*.

- **Central**: `Secrets:Store:Provider = SqlServer` on both central nodes — see *Shared State*
  above. `SiteSecretsCollector` resolves each site's referenced secrets from that store at deploy
  time; a missing/tombstoned secret fails that site's deploy before anything is sent.
- **Every node must share the same KEK** (`ZB_SECRETS_MASTER_KEY` / master-key file) — **every
  central node and every site node**, not just the central pair. This is the same requirement the
  retired gRPC secrets hub had; it did not change. **Nothing checks this at boot or at the next
  reconnect** — a mismatch surfaces only at **deploy time**, on the site: `DeployedSecretsMirror`'s
  verify step trial-decrypts every applied secret and reports a distinct error naming the secret
  and its `KekId` if the local master key cannot open it. Verify the KEK is identical everywhere
  **before** turning this on for a site — the rig's `docker/docker-compose.yml` deliberately runs
  site-b/site-c on a mismatched KEK precisely to exercise this failure path live.
- **Standby / passive-node convergence.** The active site node applies a deploy's secrets
  synchronously (so the deploy response reflects that apply immediately); the standby node never
  receives that synchronous call — it converges through ordinary LocalDb replication of the
  `deployed_secrets` table plus its **own** `DeployedSecretsMirror` running on its own schedule,
  `ScadaBridge:SiteRuntime:DeployedSecretsMirrorInterval` (default **15s**). A freshly-promoted
  standby is therefore current on secrets within one interval of the last deploy, not instantly.
- **A site pair that uses deployed secrets MUST run LocalDb replication** (`LocalDb:Replication:*`
  configured on both nodes — see *Reading the replication backlog*). The standby's only path to the
  secrets is the replicated `deployed_secrets` table. On an **unreplicated** pair only the node that
  was active at deploy time holds the secrets; after a failover the new active node has **none**
  (or a stale hand-seeded copy) until the next *Deploy Artifacts*, and a removal applied while one
  node was active leaves the secret live on the other. The Artifacts badge still reads *Current* —
  the stamp is per site, not per node. If a pair cannot be replicated, re-run *Deploy Artifacts*
  after every failover.
- **Secret rotation and live connections.** An artifact deploy writes and mirrors the shipped
  secrets **before** it pushes data-connection definitions to the DCL, so a connection created or
  reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation
  that leaves the connection's config unchanged does **not** recreate the connection (the push is
  hash-guarded): the new value takes effect on that connection's **next (re)connect**.
- **Rollout is a deployment record, not a code change** — see the deferred-work register (row 127)
  for the vd03/QA prerequisites (shared store provisioning, `Provider=SqlServer` on both centrals,
  the KEK alignment check above, LocalDb replication on every site pair that uses secrets,
  recreating any hand-seeded site secrets centrally, then *Deploy Artifacts*).
- **Upgrade impact — do the prerequisites in the same window as the upgrade.** Once a central runs
  this build, any site whose data connections carry a `secret:` reference fails its **whole**
  artifact deploy (shared scripts, external systems and DB connections included, not just secrets)
  and its Artifacts badge reads *Out of date*, until every referenced secret exists in the central
  store. Before upgrading an environment, check whether its connection configs use `secret:`
  references and whether its site pairs run LocalDb replication.
- **The gRPC secrets hub is retired in ScadaBridge** — `Secrets:Replication:Enabled` now fails
  startup on either role (Component-Host.md → *Secrets*). QA and vd03 currently run
  `Replication:Enabled=false`, so the boot itself will not fail fast — the deploy-time impact above
  is what bites.

## Scaling Guidelines

### Target Scale

- 10 sites maximum per central cluster
- 500 machines (instances) total across all sites
- 75 tags per machine (37,500 total tag subscriptions)

### Resource Requirements

| Component | CPU | RAM | Disk | Notes |
|-----------|-----|-----|------|-------|
| Central node | 4 cores | 8 GB | 50 GB | SQL Server is separate |
| Site node | 2 cores | 4 GB | 20 GB | SQLite databases grow with S&F |
| SQL Server | 4 cores | 16 GB | 100 GB | Shared across central cluster |

#### Sizing a site's data volume — `MaxStorageMb` is not the whole story

`ScadaBridge:SiteEventLog:MaxStorageMb` (default **1 GB**) bounds **`site_events` and its four
indexes**, and nothing else. It is deliberately *not* a cap on the consolidated LocalDb file
(register row 81 — decision record
`docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). Before that change it did
measure the whole file, which on a replicated node made the cap unenforceable: the trim's own CDC
oplog rows and tombstones grew the file as fast as the deletes shrank it, and the trim stopped
2.5x over cap. Tying the event log's budget to the peer's liveness was the defect, not a feature
worth preserving.

So budget the volume as the **sum** of:

- **`MaxStorageMb`** — the event log proper.
- **LocalDb replication bookkeeping** — `__localdb_oplog` and `__localdb_row_version`, bounded
  separately by `LocalDb:Replication:MaxOplogRows` (default 1 000 000) and `TombstoneRetention`
  (default 7 days). This is transient: it drains as the peer acks and as tombstones expire, but it
  is not small during a burst. Measured worst case (register row 34): purging a full 1 GB event log
  left **707 MB** of bookkeeping in the file. Unreplicated sites carry none of this — they have no
  capture triggers at all.
- **The other eight consolidated tables** plus the separate, unreplicated audit database at
  `AuditLog:SiteWriter:DatabasePath`.

The 20 GB figure above has ample headroom for all three at default settings. Raising
`MaxStorageMb` on a replicated site should be paired with a look at the oplog settings, not done
in isolation.

### Network Bandwidth

- Health reports: ~1 KB per site per 30 seconds = negligible
- Tag value updates: Depends on data change rate; OPC UA subscription-based
- Deployment artifacts: One-time burst per deployment (varies by config size)
- Debug view streaming: ~500 bytes per attribute change per subscriber

## Dual-Node Failure Recovery

### Scenario: Both Nodes Down

1. **First node starts**: Forms a single-node cluster (`min-nr-of-members = 1`).
2. **Central**: Reconnects to SQL Server, reads deployment state, becomes operational.
3. **Site**: Opens SQLite databases, rebuilds Instance Actors from persisted configs, resumes S&F retries.
4. **Second node starts**: Joins the existing cluster as standby.

### Automatic Recovery

No manual intervention required for dual-node failure. The first node to start will:
- Form the cluster
- Take over all singletons
- Begin processing immediately
- Accept the second node when it joins