Joseph Doherty
bdee12f4e9
Replace ASCII-art diagrams across the README and docs/ with editable .drawio sources plus exported PNGs, so the diagrams render clearly in rendered markdown and can be maintained/regenerated instead of being hand-edited as fragile text art. Non-diagram blocks (code, folder trees, UI wireframes) were left as text.
132 lines
5.3 KiB
Markdown
132 lines
5.3 KiB
Markdown
# Primary/Backup Data Connection Endpoints — Design
|
|
|
|
**Date:** 2026-03-22
|
|
**Status:** Approved
|
|
|
|
## Problem
|
|
|
|
Data connections currently support a single endpoint. If that endpoint goes down, the connection retries indefinitely at 5s intervals against the same address. When redundant infrastructure exists (e.g., two OPC UA servers), there is no way to automatically fail over to a backup.
|
|
|
|
## Design Decisions
|
|
|
|
| Decision | Choice |
|
|
|----------|--------|
|
|
| Failover mode | Automatic after N failed retries |
|
|
| Failback | No auto-failback; stay on active until it fails (round-robin) |
|
|
| Backup required? | Optional — single-endpoint connections work unchanged |
|
|
| Failover trigger | After configurable retry count (default 3) |
|
|
| Entity model | Separate `PrimaryConfiguration` and `BackupConfiguration` columns |
|
|
| UI approach | Two JSON text areas; backup collapsible |
|
|
| Failover logic location | DataConnectionActor (adapters stay single-endpoint) |
|
|
| Observability | Health reports + site event log entries |
|
|
|
|
## Entity Model
|
|
|
|
**`DataConnection` changes:**
|
|
|
|
| Field | Type | Notes |
|
|
|-------|------|-------|
|
|
| `PrimaryConfiguration` | string? (max 4000) | Renamed from `Configuration` |
|
|
| `BackupConfiguration` | string? (max 4000) | New. Null = no backup |
|
|
| `FailoverRetryCount` | int (default 3) | New. Retries before switching |
|
|
|
|
Both endpoints use the same `Protocol`. EF Core migration renames `Configuration` → `PrimaryConfiguration` (data-preserving).
|
|
|
|
**`DataConnectionArtifact` changes:**
|
|
- `ConfigurationJson` → `PrimaryConfigurationJson` + `BackupConfigurationJson`
|
|
|
|
## Failover State Machine
|
|
|
|
The `DataConnectionActor` Reconnecting state is extended:
|
|
|
|
|
|
<!-- source: diagrams/primary-backup-failover-state-machine.drawio — edit, then re-export with export-drawio.sh -->
|
|
|
|
**On successful reconnect (either endpoint):**
|
|
1. Reset `_consecutiveFailures = 0`
|
|
2. `ReSubscribeAll()` — re-create all subscriptions on the new adapter
|
|
3. Transition to Connected
|
|
4. Log failover event if endpoint changed
|
|
5. Report active endpoint in health metrics
|
|
|
|
**Round-robin on failure:** primary → backup → primary → backup...
|
|
|
|
**Adapter lifecycle on failover:** Actor disposes current `IDataConnection` adapter and creates a fresh one via `DataConnectionFactory.Create()` with the other endpoint's config. Clean slate — no stale state.
|
|
|
|
## Actor State
|
|
|
|
New fields in `DataConnectionActor`:
|
|
|
|
- `IDictionary<string, string> _primaryConfig`
|
|
- `IDictionary<string, string>? _backupConfig`
|
|
- `ActiveEndpoint _activeEndpoint` (enum: Primary, Backup)
|
|
- `int _consecutiveFailures`
|
|
- `int _failoverRetryCount`
|
|
|
|
`CreateConnectionCommand` gains: `primaryConfig`, `backupConfig`, `failoverRetryCount`.
|
|
|
|
`DataConnectionFactory` is unchanged — still creates single-endpoint adapters.
|
|
|
|
## Health & Observability
|
|
|
|
**`DataConnectionHealthReport`** gains:
|
|
- `ActiveEndpoint` (string): `"Primary"`, `"Backup"`, or `"Primary (no backup)"`
|
|
|
|
**Site event log entries:**
|
|
- `DataConnectionFailover` — connection name, from-endpoint, to-endpoint, reason
|
|
- `DataConnectionRestored` — connection name, active endpoint
|
|
|
|
Uses existing `ISiteEventLogger`.
|
|
|
|
## Central UI
|
|
|
|
**List page:** Add `Active Endpoint` column from health reports.
|
|
|
|
**Form (Create/Edit):**
|
|
- "Primary Endpoint Configuration" label (renamed from "Configuration")
|
|
- "Add Backup Endpoint" button reveals second JSON text area
|
|
- "Remove Backup" button in edit mode when backup exists
|
|
- "Failover Retry Count" numeric input (default 3, min 1, max 20) — visible only when backup configured
|
|
- Vertical stacking, collapsible backup subsection
|
|
|
|
## CLI
|
|
|
|
- `--configuration` renamed to `--primary-config` (hidden alias for backwards compat)
|
|
- `--backup-config` (optional)
|
|
- `--failover-retry-count` (optional, default 3)
|
|
- `data-connection get` shows both configs and active endpoint
|
|
|
|
## Management API
|
|
|
|
- `CreateDataConnectionCommand` / `UpdateDataConnectionCommand` gain `PrimaryConfiguration`, `BackupConfiguration`, `FailoverRetryCount`
|
|
- Setting `BackupConfiguration` to null removes the backup
|
|
- `GetDataConnectionResponse` returns both configs
|
|
|
|
## Deployment Flow
|
|
|
|
`DataConnectionArtifact` carries `PrimaryConfigurationJson` and `BackupConfigurationJson`. Site-side deployment handler passes both to `CreateConnectionCommand`.
|
|
|
|
## Testing
|
|
|
|
**Unit tests:**
|
|
- Actor: failover after N failures, round-robin, single-endpoint retries forever, counter reset, ReSubscribeAll on failover
|
|
- Manager actor: updated CreateConnectionCommand
|
|
- Factory: unchanged registration
|
|
|
|
**Integration test (manual with test infra):**
|
|
1. Primary=`opc.tcp://localhost:50000`, backup=`opc.tcp://localhost:50010`
|
|
2. Subscribe to `Motor.Speed`
|
|
3. `docker compose stop opcua` → verify failover to opcua2 after 3 retries
|
|
4. `docker compose stop opcua2 && docker compose start opcua` → verify round-robin back
|
|
|
|
## Implementation Tasks
|
|
|
|
1. **#4** Entity model & database (foundation)
|
|
2. **#6** CreateConnectionCommand & DataConnectionManagerActor (blocked by #4)
|
|
3. **#5** DataConnectionActor failover state machine (blocked by #4, #6)
|
|
4. **#7** Health reporting & site event log (blocked by #5)
|
|
5. **#8** Central UI (blocked by #4)
|
|
6. **#9** CLI, Management API, deployment (blocked by #4)
|
|
7. **#10** Documentation (blocked by #5)
|
|
8. **#11** Tests (blocked by #5)
|