dohertj2/lmxopcua
1
0
Fork 0
Code Issues 1 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add configurable non-transparent OPC UA server redundancy

Browse Source 
Separates ApplicationUri from namespace identity so each instance in a
redundant pair has a unique server URI while sharing the same Galaxy
namespace. Exposes RedundancySupport, ServerUriArray, and dynamic
ServiceLevel through the standard OPC UA server object. ServiceLevel
is computed from role (Primary/Secondary) and runtime health (MXAccess
and DB connectivity). Adds CLI redundancy command, second deployed
service instance, and 31 new tests including paired-server integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-03-28 13:32:17 -04:00
parent a3c2d9b243
commit a55153d7d5
27 changed files with 1475 additions and 248 deletions
Download Patch File Download Diff File Expand all files Collapse all files

561
redundancy.md
View File

@@ -2,75 +2,117 @@
## Summary
Add configurable non-transparent warm/hot redundancy to the LmxOpcUa server so that two instances sharing the same Galaxy repository can operate as a redundant pair. Each instance advertises itself and its partner through the OPC UA `ServerRedundancy` node, publishes a dynamic `ServiceLevel` reflecting runtime health, and allows clients to discover the redundant set and fail over between instances. The CLI tool gains a `redundancy` command for inspecting the redundant server set.
Add configurable non-transparent warm/hot redundancy to the LmxOpcUa server so that two instances sharing the same Galaxy repository can operate as a redundant pair. Each instance should advertise the redundant set through the standard OPC UA redundancy nodes, publish a dynamic `ServiceLevel` based on runtime health, and allow clients to discover and fail over between the instances. The CLI tool should gain a `redundancy` command for inspecting the redundant server set.
This plan covers server-side redundancy exposure, client-side discovery, a second deployed service instance, documentation, and tests. It does **not** implement automatic server-side failover or subscription transfer — those are client responsibilities per the OPC UA specification.
This review tightens the original draft in a few important ways:
- It separates **namespace identity** from **application identity**. The current host uses `urn:{GalaxyName}:LmxOpcUa` as both the namespace URI and `ApplicationUri`; that must change for redundancy because each server in the pair needs a unique server URI.
- It avoids hand-wavy "write the redundancy nodes directly" language and instead targets the OPC UA SDK's built-in `ServerObjectState` / `ServerRedundancyState` model.
- It removes a few inaccurate hardcoded assumptions, including the `ServerUriArray` node id and the deployment port examples.
- It fixes execution order so test-builder and helper changes happen before integration coverage depends on them.
This plan still covers server-side redundancy exposure, client-side discovery, a second deployed service instance, documentation, and tests. It does **not** implement automatic server-side failover or subscription transfer; those remain client responsibilities per the OPC UA specification.
---
## Background: OPC UA Redundancy Model
OPC UA defines redundancy through three address-space nodes under `Server/ServerRedundancy`:
OPC UA exposes redundancy through standard nodes under `Server/ServerRedundancy` plus the `Server/ServiceLevel` property:
| Node | Type | Purpose |
|---|---|---|
| `RedundancySupport` | `RedundancySupport` enum | Declares the redundancy mode: `None`, `Cold`, `Warm`, `Hot`, `Transparent`, `HotAndMirrored` |
| `ServerUriArray` | `String[]` | Lists the `ApplicationUri` values of all servers in the redundant set (non-transparent modes) |
| `ServiceLevel` | `Byte` (0–255) | Indicates current operational quality; clients prefer the server with the highest value |
| `ServerUriArray` | `String[]` | Lists the `ApplicationUri` values of all servers in the redundant set for non-transparent redundancy |
| `ServiceLevel` | `Byte` (0-255) | Indicates current operational quality; clients prefer the server with the highest value |
### Non-Transparent Redundancy (our target)
In non-transparent redundancy (`Warm` or `Hot`), both servers run independently with their own sessions and subscriptions. Clients discover the redundant set by reading `ServerUriArray`, monitor `ServiceLevel` on each server, and manage their own failover. This model fits our architecture where each instance connects to the same Galaxy repository and MXAccess runtime independently.
In non-transparent redundancy (`Warm` or `Hot`), both servers run independently with their own sessions and subscriptions. Clients discover the redundant set by reading `ServerUriArray`, monitor `ServiceLevel` on each server, and manage their own failover. This fits the current architecture, where each instance independently connects to the same Galaxy repository and MXAccess runtime.
### ServiceLevel Semantics
### ServiceLevel semantics
| Range | Meaning |
|---|---|
| 0 | Server is not operational |
| 1–99 | Degraded (e.g., MXAccess disconnected, DB unreachable) |
| 100–199 | Healthy secondary |
| 200–255 | Healthy primary (preferred) |
| 1-99 | Degraded |
| 100-199 | Healthy secondary |
| 200-255 | Healthy primary |
The primary server should advertise a higher `ServiceLevel` than the secondary so clients prefer it when both are healthy.
The primary should advertise a higher `ServiceLevel` than the secondary so clients prefer it when both are healthy.
---
## Current State
- `LmxOpcUaServer` extends `StandardServer` but does not override any redundancy-related methods
- `ServerRedundancy/RedundancySupport` defaults to `None` (SDK default)
- `ServiceLevel` defaults to `255` (SDK default — "fully operational")
- No configuration for redundant partner URIs or role designation
- Single deployed instance at `C:\publish\lmxopcua\instance1` on port 4840
- No CLI support for reading redundancy information
- `LmxOpcUaServer` extends `StandardServer` but does not expose redundancy state
- `ServerRedundancy/RedundancySupport` remains the SDK default (`None`)
- `Server/ServiceLevel` remains the SDK default (`255`)
- No configuration exists for redundancy mode, role, or redundant partner URIs
- `OpcUaServerHost` currently sets `ApplicationUri = urn:{GalaxyName}:LmxOpcUa`
- `LmxNodeManager` uses the same `urn:{GalaxyName}:LmxOpcUa` as the published namespace URI
- A single deployed instance is documented in [service_info.md](C:\Users\dohertj2\Desktop\lmxopcua\service_info.md)
- No CLI command exists for reading redundancy information
## Key gap to fix first
For redundancy, each server in the set must advertise a unique `ApplicationUri`, and `ServerUriArray` must contain those unique values. The current implementation cannot do that because it reuses the namespace URI as the server `ApplicationUri`. Phase 1 therefore needs an application-identity change before the redundancy nodes can be correct.
---
## Scope
### In Scope (Phase 1)
### In scope (Phase 1)
1. **Redundancy configuration model** — role, partner URIs, ServiceLevel weights
2. **Server redundancy node exposure** — `RedundancySupport`, `ServerUriArray`, dynamic `ServiceLevel`
3. **ServiceLevel computation** — based on runtime health (MXAccess state, DB connectivity, role)
4. **CLI redundancy command** — read `RedundancySupport`, `ServerUriArray`, `ServiceLevel` from a server
5. **Second service instance** — deployed at `C:\publish\lmxopcua\instance2` with non-overlapping ports
6. **Documentation** — new `docs/Redundancy.md` component doc, updates to existing docs
7. **Unit tests** — config, ServiceLevel computation, resolver tests
8. **Integration tests** — two-server redundancy E2E test in the integration test project
1. Add explicit application-identity configuration so each instance can have a unique `ApplicationUri`
2. Add redundancy configuration for mode, role, and server URI membership
3. Expose `RedundancySupport`, `ServerUriArray`, and dynamic `ServiceLevel`
4. Compute `ServiceLevel` from runtime health and preferred role
5. Add a CLI `redundancy` command
6. Document two-instance deployment
7. Add unit and integration coverage
### Deferred
- Automatic subscription transfer (client-side responsibility)
- Server-initiated failover (Galaxy `redundancy` table / engine flags)
- Automatic subscription transfer
- Server-initiated failover
- Transparent redundancy mode
- Health-check HTTP endpoint for load balancers
- Load-balancer-specific HTTP health endpoints
- Mirrored data/session state
---
## Configuration Design
### New `Redundancy` section in `appsettings.json`
### 1. Add explicit `OpcUa.ApplicationUri`
**File:** `src/ZB.MOM.WW.LmxOpcUa.Host/Configuration/OpcUaConfiguration.cs`
Add:
```csharp
public string? ApplicationUri { get; set; }
```
Rules:
- `ApplicationUri = null` preserves the current behavior for non-redundant deployments
- when `Redundancy.Enabled = true`, `ApplicationUri` must be explicitly set and unique per instance
- `LmxNodeManager` should continue using `urn:{GalaxyName}:LmxOpcUa` as the namespace URI so both redundant servers expose the same namespace
- `Redundancy.ServerUris` must contain the exact `ApplicationUri` values for all servers in the redundant set
Example:
```json
{
"OpcUa": {
"ServerName": "LmxOpcUa",
"GalaxyName": "ZB",
"ApplicationUri": "urn:localhost:LmxOpcUa:instance1"
}
}
```
### 2. New `Redundancy` section in `appsettings.json`
```json
{
@@ -84,7 +126,7 @@ The primary server should advertise a higher `ServiceLevel` than the secondary s
}
```
### Configuration model
### 3. Configuration model
**File:** `src/ZB.MOM.WW.LmxOpcUa.Host/Configuration/RedundancyConfiguration.cs` (new)
@@ -99,13 +141,17 @@ public class RedundancyConfiguration
}
```
### Configuration rules
### 4. Configuration rules
- `Enabled` defaults to `false` for backward compatibility. When `false`, `RedundancySupport = None` and `ServiceLevel = 255` (SDK defaults).
- `Mode` must be `Warm` or `Hot` (Phase 1). Maps to `RedundancySupport.Warm` or `RedundancySupport.Hot`.
- `Role` must be `Primary` or `Secondary`. Controls the base `ServiceLevel` (Primary gets `ServiceLevelBase`, Secondary gets `ServiceLevelBase - 50`).
- `ServerUris` lists the `ApplicationUri` values for **all** servers in the redundant set, including the local server. The OPC UA spec requires this to contain the full set. These are namespace URIs like `urn:ZB:LmxOpcUa`, not endpoint URLs.
- `ServiceLevelBase` is the starting ServiceLevel when the server is fully healthy. Degraded conditions subtract from this value.
- `Enabled` defaults to `false`
- `Mode` supports `Warm` and `Hot` in Phase 1
- `Role` supports `Primary` and `Secondary`
- `ServerUris` must contain the local `OpcUa.ApplicationUri` when redundancy is enabled
- `ServerUris` should contain at least two unique entries when redundancy is enabled
- `ServiceLevelBase` should be in the range `1-255`
- Effective baseline:
- Primary: `ServiceLevelBase`
- Secondary: `max(0, ServiceLevelBase - 50)`
### App root updates
@@ -117,27 +163,48 @@ public class RedundancyConfiguration
## Implementation Steps
### Step 1: Add RedundancyConfiguration model and bind it
### Step 1: Separate application identity from namespace identity
**Files:**
- `src/.../Configuration/OpcUaConfiguration.cs`
- `src/.../OpcUa/OpcUaServerHost.cs`
- `docs/OpcUaServer.md`
- `tests/.../Configuration/ConfigurationLoadingTests.cs`
Changes:
1. Add optional `OpcUa.ApplicationUri`
2. Keep `urn:{GalaxyName}:LmxOpcUa` as the namespace URI used by `LmxNodeManager`
3. Set `ApplicationConfiguration.ApplicationUri` from `OpcUa.ApplicationUri` when supplied
4. Keep `ApplicationUri` and namespace URI distinct in docs and tests
This step is required before redundancy can be correct.
### Step 2: Add `RedundancyConfiguration` and bind it
**Files:**
- `src/.../Configuration/RedundancyConfiguration.cs` (new)
- `src/.../Configuration/AppConfiguration.cs`
- `src/.../OpcUaService.cs`
Changes:
1. Create `RedundancyConfiguration` class with properties above
2. Add `Redundancy` property to `AppConfiguration`
1. Create `RedundancyConfiguration`
2. Add `Redundancy` to `AppConfiguration`
3. Bind `configuration.GetSection("Redundancy").Bind(_config.Redundancy);`
4. Pass `_config.Redundancy` through to `OpcUaServerHost` and `LmxOpcUaServer`
### Step 2: Add RedundancyModeResolver
### Step 3: Add `RedundancyModeResolver`
**File:** `src/.../OpcUa/RedundancyModeResolver.cs` (new)
Responsibilities:
- Map `Mode` string to `RedundancySupport` enum value
- Validate against supported Phase 1 modes (`Warm`, `Hot`)
- Fall back to `None` with warning for unknown modes
- map `Mode` to `RedundancySupport`
- validate supported Phase 1 modes
- fall back safely when disabled or invalid
```csharp
public static class RedundancyModeResolver
@@ -146,169 +213,199 @@ public static class RedundancyModeResolver
}
```
### Step 3: Add ServiceLevelCalculator
### Step 4: Add `ServiceLevelCalculator`
**File:** `src/.../OpcUa/ServiceLevelCalculator.cs` (new)
Computes the dynamic `ServiceLevel` byte from runtime health:
Purpose:
- compute the current `ServiceLevel` from a baseline plus health inputs
Suggested signature:
```csharp
public class ServiceLevelCalculator
public sealed class ServiceLevelCalculator
{
public byte Calculate(int baseLine, bool mxAccessConnected, bool dbConnected, bool isPrimary);
public byte Calculate(int baseLevel, bool mxAccessConnected, bool dbConnected);
}
```
Logic:
- Start with `baseLine` (from config, e.g., 200 for Primary, 150 for Secondary)
- Subtract 100 if MXAccess is disconnected
- Subtract 50 if Galaxy DB is unreachable
- Clamp to 0–255
- Return 0 if both MXAccess and DB are down
Suggested logic:
### Step 4: Extend ConfigurationValidator for redundancy
- start with the role-adjusted baseline supplied by the caller
- subtract 100 if MXAccess is disconnected
- subtract 50 if the Galaxy DB is unreachable
- return `0` if both are down
- clamp to `0-255`
### Step 5: Extend `ConfigurationValidator`
**File:** `src/.../Configuration/ConfigurationValidator.cs`
Add validation/logging for:
- `Redundancy.Enabled`, `Mode`, `Role`
- `ServerUris` should not be empty when `Enabled = true`
- `ServiceLevelBase` should be 1–255
- Warning when `Enabled = true` but `ServerUris` has fewer than 2 entries
- Log effective redundancy configuration at startup
### Step 5: Update LmxOpcUaServer to expose redundancy state
- `OpcUa.ApplicationUri`
- `Redundancy.Enabled`, `Mode`, `Role`
- `ServerUris` membership and uniqueness
- `ServiceLevelBase`
- local `OpcUa.ApplicationUri` must appear in `Redundancy.ServerUris` when enabled
- warning when fewer than 2 unique server URIs are configured
### Step 6: Expose redundancy through the standard OPC UA server object
**File:** `src/.../OpcUa/LmxOpcUaServer.cs`
Changes:
1. Accept `RedundancyConfiguration` in the constructor
2. Override `OnServerStarted` to write redundancy nodes:
- Set `Server/ServerRedundancy/RedundancySupport` to the resolved mode
- Set `Server/ServerRedundancy/ServerUriArray` to the configured URIs
3. Override `SetServerState` or use a timer to update `Server/ServiceLevel` periodically based on `ServiceLevelCalculator`
4. Expose a method `UpdateServiceLevel(bool mxConnected, bool dbConnected)` that the service layer can call when health state changes
### Step 6: Update OpcUaServerHost to pass redundancy config
1. Accept `RedundancyConfiguration` and local `ApplicationUri`
2. On startup, locate the built-in `ServerObjectState`
3. Configure `ServerObjectState.ServiceLevel`
4. Configure the server redundancy object using the SDK's standard server-state types instead of writing guessed node ids directly
5. If the default `ServerRedundancyState` does not expose `ServerUriArray`, replace or upgrade it with the appropriate non-transparent redundancy state type from the SDK before populating values
6. Expose an internal method such as `UpdateServiceLevel(bool mxConnected, bool dbConnected)` for service-layer health updates
Important: the implementation should use SDK types/constants (`ServerObjectState`, `ServerRedundancyState`, `NonTransparentRedundancyState`, `VariableIds.*`) rather than hand-maintained numeric literals.
### Step 7: Update `OpcUaServerHost`
**File:** `src/.../OpcUa/OpcUaServerHost.cs`
Changes:
1. Accept `RedundancyConfiguration` in the constructor
2. Pass it through to `LmxOpcUaServer`
3. Log active redundancy mode at startup
### Step 7: Wire ServiceLevel updates in OpcUaService
1. Accept `RedundancyConfiguration`
2. Pass redundancy config and resolved local `ApplicationUri` into `LmxOpcUaServer`
3. Log redundancy mode/role/server URIs at startup
### Step 8: Wire health updates in `OpcUaService`
**File:** `src/.../OpcUaService.cs`
Changes:
1. Bind redundancy config section
2. Pass redundancy config to `OpcUaServerHost`
3. Subscribe to `MxAccessClient.ConnectionStateChanged` to trigger `ServiceLevel` updates
4. After Galaxy DB health checks, trigger `ServiceLevel` updates
5. Use a periodic timer (e.g., every 5 seconds) to refresh `ServiceLevel` based on current component health
### Step 8: Update appsettings.json
1. Bind and pass redundancy config
2. After startup, initialize the starting `ServiceLevel`
3. Subscribe to `IMxAccessClient.ConnectionStateChanged`
4. Update DB health whenever startup repository checks, change-detection work, or rebuild attempts succeed/fail
5. Prefer event-driven updates; add a lightweight periodic refresh only if necessary
Avoid introducing a second large standalone polling loop when existing connection and repository activity already gives most of the needed health signals.
### Step 9: Update test builders and helpers before integration coverage
**Files:**
- `src/.../OpcUaServiceBuilder.cs`
- `tests/.../Helpers/OpcUaServerFixture.cs`
- `tests/.../Helpers/OpcUaTestClient.cs`
Changes:
- add `WithRedundancy(...)`
- add `WithApplicationUri(...)` or allow full `OpcUaConfiguration` override
- ensure two in-process redundancy tests can run with distinct `ServerName`, `ApplicationUri`, and certificate identity
- when needed, use separate PKI roots in tests so paired fixtures do not collide on certificate state
### Step 10: Update `appsettings.json`
**File:** `src/.../appsettings.json`
Add the `Redundancy` section with backward-compatible defaults (`Enabled: false`).
Add:
### Step 9: Update OpcUaServiceBuilder for test injection
- `OpcUa.ApplicationUri` example/commentary in docs
- `Redundancy` section with `Enabled = false` defaults
**File:** `src/.../OpcUaServiceBuilder.cs`
Add `WithRedundancy(RedundancyConfiguration)` builder method so tests can inject redundancy configuration.
### Step 10: Add CLI `redundancy` command
### Step 11: Add CLI `redundancy` command
**Files:**
- `tools/opcuacli-dotnet/Commands/RedundancyCommand.cs` (new)
- `tools/opcuacli-dotnet/README.md`
- `docs/CliTool.md`
Command: `redundancy`
Reads from the target server:
- `Server/ServerRedundancy/RedundancySupport` (i=11314)
- `Server/ServiceLevel` (i=2267)
- `Server/ServerRedundancy/ServerUriArray` (i=11492, if non-transparent redundancy)
Read:
Output format:
```
- `VariableIds.Server_ServerRedundancy_RedundancySupport`
- `VariableIds.Server_ServiceLevel`
- `VariableIds.Server_ServerRedundancy_ServerUriArray`
Output example:
```text
Redundancy Mode: Warm
Service Level: 200
Server URIs:
- urn:ZB:LmxOpcUa
- urn:ZB:LmxOpcUa2
- urn:localhost:LmxOpcUa:instance1
- urn:localhost:LmxOpcUa:instance2
```
Options: `--url`, `--username`, `--password`, `--security` (same shared options as other commands).
Use SDK constants instead of hardcoded numeric ids in the command implementation.
### Step 11: Deploy second service instance
### Step 12: Deploy the second service instance
**Deployment target:** `C:\publish\lmxopcua\instance2`
Configuration differences from instance1:
Suggested configuration differences:
| Setting | instance1 | instance2 |
|---|---|---|
| `OpcUa.Port` | `4840` | `4841` |
| `Dashboard.Port` | `8081` | `8082` |
| `OpcUa.ServerName` | `LmxOpcUa` | `LmxOpcUa2` |
| `Dashboard.Port` | `8083` | `8084` |
| `OpcUa.ApplicationUri` | `urn:localhost:LmxOpcUa:instance1` | `urn:localhost:LmxOpcUa:instance2` |
| `Redundancy.Enabled` | `true` | `true` |
| `Redundancy.Role` | `Primary` | `Secondary` |
| `Redundancy.Mode` | `Warm` | `Warm` |
| `Redundancy.ServerUris` | `["urn:ZB:LmxOpcUa", "urn:ZB:LmxOpcUa2"]` | `["urn:ZB:LmxOpcUa", "urn:ZB:LmxOpcUa2"]` |
| `Redundancy.ServiceLevelBase` | `200` | `200` |
| `Redundancy.ServerUris` | same two-entry set | same two-entry set |
Windows service for instance2:
- Name: `LmxOpcUa2`
- Display name: `LMX OPC UA Server (Instance 2)`
- Executable: `C:\publish\lmxopcua\instance2\ZB.MOM.WW.LmxOpcUa.Host.exe`
Deployment notes:
Both instances share the same Galaxy DB (`ZB`) and MXAccess runtime. The `GalaxyName` remains `ZB` for both so they expose the same namespace.
- both instances should share the same `GalaxyName` and namespace URI
- each instance must have a distinct application certificate identity
- if certificate handling is sensitive, give each instance an explicit `Security.CertificateSubject` or separate PKI root
Update `service_info.md` with the second instance details.
Update [service_info.md](C:\Users\dohertj2\Desktop\lmxopcua\service_info.md) with the second instance details after deployment is real, not speculative.
---
## Test Plan
### Unit tests — RedundancyModeResolver
### Unit tests: `RedundancyModeResolver`
**New file:** `tests/ZB.MOM.WW.LmxOpcUa.Tests/Redundancy/RedundancyModeResolverTests.cs`
| Test | Description |
|---|---|
| `Resolve_Disabled_ReturnsNone` | `Enabled=false` always returns `RedundancySupport.None` |
| `Resolve_Warm_ReturnsWarm` | `Mode="Warm"` maps to `RedundancySupport.Warm` |
| `Resolve_Hot_ReturnsHot` | `Mode="Hot"` maps to `RedundancySupport.Hot` |
| `Resolve_Disabled_ReturnsNone` | `Enabled=false` returns `None` |
| `Resolve_Warm_ReturnsWarm` | `Mode="Warm"` maps correctly |
| `Resolve_Hot_ReturnsHot` | `Mode="Hot"` maps correctly |
| `Resolve_Unknown_FallsBackToNone` | Unknown mode falls back safely |
| `Resolve_CaseInsensitive` | `"warm"` and `"WARM"` both resolve |
| `Resolve_CaseInsensitive` | Case-insensitive parsing works |
### Unit tests — ServiceLevelCalculator
### Unit tests: `ServiceLevelCalculator`
**New file:** `tests/ZB.MOM.WW.LmxOpcUa.Tests/Redundancy/ServiceLevelCalculatorTests.cs`
| Test | Description |
|---|---|
| `FullyHealthy_Primary_ReturnsBase` | All healthy, primary role → `ServiceLevelBase` |
| `FullyHealthy_Secondary_ReturnsBaseMinusFifty` | All healthy, secondary role → `ServiceLevelBase - 50` |
| `MxAccessDown_ReducesServiceLevel` | MXAccess disconnected subtracts 100 |
| `DbDown_ReducesServiceLevel` | DB unreachable subtracts 50 |
| `BothDown_ReturnsZero` | MXAccess + DB both down → 0 |
| `ClampedTo255` | Base of 255 with healthy → 255 |
| `ClampedToZero` | Heavy penalties don't go negative |
| `FullyHealthy_Primary_ReturnsBase` | Healthy primary baseline is preserved |
| `FullyHealthy_Secondary_ReturnsBaseMinusFifty` | Healthy secondary baseline is lower |
| `MxAccessDown_ReducesServiceLevel` | MXAccess failure reduces score |
| `DbDown_ReducesServiceLevel` | DB failure reduces score |
| `BothDown_ReturnsZero` | Both unavailable returns 0 |
| `ClampedTo255` | Upper clamp works |
| `ClampedToZero` | Lower clamp works |
### Unit tests — RedundancyConfiguration defaults
### Unit tests: `RedundancyConfiguration`
**New file:** `tests/ZB.MOM.WW.LmxOpcUa.Tests/Redundancy/RedundancyConfigurationTests.cs`
| Test | Description |
|---|---|
| `DefaultConfig_Disabled` | `Enabled` defaults to `false` |
| `DefaultConfig_ModeWarm` | `Mode` defaults to `"Warm"` |
| `DefaultConfig_RolePrimary` | `Role` defaults to `"Primary"` |
| `DefaultConfig_ModeWarm` | `Mode` defaults to `Warm` |
| `DefaultConfig_RolePrimary` | `Role` defaults to `Primary` |
| `DefaultConfig_EmptyServerUris` | `ServerUris` defaults to empty |
| `DefaultConfig_ServiceLevelBase200` | `ServiceLevelBase` defaults to `200` |
@@ -316,64 +413,67 @@ Update `service_info.md` with the second instance details.
**File:** `tests/ZB.MOM.WW.LmxOpcUa.Tests/Configuration/ConfigurationLoadingTests.cs`
Add:
- `Redundancy_Section_BindsCorrectly` — verify binding from appsettings.json
- `Redundancy_Section_BindsCustomValues` — in-memory override test
- `Validator_RedundancyEnabled_EmptyServerUris_ReturnsTrue_WithWarning` — validates but warns
- `Validator_RedundancyEnabled_InvalidServiceLevelBase_ReturnsFalse` — rejects 0 or >255
Add coverage for:
### Integration tests — redundancy E2E
- `OpcUa.ApplicationUri`
- `Redundancy` section binding
- redundancy validation when `ApplicationUri` is missing
- redundancy validation when local `ApplicationUri` is absent from `ServerUris`
- invalid `ServiceLevelBase`
### Integration tests
**New file:** `tests/ZB.MOM.WW.LmxOpcUa.Tests/Integration/RedundancyTests.cs`
These tests start two in-process OPC UA servers with redundancy enabled and verify client-visible behavior:
Cover:
| Test | Description |
|---|---|
| `Server_WithRedundancyDisabled_ReportsNone` | Default config → `RedundancySupport.None`, `ServiceLevel=255` |
| `Server_WithRedundancyEnabled_ReportsConfiguredMode` | `Enabled=true, Mode=Warm` → `RedundancySupport.Warm` |
| `Server_WithRedundancyEnabled_ExposesServerUriArray` | Client can read `ServerUriArray` and it matches config |
| `Server_Primary_HasHigherServiceLevel_ThanSecondary` | Primary server reports higher `ServiceLevel` than secondary |
| `TwoServers_BothExposeSameRedundantSet` | Two server fixtures, both report the same `ServerUriArray` |
| `Server_ServiceLevel_DropsWith_MxAccessDisconnect` | Simulate MXAccess disconnect → `ServiceLevel` decreases |
- redundancy disabled reports `None`
- warm redundancy reports configured mode
- `ServerUriArray` matches configuration
- primary reports higher `ServiceLevel` than secondary
- both servers expose the same namespace URI but different `ApplicationUri` values
- service level drops when MXAccess disconnects
Pattern: Use `OpcUaServerFixture.WithFakeMxAccessClient()` with redundancy config injected, connect with `OpcUaTestClient`, read the standard OPC UA redundancy nodes.
Pattern:
- use two fixture instances
- give each fixture a distinct `ServerName`, `ApplicationUri`, and port
- if secure transport is enabled in those tests, isolate PKI roots to avoid certificate cross-talk
---
## Documentation Plan
### New file: `docs/Redundancy.md`
### New file
- `docs/Redundancy.md`
Contents:
1. Overview of OPC UA non-transparent redundancy
2. Redundancy configuration section reference (`Enabled`, `Mode`, `Role`, `ServerUris`, `ServiceLevelBase`)
3. ServiceLevel computation logic and degraded-state penalties
4. How clients discover and fail over between instances
5. Deployment guide for a two-instance redundant pair (ports, service names, shared Galaxy DB)
1. overview of OPC UA non-transparent redundancy
2. difference between namespace URI and server `ApplicationUri`
3. redundancy configuration reference
4. service-level computation
5. two-instance deployment guide
6. CLI `redundancy` command usage
7. Troubleshooting: mismatched `ServerUris`, ServiceLevel stuck at 0, etc.
7. troubleshooting
### Updates to existing docs
| File | Changes |
|---|---|
| `docs/Configuration.md` | Add `Redundancy` section table, example JSON, add to validation rules list, update example appsettings.json |
| `docs/OpcUaServer.md` | Add redundancy state exposure section, link to `Redundancy.md` |
| `docs/CliTool.md` | Add `redundancy` command documentation |
| `docs/Configuration.md` | Add `OpcUa.ApplicationUri` and `Redundancy` sections |
| `docs/OpcUaServer.md` | Correct the current `ApplicationUri == namespace` description and add redundancy behavior |
| `docs/CliTool.md` | Add `redundancy` command |
| `docs/ServiceHosting.md` | Add multi-instance deployment notes |
| `README.md` | Add `Redundancy` to the component documentation table, mention redundancy in Quick Start |
| `README.md` | Mention redundancy support and link docs |
| `CLAUDE.md` | Add redundancy architecture note |
### Update: `service_info.md`
### Update after real deployment
Add a second section documenting `instance2`:
- Path: `C:\publish\lmxopcua\instance2`
- Windows service name: `LmxOpcUa2`
- Port: `4841`
- Dashboard port: `8084`
- Redundancy role: `Secondary`
- Endpoint: `opc.tcp://localhost:4841/LmxOpcUa`
- `service_info.md`
Only update this once the second instance is actually deployed and verified.
---
@@ -381,128 +481,117 @@ Add a second section documenting `instance2`:
| File | Action | Description |
|---|---|---|
| `src/.../Configuration/OpcUaConfiguration.cs` | Modify | Add explicit `ApplicationUri` |
| `src/.../Configuration/RedundancyConfiguration.cs` | New | Redundancy config model |
| `src/.../Configuration/AppConfiguration.cs` | Modify | Add `Redundancy` section |
| `src/.../Configuration/ConfigurationValidator.cs` | Modify | Validate/log redundancy settings |
| `src/.../OpcUa/RedundancyModeResolver.cs` | New | Mode string → `RedundancySupport` enum |
| `src/.../OpcUa/ServiceLevelCalculator.cs` | New | Dynamic ServiceLevel from health state |
| `src/.../OpcUa/LmxOpcUaServer.cs` | Modify | Expose redundancy nodes, accept ServiceLevel updates |
| `src/.../OpcUa/OpcUaServerHost.cs` | Modify | Pass redundancy config through |
| `src/.../OpcUaService.cs` | Modify | Bind redundancy config, wire ServiceLevel updates |
| `src/.../OpcUaServiceBuilder.cs` | Modify | Add `WithRedundancy()` builder |
| `src/.../appsettings.json` | Modify | Add `Redundancy` section |
| `tools/opcuacli-dotnet/Commands/RedundancyCommand.cs` | New | CLI command to read redundancy info |
| `tests/.../Redundancy/RedundancyModeResolverTests.cs` | New | Mode resolver unit tests |
| `tests/.../Redundancy/ServiceLevelCalculatorTests.cs` | New | ServiceLevel computation tests |
| `tests/.../Redundancy/RedundancyConfigurationTests.cs` | New | Config defaults tests |
| `tests/.../Configuration/ConfigurationLoadingTests.cs` | Modify | Binding + validation tests |
| `tests/.../Integration/RedundancyTests.cs` | New | E2E two-server redundancy tests |
| `tests/.../Helpers/OpcUaServerFixture.cs` | Modify | Accept redundancy config |
| `docs/Redundancy.md` | New | Dedicated redundancy component doc |
| `docs/Configuration.md` | Modify | Add Redundancy section |
| `docs/OpcUaServer.md` | Modify | Add redundancy state section |
| `docs/CliTool.md` | Modify | Add redundancy command |
| `docs/ServiceHosting.md` | Modify | Multi-instance notes |
| `README.md` | Modify | Add Redundancy to component table |
| `CLAUDE.md` | Modify | Add redundancy architecture note |
| `service_info.md` | Modify | Add instance2 details |
| `src/.../Configuration/ConfigurationValidator.cs` | Modify | Validate/log redundancy and application identity |
| `src/.../OpcUa/RedundancyModeResolver.cs` | New | Map config mode to `RedundancySupport` |
| `src/.../OpcUa/ServiceLevelCalculator.cs` | New | Compute `ServiceLevel` from health inputs |
| `src/.../OpcUa/LmxOpcUaServer.cs` | Modify | Expose redundancy state via SDK server object |
| `src/.../OpcUa/OpcUaServerHost.cs` | Modify | Pass local application identity and redundancy config |
| `src/.../OpcUaService.cs` | Modify | Bind config and wire health updates |
| `src/.../OpcUaServiceBuilder.cs` | Modify | Support redundancy/application identity injection |
| `src/.../appsettings.json` | Modify | Add redundancy settings |
| `tools/opcuacli-dotnet/Commands/RedundancyCommand.cs` | New | Read redundancy state from a server |
| `tests/.../Redundancy/*.cs` | New | Unit tests for redundancy config and calculators |
| `tests/.../Configuration/ConfigurationLoadingTests.cs` | Modify | Bind/validate new settings |
| `tests/.../Integration/RedundancyTests.cs` | New | Paired-server integration tests |
| `tests/.../Helpers/OpcUaServerFixture.cs` | Modify | Support paired redundancy fixtures |
| `tests/.../Helpers/OpcUaTestClient.cs` | Modify | Read redundancy nodes in integration tests |
| `docs/Redundancy.md` | New | Dedicated redundancy guide |
| `docs/Configuration.md` | Modify | Document new config |
| `docs/OpcUaServer.md` | Modify | Correct application identity and add redundancy details |
| `docs/CliTool.md` | Modify | Document `redundancy` command |
| `docs/ServiceHosting.md` | Modify | Multi-instance deployment notes |
| `README.md` | Modify | Link redundancy docs |
| `CLAUDE.md` | Modify | Architecture note |
| `service_info.md` | Modify later | Document real second-instance deployment |
---
## Verification Guardrails
Each step must pass these gates before proceeding to the next:
### Gate 1: Build
### Gate 1: Build (after each implementation step)
```bash
dotnet build ZB.MOM.WW.LmxOpcUa.slnx
```
Must produce 0 errors. Proceed only when green.
### Gate 2: Unit tests (after steps 1–4, 9)
### Gate 2: Unit tests
```bash
dotnet test tests/ZB.MOM.WW.LmxOpcUa.Tests
```
All existing + new tests must pass. No regressions.
### Gate 3: Integration tests (after steps 5–7)
### Gate 3: Redundancy integration tests
```bash
dotnet test tests/ZB.MOM.WW.LmxOpcUa.Tests --filter "FullyQualifiedName~Integration.RedundancyTests"
dotnet test tests/ZB.MOM.WW.LmxOpcUa.Tests --filter "FullyQualifiedName~Redundancy"
```
All redundancy E2E tests must pass.
### Gate 4: CLI tool builds (after step 10)
### Gate 4: CLI build
```bash
cd tools/opcuacli-dotnet && dotnet build
cd tools/opcuacli-dotnet
dotnet build
```
Must compile without errors.
### Gate 5: Manual verification — single instance (after step 8)
### Gate 5: Manual single-instance check
```bash
# Publish and start with Redundancy.Enabled=false
opcuacli-dotnet.exe connect -u opc.tcp://localhost:4840/LmxOpcUa
opcuacli-dotnet.exe redundancy -u opc.tcp://localhost:4840/LmxOpcUa
# Should report: RedundancySupport=None, ServiceLevel=255
```
### Gate 6: Manual verification — redundant pair (after step 11)
Expected:
- `RedundancySupport=None`
- `ServiceLevel=255`
### Gate 6: Manual paired-instance check
```bash
# Start both instances
sc start LmxOpcUa
sc start LmxOpcUa2
# Verify instance1 (Primary)
opcuacli-dotnet.exe redundancy -u opc.tcp://localhost:4840/LmxOpcUa
# Should report: RedundancySupport=Warm, ServiceLevel=200, ServerUris=[urn:ZB:LmxOpcUa, urn:ZB:LmxOpcUa2]
# Verify instance2 (Secondary)
opcuacli-dotnet.exe redundancy -u opc.tcp://localhost:4841/LmxOpcUa
# Should report: RedundancySupport=Warm, ServiceLevel=150, ServerUris=[urn:ZB:LmxOpcUa, urn:ZB:LmxOpcUa2]
# Both instances should serve the same Galaxy address space
opcuacli-dotnet.exe browse -u opc.tcp://localhost:4840/LmxOpcUa -r -d 2
opcuacli-dotnet.exe browse -u opc.tcp://localhost:4841/LmxOpcUa -r -d 2
```
### Gate 7: Full test suite (final)
Expected:
- both report the same `ServerUriArray`
- each reports its own unique local `ApplicationUri`
- primary reports a higher `ServiceLevel`
### Gate 7: Full test suite
```bash
dotnet test ZB.MOM.WW.LmxOpcUa.slnx
```
All tests across all projects must pass.
### Gate 8: Documentation review
- All new/modified doc files render correctly in Markdown
- Example JSON snippets match the actual `appsettings.json`
- CLI examples use correct flags and expected output
- `service_info.md` accurately reflects both deployed instances
---
## Risks and Considerations
1. **Backward compatibility**: `Redundancy.Enabled = false` must be the default so existing single-instance deployments are unaffected.
2. **ServiceLevel timing**: Updates must not race with OPC UA publish cycles. Use the server's internal lock or `ServerInternal` APIs.
3. **ServerUriArray immutability**: The OPC UA spec expects this to be static during a server session. Changes require a server restart.
4. **MXAccess shared state**: Both instances connect to the same MXAccess runtime. If MXAccess has per-client registration limits, verify that two clients can coexist.
5. **Galaxy DB contention**: Both instances poll for deploy changes. Ensure change detection doesn't trigger duplicate rebuilds or locking issues.
6. **Port conflicts**: The second instance must use different ports for OPC UA (4841) and Dashboard (8084).
7. **Certificate identity**: Each instance needs its own application certificate with a distinct `SubjectName` matching its `ServerName`.
1. **Application identity is the main correctness risk.** Without unique `ApplicationUri` values, the redundant set is invalid even if `ServerUriArray` is populated.
2. **SDK wiring may require replacing the default redundancy state node.** The base `ServerRedundancyState` does not expose `ServerUriArray`; the implementation may need the non-transparent subtype from the SDK.
3. **Two in-process servers can collide on certificates.** Tests and deployment need distinct application identities and, when necessary, isolated PKI roots.
4. **Both instances hit the same MXAccess runtime and Galaxy DB.** Verify client-registration and polling behavior under paired load.
5. **`ServiceLevel` should remain meaningful, not noisy.** Prefer deterministic role + health inputs over frequent arbitrary adjustments.
6. **`service_info.md` is deployment documentation, not design.** Do not prefill it with speculative values before the second instance actually exists.
---
## Execution Order
1. Steps 1–4: Config model, resolver, calculator, validator (unit-testable in isolation)
2. **Gate 1 + Gate 2**: Build + unit tests pass
3. Steps 5–7: Server integration (redundancy nodes, ServiceLevel wiring)
4. **Gate 1 + Gate 2 + Gate 3**: Build + all tests including E2E
5. Step 8: Update appsettings.json
6. **Gate 5**: Manual single-instance verification
7. Step 9: Update service builder for tests
8. Step 10: CLI redundancy command
9. **Gate 4**: CLI builds
10. Step 11: Deploy second instance + update service_info.md
11. **Gate 6**: Manual two-instance verification
12. Documentation updates (all doc files)
13. **Gate 7 + Gate 8**: Full test suite + documentation review
14. Commit and push
1. Step 1: add `OpcUa.ApplicationUri` and separate it from namespace identity
2. Steps 2-5: config model, resolver, calculator, validator
3. Gate 1 + Gate 2
4. Step 9: update builders/helpers so tests can express paired servers cleanly
5. Step 6-8: server exposure and service-layer health wiring
6. Gate 1 + Gate 2 + Gate 3
7. Step 10: update `appsettings.json`
8. Step 11: add CLI `redundancy` command
9. Gate 4 + Gate 5
10. Step 12: deploy and verify the second instance
11. Update `service_info.md` with real deployment details
12. Documentation updates
13. Gate 7