Corrects two claims made while writing the Phase 2 gate record.
The docker-dev rig is NOT misconfigured: central-1/central-2 carry admin,driver
deliberately — the compose says so — because they are themselves a pair that runs
the MAIN cluster's drivers.
And a redundant pair is not a missing concept. The application Cluster (ClusterId)
already is one: it owns drivers, devices and ClusterNode rows, and Phase 1's
deployment-artifact cache is keyed by it precisely so a pair shares one entry.
So the root cause is wider than the alarm drain. RedundancyStateActor elects one
Primary per AKKA cluster while every real unit of redundancy is an APPLICATION
cluster, and ServiceLevel, the alerts emit gate and the inbound device-write gate
all inherit that mis-scoping. On a fleet of pairs in one Akka cluster, exactly one
driver node cluster-wide services Primary-gated work.
Recommendation revised in the gate doc: fix the election scope (its own change,
its own gate, because it touches the safety-critical write gate), and explicitly
do NOT derive pair identity from LocalDb:Replication:PeerAddress — that would add
a second parallel notion of 'my partner' that the real fix obsoletes, and the
library has no initiator flag, so both halves would have to dial just to obtain a
config value.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.
Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.
Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
New operations runbook (enable/disable, fail-closed ApiKey rule, stop/start-
together, tombstone-retention window, MaxBatchSize row-count-vs-4MB, never
sqlite3-a-live-WAL-DB cp-triplet recipe). Configuration.md gains a LocalDb
section; Redundancy.md gains a pair-local config cache section (what boot-from-
cache does and does not cover); CLAUDE.md gains the Phase 1 scope paragraph.
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.
Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.
Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
process exits (supervisor restarts it) instead of idling with a dead actor system.
Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
ANY single loss needs 3+ members.
Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).
Akka.Cluster.Hosting's WithClustering enables an SBR downing provider by default
(applies SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is
null), which reads the pre-existing akka.conf keep-oldest block. So the cluster was
NOT running NoDowning before Critical 1 and hard-crash failover already worked — the
typed KeepOldestOption is reinforcing/explicit-in-code, not the sole activator.
Corrects the inaccurate 'HOCON inert / NoDowning / never fails over' framing in:
- ServiceCollectionExtensions.BuildClusterOptions XML comment
- akka.conf split-brain-resolver comment
- docs/Redundancy.md Split-brain section
- SplitBrainResolverActivationTests summary + assertion message (+ method rename)
No code revert (the typed option is correct belt-and-suspenders). Cluster.Tests 29/29.
Surfaced by the #9 hard-kill failover negative control.
The split-brain-resolver HOCON block in akka.conf was inert — nothing
registered a downing provider, so the cluster ran Akka's default NoDowning:
a hard-crashed node was never downed, cluster singletons + the driver
role-leader never failed over, and a partition left both redundancy sides
at ServiceLevel 240 indefinitely.
Activate the resolver via the typed ClusterOptions.SplitBrainResolver
(KeepOldestOption { DownIfAlone = true }) in a new testable
BuildClusterOptions helper. keep-oldest + down-if-alone is correct for a
2-node warm-redundancy pair. HOCON keeps stable-after (typed option can't
express it) + a cross-reference comment. Deterministic unit guard added;
end-to-end verified the resolved ActorSystem config carries the provider
and stable-after survives the HOCON/Hosting merge. Redundancy.md corrected.
Hard-kill failover integration test tracked as a follow-up (effort-M harness).
- Redundancy.md: full rewrite — Akka-leader-driven ServiceLevel replaces
operator-managed RedundancyRole. Documents the 5-tier ServiceLevelCalculator,
RedundancyStateActor cluster singleton, and the DPS data flow.
- ServiceHosting.md: full rewrite — single fused OtOpcUa.Host binary with
OTOPCUA_ROLES env gating. Documents the conditional DI graph and the new
health endpoints (/health/ready, /health/active, /healthz).
- security.md: v2 banner at top covering path/project renames + new JWT bearer
+ DataProtection persisted to ConfigDb. Body unchanged because the 4-concern
security model is unchanged in v2; full per-section rewrite waits for F15
(Admin pages migration) since security.md references many pages that move.
- README.md: platform overview updated to v2 (fused Host + role gating).
Rewrite src/ and tests/ project paths in docs, CLAUDE.md, README.md, and
test-fixture READMEs to the new module-folder layout (Core/Server/Drivers/
Client/Tooling). References to retired v1 projects (Galaxy.Host/Proxy/Shared,
the legacy monolithic test projects) are left untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five operational docs rewritten for v2 (multi-process, multi-driver, Config-DB authoritative):
- docs/Configuration.md — replaced appsettings-only story with the two-layer model.
appsettings.json is bootstrap only (Node identity, Config DB connection string,
transport security, LDAP bind, logging). Authoritative config (clusters, namespaces,
UNS, equipment, tags, driver instances, ACLs, role grants, poll groups) lives in
the Config DB accessed via OtOpcUaConfigDbContext and edited through the Admin UI
draft/publish workflow. Added v1-to-v2 migration index so operators can locate where
each old section moved. Cross-links to docs/v2/config-db-schema.md + docs/v2/admin-ui.md.
- docs/Redundancy.md — Phase 6.3 rewrite. Named every class under
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/: RedundancyCoordinator, RedundancyTopology,
ApplyLeaseRegistry (publish fencing), PeerReachabilityTracker, RecoveryStateManager,
ServiceLevelCalculator (pure function), RedundancyStatePublisher. Documented the
full 11-band ServiceLevel matrix (Maintenance=0 through AuthoritativePrimary=255)
from ServiceLevelCalculator.cs and the per-ClusterNode fields (RedundancyRole,
ServiceLevelBase, ApplicationUri). Covered metrics
(otopcua.redundancy.role_transition counter + primary/secondary/stale_count gauges
on meter ZB.MOM.WW.OtOpcUa.Redundancy) and SignalR RoleChanged push from
FleetStatusPoller to RedundancyTab.razor.
- docs/security.md — preserved the transport-security section (still accurate) and
added Phase 6.2 authorization. Four concerns now documented in one place:
(1) transport security profiles, (2) OPC UA auth via LdapUserAuthenticator
(note: task spec called this LdapAuthenticationProvider — actual class name is
LdapUserAuthenticator in Server/Security/), (3) data-plane authorization via
NodeAcl + PermissionTrie + AuthorizationGate — additive-only model per decision
#129, ClusterId → Namespace → UnsArea → UnsLine → Equipment → Tag hierarchy,
NodePermissions bundle, PermissionProbeService in Admin for "probe this permission",
(4) control-plane authorization via LdapGroupRoleMapping + AdminRole
(ConfigViewer / ConfigEditor / FleetAdmin, CanEdit / CanPublish policies) —
deliberately independent of data-plane ACLs per decision #150. Documented the
OTOPCUA0001 Roslyn analyzer (UnwrappedCapabilityCallAnalyzer) as the compile-time
guard ensuring every driver-capability async call is wrapped by CapabilityInvoker.
- docs/ServiceHosting.md — three-process rewrite: OtOpcUa Server (net10 x64,
BackgroundService + AddWindowsService, hosts OPC UA endpoint + all non-Galaxy
drivers), OtOpcUa Admin (net10 x64, Blazor Server + SignalR + /metrics via
OpenTelemetry Prometheus exporter), OtOpcUa Galaxy.Host (.NET Framework 4.8 x86,
NSSM-wrapped, env-variable driven, STA thread + MXAccess COM). Pipe ACL
denies-Admins detail + non-elevated shell requirement captured from feedback memory.
Divergence from CLAUDE.md: task spec said "TopShelf is still the service-installer
wrapper per CLAUDE.md note" but no csproj in the repo references TopShelf — decision
#30 replaced it with the generic host's AddWindowsService wrapper (per the doc
comment on OpcUaServerService). Reflected the actual state + flagged this divergence
here so someone can update CLAUDE.md separately.
- docs/StatusDashboard.md — replaced the full v1 reference (dashboard endpoints,
health check rules, StatusData DTO, etc.) with a short "superseded by Admin UI"
pointer that preserves git-blame continuity + avoids broken links from other docs
that reference it.
Class references verified by reading:
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/{RedundancyCoordinator, ServiceLevelCalculator,
ApplyLeaseRegistry, RedundancyStatePublisher}.cs
src/ZB.MOM.WW.OtOpcUa.Core/Authorization/{PermissionTrie, PermissionTrieBuilder,
PermissionTrieCache, TriePermissionEvaluator, AuthorizationGate}.cs
src/ZB.MOM.WW.OtOpcUa.Server/Security/{AuthorizationGate, LdapUserAuthenticator}.cs
src/ZB.MOM.WW.OtOpcUa.Admin/{Program.cs, Services/AdminRoles.cs,
Services/RedundancyMetrics.cs, Hubs/FleetStatusPoller.cs}
src/ZB.MOM.WW.OtOpcUa.Server/Program.cs + appsettings.json
src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Host/{Program.cs, Ipc/PipeServer.cs}
src/ZB.MOM.WW.OtOpcUa.Configuration/Entities/{ClusterNode, NodeAcl,
LdapGroupRoleMapping}.cs
src/ZB.MOM.WW.OtOpcUa.Analyzers/UnwrappedCapabilityCallAnalyzer.cs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames all 11 projects (5 src + 6 tests), the .slnx solution file, all source-file namespaces, all axaml namespace references, and all v1 documentation references in CLAUDE.md and docs/*.md (excluding docs/v2/ which is already in OtOpcUa form). Also updates the TopShelf service registration name from "LmxOpcUa" to "OtOpcUa" per Phase 0 Task 0.6.
Preserves runtime identifiers per Phase 0 Out-of-Scope rules to avoid breaking v1/v2 client trust during coexistence: OPC UA `ApplicationUri` defaults (`urn:{GalaxyName}:LmxOpcUa`), server `EndpointPath` (`/LmxOpcUa`), `ServerName` default (feeds cert subject CN), `MxAccessConfiguration.ClientName` default (defensive — stays "LmxOpcUa" for MxAccess audit-trail consistency), client OPC UA identifiers (`ApplicationName = "LmxOpcUaClient"`, `ApplicationUri = "urn:localhost:LmxOpcUaClient"`, cert directory `%LocalAppData%\LmxOpcUaClient\pki\`), and the `LmxOpcUaServer` class name (class rename out of Phase 0 scope per Task 0.5 sed pattern; happens in Phase 1 alongside `LmxNodeManager → GenericDriverNodeManager` Core extraction). 23 LmxOpcUa references retained, all enumerated and justified in `docs/v2/implementation/exit-gate-phase-0.md`.
Build clean: 0 errors, 30 warnings (lower than baseline 167). Tests at strict improvement over baseline: 821 passing / 1 failing vs baseline 820 / 2 (one flaky pre-existing failure passed this run; the other still fails — both pre-existing and unrelated to the rename). `Client.UI.Tests`, `Historian.Aveva.Tests`, `Client.Shared.Tests`, `IntegrationTests` all match baseline exactly. Exit gate compliance results recorded in `docs/v2/implementation/exit-gate-phase-0.md` with all 7 checks PASS or DEFERRED-to-PR-review (#7 service install verification needs Windows service permissions on the reviewer's box).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The standalone CLI tool is superseded by src/ZB.MOM.WW.LmxOpcUa.Client.CLI
which uses the shared IOpcUaClientService abstraction. Renames CliTool.md to
Client.CLI.md and updates README, CLAUDE.md, Security, and Redundancy docs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All commands gain --failover-urls (-F) to specify alternate endpoints.
Short-lived commands try each URL in order on initial connect. The
subscribe command monitors KeepAlive and automatically reconnects to
the next available server, re-creating the subscription on failover.
Verified with live service start/stop: primary down triggers failover
to secondary, primary restart allows failback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Separates ApplicationUri from namespace identity so each instance in a
redundant pair has a unique server URI while sharing the same Galaxy
namespace. Exposes RedundancySupport, ServerUriArray, and dynamic
ServiceLevel through the standard OPC UA server object. ServiceLevel
is computed from role (Primary/Secondary) and runtime health (MXAccess
and DB connectivity). Adds CLI redundancy command, second deployed
service instance, and 31 new tests including paired-server integration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Joseph Doherty