- New Commons.Messages.Fleet.GetDiagnostics request record.
- DriverHostActor handles GetDiagnostics in all three states (Steady, Applying,
Stale); replies with a NodeDiagnosticsSnapshot built from _currentRevision
+ the local NodeId. Drivers list is empty until F7 wires the per-instance
children.
- FleetDiagnosticsClient now resolves the target via ActorSelection at
akka.tcp://{system}@{nodeId}/user/driver-host and Asks with a 3s timeout.
On timeout/peer-down it returns an empty snapshot so the UI degrades
gracefully rather than throwing.
Two new integration tests in Host.IntegrationTests:
- GetDiagnostics_returns_snapshot_with_target_NodeId verifies the
cross-node Ask/Reply works.
- GetDiagnostics_after_deploy_reports_current_revision exercises the
end-to-end path: AdminOps starts a deployment, both DriverHostActors
apply, then diagnostics reports the new revision on both nodes.
All 98 v2 tests pass (was 96 + 2 new).
Replaces the Ok=true stub with a TCP connect to the peer's OPC UA port (4840
default) with a 2s timeout. A successful connect indicates the OPC UA server
process is up + accepting connections — enough for the redundancy calculator
to treat the peer as live. A full secure-channel Hello/Acknowledge handshake
is overkill for what the redundancy calc consumes and would pull in the OPC
UA Client SDK + a PKI setup. Upgrade later if a deeper liveness signal is ever
required.
Probe extracts the host from NodeId by stripping the :port suffix (commit
5cfbe8b encoded host:port into NodeId for cluster-member identity).
Tests: 2 new tests — Ok=true against a live TcpListener on a chosen port,
Ok=false against an unreachable endpoint. All 17 Runtime tests pass (was 16
covering only the message-contract surface).
ConfigAuditLog gains two nullable columns (EventId, CorrelationId) + a filtered
unique index UX_ConfigAuditLog_EventId. EF migration
20260526105027_AddConfigAuditLogEventIdColumns is additive (nullable + filtered
index = legacy rows backfill cleanly).
AuditWriterActor now writes EventId + CorrelationId into the dedicated columns
instead of synthesising a JSON wrapper into DetailsJson. Cross-restart dedup
is now real: a retry of an already-flushed batch hits the unique index and
SaveChanges throws; the existing catch drops the duplicate without losing the
rest of the batch.
WrapDetails helper deleted — F4 (its JSON hardening) becomes moot.
AuditWriterActorTests.Details_wrapper_embeds_eventId_and_correlationId renamed
+ rewritten to assert against the columns. All 29 ControlPlane tests pass,
all 95 v2 tests green.
DeployHappyPathTests exercises the full deploy pipeline on the 2-node harness:
AdminOperationsActor → ConfigPublishCoordinator → DistributedPubSub →
DriverHostActor on both nodes → ApplyAck → coordinator seals. Verifies both
NodeDeploymentState rows reach Applied and Deployment.Status reaches Sealed.
Exposed + fixed two production bugs along the way:
1. Coordinator was publishing DispatchDeployment on the "deployments" topic but
never subscribed to anything — DriverHostActor ACKs published on the same
topic could not reach it. Added dedicated "deployment-acks" topic with
coordinator subscription in PreStart, and DriverHostActor publishes ACKs
there.
2. NodeId derivation used member.Address.Host only — two cluster members on a
shared loopback host (test harness, dev VMs) collided to one identity. The
coordinator's expected-ack set became {1} and the system sealed after only
half the nodes acked. Switched to host:port everywhere (ClusterRoleInfo +
coordinator) so loopback nodes stay distinct and production identities are
harmlessly more specific.
Tests: 95 v2 tests pass (was 93 + 2 deploy tests), 0 skipped.
Failover scenarios (design §8 cases 3-7: node-kill-mid-apply, split-brain,
restart-during-deploy) deferred — they need controlled node-down primitives
on the harness. Tracked as F22 (failover scenario test cases).
Builds TwoNodeClusterHarness: two in-process Host-equivalent nodes sharing
an in-memory ConfigDb. Forms a 2-member Akka cluster. ClusterFormationTests
proves both nodes see each other as admin+driver role members.
Fixes a real production bug uncovered while wiring the harness — Program.cs
ran two separate ActorSystems (one from AddOtOpcUaCluster.AkkaHostedService
with cluster HOCON, one from Akka.Hosting.AddAkka with bare HOCON). Cluster
singletons landed on the bare ActorSystem and could not actually form a
cluster ("Configuration does not contain `akka.cluster` node").
Consolidation:
- AddOtOpcUaCluster now only binds AkkaClusterOptions + registers IClusterRoleInfo
- New WithOtOpcUaClusterBootstrap pushes embedded HOCON + Remote/Cluster options
into Akka.Hosting's AkkaConfigurationBuilder
- AkkaHostedService.cs deleted — Akka.Hosting now owns the lifecycle
- Program.cs + harness call WithOtOpcUaClusterBootstrap inside AddAkka
Why not WebApplicationFactory<Program>? Program.cs reads OTOPCUA_ROLES from
process env (shared across in-process WAFs); the harness replays Program.cs's
DI graph from a clean WebApplicationBuilder per node with per-node config
overrides. Same production extensions, isolated config + Kestrel + Akka ports.
Tests: 93 v2 tests pass (was 91 + 2 new cluster formation), 0 skipped.
Eliminates the services.BuildServiceProvider() captive-provider antipattern
(ASP0000) inside AddJwtBearer. The new ConfigureJwtBearerFromTokenService
resolves JwtTokenService from the real DI container at runtime and stays
in lock-step with JwtTokenService.BuildValidationParameters.
All 27 Security.Tests stay green, including the F1 integration tests that
exercise /auth/token through the real bearer pipeline.
Injects AuthenticationStateProvider and reads the current user's identity
name on Deploy click, replacing the "(current user)" placeholder.
Anonymous case falls back to "(anonymous)" — should never hit in practice
since the page requires FleetAdmin/ConfigEditor.
Mirrors the publisher-injection pattern from FleetStatusBroadcaster and
PeerOpcUaProbeActor: Props accepts an optional Action<object> override so
tests can use a TestProbe sink instead of bootstrapping DistributedPubSub
(unreliable single-node in TestKit).
Un-skips the two RedundancyStateActor tests deferred under F6.
Mirrors WithOtOpcUaControlPlaneSingletons for the driver role. Spawns
DriverHostActor + DbHealthProbeActor on the host's ActorSystem and
registers both under marker keys. Host's Program.cs now calls it when
the node carries the driver role, so driver-only and admin+driver
deployments both auto-bootstrap the per-node actors.
Integration test covers the registration round-trip via Microsoft.Extensions.Hosting
+ Akka.Hosting AddAkka.
Task 56: removes the legacy in-process Server + Admin Web project + their test
projects (Server.Tests, Admin.Tests, Admin.E2ETests). The fused OtOpcUa.Host
binary built across Phases 1-9 is now the sole production entry point.
What happened to the 47 legacy Admin Blazor pages: per follow-up F15, the
v1 architecture's draft/publish UX is replaced by v2's live-edit + snapshot-
deploy model, so a 1:1 migration is not meaningful. The mechanical move via
git mv preserves the history; service classes + page bodies that referenced
removed v1 types (ConfigGeneration, RedundancyRole, GenerationId) were
deleted. AdminUI now ships a minimal Home page + the v2 Deployments page.
Per-page rebuild against the v2 surface is tracked as F15. The v2 Deployments
page (Task 52) is the only first-party UI shipping in this PR.
Task 57: solution build green; 84+ tests green across active v2 + legacy
driver test projects.
Phase 1f — the consolidator migration. Closes out the v2 entity-model
rewrite by emitting a single EF migration that captures the cumulative
schema delta from 14a (RowVersion) through 14e (drop generation entities).
Generated: src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/
20260526081556_V2HostingAlignment.cs (1562 lines)
20260526081556_V2HostingAlignment.Designer.cs
Migration shape (per `grep -nE migrationBuilder.\(...)`):
Drop 12 ForeignKey constraints (one per live-edit entity's GenerationId FK)
Drop 2 Tables (ConfigGeneration, ClusterNodeGenerationState)
Drop 45 Indexes (every UX_*_Generation_* and IX_*_Generation_* across the
13 live-edit tables — 1 also dropped the unique-Primary
filtered index UX_ClusterNode_Primary_Per_Cluster)
Drop 13 Columns (12 GenerationId + 1 RedundancyRole)
Add 12 RowVersion columns (one per live-edit entity)
Create 4 Tables (Deployment, NodeDeploymentState, ConfigEdit,
DataProtectionKeys)
Create ~45 Indexes (recreated under the new naming pattern
UX_<Table>_LogicalId / UX_<Table>_<X> with the
GenerationId column stripped from composite keys)
Notable EF quirks accepted:
Unique-on-required-column indexes (UX_VirtualTag_LogicalId etc.) ship a
`filter: "[VirtualTagId] IS NOT NULL"` clause that EF auto-inserts for
SQL Server. Harmless — the column is C#-side `required` so NULL never
appears.
Verification:
dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Configuration -> 0 errors
dotnet ef migrations script --idempotent (against placeholder DSN)
-> 3259-line
.sql produced
OK
tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests -> 0 errors
Live `dotnet ef database update` against a scratch SQL Server deferred to
Task 15 (Migrate-To-V2.ps1) — SSH to the docker host needs a key/password I
don't have, and the always-on SQL at 10.100.0.35,14330 uses Integrated
Security (Windows auth, unreachable from this macOS dev). The migration
itself is structurally correct by construction (EF tooling generated it
against the live DbContext model); the live-DB confidence step is the
PowerShell wrapper's job.
SchemaComplianceTests updates:
- All_expected_tables_exist: removed ConfigGeneration +
ClusterNodeGenerationState; added Deployment, NodeDeploymentState,
ConfigEdit, DataProtectionKeys.
- Filtered_unique_indexes_match_schema_spec: removed entries for
UX_ClusterNode_Primary_Per_Cluster (Task 14d) and
UX_ConfigGeneration_Draft_Per_Cluster (Task 14e). Two filtered uniques
remain (UX_ClusterNodeCredential_Value, UX_ExternalIdReservation_KindValue_Active).
- Check_constraints_match_schema_spec: added CK_ConfigEdit_FieldsJson_IsJson.
StoredProceduresTests update:
- Removed RedundancyRole + 'Primary' from the raw INSERT into ClusterNode
so the DB-backed test runs against the new schema.
Phase 1e of the v2 entity-model rewrite. With the FKs gone (Task 14b) and
the apply pipeline replaced (Task 14c), the v1 draft/publish entities have
no remaining v2 consumers.
Deleted entity classes:
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ConfigGeneration.cs
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNodeGenerationState.cs
Deleted enum classes (no v2 consumers):
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Enums/GenerationStatus.cs
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Enums/NodeApplyStatus.cs
OtOpcUaConfigDbContext changes:
- Removed DbSet<ConfigGeneration> ConfigGenerations
- Removed DbSet<ClusterNodeGenerationState> ClusterNodeGenerationStates
- Removed ConfigureConfigGeneration(modelBuilder) call + method body
- Removed ConfigureClusterNodeGenerationState(modelBuilder) call + body
- Tidied the "v2 deploy-model tables" header comment
Navigation property cleanup:
- ServerCluster.Generations collection -> removed
- ClusterNode.GenerationState navigation -> removed
doc-comment cref cleanup (replaced <see cref="X"/> with <c>X</c> for the
deleted types so the C# XML comment compiler doesn't fail with CS1574):
- Deployment.cs (cref to ConfigGeneration)
- NodeDeploymentState.cs (cref to ClusterNodeGenerationState)
- Core/OpcUa/EquipmentNodeWalker.cs (cref to ConfigGeneration in the
EquipmentNamespaceContent record's doc-comment; while there, removed
"All four collections are scoped to the same ConfigGeneration" since
that's no longer true in v2)
Verification:
src/Core/ZB.MOM.WW.OtOpcUa.Configuration -> 0 errors
src/Core/ZB.MOM.WW.OtOpcUa.Core -> 0 errors
tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests -> 0 errors
tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests -> 0 errors
whole solution -> 15 errors
(all in Server/Admin; transitive Server.Tests/Admin.Tests skip per the
parent's failure, so the per-project count dropped vs Task 14d's 71)
Phase 1d of the v2 entity-model rewrite. The static RedundancyRole column
is replaced by Akka cluster's role-leader-of-"driver" election at runtime
(see RedundancyStateActor + ServiceLevelCalculator in Task 35).
Changes:
- Removed `public required RedundancyRole RedundancyRole` from
ClusterNode entity.
- Removed `e.Property(x => x.RedundancyRole).HasConversion<string>()...`
mapping from OtOpcUaConfigDbContext.ConfigureClusterNode.
- Removed the `UX_ClusterNode_Primary_Per_Cluster` filtered unique index
(filter referenced [RedundancyRole]='Primary').
- Dropped `using ZB.MOM.WW.OtOpcUa.Configuration.Enums` from ClusterNode.cs
(no longer needed).
- Deleted `Enums/RedundancyRole.cs` — the enum is unused in v2-kept code.
- DraftValidator: dropped the "exactly one Primary per cluster"
validation block. Comment in place explaining v2 picks primary at
runtime via Akka.
- DraftValidatorTests: dropped ValidateClusterTopology_flags_multiple_Primary
test; reworked BuildNode helper to no longer take a `role` argument.
Untouched (Server + Admin still reference RedundancyRole; accepted broken
per Task 56 policy):
src/Server/ZB.MOM.WW.OtOpcUa.Server/Redundancy/{ClusterTopologyLoader,
RedundancyStatePublisher, RedundancyTopology, ServiceLevelCalculator}.cs
src/Server/ZB.MOM.WW.OtOpcUa.Admin/Services/RedundancyMetrics.cs
DB-runtime tests will fail against the new schema (Task 14f's migration
drops the column) — to be updated in Task 14f's SchemaComplianceTests
update:
- SchemaComplianceTests.cs:55 (expected filtered index list)
- StoredProceduresTests.cs:263 (raw INSERT names the column)
Verification:
src/Core/ZB.MOM.WW.OtOpcUa.Configuration -> 0 errors
tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests -> 0 errors
whole solution -> 71 errors
(70 from Task 14b in Server/Admin, +1 new Server/Redundancy reference)
Joseph Doherty