A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).
Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
- I-1: QuestDbConnection caps Npgsql Maximum Pool Size at 24 (DefaultMaxPoolSize)
when the connection string does not set it; an explicit value is kept. The
effective value is logged once at Information.
- M-1: QuestDbTagCatalog.EnsureAsync waits for the tag table's sequenced commits
and re-checks before inserting, so it no longer overwrites a provisioner row
(unit/description) that is committed but not yet applied. Integration test
reproduces the clobber against real QuestDB (falsified).
- M-3: the pre-1970 drop Warning is rate-limited per tag (once per 10 min);
DroppedPre1970 still counts every sample.
- M-5: QuestDbHistorianProvider is IAsyncDisposable/IDisposable and disposes the
shared Npgsql pool; Program.cs registers the provider through a factory and
every seam resolves it from the container, so DI disposes it at shutdown.
- M-8, M-9: docs fixes (HistoryUpdate transaction wording; folded duplicate
HistoryUpdate paragraph). Pool-size docs updated for the new default cap.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
CentralCommunicationActor: while the PSK provider is unresolved or the first
ClusterNode load failed, re-arm RefreshContacts every 2 s (max 15) until the
first publish; Information log on first publish; 'NEVER be published' Error
now only after the retry budget is exhausted. Corrected false comments.
Tests: late provider (Grpc+Dps), failed first load, retries stop, bounded budget;
ControlPlaneStarterOrderTests back to the default 60 s refresh.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The old AdminUI cannot author the per-driver transport key (#576), so the
'pin before upgrading' recipe had no working path. A node-level appsetting
(env Drivers__AbCip__DefaultTransport) now supplies the default for AbCip
drivers with no key: explicit key > node default > compiled Native. Bound
and validated unconditionally (bad value fails host start); the driver
factory and the Test Connect probe both read it so they agree. AbCip.md
upgrade recipe rewritten around the key; Configuration.md documents it.
Also documents first-wins for AllowPacking/probe and host-alias throttles
(final review M6) and moves the status-bit note out of AbCip.md (M5).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Additive/wire-compatible releases across all three families; Directory.Packages.props
is the only pin surface (no csproj-level overrides, no lock files). Auth 0.3.0 adds
IApiKeyStoreMigrator and the optional Auth.ApiKeys.SqlServer package, neither needed
here. LocalDb 0.3.4's on-disk bookkeeping schema goes v2->v3 (upgrades in place on
open); 0.3.0 made the snapshot-resync flag per-table instead of per-database; 0.3.2
added validated Replication:KeepAlivePingDelay/KeepAlivePingTimeout config (defaults
60s/20s, left unset here) and a public LocalDbOplogGapException; 0.3.3 makes
RegisterReplicated throw LocalDbRegistrationException for a table that is the parent
of a row-mutating ON DELETE FK, which does not apply to this repo's four replicated
tables (no FKs). MxGateway.Client/Contracts 0.2.1 publishes the alarm snapshot_status
truncation frame and live alarm watch-list refresh; vendored the new nupkgs into
nuget-packages/ (the local-mxgw source MxGateway.* is exclusively package-source-mapped
to), matching the existing pattern from the 0.1.0->0.1.1 and 0.1.1->0.2.0 bumps.
Full solution builds clean (0 errors, 908 pre-existing analyzer warnings, unchanged
count). Full test suite: one failure, ClusterRoleInfoRoleLeaderCaseTests.
RoleLeaderChanged_for_driver_then_Driver_keeps_distinct_entries, reproduces
identically on the unbumped 0.2.1/0.2.1/0.2.0 tree under the same full-suite parallel
run (and passes standalone in isolation in 2s on the bumped tree) - a pre-existing
timing/parallelism flake, not a bump regression, so left as-is.
Updated CLAUDE.md and two docs (Configuration.md, operations/2026-07-20-localdb-pair-
replication.md) where they stated the old pin numbers and the now-superseded
per-database snapshot-resync behavior as current fact; dated historical records
(archreview snapshots, dated plan docs) were left untouched.
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
New docs/security.md section "Mesh gRPC ingress is AllowAnonymous by design"
(archreview 03/S14): what the cookie FallbackPolicy covers, why it can block a
correctly-PSK'd call before the interceptor's verdict matters, why the
AllowAnonymous opt-out is safe (all five interceptors fail-closed), the
affected-endpoint table (fused-vs-driver-only distinction), and the two
standing guards (MeshGrpcAnonymousMetadataTests, MeshGrpcMapConventionTests)
plus the auth-stack/driver-only boundary tests.
One cross-reference line each in docs/Configuration.md (§MeshTransport,
§ConfigSource/ConfigServe) and docs/Telemetry.md, so an operator reading
about the listener ports meets the constraint where they'd look for it.
LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.
Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.
configured => RegisterReplicated(table, baselineExistingRows: true), per
table, in the unchanged load-bearing DDL->register order
unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
drops the capture triggers an earlier build left behind and
prunes that table's oplog/row-version rows. One Information
line, only when it actually cleaned something.
PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").
Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.
Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
CLAUDE.md — a new "How a deploy reaches that address space" block under the v3
Batch 4 address-space section. The blanket ChangedRawTags/ChangedUnsReferenceVariables
=> Rebuild rule is gone, so the doc now names both halves explicitly: what still
rebuilds (alarm-bearing raw-tag deltas, raw-tag identity changes, raw container
renames/removals, changed equipment/alarms) and what is now surgical (non-alarm
attribute deltas, every UnsReferenceDelta). Also records the three semantics that
are easy to break by accident: mirror repair rides UnsReferenceVariable's inherited
backing fields (omitting a newly-inherited field silently reopens the staleness),
a degraded non-rebuild apply deliberately does not advance _lastApplied, and the
raw-realm announce gap is no longer neutral now that surgical adds keep
subscriptions alive.
CLAUDE.md ConfigSource — the RevisionHash == SHA-256(artifact bytes) paragraph is
still true and stays; it gains the clause that the blob is gzip-compressed since
2026-08-15 and the hash covers the COMPRESSED bytes, which is precisely why the
fetcher, cache and serve path were untouched.
docs/Uns.md + docs/Raw.md — one section each on what a deploy now tears down, from
the authoring surface's point of view, including the mirror repair semantics
(identity-preserving backing edits update in place and keep their MonitoredItems;
re-points and effective-name edits remove+recreate that one node).
docs/Configuration.md — "Artifact compression (2026-08-15)" under ConfigSource/
ConfigServe: gzip inside the hashed bytes, magic-sniffed on read, pre-upgrade
uncompressed rows still parse, plus the two operational facts an operator needs.
The first deploy after upgrade re-hashes every revision (same config, new bytes,
new RevisionHash) so it is a one-time full re-dispatch rather than a no-op; and
rollout is ordered — nodes must run artifact-sniffing code BEFORE central starts
emitting compressed blobs, since a pre-upgrade node handed a gzip artifact fails
the apply and keeps last-known-good until it is upgraded.
Plan doc — a "Follow-ups (recorded, not implemented)" section covering the seven
items surfaced during implementation and review: the raw-realm announce gap, the
RemoveEquipmentSubtree dangling-inverse-edge exposure, the missing end-to-end test
for a degraded remove-bearing apply's ratcheted rebuild, the scripted-alarm
thin-plan-ids-against-rich-rows blindness, the absent monotonic dispatch sequence,
LocalDb store ordering under concurrent StoreAsync, and threading the in-hand blob
through RestoreApplied.
tasks.json — all nine tasks completed.
Docs only; no source changes.
Closes the pin skew against ScadaBridge (which already pins 0.2.1). Pure pin
bump — no code change was required:
- `RegisterReplicated(string)` still delegates to `baselineExistingRows: false`,
so `LocalDbSetup.OnReady` is byte-identical in behaviour.
- Everything 0.2.x added is additive (`DeregisterReplicated`, `MaxBatchBytes`,
the `baselineExistingRows` overload); none of it is called here.
- The wire protocol is unchanged (0.1.x peers still sync) and the on-disk
bookkeeping schema v1 -> v2 upgrades in place on open, so no rig, volume or
operator action is needed.
The one thing that DID need reconciling is documentation. 0.2.0 made batching
byte-budgeted: `MaxBatchBytes` (2 MB default, under the 4 MB gRPC cap) now
bounds the wire message, and `MaxBatchSize` is demoted to a secondary row cap
plus the DB read-page size. The rig's `MaxBatchSize = 16` was chosen as a
hand-computed byte proxy ("row-count-only against gRPC's 4 MB cap"), and that
rationale is now wrong everywhere it appeared — appsettings, Configuration.md,
the pair-replication runbook, and both docker-dev site-a services.
The VALUE stays 16, for a different and still-real reason: OplogStore /
SnapshotStreamer materialise a whole read page in memory *before* the
byte-budget split runs, so 16 x ~171 KB artifact chunk rows bounds that
transient allocation at ~2.7 MB. Documented as such, with `MaxBatchBytes`
added to both config tables.
CLAUDE.md records the pin and two deliberate non-adoptions: this repo does not
take ScadaBridge's conditional-registration + boot-time `DeregisterReplicated`
self-heal (OnReady registers all four tables unconditionally, and
LocalDbSetupTests pins that set), and consequently does not use
`baselineExistingRows` either — so enabling replication on a long-running
default-OFF node converges on subsequent writes, not on rows already in the
file. Both are behaviour changes, out of scope for a pin bump.
Verification: solution builds clean; full suite run (57 projects). All 66
LocalDb tests in Host.IntegrationTests green. One unrelated failure,
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_
fails_with_disposed — a Roslyn evaluator disposal race with no LocalDb in its
path, whose own comment documents that its "at most one clear per evaluation"
assumption depends on thread-pool pressure. It passed 3/3 re-runs in isolation
and fails only under full-solution parallel load; pre-existing test-design
fragility, not a regression from this bump.
Foundation task of the AdminUI UI sweep. CSS/link/doc only — no behavior change
beyond restoring styles that were built but never served.
- App.razor: link the RCL's scoped-CSS bundle
(_content/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.bundle.scp.css).
It is produced on every compile and served by the Host, but nothing referenced
it, so ContextMenu.razor.css was dead at runtime and context menus rendered
in-flow with no position:fixed.
- site.css: three additions.
* .btn / .btn-sm sizing as Bootstrap CSS-VARIABLE overrides (never box
properties — .btn-group depends on Bootstrap's own .btn rules for its seams
and end-cap radii). Theme 0.3.1 ships no .btn rule, so Bootstrap's 1rem
applied against the theme's 0.9rem body. Upstreaming candidate, same as the
ignitionoee router's app.css.
* .notice-alert — red variant of the kit's amber .notice, on the --bad/
--bad-bg/--bad-border tokens. The 13 existing call sites hack this with
style="border-color:var(--alert)", and no --alert token exists anywhere, so
the tint silently no-ops today. Call-site migration is a separate task.
* .tree-twist — shared 18px expander column for the tree components, replacing
repeated inline style="width:18px". Rows are d-flex, so flex-basis is what
actually holds the column. Adoption is a separate task.
- ContextMenu.razor.css: correct the header comment — the kit is light-only
(no dark block in Theme 0.3.1, site.css, or this file). No rule changes.
- docs/Telemetry.md: record that driver-health is keyed per driver instance
across the cluster, not per Akka member (the wire record carries no per-member
identity). The fact previously lived only in Hosts.razor page copy.
- docs/Configuration.md: new audit-log subsection recording that EventId gives
cross-restart dedup, enforced by the filtered unique index
UX_ConfigAuditLog_EventId, so a retried apply inserts no duplicate row;
CorrelationId joins a row to its originating workflow. Previously only in the
historical v2 rebuild plan and ClusterAudit page copy.
#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh
split: it derives "the fleet" from central's own Akka gossip ring, which after the split is
just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole
chain is dead end to end, so this deletes it rather than reviving it.
- Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS
"fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub
("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting
HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads
NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead.
- Its second input never existed. DriverHostStatusHeartbeat had no production sender —
only the broadcaster's own test told it — so CurrentRevision was always null and the
30 s staleness -> Degraded flip could never fire.
- Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an
accident of topology rather than a design.
The fleet visibility that actually works is unchanged and already cross-mesh-correct:
/fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials.
Precedent for deleting inert machinery rather than activating it: #522 (driver stability
tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint).
Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin
singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its
key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus /
FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps
ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed
a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels,
Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are
left as the historical record they are.
Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline.
ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
OtOpcUa runs five internal gRPC planes between its own processes, all of
them cleartext HTTP/2 (h2c) with a shared-bearer interceptor: the live
telemetry stream, config-serve artifact fetch, LocalDb pair sync, mesh
commands and mesh deploy acks. The bearer therefore crosses the wire in
the clear on every one of them. That was true and undocumented; the two
mesh planes are command surfaces, so it is worth saying out loud rather
than leaving to be rediscovered per plane.
docs/security.md gains "Internal gRPC planes (h2c) — accepted cleartext
posture" under Transport Security (whose existing content is entirely
about the OPC UA endpoint). It carries a five-row table — service name,
listener config key, credential config key, interceptor — the honest
read-vs-write breakdown per plane, and the decision itself.
Decision (issue #545, option 1): risk ACCEPTED 2026-08-06, once, for all
five planes together, on the trusted-link assumption every plane already
makes (VPN / dedicated WAN). Per-cluster PSKs already bound blast radius
on the two command planes — a captured SITE-A key cannot drive SITE-B —
and production keys arrive as ${secret:...} references that must never be
reused across clusters. Recorded future work: a UseTls/scheme option,
which if built must cover all five planes together, since a mixed posture
is harder to reason about than either uniform answer.
docs/Configuration.md's MeshTransport section gains one cross-reference
paragraph pointing at the new subsection.
Docs only — no code, no behaviour change. Every service name, config key,
interceptor name and shared-listener claim was verified against the
source; ports 4056/4055 are phrased as the docker-dev rig's values of
Telemetry:GrpcListenPort / ConfigServe:GrpcListenPort, not as hardcoded
ports.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 3.2 of the ClusterClient->gRPC migration. Sweeps the docs that still
described the Akka ClusterClient transport as live, following its Phase 3
deletion (this branch): MeshTransport:Mode is now Dps (pair-local DPS) or
Grpc (cross-mesh unary calls) only, ClusterClient fails validation loudly,
CentralContactPoints is deleted (not superseded), and the split-topology
validator requires exactly Grpc rather than ClusterClient.
- docs/Configuration.md: rewrite the MeshTransport section — table,
dark-switch/PSK-seeding prose, listener sharing (:4056 driver-role,
:4055 admin-role), remove the "shipped dark, cutover pending" framing.
- docs/Redundancy.md: replace the Dps/ClusterClient transport table with
Dps/Grpc, fix the Phase 6 CentralCommunicationActor + validator bullets.
- docs/Telemetry.md: fix stale ClusterClient cross-refs, add a note that
the driver-role telemetry listener also hosts NodeCommandService.
- docs/v2/config-db-schema.md: one-line "(retired: ...)" annotation on the
historical AkkaPort/GrpcPort note — left as a historical record otherwise.
- CLAUDE.md: rewrite "## Mesh command transport (MeshTransport)" for the
post-deletion world (no ClusterClient/receptionist, per-node gRPC dial,
fire-and-forget semantics unchanged) and fix two Phase 6 bullets.
deferment.md has no mesh-transport mentions; left untouched.
docs-only change; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` stays green (0
errors) after the edit.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Phase 0 task 0.6 of the ClusterClient->gRPC migration: wire the dark
MeshTransport Grpc-mode keys into the docker-dev rig (GrpcPsk +
CentralGrpcEndpoints on all six nodes, ClusterPsks on the central pair
only), document the new keys in docs/Configuration.md as shipped
dark/cutover pending, and record the Phase 0 live-gate evidence
(rig boot with no validator fires, two deliberate-misconfig refusals,
full suite green).
Files gitea.dohertylan.com/dohertj2/lmxopcua#542 (FleetStatusBroadcaster
mesh-blind post-Phase-6) and #543 (three pre-existing interceptors accept
a whitespace-only key as set) as follow-ups out of Phase 0's scope.
`Security:Acl:Enabled` (default false) makes scoped `NodeAcl` grants decide
data-plane authorization at the three seams that have one, and gives HistoryRead
a check it never had. Read, Browse and Subscribe stay ungated — stage 6.
The ACL decision REPLACES the server-wide role check rather than chaining with
it. Chained, a grant would have to confer the role as well as pass the scope
check, and only Cluster-scoped grants confer a role — so every narrow grant
would have stayed inert and the flag would have looked inert too. The
load-bearing test is an ALLOW: a session carrying no roles at all writing
successfully on the strength of its grant. A deny-only test passes against a
build where the ACL is never consulted, because the role gate denies too.
Found en route: the principal never reached the session. The authenticator
passed it as OpcUaUserAuthResult.DisplayName, but RoleCarryingUserIdentity
derives DisplayName from the presented token — for a key-bearing client that is
the static username `apikey`, identical fleet-wide. So the trie's matching key
was dropped at the identity boundary, and the audit ClientUserId on a rejected
write recorded `apikey` for every consuming system. Principal and ApiKeyId are
now carried explicitly.
Beyond the stage as sketched:
- Revocation reaches an OPEN session. Every decision re-checks the
authenticating key against the current snapshot; without it a key revoked by a
deployment kept its authority until the client chose to reconnect, which for a
machine-to-machine consumer can be indefinitely.
- UserAuthorizationState is built per CALL, not per session. The per-session
shape exists so an LDAP session can re-resolve membership that changed out of
band; nothing can change out of band under API keys, so the generation stamp is
the whole invalidation signal and a per-call record picks it up for free.
IsStale/NeedsRefresh therefore cannot fire here and are documented as vestigial.
- The trie is built inside DataPlaneAuthorityState.Apply, under the same
generation as the snapshot, rather than by a second wiring at the publish
actor. An uninstalled trie denies everything — indistinguishable from working
enforcement. Deleting the install turns 13 of the 22 new tests red.
- The deploy log reports what the BUILD dropped instead of re-deriving it. The
old count agreed with the builder on one of its four drop reasons; a rule with
a blank ScopeId, a retired scope kind, or a path not starting at a realm
vanished with the log reading zero.
Both IApiKeyStore implementations widened to the full grant set. The narrowing
was redundant — ApiKeyInterimRoleMapper applies the same filter itself — and
became a trap once scope was enforced. Pinned by asserting the roles are
unchanged, with the two grants carrying different flags so it cannot pass by
accident.
Falsified by mutation: enforcement never engaging, the write seam ignoring the
ACL, the realm dropped from the scope, the revocation check removed, and the
trie never installed each turn the intended tests red and no others.
Full solution suite green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on 3a6b2a1d (PrimaryGateFailoverTests, and
DriverProbeRegistrationTests expecting 10 probe types where 12 now register)
plus the AbLegacy integration suite, which needs a docker fixture.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient
(OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport:
deploy seals on ClusterClient, a stopped node fails the deploy naming it, the
MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both
centrals independently), and the frame-size canary stayed silent.
Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md:
A. A 10-30s post-restart delivery window exists, but the DPS control fails
identically -- it is a cluster-convergence property, not a Phase 2
regression. Running the control is the only reason this is a note and not
a false attribution.
B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut
deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap
replaying the orphan Applying row on restart. Zero buffering removes the
silent, timer-driven replay, not the boot-time DB one. Corrected in the
plan decisions table, Configuration.md, and CLAUDE.md; the decision stands,
the justification narrowed.
C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership
layer (frozen mid-Exiting, not Unreachable, so auto-down does not help).
Transport-independent; a Phase 7 downing/rejoin drill item.
D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are
rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/
drivers. So gate steps 6/7 could not run; the driver-control node-side leg
is proven live only because DispatchDeployment shares its EventStream path.
Step 8's planned log-signal does not exist (HandleApplyAck is silent on
success); proven instead by both centrals independently logging per-instance
ClusterClient creation, which a singleton cannot.
Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which
was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj)
into the build context and exhausting the Docker VM disk mid-COPY.
Rig restored to Mode=Dps, six members, no leftover maintenance flags.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central
contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up
on the transport it has always used and the live gate can flip the whole fleet
at `docker compose up` without a compose edit or a rebuild.
Docs: a MeshTransport section in Configuration.md (including why contact points
are addresses only, why the contact set does not scope delivery, and why
buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command
transport" section in Redundancy.md contrasting the two modes, and a summary
beside the Redundancy notes in CLAUDE.md.
Corrects the design doc's claim that ScadaBridge replies with a typed failure to
every unhandled message. Neither of their comm actors has a ReceiveAny or an
Unhandled override; the idiom is per message type and fires on a missing
registration, and an unknown type dead-letters there as it does anywhere else.
Records the exit-gate deviation in the program plan: "an Ask timing out cleanly
against a stopped node" cannot be run, because no Ask crosses this boundary. The
honest equivalents are a stopped node failing the deploy at the apply deadline
and an unreachable contact dropping the command with a Warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
- docker-dev seed: AkkaPort = 4053 on all six ClusterNode rows, so a freshly
seeded rig matches a migrated one. GrpcPort left null.
- config-db-schema.md: both columns, why AkkaPort is NOT NULL/4053 and GrpcPort
is nullable, the unenforced duplication + its reconciler, and Enabled's new
second meaning as the deploy path's expected-ack set.
- Configuration.md: Cluster:Port / PublicHostname now flag that they are stored
twice, with the "update the row too" instruction and why the drift is silent.
- design doc §7: Phase 1 marked done, plus a "Phase 1 as shipped" note recording
both deviations rather than leaving the sketch reading as what happened.
- program plan: Phase 1 marked done; AdminUI node edit explicitly deferred.
- CLAUDE.md: the deploy-path behaviour change and its three consequences.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.
The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.
Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.
Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.
Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.
Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
New operations runbook (enable/disable, fail-closed ApiKey rule, stop/start-
together, tombstone-retention window, MaxBatchSize row-count-vs-4MB, never
sqlite3-a-live-WAL-DB cp-triplet recipe). Configuration.md gains a LocalDb
section; Redundancy.md gains a pair-local config cache section (what boot-from-
cache does and does not cover); CLAUDE.md gains the Phase 1 scope paragraph.
HistorianGateway is now the sole historian backend (read + alarm SendEvent +
continuous WriteLiveValues). Document the final state and retire the Wonderware
sidecar from the docs/config/labels:
- CLAUDE.md: rewrite the Historian section — ServerHistorian /
ContinuousHistorization / AlarmHistorian config keys, the IHistorianProvisioning
EnsureTags hook, the GatewayAlarmHistorianWriter SendEvent path + ReadEvents
dependency on gateway RuntimeDb:EventReadsEnabled=true, gateway-side
prerequisites (RuntimeDb flags + historian:read/write/tags:write scopes),
migration note, and two KNOWN-LIMITATION callouts (live-validation gate +
empty historized-ref-set recorder follow-on).
- appsettings.json: fix the stale ServerHistorian block (Host/Port/SharedSecret/
ServerCertThumbprint -> Endpoint/ApiKey/UseTls/AllowUntrustedServerCertificate/
CaCertificatePath/CallTimeout, keep MaxTieClusterOverfetch); add a disabled
ContinuousHistorization block; prune the orphaned Wonderware keys from
AlarmHistorian (keep the SQLite knobs). ApiKey env-supplied via
ServerHistorian__ApiKey (commented; valid strict JSON via _comment keys).
- README.md + docs (Historian.md, AlarmHistorian.md, Configuration.md,
ServiceHosting.md, DriverLifecycle.md, drivers/README.md, Uns.md, VirtualTags.md,
AlarmTracking.md, Client.UI.md, README.md, TestConnectProbes.md): retire the
Wonderware historian backend from current-backend descriptions; fix the stale
ServerHistorian/AlarmHistorian config tables (now gateway shape); convert
drivers/Historian.Wonderware.md to a retired stub pointing at the gateway.
- Source/UI labels (descriptive text only, no behavior change):
OtOpcUaServerHostedService.cs, HistoryPaging.cs, OtOpcUaSdkServer.cs,
HistorianAdapterActor.cs, VirtualTagModal.razor, ScriptedAlarmModal.razor,
AlarmsHistorian.razor now name the HistorianGateway backend.
Build clean (0 errors); AdminUI.Tests green (514 passed).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Audit (three parallel agent passes) found 43 markdown files carrying
stale references to the deleted Galaxy.Host/Proxy/Shared projects
after the v2-mxgw merge. This commit lands the prioritized fixes.
Track 1 — high-traffic in-place rewrites (3 files, ~454 lines deleted)
- README.md (202 → 91 lines): drops .NET 4.8 / x86 / TopShelf install
text; leads with the multi-driver .NET 10 server identity and points
at scripts/install/Install-Services.ps1 and the parity rig.
- docs/v2/driver-specs.md §1 Galaxy (~289 → ~66 lines): replaces the
Tier-C out-of-process spec with a Tier-A in-process description
matching the current GalaxyDriver code, with the four-section
GalaxyDriverOptions JSON shape pulled verbatim from
Config/GalaxyDriverOptions.cs.
- docs/drivers/Galaxy.md (211 → 92 lines): full rewrite around the
current Browse/Runtime/Health/Config sub-folders.
Track 2 — historical banners (5 files)
- lmx_mxgw.md, lmx_mxgw_impl.md, lmx_backend.md,
docs/v2/Galaxy.ParityMatrix.md,
docs/v2/implementation/phase-2-galaxy-out-of-process.md each get a
"✅ Completed 2026-04-30 — historical record" banner block. lmx_mxgw.md
also fixes two dead links (`docs/Galaxy.Driver.md` and
`docs/v2/Galaxy.Driver.md`) → `docs/drivers/Galaxy.md`.
Track 3 — v1 archive sweep (10 git mv + 1 new index + 2 in-place scrubs)
- Moved 10 v1 docs under docs/v1/ preserving subpath structure:
AlarmTracking, Configuration, DataTypeMapping, HistoricalDataAccess,
Subscriptions (top-level); drivers/Galaxy-Repository,
drivers/Galaxy-Test-Fixture; reqs/GalaxyRepositoryReqs,
reqs/MxAccessClientReqs, reqs/ServiceHostReqs.
- New docs/v1/README.md is the shared archive banner + per-file table.
- docs/README.md repointed to the v1 paths and updated to reflect the
v2 two-process deploy shape (Server + Admin + optional
OtOpcUaWonderwareHistorian).
- docs/v2/Galaxy.ParityRig.md got a historical banner + four inline
scrubs marking the OtOpcUaGalaxyHost service / Driver.Galaxy.Host
EXE / Driver.Galaxy.ParityTests project as deleted-in-PR-7.2.
The repo's live-reading surface (README + CLAUDE.md + docs/v2/) now
describes only the post-PR-7.2 architecture. v1 docs are preserved as
a labelled archive under docs/v1/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five operational docs rewritten for v2 (multi-process, multi-driver, Config-DB authoritative):
- docs/Configuration.md — replaced appsettings-only story with the two-layer model.
appsettings.json is bootstrap only (Node identity, Config DB connection string,
transport security, LDAP bind, logging). Authoritative config (clusters, namespaces,
UNS, equipment, tags, driver instances, ACLs, role grants, poll groups) lives in
the Config DB accessed via OtOpcUaConfigDbContext and edited through the Admin UI
draft/publish workflow. Added v1-to-v2 migration index so operators can locate where
each old section moved. Cross-links to docs/v2/config-db-schema.md + docs/v2/admin-ui.md.
- docs/Redundancy.md — Phase 6.3 rewrite. Named every class under
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/: RedundancyCoordinator, RedundancyTopology,
ApplyLeaseRegistry (publish fencing), PeerReachabilityTracker, RecoveryStateManager,
ServiceLevelCalculator (pure function), RedundancyStatePublisher. Documented the
full 11-band ServiceLevel matrix (Maintenance=0 through AuthoritativePrimary=255)
from ServiceLevelCalculator.cs and the per-ClusterNode fields (RedundancyRole,
ServiceLevelBase, ApplicationUri). Covered metrics
(otopcua.redundancy.role_transition counter + primary/secondary/stale_count gauges
on meter ZB.MOM.WW.OtOpcUa.Redundancy) and SignalR RoleChanged push from
FleetStatusPoller to RedundancyTab.razor.
- docs/security.md — preserved the transport-security section (still accurate) and
added Phase 6.2 authorization. Four concerns now documented in one place:
(1) transport security profiles, (2) OPC UA auth via LdapUserAuthenticator
(note: task spec called this LdapAuthenticationProvider — actual class name is
LdapUserAuthenticator in Server/Security/), (3) data-plane authorization via
NodeAcl + PermissionTrie + AuthorizationGate — additive-only model per decision
#129, ClusterId → Namespace → UnsArea → UnsLine → Equipment → Tag hierarchy,
NodePermissions bundle, PermissionProbeService in Admin for "probe this permission",
(4) control-plane authorization via LdapGroupRoleMapping + AdminRole
(ConfigViewer / ConfigEditor / FleetAdmin, CanEdit / CanPublish policies) —
deliberately independent of data-plane ACLs per decision #150. Documented the
OTOPCUA0001 Roslyn analyzer (UnwrappedCapabilityCallAnalyzer) as the compile-time
guard ensuring every driver-capability async call is wrapped by CapabilityInvoker.
- docs/ServiceHosting.md — three-process rewrite: OtOpcUa Server (net10 x64,
BackgroundService + AddWindowsService, hosts OPC UA endpoint + all non-Galaxy
drivers), OtOpcUa Admin (net10 x64, Blazor Server + SignalR + /metrics via
OpenTelemetry Prometheus exporter), OtOpcUa Galaxy.Host (.NET Framework 4.8 x86,
NSSM-wrapped, env-variable driven, STA thread + MXAccess COM). Pipe ACL
denies-Admins detail + non-elevated shell requirement captured from feedback memory.
Divergence from CLAUDE.md: task spec said "TopShelf is still the service-installer
wrapper per CLAUDE.md note" but no csproj in the repo references TopShelf — decision
#30 replaced it with the generic host's AddWindowsService wrapper (per the doc
comment on OpcUaServerService). Reflected the actual state + flagged this divergence
here so someone can update CLAUDE.md separately.
- docs/StatusDashboard.md — replaced the full v1 reference (dashboard endpoints,
health check rules, StatusData DTO, etc.) with a short "superseded by Admin UI"
pointer that preserves git-blame continuity + avoids broken links from other docs
that reference it.
Class references verified by reading:
src/ZB.MOM.WW.OtOpcUa.Server/Redundancy/{RedundancyCoordinator, ServiceLevelCalculator,
ApplyLeaseRegistry, RedundancyStatePublisher}.cs
src/ZB.MOM.WW.OtOpcUa.Core/Authorization/{PermissionTrie, PermissionTrieBuilder,
PermissionTrieCache, TriePermissionEvaluator, AuthorizationGate}.cs
src/ZB.MOM.WW.OtOpcUa.Server/Security/{AuthorizationGate, LdapUserAuthenticator}.cs
src/ZB.MOM.WW.OtOpcUa.Admin/{Program.cs, Services/AdminRoles.cs,
Services/RedundancyMetrics.cs, Hubs/FleetStatusPoller.cs}
src/ZB.MOM.WW.OtOpcUa.Server/Program.cs + appsettings.json
src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Host/{Program.cs, Ipc/PipeServer.cs}
src/ZB.MOM.WW.OtOpcUa.Configuration/Entities/{ClusterNode, NodeAcl,
LdapGroupRoleMapping}.cs
src/ZB.MOM.WW.OtOpcUa.Analyzers/UnwrappedCapabilityCallAnalyzer.cs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames all 11 projects (5 src + 6 tests), the .slnx solution file, all source-file namespaces, all axaml namespace references, and all v1 documentation references in CLAUDE.md and docs/*.md (excluding docs/v2/ which is already in OtOpcUa form). Also updates the TopShelf service registration name from "LmxOpcUa" to "OtOpcUa" per Phase 0 Task 0.6.
Preserves runtime identifiers per Phase 0 Out-of-Scope rules to avoid breaking v1/v2 client trust during coexistence: OPC UA `ApplicationUri` defaults (`urn:{GalaxyName}:LmxOpcUa`), server `EndpointPath` (`/LmxOpcUa`), `ServerName` default (feeds cert subject CN), `MxAccessConfiguration.ClientName` default (defensive — stays "LmxOpcUa" for MxAccess audit-trail consistency), client OPC UA identifiers (`ApplicationName = "LmxOpcUaClient"`, `ApplicationUri = "urn:localhost:LmxOpcUaClient"`, cert directory `%LocalAppData%\LmxOpcUaClient\pki\`), and the `LmxOpcUaServer` class name (class rename out of Phase 0 scope per Task 0.5 sed pattern; happens in Phase 1 alongside `LmxNodeManager → GenericDriverNodeManager` Core extraction). 23 LmxOpcUa references retained, all enumerated and justified in `docs/v2/implementation/exit-gate-phase-0.md`.
Build clean: 0 errors, 30 warnings (lower than baseline 167). Tests at strict improvement over baseline: 821 passing / 1 failing vs baseline 820 / 2 (one flaky pre-existing failure passed this run; the other still fails — both pre-existing and unrelated to the rename). `Client.UI.Tests`, `Historian.Aveva.Tests`, `Client.Shared.Tests`, `IntegrationTests` all match baseline exactly. Exit gate compliance results recorded in `docs/v2/implementation/exit-gate-phase-0.md` with all 7 checks PASS or DEFERRED-to-PR-review (#7 service install verification needs Windows service permissions on the reviewer's box).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty