Commit Graph

125 Commits

Author SHA1 Message Date
Joseph Doherty 7829b937d1 feat(717): Warm non-transparent redundancy, default-off (T1-T10)
OpcUa:Redundancy:Mode (Unset default = today's server exactly; None; Warm; Hot and unknown values fail start).
- T1 options + validator + effective-URI check + boot banner; Warm needs the driver role.
- T2 metadata in OnNodeManagerStartedAsync: Warm -> NonTransparentRedundancyType, read-only
  ServerUriArray [self, peers], RedundancySupport=Warm, boot ServiceLevel 1; None -> RedundancySupport=None
  and the SDK's orphan i=11314 deleted; Unset untouched.
- T3 RedundancyStateChanged.Revision (up-number, incarnation tick, sequence); subscribers drop older
  revisions under Warm (OpcUaPublishActor, DriverHostActor); unrevisioned never makes a node Active.
- T4 WarmRoleAuthority + ComputeWarm bands (Standby 90/50, Active 250/200/100), PromoteWhilePrimaryUnreachable.
- T5 IStandbyProjection + DeferredStandbyProjection; OpcUaPublishActor runs promotion/demotion in the spec
  4.7 order (Active level last and retried; Standby level first, data degraded regardless).
- T6 EquipmentVariableState read projection (BadOutOfService/null); Warm Standby refuses writes BadInvalidState.
- T7 D4 mechanism adopted after T0: ordinary-tag marker on MonitoredItemNotification.Message validated by
  (ClientHandle, DataValue); first-delivery decision keyed by (SubscriptionId, SequenceNumber); the SDK's
  cached copy is never mutated.
- T8 queue current/BadOutOfService to live ordinary-tag items ignoring filters.
- T9 alarm stamp/deliver split (suppressed on Standby), Part 9 methods refused on Standby, promotion
  refresh via ISubscription.ConditionRefresh2Async on every live event item.
- T10 deploy-time RedundancyMode cross-check + six instruments.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:39:50 -05:00
Joseph Doherty ec3b835e4a fix(controlplane,runtime): #868 fourth review — SQL poison allowlist, memory-counted acks survive restart, adopted revisions unconfirmed
M1 IsPoison classified every SQL Server transient as poison:
   Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
   (1205), timeout (-2 / Win32 258), dropped connection or AG failover was
   counted from memory and the node's row left Applying. Poison is now an
   ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
   whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
   retries, bounded by the deadline deferral cap. IsPoison is public for tests.
   Acks counted only from memory (poison branch and the deferral cap) are kept
   in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
   restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
   same-revision dispatch re-applies instead of short-circuiting a clean
   Applied over an empty address space: TryRecoverFromStale (restores only the
   revision marker) and RestoreApplied when the artifact read fails (null
   blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
   restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
   TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
   reach the ConfigDb acks a same-revision re-dispatch Failed
   (PartiallyFailed, not Sealed); Stale recovery and failed restores are
   unconfirmed too.

Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.

Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:31:49 -05:00
Joseph Doherty 780c87eba3 fix(controlplane,runtime): #868 third review — bounded failure reasons, poison ack writes, cache-booted revisions unconfirmed
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
  nvarchar(2048), so on SQL Server the write failed forever, the ack retried
  forever, the deadline deferred forever and every later deploy was refused.
  - One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
    used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
    (exception messages and degraded reasons).
  - A non-transient DbUpdateException (not concurrency, no transient
    DbException / timeout / socket / IO underneath) is POISON: the ack is
    counted from memory with an Error, never retried.
  - Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
    past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
  written before the outcome) and never acks. Cache-booted revisions
  (BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
  re-dispatch re-applies and acks the real outcome instead of a clean
  short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
  path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
  needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
  remainder timed out nodes that applied). No dispatched-at column: a second
  restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
  rolling-upgrade across this change.

Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:13:36 -05:00
Joseph Doherty 494dcb5b87 fix(controlplane,runtime): #868 second review — final-outcome node rows, unpersisted acks survive deadline and restart
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
   as the revision advanced, up to ~105 s before the address-space outcome.
   The coordinator's recovery counts any non-Applying row as an ack, so a
   coordinator restart in that window sealed CLEAN while the node was about to
   report degraded (#601). The row now stays Applying until ReleasePendingAck
   (or the no-publish-actor path) writes the final Applied, clean or with the
   degraded reason. A node restart in the window replays the orphan Applying
   row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
   HandleDeadline persists that backlog first and defers its decision while
   any remain, so a node that acked is never timed out as silent; PreRestart
   re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
   pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
   the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
   a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
   closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
   health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
   failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
   apply to a row still Dispatching/AwaitingApplyAcks.

Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:54:30 -05:00
Joseph Doherty 19c7b5d7e4 fix(controlplane): #868 review — complete fully-acked recoveries, replay queued acks before a deadline, retry failed ack persists
Review findings on 02503a4a (ConfigPublishCoordinator):

1 (HIGH) A recovered deployment whose every expected ack was already in
  NodeDeploymentState (a seal whose SaveChanges failed, a fused node's own row)
  was never completed: inside the deadline it waited for acks that had landed,
  past it HandleDeadline's "race" guard returned without sealing — stuck in
  AwaitingApplyAcks, refusing every later deploy. ApplyRecovery now completes
  it (Seal / PartiallyFailed), and HandleDeadline completes a full ack set
  instead of returning.
2 (MED-HIGH) The past-deadline recovery of a populated expected set ran
  HandleDeadline inline before the stash replay, discarding on-time acks queued
  during the outage. It is now self-sent, behind the replay. The zero-rows
  past-deadline branch stays inline on purpose: no ack can count there, and a
  self-sent deadline would time out a queued dispatch of that same row right
  after publishing it.
3 (MED) A failed PersistNodeAck threw, restarted the actor and dropped the ack
  (driver-only nodes never write their own row and do not resend). Acks are now
  persisted before being counted, and a failed persist is re-queued on a timer.
  The class remarks document that the Loading stash is node-local (lost on a
  hand-over; queued dispatches are recovered from their rows).
Minor: MarkReady/Become(Ready) only after the recovery applied; a throw while
applying is a failed load (backoff) rather than a tight restart loop; a queued
dispatch matching the recovered Dispatching row logs Info, not the "lost
dispatch" Warning.

Tests (ConfigPublishCoordinatorDbOutageTests): fully-acked recovery seals /
partially fails inside and past the deadline (4 cases); queued ack applied
before a past-deadline recovery; failed ack persist retried. All 6 fail on
02503a4a.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:38:45 -05:00
Joseph Doherty 02503a4a9f fix(controlplane): ConfigPublishCoordinator survives an unreachable ConfigDb at singleton start (#868)
PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.

- PreStart now only subscribes and kicks off an async recovery load piped back
  to the actor. A failure retries with exponential backoff (1 s doubling to
  30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
  stashes every message (DispatchDeployment logged as queued, ApplyAck too);
  they are replayed in order after the recovered state is applied. Recovery
  decisions (re-dispatch / time out) now run inline before the replay instead
  of via self-tells, and a duplicate dispatch of the in-flight deployment with
  a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
  "config-publish" ready-tier health check: Degraded while loading (with the
  attempt count and last error), Healthy when ready or not hosted here.

Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:28:39 -05:00
Joseph Doherty 4a6229e4d2 fix(security): env: credential references follow the per-instance rule (#849)
env:<PREFIX><INSTANCE>_... is the per-instance convention for every prefix of a
driver's scope (OTOPCUA_GALAXY_, GALAXY_MXGW_, OTOPCUA_OPCUACLIENT_,
OTOPCUA_HEIDENHAIN_), INSTANCE = a driver id or name upper-cased with every other
character as '_'. A variable whose part after the prefix equals or starts with
(at a '_' boundary) ANOTHER same-type instance's id or name is refused with
CredentialReferenceOtherInstance at save, deploy, Test Connect and browse - the
same DriverCredentialPolicy.DescribeOtherInstanceProblems the secret: rule uses
(now over CollectInstanceReferences / ReferenceNames). Another-instance matches
win over the owner's, as for secrets. Shared variables that name no instance
(GALAXY_MXGW_API_KEY) keep working. HasInstanceSegments now also gates on env:,
so an env:-only browse needs the ConfigDb (fail-closed without it).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 06:24:47 -05:00
Joseph Doherty 271b96a644 fix(security): #835/#836 review — unsaved endpoints never borrow the owner's secrets; audit credential use; path rule follows AuthType
I-1: Test Connect / browse configs may carry unsaved edits (the device modal merges
the stored DriverConfig with the typed device config). The server
(DriverInstanceIdentities.EffectiveOwnerAsync, used by AdminOperationsActor and
BrowserSessionService) honours the named owner only when every endpoint value
(new DriverCredentialRules.EndpointFields: Galaxy Endpoint, OpcUaClient
EndpointUrl/EndpointUrls/DiscoveryUrl, Heidenhain Host/Port) is one that driver
stores; otherwise the owner is null and its instance-named secrets are refused.
Every Test Connect / browse whose config carries an in-scope secret:/env: reference
is recorded (CredentialUseAudit: IAuditWriter event category CredentialUse + a
structured Information line: user, driver, endpoint hosts, outcome).

L-1: the UserCertificatePath rule is skipped only in an object that explicitly sets
AuthType to non-Certificate (string or ordinal); the deploy gate also checks the
runtime-merged driver+device config so the switch cannot be split from the path; the
form no longer persists the hidden field.
L-2: a ConfigDb fault in the browse check is a refusal; RawBrowseModal catches open
failures into its error message.
L-4: a relative UserCertificateDirectory resolves against AppContext.BaseDirectory.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 04:12:40 -05:00
Joseph Doherty 87416a230f Merge #835 #836 per-instance credential references, confined user-certificate path 2026-11-11 03:53:12 -05:00
Joseph Doherty 63331b0416 fix(security): per-instance credential references (#835) and confined OpcUaClient user-certificate path (#836)
#835: inside one driver namespace, a secret:<ns>/<X>/... reference whose first
segment names ANOTHER existing same-type driver instance (id or name,
case-insensitive; another-instance match beats an own-name match, so renaming
one's driver after a victim's unlocks nothing) is refused at save
(DriverConfigSaveGate.ValidateAsync), deploy (CredentialReferenceOtherInstance,
Heidenhain included), Test Connect (TestDriverConnect.DriverInstanceId, ConfigDb
read off the mailbox) and live browse (BrowserSessionService, fail-closed without
a ConfigDb). One rule: DriverCredentialPolicy.DescribeOtherInstanceProblems +
DriverInstanceIdentities. Shared names and the driver's own id stay valid; env:
is not instance-scoped. Running drivers are unchanged (warn-only window: the
deploy gate and the upgrade preflight / /raw banner list them).

#836: OpcUaClient UserCertificatePath is a relative path inside
Drivers:OpcUaClient:UserCertificateDirectory (default
<LocalAppData>/OtOpcUa/pki/user-certificates). Syntactic rule (CredentialFilePaths,
a DriverCredentialPolicy path field) at every #815 seam; at load time
ResolveSecretRefsAsync joins, canonicalises and requires lexical AND
link-resolved containment before the PFX password is resolved, and
LoadCertificate refuses an unresolved path. A running driver's pre-#836 absolute
path keeps loading with a rate-limited warning (LegacyCredentialReferenceTracker.ReportPath).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 03:52:48 -05:00
Joseph Doherty 038aa007ac fix(hosts): driver health is per (driver, node) — ReportingNodeId on DriverHealthChanged + telemetry proto (#847) 2026-11-11 03:41:59 -05:00
Joseph Doherty a224fd6391 fix(security): scope operator-authored credential references per driver (#815)
A Designer could point a Galaxy / OpcUaClient endpoint at a host they control,
set a credential field to any reference the node can resolve
(env:ConnectionStrings__ConfigDb, env:ServerHistorian__ApiKey, secret:<any>,
Galaxy file:<any path>) and click Test Connect / live browse; the node resolved
the value and sent it to that host.

One shared rule in Core.Abstractions (CredentialReferences +
DriverCredentialPolicy) generalises the #803 Heidenhain namespace:
- Galaxy: secret:galaxy/... | secret:otopcua/galaxy/... | env:OTOPCUA_GALAXY_... |
  legacy env:GALAXY_MXGW_...; file: refused; dev:/literal API key unchanged.
- OpcUaClient: secret:opcuaclient/... | env:OTOPCUA_OPCUACLIENT_...; literals unchanged.
- Heidenhain adopts the shared parser/resolver (namespace unchanged).
Enforced at the AdminUI save path (DriverConfigSaveGate), the deploy gate
(CredentialReferenceDeployChecks, code CredentialReferenceOutOfScope), Test
Connect (AdminOperationsActor, before any probe), live browse
(BrowserSessionService, before any browser) and again at resolve time in every
driver/browser (fail closed, before any store/env read).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-11 01:28:58 -05:00
Joseph Doherty 2b45694f91 feat(heidenhain): read-only, credential and endpoint deploy gates + TagConfig inspector (#803)
- HeidenhainDeployChecks (wired into DraftValidator): HeidenhainTagWritable refuses a
  ReadWrite access level on any Heidenhain tag, so no node is ever advertised writable;
  HeidenhainCredentialPersisted refuses a literal password in DriverConfig/DeviceConfig
  (only blank / secret: / env: references; the value is never echoed);
  HeidenhainConfigInvalid for out-of-range settings or a tagged device without an endpoint.
- EquipmentTagConfigInspector maps Heidenhain to HeidenhainTagDefinitionFactory.Inspect.
- Driver validates only devices that carry tags (an unconfigured default Device1 no longer
  faults the driver).
2026-11-10 23:10:36 -05:00
Joseph Doherty 80118a01c5 fix(deploy): persist which nodes never acked on a deploy timeout (lmxopcua#784)
In the normal timeout branch (some acks landed) the deployment row was set to
TimedOut with a null FailureReason, so GET /api/deployments/{id} could not say
which node to fix; only the central log named it. The branch now persists a
reason naming the silent nodes ("Timed out waiting for acks (n/m landed). No ack
from: ..."), bounded to 20 named nodes plus a count of the rest, and still
backstopped by the 2048-char Truncate.

Tests: the reason names the silent node and not the acked one; on a 25-node
silence it names 20, counts 5 and fits the column. Mutation-checked: dropping
the SetFailureReason fails both, dropping the bound fails the second.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 21:30:17 -05:00
Joseph Doherty 43f61eba2e fix(mesh): stop reporting override-keyed clusters as REVOKED (lmxopcua#784)
The docker-dev centrals logged "Mesh PSK secret ot-mesh-psk-<cluster> ... is
absent (or blank) in the store; the key is REVOKED" for every cluster on every
contact refresh, while every gRPC dispatch and ack authenticated.

Investigated: not an auth bypass. The rig supplies each cluster's key through
MeshTransport:ClusterPsks (central side, equal to that cluster's nodes'
MeshTransport:GrpcPsk) and seeds nothing in the store. TryGetPsk/Matches consult
the override first on every call, so enforcement was on; only RefreshAsync read
the store for those clusters regardless and logged a revocation that never took
effect. RefreshAsync now skips clusters covered by a non-blank override, so the
REVOKED warning is only logged for a cluster that genuinely has no key.

Test pins it with a positive control (a cluster with no override is still read,
warned about and refused) and enforcement on the override key (wrong token
rejected). Mutation-checked: removing the skip fails the test.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 21:30:17 -05:00
Joseph Doherty 241318e7e5 feat(deploy): reject Sql Query tags naming undeclared queries; warn on Writable/writable disagreement (lmxopcua#781)
v2-ci / unit-tests (push) Has been cancelled
Item 5: DraftValidator now emits SqlQueryNotDeclared (always blocking) for a Query tag whose
query is absent from the merged driver config's queries map, via the driver's own tag parse.
Item 3: StartDeployment appends a non-blocking warning to the accepted message when the Tag
entity's AccessLevel and the Sql blob's writable flag disagree.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 18:08:54 -05:00
Joseph Doherty 919009c5ef fix(ablegacy): stamp health from the injectable Clock; drop the redundant per-project TreatWarningsAsErrors (lmxopcua#768)
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).

Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:40:07 -05:00
Joseph Doherty 5097444d56 fix(controlplane): condition the dispatch flip on the guard's RowVersion; one deadline source; align IdentityModel pins
Re-review 1: HandleDispatch's terminal guard and its AwaitingApplyAcks flip read the row separately, so a
lost-dispatch closure committed between them was silently reopened. The flip now carries the RowVersion the
guard read; a change in between is a concurrency conflict (nothing seeded, nothing published, coordinator
reset). Pinned on real SQL Server with a closure injected between the two reads (fails without the fix).

AdminOps' apply deadline and the coordinator's now both read DeployTimings.ApplyDeadline, passed explicitly
to both singletons. Microsoft.IdentityModel.Protocols(.OpenIdConnect) pinned to 8.11.0 and referenced
directly by the Security project, so the whole IdentityModel family resolves 8.11.0 (fresh restore clean
of NU190x/NU1605).

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 14:08:42 -04:00
Joseph Doherty 896aaa2b02 fix(controlplane): a PSK secret deleted from the store revokes the key; bound refresh concurrency
RefreshAsync now tells "the store answered: no such secret" (revocation - drop the refreshed key AND the
lazily cached one) apart from "the read failed" (keep the previous key). Store reads fan out through
Parallel.ForEachAsync capped at MaxConcurrentRefreshReads = 8 instead of one read per cluster at once.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:49:56 -04:00
Joseph Doherty 2adeacc655 fix(controlplane): keep the revision hash byte-identical to the pre-refactor composer
The one-snapshot refactor re-sorted every collection in memory with an ordinal comparer, while the old
composer's SQL ORDER BY used the database's case-insensitive collation; mixed-case keys therefore re-hashed
an unchanged configuration (spurious drift fleet-wide, a "new" revision on a no-op redeploy). The snapshot
factory now reads every composed table ORDER BY its natural key server-side and Compose serialises in the
order given. Pinned byte-for-byte against a verbatim replica of the old composer on the real SQL Server.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:48:40 -04:00
Joseph Doherty 9466b8b242 fix(controlplane): close a lost dispatch from the in-flight check; guard terminal rows; blob-free status writes
I7: a Dispatching row committed after the coordinator's PreStart whose Tell never arrived had no owner and
refused every later deploy. The in-flight check (now fully off the admin singleton's thread, serialised by the
actor-local compose flag alone) closes a Dispatching row with zero NodeDeploymentState rows that is past the
apply deadline as TimedOut (optimistic on RowVersion) and proceeds.

HandleDispatch ignores a dispatch for a Sealed/PartiallyFailed/TimedOut row, so a late or duplicated dispatch
cannot flip it back. Coordinator status/reason writes attach a key+RowVersion stub instead of loading the
multi-MB ArtifactBlob; proven on the real SQL Server (blob untouched, seal lands).

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:46:54 -04:00
Joseph Doherty f7e7ba0263 fix(controlplane): audit writer drops only already-persisted EventIds, never the whole SQL Server batch
SQL Server's DbUpdateException for one duplicate names every entry in the statement batch, so detaching
ex.Entries dropped the whole flush. On a row-level rejection the writer now asks the database which of the
batch's EventIds it already holds, drops exactly those, re-saves the rest, and falls back to row-by-row only
if a non-duplicate row is still refused. Fresh context + entities per attempt. Pinned against the real
central SQL Server (new SqlServerTestDatabase harness) and a faithful whole-batch interceptor.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:42:29 -04:00
Joseph Doherty f1c52385ed fix(controlplane): retention keep window counts Sealed rows; the newest Sealed blob is never pruned
The window used to count every terminal row, so 50+ failed deploys pushed the last good artifact out of it.
Now the newest KeepSealed (>= 1) Sealed rows keep their blob, failed rows are prunable unless referenced in
the ReferencedDays window. Pinned: 60 TimedOut + 1 old Sealed -> Sealed retained, 60 TimedOut emptied.

Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:39:34 -04:00
Joseph Doherty 63714e430a fix(controlplane): evict retired mesh channels and refresh PSKs off the request path
- MeshGrpcCommandDialer.RetainOnly (IMeshCommandDialer, default no-op): CentralCommunicationActor
  hands the dialer the current endpoint set on every contact refresh, so a renamed host or moved
  port releases its cached GrpcChannel and idle HTTP/2 connection instead of living for the
  process (R03:1.3).
- IMeshPskProvider.RefreshAsync (default = the old InvalidateAll): MeshPskProvider pre-resolves
  every known cluster's key concurrently, bounded like the sync path, and swaps an immutable
  snapshot in atomically; a failed read keeps that cluster's previous key. The actor calls it from
  ContactsLoaded (after SetKnownClusters, off-actor via PipeTo) instead of InvalidateAll, so the
  first ack per cluster per ContactRefreshSeconds no longer blocks a gRPC thread on a 3 s-bounded
  store read or gets denied past the bound (R03:3.4).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:34:39 -04:00
Joseph Doherty 18352442ab fix(controlplane): audit writer drops only the rejected row, not the whole batch
AuditWriterActor documented per-row dedup on the UX_ConfigAuditLog_EventId index but flushed one
statement batch: a single duplicate EventId (any at-least-once producer across a singleton
hand-over) threw for all 500 rows (R03:2.6). SaveChanges now detaches the entries a
DbUpdateException names (2601/2627 logged at Debug as the expected duplicate, anything else at
Warning) and re-saves, bounded by the batch size; a failure naming no entry stays a whole-batch
failure.

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:30:03 -04:00
Joseph Doherty 1bf5ef916c feat(controlplane): deployment artifact retention on the admin singleton
Every deploy stored the whole gzip'd fleet snapshot in Deployment.ArtifactBlob forever (R03:1.2).
DeploymentRetentionActor (admin singleton, proxy-less) sweeps every Deployments:Retention:
IntervalMinutes (60): terminal rows (Sealed / PartiallyFailed / TimedOut) older than the newest
KeepSealed (50) and not referenced by a NodeDeploymentState started/applied within ReferencedDays
(30) get their blob EMPTIED — never deleted, never NULLed: the column stays NOT NULL and every
reader already treats zero bytes as "no artifact" (#485), so no schema migration and no reader
change. In-flight rows are never touched; MaxRowsPerSweep (200) bounds one sweep.

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:27:59 -04:00
Joseph Doherty f72af76e99 fix(security): validate the API-key pepper at start, finish JWT bearer, circuit-aware audit actor
- ApiKeyOptionsValidator (ValidateOnStart on any driver/admin node): a blank or placeholder
  Security:ApiKeys:Pepper fails host start instead of rejecting every key after a green boot (R03:2.3).
- JWT is finished and opt-in by key (R03:2.4): with Security:Jwt:SigningKey set, a Bearer scheme
  (JwtBearerAuthenticationHandler over JwtTokenService.TryValidate / BuildValidationParameters, fail
  closed) accepts the tokens /auth/token mints; the default scheme is a cookie-or-bearer policy scheme
  and the fallback policy names it, so /api/* and every RequireAuthorization endpoint take a browser
  session OR a machine token. JwtOptions are validated at start (JwtOptionsValidator). Without a key,
  /auth/token is not mapped (404, not the former constructor 500) and no bearer scheme exists.
  A direct AddJwtBearer needs Microsoft.AspNetCore.Authentication.JwtBearer in
  Directory.Packages.props (not this lane's file) — the handler is a drop-in to swap.
- AuthenticationStateAuditActorAccessor is the registered IAuditActorAccessor: the Blazor circuit's
  principal first, HttpContext as the fallback (R03:2.5). HttpAuditActorAccessor shares its resolver.
- CookieAuthenticationStateProvider (unregistered, fire-and-forget ping loop) is deleted (R03:1.5/2.5).
- /auth/login logs the LDAP exception at Warning before answering 503 (R03:2.7).
- Admin operations reply with a fixed message + correlation id instead of ex.Message (R03:4.1).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:23:14 -04:00
Joseph Doherty ae84253773 fix(controlplane): keep the admin singleton's mailbox free during Test Connect and deploy compose
ReceiveAsync suspended the mailbox for the whole of every handler, so one Test Connect against a
black-holed PLC (60 s clamp) or one large deploy compose stalled every other admin operation
fleet-wide — AdminUI alarm acks/shelves, driver restarts, the next deploy.

- TestDriverConnect runs the probe on the thread pool and pipes a TestDriverConnectResult
  straight to the requester (every path yields a result, never Status.Failure).
- StartDeployment serialises only the in-flight check (DB row + an actor-local compose-in-flight
  flag, because the Deployment row is written at the END of the compose); the snapshot read,
  validation, gzip and row write run off-actor and return as DeployComposed /
  DeployComposeFailed, where the coordinator is told and the requester answered.

Tests: an alarm ack completes in < 1 s while a 60 s probe is pending and while a compose is
parked on a gated DbContext (positive controls: both still complete afterwards); a second
StartDeployment during an in-flight compose is refused.

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:41:35 -04:00
Joseph Doherty 73e74e270d fix(controlplane): validate and compose a deployment from ONE snapshot read
StartDeployment read the whole config DB twice with no transaction — DraftSnapshotFactory for
the validator, then ConfigComposer for the artifact, plus clusters/nodes and Script.SourceCode
again in between. An AdminUI save landing between the two shipped an artifact the DraftValidator
never saw.

- DraftSnapshot carries ServerClusters, ClusterNodes and ApiKeys; DraftSnapshotFactory reads
  every table inside one SNAPSHOT transaction on a relational provider (no-op on InMemory or
  under an ambient transaction).
- ConfigComposer.Compose(DraftSnapshot) is pure over the snapshot (ordinal-sorted keys);
  SnapshotAndFlattenAsync(db) delegates to it, byte-identical.
- AdminOperationsActor validates topology, estimates compile cost and composes the artifact
  from the one snapshot it holds.
- Migration EnableSnapshotIsolation: ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON
  (suppressTransaction; no model change).

Tests: the SQL Server snapshot read ignores a row committed mid-read (positive control: a later
read sees it); an edit injected between the Equipment and Tags reads is absent from the shipped
artifact and rejected by the next deploy; Compose(snapshot) == SnapshotAndFlattenAsync(db).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:36:12 -04:00
Joseph Doherty b437c34f92 fix(controlplane): recover a lost dispatch instead of wedging the deploy pipeline
A Deployment row left in Dispatching with no NodeDeploymentState rows (HandleDispatch threw
before its SaveChanges, or the process died between the AdminOps row write and the
coordinator's Tell) was recovered with an empty expected set; the deadline then hit the
0 == 0 "race" branch and returned without touching the row, so every later StartDeployment was
refused AnotherDeploymentInFlight until someone hand-edited SQL.

- PreStart: a recovered row with no expected-ack rows is a lost dispatch. Inside the apply
  deadline it is re-driven through HandleDispatch (bounded: CreatedAtUtc never moves); past it
  the row is closed TimedOut with a reason naming the condition.
- HandleDispatch is idempotent on a re-dispatch (rows already seeded ARE the expected set).
- HandleDeadline takes the race branch only when _expectedAcks.Count > 0.
- The PreStart recovery read projects ArtifactBlob away (R03:3.5).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:22:20 -04:00
Joseph Doherty 826312befe docs: fill missing XML docs and strip tracking IDs from src comments
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
2026-10-15 06:14:58 -04:00
Joseph Doherty 2659a690a6 fix(#654): review fixes - non-vacuous stop-on-publish test, Error only when retry budget refuses
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 23:39:06 -04:00
Joseph Doherty 9dc6390d50 fix(#654): bounded fast retry while the known-cluster gate is unpublished
CentralCommunicationActor: while the PSK provider is unresolved or the first
ClusterNode load failed, re-arm RefreshContacts every 2 s (max 15) until the
first publish; Information log on first publish; 'NEVER be published' Error
now only after the retry budget is exhausted. Corrected false comments.
Tests: late provider (Grpc+Dps), failed first load, retries stop, bounded budget;
ControlPlaneStarterOrderTests back to the default 60 s refresh.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 23:32:38 -04:00
Joseph Doherty 0a821417ea fix(deploy): derive the apply-outcome timeout from central's apply deadline (105 s) and name a timeout reason distinctly (#601 review)
60 s labelled a slow-but-clean large rebuild degraded. The deadline moves to a
shared Commons DeployTimings constant that ConfigPublishCoordinator reads, and
the node's timeout is deadline - 15 s margin, so the held ack still lands in
time. A timed-out ack's reason now says the outcome was not reported in time,
distinct from a real sink failure.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:54:09 -04:00
Joseph Doherty 345f40550b fix(deploy): a node with a ConfigDb records its degraded apply on its own NodeDeploymentState row (#601 review)
The apply writes the node's row Applied/null before the address-space outcome
is known; central adds the reason only when it ACCEPTS the ack, so a discarded
(stale/timed-out) ack left a fused node's row reading clean. The prefix moves
to ApplyAck.DegradedReasonPrefix so both writers spell the row identically.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:46:18 -04:00
Joseph Doherty 49dfabc141 feat(deploy): coordinator records degraded acks; deployment seals but names degraded nodes (#601)
An Applied+Degraded ack keeps NodeDeploymentState.Status = Applied with
FailureReason 'Applied degraded: <reason>' (no enum/migration change). The
deployment still seals (the revision is running; latest-sealed drives drift)
but Deployment.FailureReason names the degraded nodes and the seal logs at
Warning. Recovered from the DB on singleton failover. Old-node acks (no field,
DPS JSON or gRPC) behave exactly as before.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:32:09 -04:00
Joseph Doherty c2bdb7d5de feat(hosts): driver diagnostics channel on /hosts, S7 negotiated PDU size first (#640)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 16:57:38 -04:00
Joseph Doherty d8f3dd7394 fix(ablegacy): review fixes - one family-aware address authority in Contracts, deploy gate + editor check, malformed address -> BadNodeIdUnknown warned once, probe normalised, octal/migration docs (#244 #245 #247)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 14:38:25 -04:00
Joseph Doherty e2497c908d test(cluster): guard that role reads live only in the authority (archreview theme 4) 2026-08-20 10:21:36 -04:00
Joseph Doherty e46c1db5e5 fix(cluster): gossiped-role predicates compare case-insensitively — a miscased peer no longer un-elects the Primary 2026-08-20 10:04:16 -04:00
Joseph Doherty 9dbd5e96f7 perf(controlplane): gzip artifact blob inside the hashed bytes (~10-20x smaller)
ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest)
immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so
the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob,
the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a
FetchAndCache node verifies against. The invariant the deploy path rests on
— RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what
leaves the fetcher, the cache and the serve path untouched: nothing between
the composer and DeploymentArtifact.Open ever looks inside the blob.

Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start
with 0x1F, so no framing of our own is needed) and inflates through a size
guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can
be exercised without inflating half a gigabyte per test run. Pre-upgrade
uncompressed blobs take the identical path they always did.

Every decompression failure — corrupt stream, truncated stream, over-limit
inflation — is reported as JsonException, the same exception family the
malformed-JSON path already owns. So the lenient static wrappers still
degrade to empty results and the apply paths still fail the apply and keep
last-known-good (#485); no caller's decision changes because the reason the
bytes are unusable changed.

gzip output is not guaranteed byte-stable across .NET releases. Harmless
here: the hash is computed FROM the emitted bytes at compose time and never
re-derived from the JSON, so the worst an implementation change costs is
one spurious "changed" revision after a runtime upgrade. Snapshot
determinism is unaffected — compression wraps the final bytes and reorders
nothing.

Tests: gzip/uncompressed parity through both the instance parses and the
statics, corrupt-gzip leniency through the wrappers, the size guard (with a
falsifiability control proving the same bytes inflate fine under the
production limit), and composer-side blob-starts-with-1F8B +
RevisionHash == SHA256(blob) with a control proving the hash is NOT over
the JSON.
2026-08-15 17:00:19 -04:00
Joseph Doherty 05b424788f fix(tagconfig): close the four #551 residuals — accept-gate-first Inspect, three authoring rules, CSV/service gate, doc sweep
Gitea #551.

1. CSV import ran no driver-typed config gate. Its typed columns were always
   strictly parsed, but the free-form TagConfigJson column reached the database
   unread, so a CSV could author a tag the driver rejects — one that deploys
   green and reads BadNodeIdUnknown forever. RawTagCsvMapper.Parse now names the
   offending row in the staged review grid, and RawTreeService enforces the same
   gate on CreateTag/UpdateTag/ImportTags: a form cannot be the guarantee (the
   API and any future caller reach none of them, and this repo has no bUnit).
   A rejected import row is an ERROR — the batch inserts nothing and the row is
   named — never a skip.

   Note the brief's premise that manual entry rides ImportTagsAsync is wrong:
   RawManualTagEntryModal commits row-by-row via CreateTagAsync, which is why
   the gate had to land on all three write paths.

2/3. The seven pre-#549 Inspect methods ran a hand-list of DescribeInvalidEnum
   checks, so every rejection cause not on the list was invisible. They are now
   ACCEPT-GATE-FIRST (the MTConnect/Sql/OpcUaClient/Calculation shape): ask the
   driver's own reader for the verdict, then explain the refusal, with a
   catch-all so "the parse rejects but Inspect said nothing" is unreachable.
   That closes S7's absent dataType, AbCip members[], Modbus stringByteOrder and
   the type-dependent guards, plus causes the issue did not name (absent
   region/address/tagPath/symbolPath/topic, TwinCAT Structure, MQTT's Sparkplug
   tuple). FOCAS and MQTT keep advisories about ACCEPTED blobs, so they
   diagnose rather than return empty.

   Authoring half: ModbusTagConfigModel gains stringByteOrder;
   AbCipTagConfigModel asks the new AbCipTagDefinitionFactory.DescribeInvalidMember;
   S7TagConfigModel captures whether the SOURCE blob declared dataType and asks
   S7TagDefinitionFactory.DataTypeKey/DeclaresDataType. One authority per rule.

4. Doc sweep: the six factory Inspect summaries, EquipmentTagConfigInspector's
   class remarks and AdminOperationsActor's comment all claimed invalid enums are
   "silently defaulted by the lenient runtime" — false since R2-11 Phase C made
   the runtime reads strict. They describe rejections now.
2026-08-10 00:08:06 -04:00
Joseph Doherty b4f8cb8de7 refactor(opcuaclient): move NamespaceMap to a .Session leaf so Contracts is dependency-light
G-7 wired OpcUaClientTagDefinitionFactory.Inspect into the deploy-time TagConfig
gate, which gave ControlPlane a ProjectReference to
Driver.OpcUaClient.Contracts — and with it, transitively, the whole OPC UA client
stack (5 OPCFoundation packages) into every host that loads the control plane.

NamespaceMap was the only reason Contracts carried
OPCFoundation.NetStandard.Opc.Ua.Client: its entire public surface is
ISession/NamespaceTable/NodeId-typed. It moves to a new leaf project,
ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Session, referenced by the runtime .Driver
and by .Browser. Contracts is left with Core.Abstractions and NOTHING else —
the same shape as Calculation.Contracts.

The namespace stays ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient, so no consumer's
`using` changed. The .Session -> .Contracts direction is deliberate: the blob
read NamespaceMap needs is OpcUaClientTagDefinitionFactory.TryReadNodeId, the
one authority it shares with the deploy gate, and that must stay in the
dependency-light half.

Why a new project rather than folding NamespaceMap into the runtime .Driver and
having .Browser reference it (the Driver.Mqtt.Browser #512-style deliberate
exception): the AdminUI project-references .Browser and NOT .Driver, so that
shape would pull the OpcUaClient driver runtime — plus Core,
Opc.Ua.Configuration and Secrets.Abstractions — into the AdminUI's build/deploy
graph. That is exactly the violation #512 spent two extractions removing from
Mqtt.Browser; re-introducing it to save one csproj is a bad trade.

Guard: ContractsIsDependencyLightTests (Driver.OpcUaClient.Tests) asserts the
Contracts manifest names no Opc.Ua* assembly, plus a counterpart asserting
.Session does. An ABSENCE assertion is sound on the manifest where a presence
one is not — the compiler elides an unused reference, but it MUST emit one the
moment Opc.Ua-typed code is added. Falsified by temporarily re-adding the
package plus an Opc.Ua.NodeId-typed member: red on Opc.Ua.Types, green again on
removal.

Evidence — `dotnet list ControlPlane package --include-transitive | grep -i opc`:
  before: OPCFoundation.NetStandard.Opc.Ua.{Client,Configuration,Core,
          Security.Certificates,Types} 1.5.378.106
  after:  none (and 0 Opc.Ua*.dll in ControlPlane's output directory)
2026-08-09 21:56:29 -04:00
Joseph Doherty 1469fbb4d1 feat(deploy): wire the four G-7 inspectors into the deploy-time TagConfig gate
Adds the Sql / MTConnect / Calculation / OpcUaClient Contracts ProjectReferences to
ControlPlane and their entries to EquipmentTagConfigInspector, closing the G-7 asymmetry: the
deploy gate now checks every driver the AdminUI checks. Galaxy is the sole remaining unmapped
type and is so by design — its dotted tag_name.AttributeName reference has no typed shape, and
the AdminUI has no typed editor for it either, so the two gates agree.

Adding the ProjectReference is what makes EquipmentTagConfigInspectorCoverageTests demand a map
entry (its sweep scans the output directory, so an unreferenced Contracts assembly is invisible
to it). Falsified: deleting the Sql entry fails
Every_driver_shipping_an_Inspect_is_wired_into_the_dispatch_map with
["SqlTagDefinitionFactory"]; restored.

Tests: three cases per driver in EquipmentTagConfigInspectorTests (canonical valid config warns
nothing / present-but-invalid field draws a warning naming it / broken JSON warns without
throwing), plus a Calculation control pinning that the two shapes the runtime accepts and the
editor refuses draw nothing. OpcUaClient moved out of the unmapped-driver theory.

New TagConfigAuthoringDeployParityTests (in AdminUI.Tests, the only suite that sees both gates)
pins the direction that matters: a config the AdminUI model's own ToJson produces and Validate
accepts draws no deploy warning. Its falsifiability control — the same driver types warning
about a blob the editor refuses — caught a pre-existing, unrelated authoring-side gap:
MTConnectTagConfigModel reads dataType leniently, so a misspelled name loads as String and
saves clean while the strict runtime rejects it. Left alone (rewriting the editors is out of
scope); the control uses the digits-only shape both gates refuse, and the case is documented
inline.
2026-08-09 21:14:49 -04:00
Joseph Doherty b5607d355a refactor(fleet): #542 review residue — drop dead proxyOptions local, retire stale compliance defer line
The proxyOptions local in WithOtOpcUaControlPlaneSingletons was already unused on master
(pre-existing dead code, not orphaned by the deletion commit — every registration passes
singletonOptions); removed while the file is open. The phase-6-1 compliance script still
Assert-Deferred'd the FleetStatusHub /hosts follow-up — an item that is now deleted, not
deferred — replaced with a retirement comment.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:28:11 -04:00
Joseph Doherty f1c188114b refactor(fleet): #542 delete the dead fleet-status chain (broadcaster, SignalR bridge, hub, messages)
#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh
split: it derives "the fleet" from central's own Akka gossip ring, which after the split is
just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole
chain is dead end to end, so this deletes it rather than reviving it.

  - Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS
    "fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub
    ("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting
    HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads
    NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead.
  - Its second input never existed. DriverHostStatusHeartbeat had no production sender —
    only the broadcaster's own test told it — so CurrentRevision was always null and the
    30 s staleness -> Degraded flip could never fire.
  - Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an
    accident of topology rather than a design.

The fleet visibility that actually works is unchanged and already cross-mesh-correct:
/fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials.

Precedent for deleting inert machinery rather than activating it: #522 (driver stability
tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint).

Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin
singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its
key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus /
FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps
ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed
a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels,
Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are
left as the historical record they are.

Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline.
ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:21:30 -04:00
Joseph Doherty 4d2dfb5b50 test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:

  1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
     point, kept as a test rather than a live-gate note: each cluster stays single-member and
     none ever holds another's address as a member.
  2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
     site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
     would be refused at the far side.
  3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
     it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
     process did". The same node then accepts the right key, so the silence is the auth gate
     and not a dead endpoint.

All three run fully offline — no Docker fixture — and pass.

Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.

The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
2026-08-06 15:58:36 -04:00
Joseph Doherty 5d7ecdf737 refactor(mesh): step 3 — delete both ClusterClientReceptionist registrations
Central's CentralCommunicationActor and a node's NodeCommunicationActor no longer register
with a receptionist; the ActorRegistry entry each already carried is now the whole
reachability story, and it is what CentralCommandGrpcService / NodeCommandGrpcService
resolve to hand an inbound message over. Both `using Akka.Cluster.Tools.Client` directives
are gone with them.
2026-08-06 15:41:25 -04:00
Joseph Doherty 0db40a5e7f refactor(mesh): step 1 — delete the central ClusterClient factory and per-cluster client cache
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).

Ripples fixed in this step because the build must be green at every commit:
  - ControlPlane/ServiceCollectionExtensions drops the factory argument.
  - CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
    Grpc counterpart that already asserted the same property independently), the three
    transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
  - MeshClusterClientFactoryTests deleted (its subject is gone).
  - MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
    DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
    replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
    (two separate meshes, no cross-mesh Akka association).
2026-08-06 15:35:54 -04:00
Joseph Doherty d7c9ead64e test(mesh): close review findings on the PreStart-race logging fix — non-vacuous once-per-incarnation pin, broken-dialer symmetry, comment idiom
Follow-up to a94eb8a3, addressing CHANGES REQUESTED on code review. Production
logic was confirmed correct on all spec points; this commit fixes the review's
test/comment findings only.

IMPORTANT-1: A_permanently_unresolvable_pskProvider_is_reported_once_per_incarnation_not_once_per_refresh
did not prove the second refresh ran — its Expect(0) window gated on a dialer
dispatch landing, but LoadContactsFromDb is fire-and-forget and the first
block's AwaitCondition already re-Tells MeshCommand per poll, so the dialer
call count could reach the threshold before PublishKnownClusters was ever
re-invoked by the second load. Fixed by having the PSK factory count its own
invocations and gating both AwaitConditions on that count directly (pskCalls
>= 2 proves the first retry; pskCalls > pskCallsAfterFirstLoad, awaited INSIDE
the Expect(0) window, proves the second one). Bite-checked: with
_pskProviderAbsenceReported's guard temporarily removed, the test now
correctly fails ("Received 1 message too many... received 1 that matched
filter"); guard restored and reverified green.

MINOR-2: reworded four comments (CentralCommunicationActor.cs ~109, ~259,
~305-309, ~853) from narrating the change ("this fix removes", "already
existed, unchanged", "added by this fix", "the old two-call-site shape") to
stating the timeless invariant.

MINOR-3: added A_supplied_dialer_factory_that_always_throws_still_drops_each_command_loudly_while_PreStart_stays_quiet
— the symmetric case to the existing never-supplied-dialer test, exercising
Resolve's catch + firstUse:true path (a factory that WAS supplied and always
throws) rather than the null-factory path. Asserts PreStart plus the first
contact load stay silent and DispatchOverGrpc's existing per-dispatch Error
still fires on every dispatch, unguarded.

MINOR-4: "Check the ControlPlane registration wires IMeshPskProvider" ->
"Check that the ControlPlane registration wires up IMeshPskProvider"; no test
pinned the old substring, so no test update was needed for this one.

Skipped item 5 (restart-re-arms pin) per instruction — out of scope.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 10:30:20 -04:00