- MeshPskProviderRefreshTests: GatedResolver.Entered was set by the priming
refresh and never reset, so Wait() returned before the held refresh reached
the store and the mid-refresh call-count snapshot raced its increment
(expected 1, was 2). Hold() now resets it.
- FailedCredentialLockoutTests all-bad burst: the 15 waiters could queue
before a slot holder ran the increment after BeginAsync (authenticated 0
vs 5). The wait now covers both conditions; the security assertions are
unchanged.
Both looped 20x locally, 0 failures.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 IsPoison classified every SQL Server transient as poison:
Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
(1205), timeout (-2 / Win32 258), dropped connection or AG failover was
counted from memory and the node's row left Applying. Poison is now an
ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
retries, bounded by the deadline deferral cap. IsPoison is public for tests.
Acks counted only from memory (poison branch and the deferral cap) are kept
in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
same-revision dispatch re-applies instead of short-circuiting a clean
Applied over an empty address space: TryRecoverFromStale (restores only the
revision marker) and RestoreApplied when the artifact read fails (null
blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
reach the ConfigDb acks a same-revision re-dispatch Failed
(PartiallyFailed, not Sealed); Stale recovery and failed restores are
unconfirmed too.
Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.
Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review findings on 02503a4a (ConfigPublishCoordinator):
1 (HIGH) A recovered deployment whose every expected ack was already in
NodeDeploymentState (a seal whose SaveChanges failed, a fused node's own row)
was never completed: inside the deadline it waited for acks that had landed,
past it HandleDeadline's "race" guard returned without sealing — stuck in
AwaitingApplyAcks, refusing every later deploy. ApplyRecovery now completes
it (Seal / PartiallyFailed), and HandleDeadline completes a full ack set
instead of returning.
2 (MED-HIGH) The past-deadline recovery of a populated expected set ran
HandleDeadline inline before the stash replay, discarding on-time acks queued
during the outage. It is now self-sent, behind the replay. The zero-rows
past-deadline branch stays inline on purpose: no ack can count there, and a
self-sent deadline would time out a queued dispatch of that same row right
after publishing it.
3 (MED) A failed PersistNodeAck threw, restarted the actor and dropped the ack
(driver-only nodes never write their own row and do not resend). Acks are now
persisted before being counted, and a failed persist is re-queued on a timer.
The class remarks document that the Loading stash is node-local (lost on a
hand-over; queued dispatches are recovered from their rows).
Minor: MarkReady/Become(Ready) only after the recovery applied; a throw while
applying is a failed load (backoff) rather than a tight restart loop; a queued
dispatch matching the recovered Dispatching row logs Info, not the "lost
dispatch" Warning.
Tests (ConfigPublishCoordinatorDbOutageTests): fully-acked recovery seals /
partially fails inside and past the deadline (4 cases); queued ack applied
before a past-deadline recovery; failed ack persist retried. All 6 fail on
02503a4a.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PreStart ran the in-flight recovery query synchronously. With the ConfigDb
unreachable when the singleton started, the throw became an
ActorInitializationException, the default strategy stopped the actor, the
ClusterSingletonManager never re-created it, and every later deploy hung.
- PreStart now only subscribes and kicks off an async recovery load piped back
to the actor. A failure retries with exponential backoff (1 s doubling to
30 s, forever) and never stops the actor.
- Until the load succeeds the coordinator is in a Loading behaviour that
stashes every message (DispatchDeployment logged as queued, ApplyAck too);
they are replayed in order after the recovered state is applied. Recovery
decisions (re-dispatch / time out) now run inline before the replay instead
of via self-tells, and a duplicate dispatch of the in-flight deployment with
a seeded expected set is ignored so queued acks are never reset.
- New ConfigPublishCoordinatorReadiness (admin-node DI singleton) +
"config-publish" ready-tier health check: Degraded while loading (with the
attempt count and last error), Healthy when ready or not hosted here.
Tests: ConfigPublishCoordinatorDbOutageTests (DB down at start -> actor alive,
Loading reported, deploy queued, DB returns -> load succeeds, queued deploy
dispatched and sealed; queued ack applied to the recovered in-flight set;
NotHosted after stop) and ConfigPublishCoordinatorHealthCheckTests.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
env:<PREFIX><INSTANCE>_... is the per-instance convention for every prefix of a
driver's scope (OTOPCUA_GALAXY_, GALAXY_MXGW_, OTOPCUA_OPCUACLIENT_,
OTOPCUA_HEIDENHAIN_), INSTANCE = a driver id or name upper-cased with every other
character as '_'. A variable whose part after the prefix equals or starts with
(at a '_' boundary) ANOTHER same-type instance's id or name is refused with
CredentialReferenceOtherInstance at save, deploy, Test Connect and browse - the
same DriverCredentialPolicy.DescribeOtherInstanceProblems the secret: rule uses
(now over CollectInstanceReferences / ReferenceNames). Another-instance matches
win over the owner's, as for secrets. Shared variables that name no instance
(GALAXY_MXGW_API_KEY) keep working. HasInstanceSegments now also gates on env:,
so an env:-only browse needs the ConfigDb (fail-closed without it).
Co-Authored-By: Claude <noreply@anthropic.com>
A frozen pre-#847 copy of DriverHealth (test-only telemetry_pre847.proto, generated via
Grpc.Tools in the test project) stands in for an old node / old central: new->new round-trips
the node id; old->new parses as absent (HasReportingNodeId false -> null, not "") with every
other field intact; new->old ignores field 15 without shifting known fields and preserves it as
an unknown field.
I-1: Test Connect / browse configs may carry unsaved edits (the device modal merges
the stored DriverConfig with the typed device config). The server
(DriverInstanceIdentities.EffectiveOwnerAsync, used by AdminOperationsActor and
BrowserSessionService) honours the named owner only when every endpoint value
(new DriverCredentialRules.EndpointFields: Galaxy Endpoint, OpcUaClient
EndpointUrl/EndpointUrls/DiscoveryUrl, Heidenhain Host/Port) is one that driver
stores; otherwise the owner is null and its instance-named secrets are refused.
Every Test Connect / browse whose config carries an in-scope secret:/env: reference
is recorded (CredentialUseAudit: IAuditWriter event category CredentialUse + a
structured Information line: user, driver, endpoint hosts, outcome).
L-1: the UserCertificatePath rule is skipped only in an object that explicitly sets
AuthType to non-Certificate (string or ordinal); the deploy gate also checks the
runtime-merged driver+device config so the switch cannot be split from the path; the
form no longer persists the hidden field.
L-2: a ConfigDb fault in the browse check is a refusal; RawBrowseModal catches open
failures into its error message.
L-4: a relative UserCertificateDirectory resolves against AppContext.BaseDirectory.
Co-Authored-By: Claude <noreply@anthropic.com>
#835: inside one driver namespace, a secret:<ns>/<X>/... reference whose first
segment names ANOTHER existing same-type driver instance (id or name,
case-insensitive; another-instance match beats an own-name match, so renaming
one's driver after a victim's unlocks nothing) is refused at save
(DriverConfigSaveGate.ValidateAsync), deploy (CredentialReferenceOtherInstance,
Heidenhain included), Test Connect (TestDriverConnect.DriverInstanceId, ConfigDb
read off the mailbox) and live browse (BrowserSessionService, fail-closed without
a ConfigDb). One rule: DriverCredentialPolicy.DescribeOtherInstanceProblems +
DriverInstanceIdentities. Shared names and the driver's own id stay valid; env:
is not instance-scoped. Running drivers are unchanged (warn-only window: the
deploy gate and the upgrade preflight / /raw banner list them).
#836: OpcUaClient UserCertificatePath is a relative path inside
Drivers:OpcUaClient:UserCertificateDirectory (default
<LocalAppData>/OtOpcUa/pki/user-certificates). Syntactic rule (CredentialFilePaths,
a DriverCredentialPolicy path field) at every #815 seam; at load time
ResolveSecretRefsAsync joins, canonicalises and requires lexical AND
link-resolved containment before the PFX password is resolved, and
LoadCertificate refuses an unresolved path. A running driver's pre-#836 absolute
path keeps loading with a rate-limited warning (LegacyCredentialReferenceTracker.ReportPath).
Co-Authored-By: Claude <noreply@anthropic.com>
User decision: save, Test Connect, live browse and the deploy gate keep
refusing out-of-namespace credential references, but an ALREADY-DEPLOYED
driver must not fault at the upgrade restart or (Galaxy) at the next
gateway reconnect, since ReopenAsync re-resolves Gateway.ApiKeySecretRef.
- LegacyCredentialReferenceTracker (Core.Abstractions): the one, deprecated,
call-site-chosen compatibility path. Running Galaxy (InitializeAsync,
ReopenAsync, ack fallback) and OpcUaClient (session open) pass it and keep
the pre-#815 semantics (case-insensitive prefixes, Galaxy file:, OpcUaClient
literal pass-through) with a rate-limited Warning (driver/field/allowed
form, never the value) and a credential-compat:<field> diagnostics entry
(both drivers now implement IDriverDiagnostics). Browsers and probes pass
none and stay hard-fail; Heidenhain has none.
- Deploy-gate message names the migration.
- Upgrade preflight: CredentialReferencePreflightService (admin nodes, once
after ApplicationStarted, ConfigDb only, resolves nothing) logs a Warning
per Draft row / Deployed-artifact finding; the /raw banner shows them plus
Running drivers reporting warn-only use (driver-health diagnostics).
- docs/security.md: exact replacement table (incl. the old form default
env:MX_API_KEY, file:, un-namespaced secret:, casing changes), the
ReopenAsync reconnect behaviour, the warn-only window and its removal.
Co-Authored-By: Claude <noreply@anthropic.com>
A Designer could point a Galaxy / OpcUaClient endpoint at a host they control,
set a credential field to any reference the node can resolve
(env:ConnectionStrings__ConfigDb, env:ServerHistorian__ApiKey, secret:<any>,
Galaxy file:<any path>) and click Test Connect / live browse; the node resolved
the value and sent it to that host.
One shared rule in Core.Abstractions (CredentialReferences +
DriverCredentialPolicy) generalises the #803 Heidenhain namespace:
- Galaxy: secret:galaxy/... | secret:otopcua/galaxy/... | env:OTOPCUA_GALAXY_... |
legacy env:GALAXY_MXGW_...; file: refused; dev:/literal API key unchanged.
- OpcUaClient: secret:opcuaclient/... | env:OTOPCUA_OPCUACLIENT_...; literals unchanged.
- Heidenhain adopts the shared parser/resolver (namespace unchanged).
Enforced at the AdminUI save path (DriverConfigSaveGate), the deploy gate
(CredentialReferenceDeployChecks, code CredentialReferenceOutOfScope), Test
Connect (AdminOperationsActor, before any probe), live browse
(BrowserSessionService, before any browser) and again at resolve time in every
driver/browser (fail closed, before any store/env read).
Co-Authored-By: Claude <noreply@anthropic.com>
Central pin 1.1.0 -> 3.2.2; AdminUI/ControlPlane/Runtime/Host.Tests no longer carry VersionOverride.
The three driver IntegrationTests fixtures' Fact/Theory attributes gain the source-info ctor (xUnit3003).
No xUnit1051 fallout remained (already cleaned). Test counts unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Akka.TestKit.Xunit2 -> Akka.TestKit.Xunit 1.5.62 (xunit.v3 TestKit; needs xunit.v3 3.2.2, set via
VersionOverride on the four projects). Removes the xunit 2.9.3 pin. IAsyncLifetime returns ValueTask,
SqlServerFactAttribute gains the source-info ctor, and xUnit1051 call sites pass
TestContext.Current.CancellationToken. Test counts unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
In the normal timeout branch (some acks landed) the deployment row was set to
TimedOut with a null FailureReason, so GET /api/deployments/{id} could not say
which node to fix; only the central log named it. The branch now persists a
reason naming the silent nodes ("Timed out waiting for acks (n/m landed). No ack
from: ..."), bounded to 20 named nodes plus a count of the rest, and still
backstopped by the 2048-char Truncate.
Tests: the reason names the silent node and not the acked one; on a 25-node
silence it names 20, counts 5 and fits the column. Mutation-checked: dropping
the SetFailureReason fails both, dropping the bound fails the second.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The docker-dev centrals logged "Mesh PSK secret ot-mesh-psk-<cluster> ... is
absent (or blank) in the store; the key is REVOKED" for every cluster on every
contact refresh, while every gRPC dispatch and ack authenticated.
Investigated: not an auth bypass. The rig supplies each cluster's key through
MeshTransport:ClusterPsks (central side, equal to that cluster's nodes'
MeshTransport:GrpcPsk) and seeds nothing in the store. TryGetPsk/Matches consult
the override first on every call, so enforcement was on; only RefreshAsync read
the store for those clusters regardless and logged a revocation that never took
effect. RefreshAsync now skips clusters covered by a non-blank override, so the
REVOKED warning is only logged for a cluster that genuinely has no key.
Test pins it with a positive control (a cluster with no override is still read,
warned about and refused) and enforcement on the override key (wrong token
rejected). Mutation-checked: removing the skip fails the test.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 5: DraftValidator now emits SqlQueryNotDeclared (always blocking) for a Query tag whose
query is absent from the merged driver config's queries map, via the driver's own tag parse.
Item 3: StartDeployment appends a non-blocking warning to the accepted message when the Tag
entity's AccessLevel and the Sql blob's writable flag disagree.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Akka TestKit's AwaitCondition without an interval polls every max/10 of the
DILATED budget. Measured here: a condition that turned true at 2 s under a
15 s budget returned after 3003 ms at timefactor 1 (3 polls) and 4502 ms at
timefactor 3 (2 polls). AwaitAssert / AwaitAssertAsync are unaffected (a
fixed 100 ms: 21 polls in ~2 s at both factors).
RuntimeActorTestBase and ControlPlaneActorTestBase now declare
AwaitCondition overloads that hide TestKit's and default the interval to
PollInterval (20 ms), so every subclass call without an interval binds to
them (measured after: returns 5-22 ms after the condition holds, at both
factors). PresenceBudget's doc no longer claims the helpers return "the
instant the condition holds" - it states the per-helper poll behaviour.
Bare-TestKit classes with AwaitCondition calls get an explicit interval:
TelemetryDialDeadlineTests / TelemetryDialSupervisorTests (next to their own
undilated 5 s / 8 s WaitForCount deadlines and a 1 s dial deadline) and
OpcUaEndpointStepDownActorTests (RuntimeActorTestBase.PollInterval, now
public like PresenceBudget). ContinuousHistorizationRecorderTypedValueTests'
AwaitConditionAsync was left alone (historian code is being changed in
parallel).
Suite results (Release, 0 warnings):
- Runtime.Tests 946/3 skipped/0 failed at timefactor 3 (1 m 45 s, was
2 m 57 s) and 1 (1 m 18 s, was 2 m 57 s);
- ControlPlane.Tests 319/0 failed at 3 (1 m 51 s) and 1 (1 m 11 s);
- AdminUI.Tests Telemetry 15/0 failed at 3 and 1.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).
Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Re-review 1: HandleDispatch's terminal guard and its AwaitingApplyAcks flip read the row separately, so a
lost-dispatch closure committed between them was silently reopened. The flip now carries the RowVersion the
guard read; a change in between is a concurrency conflict (nothing seeded, nothing published, coordinator
reset). Pinned on real SQL Server with a closure injected between the two reads (fails without the fix).
AdminOps' apply deadline and the coordinator's now both read DeployTimings.ApplyDeadline, passed explicitly
to both singletons. Microsoft.IdentityModel.Protocols(.OpenIdConnect) pinned to 8.11.0 and referenced
directly by the Security project, so the whole IdentityModel family resolves 8.11.0 (fresh restore clean
of NU190x/NU1605).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
RefreshAsync now tells "the store answered: no such secret" (revocation - drop the refreshed key AND the
lazily cached one) apart from "the read failed" (keep the previous key). Store reads fan out through
Parallel.ForEachAsync capped at MaxConcurrentRefreshReads = 8 instead of one read per cluster at once.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The one-snapshot refactor re-sorted every collection in memory with an ordinal comparer, while the old
composer's SQL ORDER BY used the database's case-insensitive collation; mixed-case keys therefore re-hashed
an unchanged configuration (spurious drift fleet-wide, a "new" revision on a no-op redeploy). The snapshot
factory now reads every composed table ORDER BY its natural key server-side and Compose serialises in the
order given. Pinned byte-for-byte against a verbatim replica of the old composer on the real SQL Server.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
I7: a Dispatching row committed after the coordinator's PreStart whose Tell never arrived had no owner and
refused every later deploy. The in-flight check (now fully off the admin singleton's thread, serialised by the
actor-local compose flag alone) closes a Dispatching row with zero NodeDeploymentState rows that is past the
apply deadline as TimedOut (optimistic on RowVersion) and proceeds.
HandleDispatch ignores a dispatch for a Sealed/PartiallyFailed/TimedOut row, so a late or duplicated dispatch
cannot flip it back. Coordinator status/reason writes attach a key+RowVersion stub instead of loading the
multi-MB ArtifactBlob; proven on the real SQL Server (blob untouched, seal lands).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
SQL Server's DbUpdateException for one duplicate names every entry in the statement batch, so detaching
ex.Entries dropped the whole flush. On a row-level rejection the writer now asks the database which of the
batch's EventIds it already holds, drops exactly those, re-saves the rest, and falls back to row-by-row only
if a non-duplicate row is still refused. Fresh context + entities per attempt. Pinned against the real
central SQL Server (new SqlServerTestDatabase harness) and a faithful whole-batch interceptor.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The window used to count every terminal row, so 50+ failed deploys pushed the last good artifact out of it.
Now the newest KeepSealed (>= 1) Sealed rows keep their blob, failed rows are prunable unless referenced in
the ReferencedDays window. Pinned: 60 TimedOut + 1 old Sealed -> Sealed retained, 60 TimedOut emptied.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- MeshGrpcCommandDialer.RetainOnly (IMeshCommandDialer, default no-op): CentralCommunicationActor
hands the dialer the current endpoint set on every contact refresh, so a renamed host or moved
port releases its cached GrpcChannel and idle HTTP/2 connection instead of living for the
process (R03:1.3).
- IMeshPskProvider.RefreshAsync (default = the old InvalidateAll): MeshPskProvider pre-resolves
every known cluster's key concurrently, bounded like the sync path, and swaps an immutable
snapshot in atomically; a failed read keeps that cluster's previous key. The actor calls it from
ContactsLoaded (after SetKnownClusters, off-actor via PipeTo) instead of InvalidateAll, so the
first ack per cluster per ContactRefreshSeconds no longer blocks a gRPC thread on a 3 s-bounded
store read or gets denied past the bound (R03:3.4).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
AuditWriterActor documented per-row dedup on the UX_ConfigAuditLog_EventId index but flushed one
statement batch: a single duplicate EventId (any at-least-once producer across a singleton
hand-over) threw for all 500 rows (R03:2.6). SaveChanges now detaches the entries a
DbUpdateException names (2601/2627 logged at Debug as the expected duplicate, anything else at
Warning) and re-saves, bounded by the batch size; a failure naming no entry stays a whole-batch
failure.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Every deploy stored the whole gzip'd fleet snapshot in Deployment.ArtifactBlob forever (R03:1.2).
DeploymentRetentionActor (admin singleton, proxy-less) sweeps every Deployments:Retention:
IntervalMinutes (60): terminal rows (Sealed / PartiallyFailed / TimedOut) older than the newest
KeepSealed (50) and not referenced by a NodeDeploymentState started/applied within ReferencedDays
(30) get their blob EMPTIED — never deleted, never NULLed: the column stays NOT NULL and every
reader already treats zero bytes as "no artifact" (#485), so no schema migration and no reader
change. In-flight rows are never touched; MaxRowsPerSweep (200) bounds one sweep.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- ApiKeyOptionsValidator (ValidateOnStart on any driver/admin node): a blank or placeholder
Security:ApiKeys:Pepper fails host start instead of rejecting every key after a green boot (R03:2.3).
- JWT is finished and opt-in by key (R03:2.4): with Security:Jwt:SigningKey set, a Bearer scheme
(JwtBearerAuthenticationHandler over JwtTokenService.TryValidate / BuildValidationParameters, fail
closed) accepts the tokens /auth/token mints; the default scheme is a cookie-or-bearer policy scheme
and the fallback policy names it, so /api/* and every RequireAuthorization endpoint take a browser
session OR a machine token. JwtOptions are validated at start (JwtOptionsValidator). Without a key,
/auth/token is not mapped (404, not the former constructor 500) and no bearer scheme exists.
A direct AddJwtBearer needs Microsoft.AspNetCore.Authentication.JwtBearer in
Directory.Packages.props (not this lane's file) — the handler is a drop-in to swap.
- AuthenticationStateAuditActorAccessor is the registered IAuditActorAccessor: the Blazor circuit's
principal first, HttpContext as the fallback (R03:2.5). HttpAuditActorAccessor shares its resolver.
- CookieAuthenticationStateProvider (unregistered, fire-and-forget ping loop) is deleted (R03:1.5/2.5).
- /auth/login logs the LDAP exception at Warning before answering 503 (R03:2.7).
- Admin operations reply with a fixed message + correlation id instead of ex.Message (R03:4.1).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
ReceiveAsync suspended the mailbox for the whole of every handler, so one Test Connect against a
black-holed PLC (60 s clamp) or one large deploy compose stalled every other admin operation
fleet-wide — AdminUI alarm acks/shelves, driver restarts, the next deploy.
- TestDriverConnect runs the probe on the thread pool and pipes a TestDriverConnectResult
straight to the requester (every path yields a result, never Status.Failure).
- StartDeployment serialises only the in-flight check (DB row + an actor-local compose-in-flight
flag, because the Deployment row is written at the END of the compose); the snapshot read,
validation, gzip and row write run off-actor and return as DeployComposed /
DeployComposeFailed, where the coordinator is told and the requester answered.
Tests: an alarm ack completes in < 1 s while a 60 s probe is pending and while a compose is
parked on a gated DbContext (positive controls: both still complete afterwards); a second
StartDeployment during an in-flight compose is refused.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
StartDeployment read the whole config DB twice with no transaction — DraftSnapshotFactory for
the validator, then ConfigComposer for the artifact, plus clusters/nodes and Script.SourceCode
again in between. An AdminUI save landing between the two shipped an artifact the DraftValidator
never saw.
- DraftSnapshot carries ServerClusters, ClusterNodes and ApiKeys; DraftSnapshotFactory reads
every table inside one SNAPSHOT transaction on a relational provider (no-op on InMemory or
under an ambient transaction).
- ConfigComposer.Compose(DraftSnapshot) is pure over the snapshot (ordinal-sorted keys);
SnapshotAndFlattenAsync(db) delegates to it, byte-identical.
- AdminOperationsActor validates topology, estimates compile cost and composes the artifact
from the one snapshot it holds.
- Migration EnableSnapshotIsolation: ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON
(suppressTransaction; no model change).
Tests: the SQL Server snapshot read ignores a row committed mid-read (positive control: a later
read sees it); an edit injected between the Equipment and Tags reads is absent from the shipped
artifact and rejected by the next deploy; Compose(snapshot) == SnapshotAndFlattenAsync(db).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
A Deployment row left in Dispatching with no NodeDeploymentState rows (HandleDispatch threw
before its SaveChanges, or the process died between the AdminOps row write and the
coordinator's Tell) was recovered with an empty expected set; the deadline then hit the
0 == 0 "race" branch and returned without touching the row, so every later StartDeployment was
refused AnotherDeploymentInFlight until someone hand-edited SQL.
- PreStart: a recovered row with no expected-ack rows is a lost dispatch. Inside the apply
deadline it is re-driven through HandleDispatch (bounded: CreatedAtUtc never moves); past it
the row is closed TimedOut with a reason naming the condition.
- HandleDispatch is idempotent on a re-dispatch (rows already seeded ARE the expected set).
- HandleDeadline takes the race branch only when _expectedAcks.Count > 0.
- The PreStart recovery read projects ArtifactBlob away (R03:3.5).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
CentralCommunicationActor: while the PSK provider is unresolved or the first
ClusterNode load failed, re-arm RefreshContacts every 2 s (max 15) until the
first publish; Information log on first publish; 'NEVER be published' Error
now only after the retry budget is exhausted. Corrected false comments.
Tests: late provider (Grpc+Dps), failed first load, retries stop, bounded budget;
ControlPlaneStarterOrderTests back to the default 60 s refresh.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and
v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3
on the test step.
- Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR
(TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget.
- Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests
RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process
(abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s.
- Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start
state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes
host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while
Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick).
- Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps
(wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log.
- Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on
Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped.
- infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
An Applied+Degraded ack keeps NodeDeploymentState.Status = Applied with
FailureReason 'Applied degraded: <reason>' (no enum/migration change). The
deployment still seals (the revision is running; latest-sealed drives drift)
but Deployment.FailureReason names the degraded nodes and the seal logs at
Warning. Recovered from the DB on singleton failover. Old-node acks (no field,
DPS JSON or gRPC) behave exactly as before.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest)
immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so
the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob,
the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a
FetchAndCache node verifies against. The invariant the deploy path rests on
— RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what
leaves the fetcher, the cache and the serve path untouched: nothing between
the composer and DeploymentArtifact.Open ever looks inside the blob.
Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start
with 0x1F, so no framing of our own is needed) and inflates through a size
guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can
be exercised without inflating half a gigabyte per test run. Pre-upgrade
uncompressed blobs take the identical path they always did.
Every decompression failure — corrupt stream, truncated stream, over-limit
inflation — is reported as JsonException, the same exception family the
malformed-JSON path already owns. So the lenient static wrappers still
degrade to empty results and the apply paths still fail the apply and keep
last-known-good (#485); no caller's decision changes because the reason the
bytes are unusable changed.
gzip output is not guaranteed byte-stable across .NET releases. Harmless
here: the hash is computed FROM the emitted bytes at compose time and never
re-derived from the JSON, so the worst an implementation change costs is
one spurious "changed" revision after a runtime upgrade. Snapshot
determinism is unaffected — compression wraps the final bytes and reorders
nothing.
Tests: gzip/uncompressed parity through both the instance parses and the
statics, corrupt-gzip leniency through the wrappers, the size guard (with a
falsifiability control proving the same bytes inflate fine under the
production limit), and composer-side blob-starts-with-1F8B +
RevisionHash == SHA256(blob) with a control proving the hash is NOT over
the JSON.
MTConnectTagConfigModel read `dataType` with TagConfigJson.GetEnum, which falls
back to String on ANY unrecognised value. A misspelled name ("Strng") from a
hand-authored blob, a CSV import or the API was therefore accepted at authoring
and silently rewritten, while the driver's strict parse rejected it and the tag
died with BadNodeIdUnknown. Found by the G-7 parity tests' falsifiability
control, which had to use the digits-only shape to have a case that failed.
The rule is not restated in the AdminUI. MTConnectTagDefinitionFactory exposes
its parse as `IsAcceptedDataTypeName`, documented as the one authority the
driver, the deploy gate and the authoring model all consult; the model's
Validate() asks it about the raw value the blob carries. The old digits-only
message is NOT retained as a special case: the runtime draws no distinction
between a bare number and a typo — one predicate, one Inspect warning — so a
second message here would be a second statement of a rule this model does not
own. The raw value is captured as the JSON TOKEN, because the runtime
deserialises the key into a `string?` DTO field and an unquoted number throws
there too.
The survey found the same asymmetry at six more editors, contrary to the
assumption that those runtimes silently default: R2-11 Phase C moved every
driver's TagConfig enum read to TagConfigJson.TryReadEnumStrict, which REJECTS
the tag on a present-but-invalid string, while the editors kept loading them
with a silent fallback. Modbus (region/dataType/byteOrder), S7, AbCip, AbLegacy,
TwinCAT and FOCAS now check their enums in Validate() through the runtime's own
TagConfigJson.DescribeInvalidEnumText — added alongside a public
ClassifyEnumText that TryReadEnum now delegates to, so the JSON-token door and
the string door reach one implementation. Mqtt and Sql already carried
equivalent strict checks and are untouched.
Care taken NOT to become stricter than the runtime: a present-but-non-string
token stays accepted for those six (TryReadEnum calls it absent), as does an
absent key, each pinned by a test.
Also adds a direct ProjectReference to Core.Abstractions from the AdminUI — the
save gate now leans on a type it was consuming only transitively.
Adds the Sql / MTConnect / Calculation / OpcUaClient Contracts ProjectReferences to
ControlPlane and their entries to EquipmentTagConfigInspector, closing the G-7 asymmetry: the
deploy gate now checks every driver the AdminUI checks. Galaxy is the sole remaining unmapped
type and is so by design — its dotted tag_name.AttributeName reference has no typed shape, and
the AdminUI has no typed editor for it either, so the two gates agree.
Adding the ProjectReference is what makes EquipmentTagConfigInspectorCoverageTests demand a map
entry (its sweep scans the output directory, so an unreferenced Contracts assembly is invisible
to it). Falsified: deleting the Sql entry fails
Every_driver_shipping_an_Inspect_is_wired_into_the_dispatch_map with
["SqlTagDefinitionFactory"]; restored.
Tests: three cases per driver in EquipmentTagConfigInspectorTests (canonical valid config warns
nothing / present-but-invalid field draws a warning naming it / broken JSON warns without
throwing), plus a Calculation control pinning that the two shapes the runtime accepts and the
editor refuses draw nothing. OpcUaClient moved out of the unmapped-driver theory.
New TagConfigAuthoringDeployParityTests (in AdminUI.Tests, the only suite that sees both gates)
pins the direction that matters: a config the AdminUI model's own ToJson produces and Validate
accepts draws no deploy warning. Its falsifiability control — the same driver types warning
about a blob the editor refuses — caught a pre-existing, unrelated authoring-side gap:
MTConnectTagConfigModel reads dataType leniently, so a misspelled name loads as String and
saves clean while the strict runtime rejects it. Left alone (rewriting the editors is out of
scope); the control uses the digits-only shape both gates refuse, and the case is documented
inline.
#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh
split: it derives "the fleet" from central's own Akka gossip ring, which after the split is
just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole
chain is dead end to end, so this deletes it rather than reviving it.
- Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS
"fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub
("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting
HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads
NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead.
- Its second input never existed. DriverHostStatusHeartbeat had no production sender —
only the broadcaster's own test told it — so CurrentRevision was always null and the
30 s staleness -> Degraded flip could never fire.
- Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an
accident of topology rather than a design.
The fleet visibility that actually works is unchanged and already cross-mesh-correct:
/fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials.
Precedent for deleting inert machinery rather than activating it: #522 (driver stability
tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint).
Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin
singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its
key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus /
FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps
ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed
a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels,
Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are
left as the historical record they are.
Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline.
ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
DeadlineElapsed_for_current_deployment_marks_TimedOut carried two races,
both test-side. ConfigPublishCoordinator itself is untouched.
(a) The NodeDeploymentState row was inserted AFTER Sys.ActorOf. The
coordinator's PreStart reads NodeDeploymentStates to recover the
expected-ack set, and it runs as soon as the actor is scheduled — so if
PreStart won, the expected-ack set was empty and the coordinator could
short-circuit to Sealed instead of ever reaching TimedOut. The row is now
seeded before the spawn.
(b) The applyDeadline was 150 ms of real wall clock, and it arms a real
Akka timer. Under load that timer could fire on its own and race the
test's explicit DeadlineElapsed tell — so the test was intermittently
exercising the timer rather than the message handler it names. The
deadline is now TimeSpan.FromMinutes(5), matching the other two tests in
this class, which makes it un-hittable inside the test's own 3 s
AwaitAssert bound and leaves the explicit tell as the only deadline
signal the coordinator ever sees. That tell, and the AwaitAssert bound,
are unchanged.
Verified: the single test 5/5 green in a row, then the whole class 3/3.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.
SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.
Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
Follow-up to a94eb8a3, addressing CHANGES REQUESTED on code review. Production
logic was confirmed correct on all spec points; this commit fixes the review's
test/comment findings only.
IMPORTANT-1: A_permanently_unresolvable_pskProvider_is_reported_once_per_incarnation_not_once_per_refresh
did not prove the second refresh ran — its Expect(0) window gated on a dialer
dispatch landing, but LoadContactsFromDb is fire-and-forget and the first
block's AwaitCondition already re-Tells MeshCommand per poll, so the dialer
call count could reach the threshold before PublishKnownClusters was ever
re-invoked by the second load. Fixed by having the PSK factory count its own
invocations and gating both AwaitConditions on that count directly (pskCalls
>= 2 proves the first retry; pskCalls > pskCallsAfterFirstLoad, awaited INSIDE
the Expect(0) window, proves the second one). Bite-checked: with
_pskProviderAbsenceReported's guard temporarily removed, the test now
correctly fails ("Received 1 message too many... received 1 that matched
filter"); guard restored and reverified green.
MINOR-2: reworded four comments (CentralCommunicationActor.cs ~109, ~259,
~305-309, ~853) from narrating the change ("this fix removes", "already
existed, unchanged", "added by this fix", "the old two-call-site shape") to
stating the timeless invariant.
MINOR-3: added A_supplied_dialer_factory_that_always_throws_still_drops_each_command_loudly_while_PreStart_stays_quiet
— the symmetric case to the existing never-supplied-dialer test, exercising
Resolve's catch + firstUse:true path (a factory that WAS supplied and always
throws) rather than the null-factory path. Asserts PreStart plus the first
contact load stay silent and DispatchOverGrpc's existing per-dispatch Error
still fires on every dispatch, unguarded.
MINOR-4: "Check the ControlPlane registration wires IMeshPskProvider" ->
"Check that the ControlPlane registration wires up IMeshPskProvider"; no test
pinned the old substring, so no test update was needed for this one.
Skipped item 5 (restart-re-arms pin) per instruction — out of scope.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Joseph Doherty