- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and
v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3
on the test step.
- Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR
(TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget.
- Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests
RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process
(abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s.
- Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start
state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes
host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while
Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick).
- Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps
(wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log.
- Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on
Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped.
- infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- I1: the upgrade restart itself moves every no-key AbCip/AbLegacy driver to Native (every process
start constructs drivers, incl. boot-from-cache). Docs + CLAUDE.md now say so and tell operators
to set "transport": "Libplctag" BEFORE upgrading (older builds ignore the unknown key).
- I2: AbLegacy release note leads with the status-bit write refusal (T4:0.DN, C5:0.CU, R6:0.EN, ...)
on defaulted SLC/MicroLogix drivers and the fallback. The real client status is
BadCommunicationError, not BadNotSupported (the driver's write catch-all maps the runtime's
AbTransportException like any transport failure, as #571 does for PLC-5), with handle eviction
and Degraded health; pinned by a driver-level test (no 0xAA write on the wire, with a positive
control). ST string byte order stated as unverified. Also corrects the docs' "T4:0/DN" examples:
the driver's address grammar is "T4:0.DN" ("/DN" is a malformed address).
- I3: DriverProbeHandshakeE2eTests' two AbCip probe tests are a [Theory] over Native and
Libplctag. Run against ab_server controllogix: all 4 rows pass with identical exact messages.
- M1: documented (docs + resolver XML doc): the AdminUI merges only the probed device, so a
mixed-family defaulted AbLegacy driver can probe Native while running Libplctag.
- M2: factory ParseEnum rejects undefined integer strings ("7") via Enum.IsDefined (both drivers).
- M3: AbCip driver-level bad value reads "AB CIP config in '<id>' has unknown Transport ..."; the
named constant is gone (tagName: null now means driver-level).
- M4: the explicit-Native limited-family Warning is once per instance, not per (re)init.
- Nit: EffectiveTransport docs note that injected factories bypass it.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Task 22 (spec section 8 P5), with the controller rulings that exclude PLC-5 and LogixPccc.
- Transport is nullable on both option records; null (key absent) means "use the default" and is
never serialized. Factory DTO parse: absent -> null; a bad value still fails via ParseEnum.
- One resolver per driver (AbCipTransportResolver / AbLegacyTransportResolver), used by the driver
constructor, the Test Connect probe and (through the driver) the CLI.
AbCip: null -> Native. AbLegacy: null -> Native unless any device is Plc5 or LogixPccc, then the
whole driver resolves to Libplctag. Explicit values always win; explicit Native with a
Plc5/LogixPccc device logs one Warning at init.
- Init log shows configured vs effective, e.g. transport=Native (default).
- AdminUI forms: "Default (...)" / "Native (managed)" / "libplctag (fallback)". Default omits the
key on save (reverses Task 19's explicit re-emit, which would have pinned every edited driver).
- CLIs: --transport default|native|libplctag, default "default" via the resolver, so
--plc-type Plc5 with no --transport runs libplctag.
- Tests pinned to Libplctag where a default-path runtime was created: found by instrumenting the
native factories with the current xUnit test name (one unit test, AbLegacyReinitConfigAdoption;
plus the two fixture-gated AbCip probe E2E tests whose exact messages were proven on libplctag).
TransportMatrix and the differential harness already pass explicit transports.
- Switch/resolver/CLI/form tests are falsified by mutating the resolver (default -> Libplctag,
exclusion removed, AbLegacy default -> Libplctag): each mutation turns them red.
- Replaces the inline role-name literal in AbCipDriverFactoryExtensions' ParseEnum call with a named
constant; RoleAuthorityGuardTests was already red on the base commit for that line.
- Docs: AbCip.md / AbLegacy.md default + release note; CLAUDE.md AB section.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
MeshGrpcMapConventionTests - source-scan of Program.cs (solution-root walk,
following FlushLoggingConventionTests.cs's idiom): zero direct
MapGrpcService</MapZbLocalDbSync( occurrences, exactly one
MapOtOpcUaMeshGrpcServices( call.
Complements the endpoint-metadata census (T1-T3): the census catches a map
added to MeshGrpcEndpoints without the AllowAnonymous convention, but cannot
see a map that bypasses the helper and goes straight into Program.cs - that
shape never reaches MapOtOpcUaMeshGrpcServices at all. Verified red against
the pre-T1 Program.cs (five inline map sites, zero mapper calls) and green
against the current extracted form.
MeshGrpcDriverOnlyPipelineTests - the driver-role shape (no AddOtOpcUaAuth, no
UseAuthentication/UseAuthorization at all) behaves identically to the fused
shape's T4 positive/negative tests for NodeCommandService and
TelemetryStreamService: a correctly-keyed call succeeds, a no-key call gets
the interceptor's PermissionDenied + detail string.
Also asserts the STRUCTURAL reason it is unaffected - no
IAuthorizationPolicyProvider registered at all - so a future "register auth
everywhere for consistency" change fails this test before it ever reaches a
live site outage. Confirmed (by temporarily swapping in the pre-T2/T3 mapper)
that this class passes identically on both builds: driver-only nodes were
never exposed to 03/S14 in the first place, with or without
.AllowAnonymous().
Extends MeshGrpcAuthStackTests (T5): a no-bearer or wrong-bearer call to
DeploymentArtifactService.Fetch, TelemetryStreamService.Subscribe and
NodeCommandService.Execute fails with the INTERCEPTOR's detail string
("Config-serve authentication failed.", "Telemetry stream authentication
failed.", "Node command authentication failed."), asserted alongside the
status code rather than in place of it - the vacuity trap the plan calls out
(ConfigServeAuthInterceptor throws Unauthenticated, and Grpc.Net.Client can
map an HTTP 401 to the same status, so status alone would not discriminate a
broken build).
Verified red-then-green against the T1 (pre-AllowAnonymous) mapper: all 6 new
theory cases fail there with StatusCode.Internal ("HTTP/2 server reset the
stream, PROTOCOL_ERROR") instead of Unauthenticated/PermissionDenied - the
cookie challenge breaking the h2c stream before any gRPC trailers are
written, never a clean auth denial. That is the exact signature the live
gate (T10) should grep for; recorded in the class remarks.
MeshGrpcAuthStackTests - the CI-shaped guard 03/S14 asked for. A real
AddOtOpcUaAuth(config) + UseAuthentication/UseAuthorization pipeline (cookie
scheme, DisableLogin=false, EnvironmentName=Production) with the production
interceptor set, driving DeploymentArtifactService.Fetch,
CentralCommandService.ReportApply and NodeCommandService.Execute through
MapOtOpcUaMeshGrpcServices.
Verified red-then-green by hand: with the T1 (pre-AllowAnonymous) mapper
swapped in, all three positive calls fail with
"HTTP/2 server reset the stream, PROTOCOL_ERROR" (RpcException/Internal) -
the cookie challenge breaking prior-knowledge h2c, not a clean auth denial.
With the current mapper (T2/T3) all three succeed. That asymmetry is the only
thing distinguishing the two builds; every other assertion here (cookie
handler identity, the falsifiability control) is unchanged by the fix and
proves nothing on its own.
Falsifiability control: LocalDbSyncService is mapped directly (bypassing
MapOtOpcUaMeshGrpcServices) with no .AllowAnonymous(), and a correctly-shaped
call to it is refused - proving the fixture's auth stack is genuinely live,
not accidentally permissive.
MapZbLocalDbSync returns IEndpointRouteBuilder, not an IEndpointConventionBuilder,
so .AllowAnonymous() cannot be chained onto it the way the other four mesh maps
are (T2). app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync() carries
the same convention while leaving the gRPC route pattern byte-identical -
verified directly (LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical).
The census's RealMethodEndpoints filter had to move off a "route pattern starts
with /" heuristic to an exact-route-pattern-set match: inside the empty-prefix
group, grpc-dotnet's own "Unimplemented service"/"Unimplemented method" fallback
endpoints also gain a leading "/", so the heuristic let them back in and
over-counted (caught by DriverOnly_WithASyncPort_AlsoMapsTheLocalDbSyncEndpoint
and LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical going red
against the naive filter before this fix).
All five mesh gRPC endpoints now carry IAllowAnonymous (archreview 03/S14, T3).
.AllowAnonymous() on DeploymentArtifactGrpcService, CentralCommandGrpcService,
TelemetryStreamGrpcService and NodeCommandGrpcService — the four MapGrpcService
maps that are chainable. On a login-enabled admin node the global cookie
FallbackPolicy (RequireAuthenticatedUser) applies to every endpoint without
explicit authorization metadata, gRPC included; the PSK bearer interceptors run
INSIDE the endpoint invocation, strictly after that middleware, so a
correctly-keyed node call was challenged before the interceptor's verdict could
matter. Safe because all four interceptors fail closed on an unset key.
The census now asserts IAllowAnonymous on all four (red before this change,
green after) plus a scope guard proving the opt-out never leaks onto an
unrelated endpoint. MapZbLocalDbSync (T3) is not chainable the same way and is
asserted absent here on purpose, so this test cannot start passing for the
wrong reason once T3 lands.
Behaviour-preserving: Program.cs:743-784 collapses to a single call, same gates,
same order, every existing per-map rationale comment carried over. A new
endpoint-metadata census (MeshGrpcAnonymousMetadataTests) drives the SAME
production mapper across all three role shapes (admin-only / driver-only /
fused), so the guard can never assert a shape production no longer has -
archreview 03/S14, T1.
Review follow-ups to 34829ea0.
IMPORTANT - the leak filter had its own flake vector. This test saturates the
thread pool by design, and TimedScriptEvaluator starts its 5s clock at WaitAsync
rather than when the inner Task.Run gets a thread, so a legitimately-enqueued
evaluation can blow the budget while still queued and surface "script timed out
after 5.0s". The reason != exhaustion filter classified that as a raw disposed
escape and hard-failed on it - handing the test back the load-fragility the
reshape removed. Timeouts are now their own bucket, tolerated and rate-bounded
exactly as exhaustions are; a raw disposed escape still fails on first occurrence.
Doc-comment names this second residual vector alongside the exhaustion calibration.
MINOR (scope extension authorised by review) - "evaluator disposed during
evaluation" existed as three unshared literals. Promoted to
ScriptEvalFailureReasons alongside VirtualTagEvalResult in Commons.Engines (the
assembly all three already reference) and referenced from both evaluators and the
test. The timeout prefix joins it, since the test now classifies on it too. String
values are byte-identical - no behaviour change.
Falsified: with a 12ms script budget the race produced real timeouts (3/2695).
Classified as leaks (the pre-review filter) they hard-failed the test 6/6 items;
bucketed as starvation they sit 0.11% under the 10% bound and the test is green.
The racing test encoded two timing assumptions and failed under full-solution
parallel load while passing in isolation. Both are removed; no production code
changed.
1. Assertion shape. failures.ShouldBeEmpty() assumed at most ONE apply-boundary
clear can land inside a single evaluation, so the evaluator's one-shot retry
always suffices. Under contention two can land and the retry legitimately
exhausts. The test now asserts the invariant the retry actually guarantees and
which is timing-independent: no raw disposed-script failure ESCAPES the retry.
Every failure must be the documented exhaustion outcome ("evaluator disposed
during evaluation") and never the pre-fix generic escape ("script threw:
Cannot access a disposed object..."), plus a deliberately loose bound on the
exhaustion rate so the check cannot go vacuous.
2. Workload shape. The fixed 12x400 evaluations against a fixed 45s WaitAsync
budget is CPU-dependent (every clear forces a recompile). Four concurrent
instances of this class reproduced a hard TimeoutException 4/4 - the likely
real shape of the reported load failure. The loops are now duration-bounded
(10s race window or 400 iterations, whichever first); the 45s WaitAsync stays
purely as a hang detector.
Renamed to Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure
so the name matches what it now asserts; the old name is cited in the doc-comment
for traceability.
Review round on 3e62f161. Two coverage gaps and three wording/comment nits.
IMPORTANT 1 — deregistration is now proven DATA-PRESERVING. The self-heal test
inserted a deployment_pointer row and rebooted unconfigured but never looked at
the row again, so a library regression that touched base-table data during
cleanup would have gone unnoticed — on the very tables that ARE the
boot-from-cache backstop, and on the default-OFF nodes least likely to notice.
The new DisablingReplication_PrunesBookkeepingButNeverTouchesTheCachedDataItself
asserts the row's CONTENT survives (not just its count — a count passes against
a cleanup that blanks columns) while __localdb_row_version and __localdb_oplog,
the bookkeeping deregistration does own, go to zero.
IMPORTANT 2 — mid-loop partial failure is documented and pinned. Both loops flip
four tables with no try/catch, so a throw on the third leaves two flipped and
propagates out of the AddZbLocalDb singleton factory, failing host start. That
is correct — a node that came up anyway would be silently capturing on an
arbitrary subset — and both library calls are idempotent, so the retried boot
completes the flip. Written into OnReady's remarks and pinned by
AFailureMidwayThroughRegistration_FailsBootLoudly_AndTheRetriedBootConverges,
which drives OnReady through a delegating ILocalDb that throws on the Nth
RegisterReplicated, asserts the throw propagates, asserts the wrapper actually
reached that call, then re-runs OnReady unwrapped and asserts all four converge.
MINOR 3 — the Boot helper now records that OnReady's null-logger fallback lands
on Serilog's static Log.Logger, which this assembly never assigns; assertions
about logging pass an explicit logger.
MINOR 5 — log wording: "are both unset" -> "are both absent — unset or blank",
matching IsNullOrWhiteSpace.
MINOR 4 — LocalDbDeploymentArtifactCacheTests said its setup "mirrors
LocalDbSetup.OnReady"; it reproduces the two tables only and registers them
unconditionally without baselining. Softened to say so and to name the
divergence as intentional and inert there.
Falsified both new pins. Wrapper set to failOnCall: 99 (never throws) => the
partial-flip pin is the SOLE failure, so it genuinely exercises the Nth-call
throw path rather than some incidental exception. Predicate forced constant-true
=> five pins red including the new data-preservation one. Both mutations
reverted.
LocalDbSetupTests 15/15. Host.IntegrationTests 226 passed / 0 failed / 5
skipped. Runtime.Tests LocalDbDeploymentArtifactCacheTests 12/12.
LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.
Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.
configured => RegisterReplicated(table, baselineExistingRows: true), per
table, in the unchanged load-bearing DDL->register order
unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
drops the capture triggers an earlier build left behind and
prunes that table's oplog/row-version rows. One Information
line, only when it actually cleaned something.
PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").
Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.
Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
Four-minor jump 0.2.3 → 0.6.2. The release line exists because a deploy
renaming the app directory took the MxGateway credential store with it
(2026-08-09, two silent days): Secrets:SqlitePath must now be absolute AND
outside the content root, rejected at boot rather than relocated. 0.6.2 over
0.6.1 is message-only (the rejection text now teaches the mounted-volume
case and forecloses the override hunt) — the live rig evidence below was
gathered at 0.6.1 and the validator behavior is identical.
Adoption changes beyond the pins:
- appsettings.json drops the relative "otopcua-secrets.db" (now boot-fatal);
dev gets the library's absolute per-user default, deployments supply their
own (wonder already pins an absolute out-of-root path in the service env).
- Program.cs's pre-host ${secret:} expander now calls the FOUR-argument
AddZbSecrets with builder.Environment.ContentRootPath. This call site is
the actual fix, not the pin: the throwaway container has no
IHostEnvironment, and without the explicit root the under-content-root
rule is skipped at exactly the moment the migrator creates the store file
(the 0.6.0 defect). Do NOT simplify back to the 3-argument overload — it
compiles and silently reopens the gap. Pinned by a Program.cs source-scan
test AND a runtime falsification control that RUNS the migration and
demonstrably creates the store at the rejected path under the 3-arg form.
- docker-dev: Secrets__SqlitePath moves /app/data → /data via a SECOND mount
of the same per-node volume (same file at the volume root, so seeded
stores carry over) because /app is the container's content root and the
new rule correctly rejects it. LocalDb stays on /app/data (secrets-only
rule).
- SecretsStorePathValidationTests: positive-control probes, artifact-first
(relative and in-content-root paths must FAIL leaving no file; outside-
root passes; the no-explicit-root control proves the file assertions can
fail) — a clean boot is equally consistent with the validator never being
wired.
Live-verified on the six-node docker-dev rig (at 0.6.1): all nodes boot
under the new validation with stores at /data; an AdminUI-written secret
converged into BOTH central nodes' independent stores; site-a held only its
own pair's secrets (Phase 6 pair-local isolation intact). Replicator wire
behavior unchanged 0.3.0–0.6.x per CHANGELOG; registration-graph pins and
the full suite pass.
TelemetryStreamAuthInterceptor, ConfigServeAuthInterceptor and
LocalDbSyncAuthInterceptor all decided "is a key configured?" with
string.IsNullOrEmpty. A whitespace-only configured value therefore passed
that guard and fell through to the constant-time comparison — where it is
satisfied by a whitespace-only bearer token. The gate would be open to
anyone who guessed a blank credential.
This backports the check the two NEW mesh interceptors already use:
MeshCommandAuthInterceptor (and its central-side clone) validate with
string.IsNullOrWhiteSpace, so the four/five planes now answer the same way
to the same misconfiguration. Nothing else changes — the denial status
codes (PermissionDenied / Unauthenticated / PermissionDenied respectively)
and every denial message are untouched, and no key material is logged.
No deployment configures a whitespace key: the docker-dev rig and
production both set real keys, so this is a fail-closed hardening of an
unreachable-today misconfiguration rather than a live exposure.
Tests: each of the three test files gains a ..._WithBlankKeyConfigured_IsDenied
fact mirroring MeshCommandAuthInterceptorTests.ExecuteMethod_WithBlankPskConfigured_IsDenied
(configured key " ", presented "Bearer "), verified red before the fix.
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor also gain the
HasExactlyOnePublicConstructor pin their two siblings already had —
Grpc.AspNetCore silently stops invoking an interceptor with more than one
public constructor, which would disable auth without failing anything.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
MeshPaths.CentralCommunication / .NodeCommunication (the "/user/..." full-path constants) had
ZERO production consumers after the ClusterClient deletion: both gRPC services resolve their
comm actor through ActorRegistry by typed key, never by ActorSelection over a path. Only the
path-probe test still read them, while the class doc went on calling them "the wire contract
of the central↔node boundary" — a claim that stopped being true the moment nothing addressed
an actor across a process boundary by name.
- The two full-path constants are deleted; the two Name constants remain.
- MeshPaths' doc now says what it is (the local actor-naming convention the registry-keyed
services are spawned under) and records what it stopped being, so the next reader does not
re-derive a contract from the file's history.
- MeshCommActorPathTests composes "/user/" + Name itself — the absolute path is that test's
own addressing detail now, not something the product declares. Renamed
Both_mesh_comm_actors_exist_at_their_contract_paths ->
..._are_spawned_under_their_declared_names, because what it still pins is that the two
registration sites actually ran on a real booted node; a rename is now a compile error, so
"contract" was the wrong word for what would break.
- NodeCommunicationActor's summary no longer cross-references the deleted constant or calls
the name an address. CentralCommunicationActor was checked and makes no such claim.
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
A browse-committed AB CIP / TwinCAT / FOCAS tag bound with its model's DEFAULT
data type, not the one the device reported. A browsed REAL committed as
`{"tagPath":"Pump_1_Speed","dataType":"DInt"}` — and those three drivers decode
the wire bytes by that field, not by the raw row's OPC UA DataType — so the tag
read four bytes of a float through an integer decoder and published a plausible
wrong number with no error raised anywhere. FOCAS was missed when #536 was filed.
The type is CARRIED, not derived. Reverse-mapping the leaf's driver-agnostic
DriverDataType looks like the smaller fix and is wrong: SInt/Int/DInt/USInt/UInt
all collapse into Int32 on the way out of the driver, so that version would bind
a 16-bit INT as a 32-bit DINT while appearing to fix the REAL. Drivers now state
their own type name on DriverAttributeInfo.NativeDataType, which reaches the
picker as AttributeInfo.AddressFields["dataType"] and the commit mapper from
there. A leaf that states nothing keeps the model default — the pre-#536
behaviour, kept deliberately as the fallback rather than a guess.
Numeric strings are refused before parsing: Enum.TryParse accepts "9" for any
enum and Enum.IsDefined then confirms it, so an unguarded parse would bind a tag
to a type nobody named. A test caught this in the first version of the guard.
--- and the other half, which the fixture was hiding ---
Test Connect on a device with NO authored tags — every AB CIP device at the
moment an operator first presses the button — reported "CIP handshake failed"
against a healthy controller. That was written down as a fixture limitation
("ab_server doesn't implement @raw_cpu_type"). Both halves of that were wrong:
* @raw_cpu_type is not a libplctag special tag at all (only @raw, @tags and
@udt/ are), so it goes out as an ordinary symbolic name that no controller
defines, and the controller's "tag does not exist" IS the successful answer.
* ab_server answered an unresolvable tag path with CIP 0x03 where a Logix
controller answers 0x04 — ErrorBadParam vs ErrorNotFound, which the probe
classifies oppositely. Fixed in vendored patch 0002.
* With the fixture then telling the truth, the real defect surfaced:
AbCipDriverProbe.IsReachableException matched the native C status spellings
("STATUS_NOT_FOUND"), but libplctag.NET throws with the MANAGED enum name
("ErrorNotFound"). Those substrings can never match, so the reachable
classification could not fire on the exception path — which is the path every
no-authored-tags probe takes. This was failing on real hardware too, not just
against the sim. It now recovers the Status and defers to IsReachableStatus,
so there is one classification rather than two that can drift.
Rejected: switching the fallback to @tags. libplctag builds a tag-listing tag
with NO network I/O, so it reports OK against any TCP listener — measured green
against a Mosquitto broker.
Verification: full solution suite green. Falsification runs confirm exactly the
intended tests go red — 9 for the type carry, 3 (the reachable statuses only) for
the probe classification. Live on docker-dev: browse-committed TestString/
Motor_Run/TestSINT now store String/Bool/SInt, alongside last session's
Pump_1_Speed still holding the wrong DInt. TestSINT is the falsifier — its row
DataType is Int32, identical to TestDINT's, so no reverse map could have produced
"SInt".
Known consequence: pre-existing browse-committed tags keep their wrong dataType;
the fix applies to new commits only.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
None of these were fixture absence. Each is a guard that went quietly wrong and
then reported the wrong thing.
PrimaryGateFailoverTests — per-cluster mesh Phase 6 (a2c5d57f) moved
RedundancyStateActor out of WithOtOpcUaControlPlaneSingletons into its own
cluster-role-scoped registration, made from Program.cs's DRIVER branch. The
harness models a fused admin,driver node but was never updated, so from that
commit onward NO node it built spawned the singleton: nothing published
redundancy-state, both nodes sat at "role unknown" forever, and the test polled
45s and timed out. The test's own summary calls this its negative control for a
broken redundancy-topic subscribe — it fired correctly, at the harness rather
than at production. Second time this shape has bitten the harness; the
AddOtOpcUaRuntime note a few lines up records the first.
DriverProbeRegistrationTests — the expected driver set was a hand-written array
kept in sync by comment, and it drifted when Sql (9b5a9687) and Mqtt (f79d13e2)
shipped. It failed as "12 where 10 expected", which names neither driver and
reads as if the probes were wrong. It also could not catch the direction that
matters: a driver type registered with no probe passed silently, because a
hand-written list only grows when someone remembers, and this test exists for
the case where nobody did. Now derived from DriverTypeNames.All, both
directions, plus an explicit case-drift test for the two types whose AdminUI key
differs in case from the constant (TwinCat/TwinCAT, Focas/FOCAS) — which the
derived tests cannot cover, since they compare the constants to themselves.
DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim — the probe config still used
the retired v2 `Tags: [{DeviceHostAddress, TagPath}]` shape, so
AbCipDriverOptions.RawTags bound empty, the probe fell through to its
@raw_cpu_type system-tag fallback, and ab_server answered ErrorBadParam. The test
has therefore spent the whole v3 migration exercising the one path its own
comment says it does not exercise. Rewritten to the v3 RawTags shape, and the
assertion tightened from ShouldContain("CIP session OK") to an exact match:
the probe's two green messages both contain that substring, so the loose form
would go green again the moment the config merely reached the controller,
whether or not the authored tag resolved.
Falsified each: drop the MTConnect probe (2 red, by name), remove the harness
singleton registration (back to the 48s "role unknown" timeout), break the
RawTags key (back to ErrorBadParam).
Host.IntegrationTests 211 passed / 4 skipped / 0 failed, against live fixtures on
10.100.0.35.
Still red and NOT addressed here: the AbLegacy integration suite (4 tests). Its
fixture is genuinely down, but it hard-fails instead of skipping because the
AbCip ControlLogix container holds :44818 — the reachability guard cannot tell
"my ab_server" from "someone else's", so SkipReason stays null and SLC500-mode
PCCC runs against a ControlLogix. Needs either distinct fixture ports or a
family-aware guard; both are shared-infrastructure changes.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Stage 2b. LdapOpcUaUserAuthenticator was left registered nowhere by stage 2a;
this removes it, its three test files, and the config surface that existed only
to serve it.
Deleted: LdapOpcUaUserAuthenticator, LdapOpcUaUserAuthenticatorTests,
LdapAuthResilienceTests, LdapGlauthOutageLiveTests (19 tests).
Also removed four Security:Ldap keys — AuthTimeoutMs,
MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds —
plus their validator rules and binding/validator test coverage. Verified first
that the ONLY consumers were the deleted class and the validator rules asserting
them; none appear in any checked-in appsettings. Leaving them would have created
exactly the dead-config surface #521/#524/#525 were filed to clear up.
Worth stating why that resilience machinery does not need relocating rather than
just deleting. It existed because the OPC UA SDK raises impersonation
SYNCHRONOUSLY under a SessionManager-wide event lock, so a hung LDAP bind there
serialized every session activation server-wide — Anonymous and X509 included.
Key verification is one indexed lookup plus one HMAC with no network I/O, so the
hazard the wall-clock bound, concurrency cap and outage circuit guarded against
no longer exists on that path. LDAP now runs only on AdminUI login, an ordinary
async request path where a slow directory affects the one request making it.
ConnectionTimeoutMs is RETAINED: it is projected into the shared library and is
not authenticator-specific.
Docs corrected rather than left to rot:
- CLAUDE.md's "LDAP Authentication" section described the deleted file as the
data-plane authenticator. Rewritten, including a warning not to "restore" it on
seeing the LDAP wiring that AdminUI legitimately still needs.
- infra/README §5b (the GLAuth outage live gate) marked RETIRED with the reason,
rather than silently dropped — the harness GLAuth is still used by
OTOPCUA_HARNESS_USE_LDAP=1 and is unaffected.
- docs/security.md's five-key resilience table cut to the one surviving key, and
a new API-key identity section added.
Historical archreview/ records are left as-is; they are dated accounts.
Host.IntegrationTests 211 tests (was 230, i.e. exactly the 19 removed) with the
same 3 pre-existing failures verified against clean master. Host 37, Security 89.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Both `IDriver` members had no production call site. Their only consumer was
`MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and
`GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which
had not been true since v3, even before #522 removed the poller.
Owner decision on #525 was (a) delete rather than (b) build the process-wide
watchdog that would consume them. The driver-side implementations are in git
history and were mostly reusable; what was always missing is the trigger, and a
member nobody calls is how the misleading doc-comment survived this long.
Removed from `IDriver`, all 12 driver implementations, and every test stub —
plus the support code that existed only to feed them, which is the part a
member-only deletion would have left behind as dead weight:
- MTConnect's three per-entry byte-weight constants, its cached
`ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run
on every cache fill to populate a number with no remaining reader.
- OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it.
- MQTT's `ApproxBytesPerAuthoredTag`.
Two behavioural notes, neither a change in what the software does:
- `AbCipTemplateCache` now has no production reset path. Its only caller was
AB CIP's flush, which itself had no caller — so nothing changes except how
visible the fact is. Recorded on the type, along with the correction that its
comment's claim about a device `ReinitializeAsync` also flushing it was never
true. A UDT redefined on a live controller keeps its stale shape until the
driver instance is rebuilt.
- MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an
Agent restart. That is now `internal` rather than `private` so the tests which
used the flush purely to empty the cache drive the real path instead — they
now cover a case that actually occurs in production. Several unrelated doc
comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are
repointed to it.
Tests: the ones that existed only to assert on the removed members are deleted
(3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was
vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a
footprint of 0 before the flush and 0 after with no discovery in between, so it
would have passed against a flush that did nothing at all.
Full suite green except 7 failures verified identical on master before and
after (4 AbLegacy integration with the compose profile down, 3
Host.IntegrationTests), checked by name rather than assumed.
Docs corrected where they described this as live: `DriverLifecycle.md` (which
carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and
`v2/driver-stability.md` — whose "polls this every 30 s" + per-driver
`DriverConfig` memory budget were never built as written and are now banner-ed
as design intent rather than description.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Register the MTConnect factory + probe with the Host, add the
DriverTypeNames.MTConnect constant, and keep DriverTypeNamesGuardTests green.
- DriverTypeNames: new MTConnect const + appended to All.
- DriverFactoryBootstrap: MTConnectDriverFactoryExtensions.Register in
Register(...) at the default Tier A (managed HttpClient/XML, in-process;
Tier C is the only tier arming process-recycle, wrong here), and
TryAddEnumerable(IDriverProbe -> MTConnectDriverProbe) in
AddOtOpcUaDriverProbes so the probe reaches admin-only nodes (the
admin-pinned Test-Connect singleton) without double-registering on a
fused admin,driver node.
- Host.csproj: ProjectReference to Driver.MTConnect (required to compile
the Register call).
- Core.Abstractions.Tests.csproj: ProjectReference to Driver.MTConnect so
the guard's bin scan discovers the factory. This and the constant MUST
land together -- verified RED (2/4 facts) with the constant alone.
- DriverProbeRegistrationTests: MTConnect added to AdminUiDriverTypeKeys;
its idempotency fact asserts distinct-probe-count and goes RED without
this (and RED if TryAddEnumerable is downgraded to AddSingleton).
Reconciles all four "MTConnect" literals onto the one constant: the
factory's DriverTypeName, MTConnectDriver.DriverType, and
MTConnectDriverProbe.DriverType now all alias DriverTypeNames.MTConnect
(no circular reference -- Core.Abstractions is a leaf both driver
projects already reference). This is the repo's TwinCat/Focas anti-drift
convention.
After the fleet splits into one Akka mesh per application Cluster, a
ClusterClientReceptionist serves only its own mesh — so the single
fleet-wide client's SendToAll reached only the mesh whose receptionist
answered, leaving every other cluster silently on its old configuration.
Central now holds one ClusterClient per ClusterId and fans SendToAll
across all of them. LoadContactsFromDb selects ClusterId and groups the
receptionist contacts per cluster (keeping the per-row TryParse guard and
the enabled/non-maintenance filter); ContactsLoaded carries
ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed,
stop+drop vanished, warn-don't-create on empty); RebuildClient builds a
per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create
gained a clusterId parameter so the per-cluster clients get distinct,
diagnosable names. ApplyAck handling and the DPS branch are unchanged; the
deploy path stays payload-free.
Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster,
fan-out SendToAll to every cluster client) via the recording factory double,
plus the real two-mesh boundary test extended to central + two separate site
meshes proving one dispatch reaches both and a client scoped to one cluster
never crosses into another. Red-before-green verified: crippling the fan-out
to a single client fails the reaches-both-meshes assertion.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Each of the four telemetry publish seams now also emits its DPS payload into the
node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub
publish untouched:
- driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health
- resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience
- alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm
- script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script
DI services take the hub as a required ctor param; actors take an optional
nullable hub threaded through their Props and the real spawn sites
(DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor
<- DriverHostActor), so existing test Props constructions keep compiling and simply
do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe
on every node.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent
(central persists acks from the ApplyAck); scripted-alarm condition state
served from the LocalDb store instead of the ConfigDb-backed Ef store.
Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 8. Stands a real h2c-only Kestrel listener serving the real
DeploymentArtifactGrpcService behind the real ConfigServeAuthInterceptor (in-memory
config DB seeded with one sealed deployment), and drives the real
GrpcDeploymentArtifactFetcher across it — proving a stream crosses the real boundary,
not just that the fetcher's logic is right (Task 4's fakes). Asserts: (1) right key +
>256 KB blob reassembles byte-equal and verifies against RevisionHash; (2) FALSIFIABILITY
CONTROL — wrong key returns null while the right key succeeds on the SAME server (so the
null is the auth gate, not a dead server); (3) unknown id -> null (NotFound-hidden);
(4) [dead-port, real-port] failover still returns the bytes. h2c GrpcChannel over http://
works out of the box in .NET 10. Tagged [Trait Category=ArtifactBoundary].
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 3. Central serves the deployment artifact over a dedicated h2c
Kestrel listener (ConfigServe:GrpcListenPort), gated by the ConfigServeAuthInterceptor
(shared bearer key, FixedTimeEquals, fail-closed, path-scoped to
/deployment_artifact.v1.DeploymentArtifactService/). The listener block is merged
with the LocalDb-sync listener so a fused admin+driver node re-applies its existing
HTTP surface exactly ONCE and adds both dedicated h2c ports on top — double-binding
would throw 'address already in use'; zero re-binding would silently unbind the
AdminUI behind Traefik. AddGrpc now registers under hasDriver || hasAdmin with both
path-scoped interceptors sharing one pipeline.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two genuinely separate single-member clusters sharing the ActorSystem name and
joined only by ClusterClient, so the mesh boundary is exercised over the wire
rather than through an in-process relay: a relay cannot fail for a missing
receptionist extension, an unregistered service, a malformed contact path or a
serialization break, which are the Phase 2 failure modes.
Covers both legs plus the control:
* central SendToAll -> node comm actor -> node EventStream
* node ApplyAck -> central comm actor, through the node's own client
* a node without the receptionist extension receives nothing
Sabotage-verified. Suppressing the node-side EventStream publish and the
outbound ack Tell reddens one leg each. Restoring the receptionist and the
service registration on the control node reddens the control, which is what
proves the control's send is live rather than silently misaddressed.
The control removes the extension AND the RegisterService call, since resolving
the receptionist to register would materialise the extension whose absence is
the point; it falsifies "delivery happens without a receptionist boundary", not
the extension line alone. Recorded in the test.
Also corrects MeshCommActorPathTests: with one node per side, per-node and
singleton registration are indistinguishable here, so the property the dropped
Task 6 assertion covered belongs to live-gate step 8, not to this class.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 6. Both comm actors are now spawned and receptionist-registered:
CentralCommunicationActor from WithOtOpcUaControlPlaneSingletons, and
NodeCommunicationActor from WithOtOpcUaRuntimeActors (before DriverHostActor,
because it is the host's ackRouter under ClusterClient mode; it has no
dependency of its own on the host, since inbound commands travel via the node's
EventStream).
Both are plain WithActors registrations, NOT singletons -- a driver node's
ClusterClient rotates across contact points and must find a live comm actor at
whichever node answers. Both register with the receptionist in BOTH transport
modes: an idle registration costs nothing, whereas registering only under
ClusterClient would mean flipping the flag on a running fleet needs a restart
before anything can be reached, turning a config change back into a deployment.
The coordinator ref is resolved lazily per message (Func<IActorRef?> +
registry.TryGet) so the comm actor never depends on the order in which
Akka.Hosting materialises the singleton proxy relative to this block.
Adds MeshCommActorPathTests, which boots the real two-node host and Identify-
probes both paths. These strings are the wire contract and nothing else asserts
they agree: a rename would compile, pass every unit test, deploy, and deliver
nothing, because a ClusterClient send to an unregistered path is dropped
silently. Sabotage-verified with an actual rename.
DELIBERATELY NOT TESTED, with the reason recorded in the file: that the actors
are per-node rather than singletons. Two discriminators were tried and both were
invalid -- "the path resolves on both nodes" passes under either shape because a
ClusterSingletonManager is also created on every node in the role, and "no
/singleton child" is null even for the KNOWN singleton /user/config-publish, so
Akka.Hosting does not lay them out that way. A sabotage re-registering the actor
via WithSingleton left both green, which is what exposed them. Rather than ship
an assertion that cannot fail, the property is left to the Task 8 boundary test,
where a send only arrives if the target really is registered.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.
Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.
Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).
Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Comparing full-suite runs on this branch against master in a clean worktree:
both fail 11 tests, 9 identical. Master's two extras are load-flaky unit tests
(AbCip Probe_loops, Galaxy EventPumpBoundedChannel); this branch's two extras
were both Host.IntegrationTests deploy-path tests — the area this change
re-times — so they are attributable here rather than to background flakiness.
Cause is margin, not correctness. These waits used to observe a coordinator
that sealed instantly on an empty expected-ack set; they now observe a real
ApplyAck round-trip from every configured node. 15s was enormous margin against
"instant" and thin against the real thing, so they failed only under full-suite
CPU contention.
Hoisted to TwoNodeClusterHarness.DeploySealTimeout (45s) so the reason is
recorded once rather than as four unexplained numbers.
DriverReconnectE2eTests is deliberately left alone: it seeds both ClusterNode
rows itself and unconditionally, so its expected-ack set is identical before and
after this change. Widening its timeout would be papering over a flake this
change did not cause.
Host.IntegrationTests 195/201; sole failure AbCip_Green_AgainstSim, verified
failing on master.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The full-suite run surfaced two load-dependent failures that were green in
isolation. Root cause: SeedDefaultClusterAsync no-opped unless
OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores
FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that.
With no ClusterNode rows the coordinator's expected-ack set is empty, so it
seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has
applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each
node's row on its own schedule, every test counting those rows after a seal
became a race. Reproducibly green alone, red under suite load, and it surfaced
in a different test each run, which is what made it look like flakiness rather
than a contract change.
Seeding in both modes restores the invariant those tests were written against
and is closer to production either way: a real fleet always has these rows,
because the FK requires them.
Deployment_started_with_node_b_down_seals_with_one_node_state asserted the
behaviour Phase 1 deliberately removed — its own comment documented membership
snapshotting, i.e. sealing green while a configured node never received the
deployment. Replaced by two tests covering the new contract: a stopped node is
still expected (both state rows exist, B still Applying, deployment does not
seal), and MaintenanceMode is what makes it seal with one.
The new "does not seal" test asserts both rows exist rather than only the
absence of a seal, so it cannot pass against a coordinator that simply died.
Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim —
verified failing on master in a clean worktree, so pre-existing fixture
baseline, not a regression.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Completes 835fc08c, which fixed the harness's DisposeAsync but left StopNodeBAsync
calling NodeB.DisposeAsync() directly — despite that commit's own message naming
it as carrying the same bug. Caught by re-reading the code rather than trusting
the commit message.
This one is not just about memory. IHost.DisposeAsync never invokes
IHostedService.StopAsync, so CoordinatedShutdown never runs and node B never
performs a graceful Cluster.Leave. Node A instead notices only when the failure
detector times the member out — a slower and materially different path from the
graceful leave the failover tests are written to exercise. The doc comment
asserted the opposite ("shuts down node B via DisposeAsync, which runs
CoordinatedShutdown"); it now says what actually happens and why.
Verified: the three failover suites that call it pass (5/5), and the project is
unchanged at 194 passed with only the AbCip fixture failure, which needs a docker
fixture that is not running.
Also checked the sibling harness while here: LocalDbPairHarness already stops each
host before disposing it, so it needed nothing.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.
The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).
Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.
The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.
The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.
Three defects crash-looped every driver node before check 1 could even run:
1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
Validator exists to turn exactly that class of failure into a named
OptionsValidationException, but its documented fail tier explicitly excluded
ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
calls". It does not: the client validates its own options at construction, so
the process dies during Akka startup and never makes a call.
2. UseTls disagreeing with the endpoint scheme kills the host too, in both
directions (both messages confirmed in the shipped client assembly). Moving an
endpoint from https to http without clearing UseTls is an ordinary migration
slip.
3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
to false, so it always sent RequireCertificateValidation=true — which the
client rejects outright when UseTls=false. Every http:// deployment crashed,
though the scheme is documented as the supported way to select h2c, and the
only workaround was to assert a certificate posture for a connection that has
no certificate.
The fourth was the blocker, and it is Phase 2's own:
4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
driver Primary is central-1 — it carries the driver Akka role, replicates
nobody's LocalDb and does not even run the alarm historian — so every driver
node logged "Historian drain suspended", including the two site-b nodes that
have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
fine. The cost is not a duplicate; it is the buffer growing to the capacity
wall and evicting the audit trail it exists to protect.
Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
drains; the two gates now deliberately disagree, and a test pins that); peer-
host matching in DriverHostActor so a node stands down only for a Primary
holding its rows; and AddAlarmHistorian short-circuiting the gate when
replication is unconfigured — testing BOTH Replication:PeerAddress and
SyncListenPort, since only the dialing half sets the former while both halves
share the queue.
Every one of these follows from the asymmetry: a false allow costs a duplicate
row, which at-least-once delivery already accepts and payload-hash ids
collapse; a false deny loses data silently.
A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.
Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.
Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty