Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.
Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.
Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).
Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Comparing full-suite runs on this branch against master in a clean worktree:
both fail 11 tests, 9 identical. Master's two extras are load-flaky unit tests
(AbCip Probe_loops, Galaxy EventPumpBoundedChannel); this branch's two extras
were both Host.IntegrationTests deploy-path tests — the area this change
re-times — so they are attributable here rather than to background flakiness.
Cause is margin, not correctness. These waits used to observe a coordinator
that sealed instantly on an empty expected-ack set; they now observe a real
ApplyAck round-trip from every configured node. 15s was enormous margin against
"instant" and thin against the real thing, so they failed only under full-suite
CPU contention.
Hoisted to TwoNodeClusterHarness.DeploySealTimeout (45s) so the reason is
recorded once rather than as four unexplained numbers.
DriverReconnectE2eTests is deliberately left alone: it seeds both ClusterNode
rows itself and unconditionally, so its expected-ack set is identical before and
after this change. Widening its timeout would be papering over a flake this
change did not cause.
Host.IntegrationTests 195/201; sole failure AbCip_Green_AgainstSim, verified
failing on master.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The full-suite run surfaced two load-dependent failures that were green in
isolation. Root cause: SeedDefaultClusterAsync no-opped unless
OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores
FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that.
With no ClusterNode rows the coordinator's expected-ack set is empty, so it
seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has
applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each
node's row on its own schedule, every test counting those rows after a seal
became a race. Reproducibly green alone, red under suite load, and it surfaced
in a different test each run, which is what made it look like flakiness rather
than a contract change.
Seeding in both modes restores the invariant those tests were written against
and is closer to production either way: a real fleet always has these rows,
because the FK requires them.
Deployment_started_with_node_b_down_seals_with_one_node_state asserted the
behaviour Phase 1 deliberately removed — its own comment documented membership
snapshotting, i.e. sealing green while a configured node never received the
deployment. Replaced by two tests covering the new contract: a stopped node is
still expected (both state rows exist, B still Applying, deployment does not
seal), and MaintenanceMode is what makes it seal with one.
The new "does not seal" test asserts both rows exist rather than only the
absence of a seal, so it cannot pass against a coordinator that simply died.
Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim —
verified failing on master in a clean worktree, so pre-existing fixture
baseline, not a regression.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.
The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:
Enabled part of the declared topology (validator, untouched)
MaintenanceMode expected to participate now (coordinator + reconciler)
Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.
Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.
Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
- docker-dev seed: AkkaPort = 4053 on all six ClusterNode rows, so a freshly
seeded rig matches a migrated one. GrpcPort left null.
- config-db-schema.md: both columns, why AkkaPort is NOT NULL/4053 and GrpcPort
is nullable, the unenforced duplication + its reconciler, and Enabled's new
second meaning as the deploy path's expected-ack set.
- Configuration.md: Cluster:Port / PublicHostname now flag that they are stored
twice, with the "update the row too" instruction and why the drift is silent.
- design doc §7: Phase 1 marked done, plus a "Phase 1 as shipped" note recording
both deviations rather than leaving the sketch reading as what happened.
- program plan: Phase 1 marked done; AdminUI node edit explicitly deferred.
- CLAUDE.md: the deploy-path behaviour change and its three consequences.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two
places and nothing made them agree. Phase 2 dials the row instead of gossiping,
so a node binding 4054 while its row says 4053 becomes unreachable from central
— and the symptom is a silent absence of acks rather than an error. That is the
shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo.
Since Phase 1 also made the rows the deploy path's expected-ack set, drift
already costs a failed deployment today.
Implemented as the plan's preferred option: an admin-role singleton comparing
rows against the membership an admin node can already see, rather than each
driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb
connection, so a self-assertion written there would have to be deleted again.
Three shapes, split by severity: a row whose dial target disagrees with its own
NodeId and a running node with no row are Errors; an enabled row with no
matching member is a Warning, because a node down for maintenance is a
legitimate state. Findings are logged only when the set changes — a check that
reprints the same warning every sweep trains operators to filter it out.
Documented limitation: Phase 2 must revisit this. Once the fleet splits into
one mesh per cluster an admin node cannot see site members, and every site row
would report EnabledRowNotInCluster forever.
Sabotage: removing the change-detection guard turns the repeat test red.
ControlPlane.Tests 100/100.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Cuts ConfigPublishCoordinator's last genuinely mesh-bound dependency: central
must be able to name the nodes a deployment is for without sharing a gossip
ring with them, because Phase 2 splits the fleet into one mesh per Cluster.
Behaviour change, confirmed before implementing: a configured node that is
switched off is now expected, so deploying while it is down fails at the apply
deadline instead of sealing green without it. Under the membership rule the
operator was told the fleet was deployed when it was not. ClusterNode.Enabled
= 0 is the maintenance hatch, and the deadline log now names the silent nodes
and points at that hatch — "4/5 acks landed" would leave an operator reading
logs on five machines.
Two assumptions are now documented on the class rather than left implicit:
every ClusterNode row is a driver node (the DB has no role column and
deliberately does not gain one — it would drift from Cluster:Roles), and
ServerCluster.Enabled is not consulted because nothing else consults it.
Dropped from the plan: cluster-scope filtering of the expected set. There is
no cluster-scoped deployment to filter on — Deployment has no ClusterId,
ConfigComposer always snapshots the whole DB, and ResolveClusterScope is
node-side self-scoping of a fleet-wide artifact.
Positive control: reverting DiscoverDriverNodes to the membership scan turns
three of the four new tests red. The fourth pins the seal-empty branch and is
documented as derivation-insensitive rather than left to look like coverage.
ControlPlane.Tests 90/90.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Additive only — two AddColumn ops, no AlterColumn, so no accumulated model
drift is riding along with this change:
ALTER TABLE [ClusterNode] ADD [AkkaPort] int NOT NULL DEFAULT 4053;
ALTER TABLE [ClusterNode] ADD [GrpcPort] int NULL;
Adds a third test covering the DB-side default specifically. The entity
round-trip cannot see it: ClusterNode.AkkaPort's CLR initializer is also 4053,
so that assertion passes with the mapping default deleted. The new test inserts
through raw SQL with the column omitted, so only the schema can supply the
value — verified by setting defaultValue: 0, which turns exactly that one test
red and leaves the other two green.
Configuration.Tests 95/95.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central
can no longer see a site node's appsettings, so the transport ports it must
dial have to live in a row central can read.
AkkaPort is non-nullable with a 4053 default — every node listens on a
remoting port, so 0 is never a truthful value and pre-existing rows must
migrate to something real. GrpcPort is nullable with no default: nothing
listens on it until Phase 5, and a non-null default would assert a port that
does not exist.
Both are documented as central's dial targets rather than the node's own
binding config; the duplication against Cluster:Port is reconciled in Task 4.
The new SchemaCompliance test is red until the migration lands (Task 2).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.
The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.
Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.
Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
islanding a node that a manual failover had bounced: InitJoinAck received, no
Welcome inside the window because the peer's ring still held the old
incarnation, watchdog fired, second cluster. The TCP reachability guard patched
that one shape; the race stayed. It was also inert for every non-seed node, so
after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
through the production bootstrap at production failure-detection timings, incl.
a falsifiability control proving the OLD peer-first ordering never forms (that
test failed at 11s against the watchdog, which is what proved the retirement),
the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
cold start converging on ONE cluster.
Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
Corrects the plan's behavior table (the mid-join-handshake row was FALSE) and
notes the mesh-program Phase 6 convergence on ScadaBridge's self-first seed
ordering, which retires the watchdog + guard entirely.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The live gate for the manual-failover control caught the self-form fallback
forming a SECOND cluster. A node bounced by a failover restarted, received
InitJoinAck from its live peer — the join was in flight and healthy — but did
not get the Welcome inside the 10s window, because the peer's ring still held
the node's previous incarnation (Exiting -> Down -> Removed). The fallback
fired on the timer and the node islanded itself until an operator restarted it.
The original design assumed Cluster.Join(SelfAddress) would be ignored
mid-handshake. It is not — it wins. And since manual failover deliberately
produces that restart, every failover could island the node it bounced.
Before self-forming, TCP-probe the other seed addresses; a reachable peer means
wait another window instead. That is also what the fallback claims to detect:
'no seed answered InitJoin (peer down at boot)'.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Both change how someone should reason about this area, and the second changes a
public message contract, so they belong in CLAUDE.md rather than only in
docs/Redundancy.md. Includes the rule the work established: assert the effective
cluster config off a running ActorSystem, never the typed options or the
akka.conf text, because several HOCON fragments compete and the losing one still
reads correctly in the file.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h,
GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against
System.Security.Cryptography.Xml 10.0.7, which Microsoft.AspNetCore.DataProtection
10.0.7 pulls in transitively for key storage. Under TreatWarningsAsErrors a fresh
restore produces 204 NU1903 errors and the solution does not build at all.
This was invisible locally: machines with cached audit data keep building, so the
break only shows on a clean clone or a docker image build. It surfaced here by
accident — a detached worktree at the pre-change baseline, created to check
whether a flaky test predated today's work, could not restore. Worth noting that
the diagnostic path was the accident, not the intent: nothing in the normal build
or test loop would have reported this before someone else hit it.
Same surgical-pin pattern as the SQLitePCLRaw entry directly above it: a direct
PackageReference at the one project where the chain enters the repo
(Core.Configuration), rather than CentralPackageTransitivePinningEnabled, which
this repo already documents as breaking the Roslyn version split. Bumping the
DataProtection parent to 10.0.10 instead was rejected for the reason the sister
repo recorded when it hit the same advisories: 10.0.10 floors
Microsoft.Extensions.* and, via the EFCore adapter, Microsoft.EntityFrameworkCore
at 10.0.10, forcing a family-wide servicing bump and an NU1605 downgrade cascade
that deserves its own reviewed change.
Verified the way the defect demanded — in a fresh worktree, not this one: restore
clean (0 errors, down from 204) and full solution build clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The design doc still described the downing gap and the RoleLeader derivation as
open. Both shipped earlier today, so 6.2 and 4 now carry the fix and its pin
instead of the diagnosis, and 7 marks 0a/0b done. 0a's live gate stays open and
is explicitly deferred to phase 7: the failure only appears in a 1-vs-1 split and
docker-dev is a single six-node mesh, so there is no rig on which to drill it
until the meshes are actually split.
The phase 1 plan lands with one finding that changes its shape. The design pairs
"ClusterNode gains address columns" with "coordinator sources its expected-ack set
from the DB" as though they were one change; they are independent, and the second
is far smaller than it reads. ClusterNode.NodeId is ALREADY in the coordinator's
identifier space — the rig seeds central-1:4053 and so on, because
NodeDeploymentState.NodeId is FK-bound to it and the coordinator's
membership-derived ids already satisfy that FK. That the deploy path works today
is standing proof the two sets agree.
What is not small is the semantics. Membership-derived and DB-derived sets differ
on a configured-but-stopped node: today the deployment seals green without it —
telling the operator the fleet is deployed when it is not — and afterwards it
would fail at the apply deadline. Failing is the better behaviour and is required
once central cannot see cluster membership, but it needs an escape hatch or a node
down for maintenance blocks every deploy. ClusterNode.Enabled already is that
hatch, so the plan filters on it and gates the phase on a live check that proves
both halves: a stopped node fails the deploy legibly, and disabling it lets the
deploy seal.
The plan also adds a task the design does not have: AkkaPort duplicates the node's
own Cluster:Port with nothing making them agree, and an unenforced duplicated
address is the same shape as the Modbus/ModbusTcp and TwinCat/Focas drifts already
recorded here.
Phases 2-7 remain unstarted. They change the running data path and rewrite the
rig, and each needs its own plan and live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two independent housekeeping defects.
DriverTypeNamesGuardTests (3 failures, pre-existing). The guard discovers each
driver's *DriverFactoryExtensions.Register by reflection and invokes it, then
asserts the registered type-name set matches the DriverTypeNames constants. It
built the argument array positionally — registry first, null for everything after
— on the assumption that the remaining parameters were optional. Galaxy's are
not: its Register takes a REQUIRED ISecretResolver and guards it with
ArgumentNullException.ThrowIfNull, so the reflective call threw and all three
parity facts failed. OpcUaClient's equivalent parameter is optional, which is why
only Galaxy tripped it.
Worth stating plainly: the guard was not partially broken, it was completely
inert. The throw happened while building the registered set, so NO driver's
parity was ever checked — a genuine drift in any of the nine would have been
invisible behind the same three red tests. Arguments are now supplied by
parameter type, and a parameter the helper cannot satisfy fails with a message
naming the factory and the parameter instead of an opaque ArgumentNullException.
The stand-in resolver reports every secret absent, so if the "factory func is
never invoked" assumption is ever broken, the driver fails closed.
Core.Abstractions.Tests: 138 passed, 0 failed.
MSBuild node reuse. MSBuild leaves worker nodes resident between builds so the
next one starts warm; across this session's repeated solution builds and test
runs that reached 55 idle processes holding ~6 GB. That is not just untidy — the
integration suites assert on timeouts, so memory pressure produces failures
indistinguishable from real regressions, the same failure mode the Workstation-GC
fix addressed from a different direction.
Directory.Build.rsp now passes -nodeReuse:false. Measured on the same incremental
solution build: 15 residual nodes without it, 2 with, and elapsed time 6.12s vs
6.11s — no cost worth the memory on this machine.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.
Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.
Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.
The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.
Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.
Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
I reported this gap as still open, citing a requirements doc line and the PR that
made the failover drill honest about it. Both were stale: ScadaBridge closed it
three hours earlier in cf3bd52f by switching to auto-down. My check missed it
because I compared only one direction (main..origin/main) and never asked whether
local main was AHEAD of origin, so two unpushed commits were invisible to me.
The substance matters more than the correction. Their commit message records a
live-proven finding: Akka.NET 1.5.62 KeepOldest.OldestDecision only lets
down-if-alone rescue a side with >= 2 members, so in a 1-vs-1 split the survivor
takes DownReachable and downs ITSELF. OtOpcUa runs precisely that configuration —
akka.conf active-strategy keep-oldest with down-if-alone on, plus the matching
typed KeepOldestOption — which means a two-node pair cannot fail over when the
oldest node crashes. That is a total-outage defect inside the redundancy feature,
independent of the mesh design, and it is now Phase 0a: most urgent, and needing a
crash-the-oldest live gate because the failure only appears 1-vs-1.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Completes 835fc08c, which fixed the harness's DisposeAsync but left StopNodeBAsync
calling NodeB.DisposeAsync() directly — despite that commit's own message naming
it as carrying the same bug. Caught by re-reading the code rather than trusting
the commit message.
This one is not just about memory. IHost.DisposeAsync never invokes
IHostedService.StopAsync, so CoordinatedShutdown never runs and node B never
performs a graceful Cluster.Leave. Node A instead notices only when the failure
detector times the member out — a slower and materially different path from the
graceful leave the failover tests are written to exercise. The doc comment
asserted the opposite ("shuts down node B via DisposeAsync, which runs
CoordinatedShutdown"); it now says what actually happens and why.
Verified: the three failover suites that call it pass (5/5), and the project is
unchanged at 194 passed with only the AbCip fixture failure, which needs a docker
fixture that is not running.
Also checked the sibling harness while here: LocalDbPairHarness already stops each
host before disposing it, so it needed nothing.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.
The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).
Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.
The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.
The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 moves the alarm store-and-forward buffer off its standalone
alarm-historian.db into the consolidated LocalDb as a replicated alarm_sf_events
table, so a node that dies holding undelivered alarm history no longer takes it to
the grave. SqliteStoreAndForwardSink becomes LocalDbStoreAndForwardSink; the
breaking AlarmHistorian:DatabasePath key is removed, surviving only as the path
AlarmSfLegacyMigrator reads to copy a pre-consolidation queue across on first boot.
Because the buffer now replicates, exactly-once delivery could no longer rest on
the enqueue-side gate alone — the Secondary holds a full copy — so the drain gate
had to land in the same commit that registered the table, not as a later
refinement. Delivery is at-least-once across a failover by design; row ids are a
hash of the payload, so an event both nodes accept in a boot window converges to
one row.
The live gate on the docker-dev rig found four production defects, three of which
crash-looped every driver node before its first check could run: an empty
ServerHistorian:ApiKey and a UseTls/scheme mismatch both kill the host (the
validator had classified them as degrading, but the gateway client validates its
own options at construction), and plaintext h2c was unreachable entirely because
the adapter forwarded TLS-only options unconditionally. The fourth was Phase 2's
own: the drain deferred to a cluster-wide elected Primary while the queue is
pair-local, so on a fleet rig every node — including unpaired ones — suspended its
drain and nothing drained anywhere.
Also carries the DoD sweep, which found eight live sites still naming the deleted
sink including user-visible AdminUI text, and the design work that followed: a
per-cluster Akka mesh design mirroring ScadaBridge, and a recorded limitation that
the redundancy election is scoped per Akka cluster rather than per application
Cluster.
Full suite verified twice against a pre-branch baseline worktree: failure sets
identical, zero regressions. The first run showed two extra deploy-test timeouts
that proved to be resource starvation from a leaked test-host process, not a
regression — they pass isolated and did not recur.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Section 5 still listed "site-local-primary storage" among the things deliberately
NOT copied from ScadaBridge — which §6.1 now reverses, since driver nodes stop
connecting to the ConfigDb and read their configuration from LocalDb. Replaced
with an explicit copied/not-copied split, and narrowed the not-copied item to what
it should have said all along: their transient-only central alarm cache, which
OtOpcUa has no reason to adopt because it persists alarm history through the
historian and Phase 2's replicated buffer already survives a failover.
Also corrected a renumbering artefact — the risk about rewriting the docker-dev
rig cited Phase 4, but the mesh partition moved to Phase 6 when the two data-path
phases were inserted.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
(1) One central DB, driver nodes disconnected from it. Verified in ScadaBridge
before adopting: AddConfigurationDatabase is called at Program.cs:264, inside the
Central branch (89-478); the Site branch starts at 479, so site nodes never
register the central database at all. There is still exactly one authoritative
configuration database — it is central-only, and sites fetch from central and
cache locally.
The consequence worth naming: this promotes LocalDb from an outage cache to the
driver node's steady-state configuration store. Boot-from-cache stops being the
exceptional path and becomes the normal one. Phase 1's chunking, SHA-256
verification, retention and pair replication all carry over — but a cache defect
that used to be a degraded-mode bug becomes a total-availability one, so the #485
unreadable-artifact class now has a larger blast radius.
Audited the five driver-side ConfigDb consumers that must be re-homed. Two are
more than mechanical: EfAlarmConditionStateStore holds pair-local Part 9 condition
state and should follow the alarm S&F buffer into LocalDb, and DbHealthProbeActor
feeds ServiceLevel — a driver node with no DB to probe changes what a
client-visible value means.
(2) The keep-oldest gap: corrected the status rather than the conclusion. What was
resolved is the documentation and the drill, not the hole. Gitea PR #12 (closed)
carried T1 badc97af, which made the drill "measure the registered keep-oldest
outage instead of pretending recovery", and T2 d5364506, which dropped "the
undeliverable ~25s active-crash promise". The gap itself is still the registered
deferred keep-oldest topology decision on the 2026-07-08 master tracker, with no
open issue. Accepted here with the same posture, recorded as a known risk.
(3) Auth matches ScadaBridge for now — unauthenticated inter-cluster transports on
a trusted network, recorded as an accepted risk with the fail-closed interceptor
already in our tree noted as the template for whenever it is revisited.
Sequencing grows from six phases to eight; the two that change a running system's
data path rather than its wiring each get their own live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Design for review, mirroring the ScadaBridge hub-and-spoke shape. Researched from
the ScadaBridge tree directly rather than from its CLAUDE.md, which understates
the boundary in five ways: there are THREE transports (ClusterClient, gRPC, plus
token-gated HTTP), the gRPC direction is inverted (central dials INTO each site),
active node is the OLDEST Up member and explicitly never the cluster leader, all
clusters share one ActorSystem name, and site nodes carry two roles.
Records a defect worth fixing regardless of the topology decision: OtOpcUa uses
two different node-selection rules at once. SBR is keep-oldest and singletons are
placed on the oldest member, but RedundancyStateActor elects Primary from
RoleLeader("driver") — lowest address. Those diverge permanently after a
restart-and-rejoin, so the node the SBR protects and hosts every singleton on need
not be the node the data-plane gates consider Primary. ScadaBridge forbids that
rule by name, for the reason it gives: both sides claim leadership during a
partition, which is the dual-primary shape archreview 03/S4 exists to prevent.
Three open questions are left explicitly undecided: whether to adopt their
autonomous-site data architecture or keep the shared ConfigDb; what to do about
the registered two-node keep-oldest total-outage gap they acknowledge but have not
closed; and whether to authenticate transports they left unauthenticated.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Corrects two claims made while writing the Phase 2 gate record.
The docker-dev rig is NOT misconfigured: central-1/central-2 carry admin,driver
deliberately — the compose says so — because they are themselves a pair that runs
the MAIN cluster's drivers.
And a redundant pair is not a missing concept. The application Cluster (ClusterId)
already is one: it owns drivers, devices and ClusterNode rows, and Phase 1's
deployment-artifact cache is keyed by it precisely so a pair shares one entry.
So the root cause is wider than the alarm drain. RedundancyStateActor elects one
Primary per AKKA cluster while every real unit of redundancy is an APPLICATION
cluster, and ServiceLevel, the alerts emit gate and the inbound device-write gate
all inherit that mis-scoping. On a fleet of pairs in one Akka cluster, exactly one
driver node cluster-wide services Primary-gated work.
Recommendation revised in the gate doc: fix the election scope (its own change,
its own gate, because it touches the safety-critical write gate), and explicitly
do NOT derive pair identity from LocalDb:Replication:PeerAddress — that would add
a second parallel notion of 'my partner' that the real fix obsoletes, and the
library has no initiator flag, so both halves would have to dial just to obtain a
config value.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.
Three defects crash-looped every driver node before check 1 could even run:
1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
Validator exists to turn exactly that class of failure into a named
OptionsValidationException, but its documented fail tier explicitly excluded
ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
calls". It does not: the client validates its own options at construction, so
the process dies during Akka startup and never makes a call.
2. UseTls disagreeing with the endpoint scheme kills the host too, in both
directions (both messages confirmed in the shipped client assembly). Moving an
endpoint from https to http without clearing UseTls is an ordinary migration
slip.
3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
to false, so it always sent RequireCertificateValidation=true — which the
client rejects outright when UseTls=false. Every http:// deployment crashed,
though the scheme is documented as the supported way to select h2c, and the
only workaround was to assert a certificate posture for a connection that has
no certificate.
The fourth was the blocker, and it is Phase 2's own:
4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
driver Primary is central-1 — it carries the driver Akka role, replicates
nobody's LocalDb and does not even run the alarm historian — so every driver
node logged "Historian drain suspended", including the two site-b nodes that
have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
fine. The cost is not a duplicate; it is the buffer growing to the capacity
wall and evicting the audit trail it exists to protect.
Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
drains; the two gates now deliberately disagree, and a test pins that); peer-
host matching in DriverHostActor so a node stands down only for a Primary
holding its rows; and AddAlarmHistorian short-circuiting the gate when
replication is unconfigured — testing BOTH Replication:PeerAddress and
SyncListenPort, since only the dialing half sets the former while both halves
share the queue.
Every one of these follows from the asymmetry: a false allow costs a duplicate
row, which at-least-once delivery already accepts and payload-hash ids
collapse; a false deny loses data silently.
A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.
Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.
Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.
Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline 2e46d054. The two failure SETS are identical -- all 13
tests, same names, zero new failures. Net +26 tests: +3 Core.AlarmHistorian
(drain gate), +6 Runtime (role view), +17 Host.IntegrationTests (migrator +
convergence). Set comparison rather than counts, because the suite carries
standing environment- and load-dependent failures a count would hide.
The greps found real drift Task 6 missed -- eight live sites still naming the
deleted SqliteStoreAndForwardSink, including the AdminUI /alarms/historian
panel text, which is user-visible, and a <see cref> in HistorianAdapterActor
that resolved to nothing without warning. All repointed at
LocalDbStoreAndForwardSink; CLAUDE.md's alarm-history paragraph now also
records the LocalDb buffer and the primary-gated drain, and drops DatabasePath
from the knob list. docs/AlarmTracking.md still promised an
AlarmHistorianOptions.Validate() startup warning for a relative DatabasePath
and an empty SharedSecret; both branches are gone, so it now says so.
Code references to AlarmHistorian:DatabasePath reduce to exactly two intentional
ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No `new SqliteConnection`
remains anywhere in Core.AlarmHistorian.
Recon doc gains the durable verification record: guard-deletion evidence for
both vacuous passes, the two exact-set replicated-table pins (both assert set
equality, so an added or a dropped registration fails), and the baseline test
comparison with a per-failure account of why each of the 13 is not this
branch's.
Stops here per the plan. Task 8's live gate needs explicit go-ahead; nothing on
this branch is to be merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
D-6 (payload-hash migrator ids over mig-{node}-{legacyId}) and D-7 (the
plan's Host.Tests project does not exist) were captured only in commit
messages and the tasks file. They belong with the other deviations, where
the next reader looks.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.
Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.
Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Moves the alarm store-and-forward buffer out of its own alarm-historian.db
and into the node's consolidated LocalDb, where it replicates to the
redundant pair peer. A node that dies holding undelivered alarm history no
longer takes it to the grave.
Tasks 2 and 3 land together, as the plan anticipated. They are not separable:
the gate is a constructor argument of the rewritten sink, and a commit that
replicated the queue without gating the drain would be a commit in which both
nodes of a pair deliver every alarm event, continuously.
That drain gate is the load-bearing part of the change, and the recon
explains why it is new work rather than a refinement. Exactly-once delivery
across a pair is enforced today on the ENQUEUE side, by
HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is
empty and its ungated drain has nothing to send. Replicating the table
destroys that invariant.
The gate is a Func<bool> the caller supplies, because the drain runs on a
timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot
reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new
IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate
verdict to -- the same verdict the inbound-write and native-ack gates use, so
there is no second notion of am-I-the-Primary to drift.
Two failure modes are deliberately closed:
- The view is seeded OPEN, matching the policy's own answer for an unknown
role with no driver peer. A deployment that runs no redundancy never
publishes to it, and defaulting closed would silently stop its alarm
history forever.
- A gate that throws is read as not-now, never as permission, and a closed
gate reports the new HistorianDrainState.NotPrimary rather than Idle. A
Secondary's rising queue is supposed to look different from a stalled
drain, and if BOTH nodes report NotPrimary the pair is misconfigured and
says so instead of quietly filling toward the capacity ceiling.
Row ids are a hash of the payload rather than fresh GUIDs. Both adapters
accept the same fanned transition in the window before the first redundancy
snapshot arrives, and under last-writer-wins an equal key converges those two
accepts into one row instead of duplicating them.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and
registers it for replication, so a node that dies with undelivered alarm
history no longer takes it to the grave.
The table keeps the legacy queue's shape rather than the status column the
plan sketched. An acknowledged row is deleted, not marked delivered: that
needs no second sweeper to keep the table bounded, leaves the capacity
semantics untouched, and the replication engine carries the delete as a
tombstone so the peer drops its copy anyway -- which is what the status
column was for. last_error is retained because it is the only operator-facing
record of why a row was dead-lettered.
The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot
replicate under last-writer-wins: two nodes would independently allocate
rowid 7 to different alarms and silently overwrite each other. The drain
index therefore orders by enqueued_at_utc rather than insertion order, with
id as a tiebreak so the ordering is total.
Tables are created unconditionally, independent of AlarmHistorian:Enabled.
An empty registered table costs three triggers; creating it lazily would mean
a node that enables the historian later writes rows before its capture
triggers exist, which is exactly the silent-loss shape OnReady's ordering
comment warns about.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Answers Task 0 with file:line citations against master 2e46d054.
The STOP condition does not fire: PayloadJson is TEXT, no BLOB column, so
alarm_sf_events can be registered for replication as-is.
Two findings reshape the plan's assumptions:
- The drain worker is not a hosted service or an actor. It is a
self-rescheduling Timer owned by the sink and started from the DI
factory, so the Primary gate has to live inside DrainOnceAsync as an
injected delegate -- Core.AlarmHistorian cannot reference
PrimaryGatePolicy, which lives in Runtime.
- A Primary gate already exists on the enqueue side
(HistorianAdapterActor.ShouldHistorize), using a different policy than
the one the plan specifies for the drain. That gate is why today's
ungated drain is safe: the Secondary never enqueues, so its queue is
empty. Replicating the table breaks that invariant, which makes the
drain gate mandatory in the same commit rather than a refinement.
Records deviations D-1 (deterministic content-hash ids so a boot-window
double-enqueue converges instead of duplicating), D-2 (gate denial must be
observable or a snapshot identity mismatch silently stops alarm history),
D-3 (Tasks 2+3 land as one commit) and D-4 (the rig will dead-letter in
~5 minutes at MaxAttempts 10 with no gateway).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2's recon carries a STOP condition: a BLOB payload column in the legacy
alarm store-and-forward table could not be registered for replication and would
have to become base64 TEXT, sized against the chunk guidance. Resolved ahead of
execution so the answer survives into the Phase 2 session:
- No BLOB. All 8 columns are TEXT/INTEGER; the payload is PayloadJson TEXT NOT
NULL. No base64 conversion, no chunk sizing needed.
- The executed DDL (SqliteStoreAndForwardSink.cs:657-667) matches the class
doc-comment at :17-26 exactly — checked because a doc comment is not evidence.
- Payload is bounded by shape: a serialized AlarmHistorianEvent is 10 scalar
fields, no collections or nesting, so realistic worst case is low single-digit
KB against a 171 KB guidance.
- The PK IS autoincrement, confirming the plan's prediction that the migrator
must mint deterministic mig-{node}-{legacyId} ids and the new table needs a
TEXT GUID PK.
- Also captured for reuse: the drain's covering index (DeadLettered, RowId), and
the exact legacy column list the migrator's pragma_table_info intersection
check compares against.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Deferred work was living in plan docs and a CLAUDE.md banner rather than the
issue tracker, so it was invisible unless you happened to read the right file.
Each item was verified against the current tree before filing — nothing
speculative, nothing already fixed:
- #487 scripted-alarm redeploy recovery (CONFIRMED pre-existing bug). Verified
still unfixed: ReassertValue exists only for VirtualTags, there is no
OnAlarmsLoaded equivalent. Carries double-alarm-history risk, hence the
careful proposed shape (node-only re-assert, never the alerts topic).
- #488 periodic desired-vs-actual subscription reconcile. Records what exists
today (Connected-entry re-assert + a health-poll tick that never touches
subscriptions) and why a true reconcile needs a new ISubscribable member
across 7 drivers, plus the cheap no-interface-change variant to try first.
- #489 raw-tag rename hot-rebind (today: deploy-THEN-recreate).
- #490 absolute-RawPath Monaco tag-path completion.
- #491 continuous-historization live end-to-end + restart-convergence gate.
CLAUDE.md's "KNOWN LIMITATION 1" was STALE and told the reader not to trust a
cutover that has in fact been live-validated (read/write-persist/alarm-send all
green; alarm-readback skipped as a confirmed protocol limitation, not a bug) —
evidence in docs/plans/2026-06-27-otopcua-historian-followups.md. Rewritten as
a passed gate, keeping the recipe for re-running it. Limitation 2 is genuinely
open and now points at #491.
The cross-repo ScadaBridge cutover was already tracked in that repo (its #14
open, #20 closed), so no duplicate was filed; the Batch 4 doc now says so, and
records that the umbrella-index half is done.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty