Both change how someone should reason about this area, and the second changes a
public message contract, so they belong in CLAUDE.md rather than only in
docs/Redundancy.md. Includes the rule the work established: assert the effective
cluster config off a running ActorSystem, never the typed options or the
akka.conf text, because several HOCON fragments compete and the losing one still
reads correctly in the file.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h,
GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against
System.Security.Cryptography.Xml 10.0.7, which Microsoft.AspNetCore.DataProtection
10.0.7 pulls in transitively for key storage. Under TreatWarningsAsErrors a fresh
restore produces 204 NU1903 errors and the solution does not build at all.
This was invisible locally: machines with cached audit data keep building, so the
break only shows on a clean clone or a docker image build. It surfaced here by
accident — a detached worktree at the pre-change baseline, created to check
whether a flaky test predated today's work, could not restore. Worth noting that
the diagnostic path was the accident, not the intent: nothing in the normal build
or test loop would have reported this before someone else hit it.
Same surgical-pin pattern as the SQLitePCLRaw entry directly above it: a direct
PackageReference at the one project where the chain enters the repo
(Core.Configuration), rather than CentralPackageTransitivePinningEnabled, which
this repo already documents as breaking the Roslyn version split. Bumping the
DataProtection parent to 10.0.10 instead was rejected for the reason the sister
repo recorded when it hit the same advisories: 10.0.10 floors
Microsoft.Extensions.* and, via the EFCore adapter, Microsoft.EntityFrameworkCore
at 10.0.10, forcing a family-wide servicing bump and an NU1605 downgrade cascade
that deserves its own reviewed change.
Verified the way the defect demanded — in a fresh worktree, not this one: restore
clean (0 errors, down from 204) and full solution build clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The design doc still described the downing gap and the RoleLeader derivation as
open. Both shipped earlier today, so 6.2 and 4 now carry the fix and its pin
instead of the diagnosis, and 7 marks 0a/0b done. 0a's live gate stays open and
is explicitly deferred to phase 7: the failure only appears in a 1-vs-1 split and
docker-dev is a single six-node mesh, so there is no rig on which to drill it
until the meshes are actually split.
The phase 1 plan lands with one finding that changes its shape. The design pairs
"ClusterNode gains address columns" with "coordinator sources its expected-ack set
from the DB" as though they were one change; they are independent, and the second
is far smaller than it reads. ClusterNode.NodeId is ALREADY in the coordinator's
identifier space — the rig seeds central-1:4053 and so on, because
NodeDeploymentState.NodeId is FK-bound to it and the coordinator's
membership-derived ids already satisfy that FK. That the deploy path works today
is standing proof the two sets agree.
What is not small is the semantics. Membership-derived and DB-derived sets differ
on a configured-but-stopped node: today the deployment seals green without it —
telling the operator the fleet is deployed when it is not — and afterwards it
would fail at the apply deadline. Failing is the better behaviour and is required
once central cannot see cluster membership, but it needs an escape hatch or a node
down for maintenance blocks every deploy. ClusterNode.Enabled already is that
hatch, so the plan filters on it and gates the phase on a live check that proves
both halves: a stopped node fails the deploy legibly, and disabling it lets the
deploy seal.
The plan also adds a task the design does not have: AkkaPort duplicates the node's
own Cluster:Port with nothing making them agree, and an unenforced duplicated
address is the same shape as the Modbus/ModbusTcp and TwinCat/Focas drifts already
recorded here.
Phases 2-7 remain unstarted. They change the running data path and rewrite the
rig, and each needs its own plan and live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two independent housekeeping defects.
DriverTypeNamesGuardTests (3 failures, pre-existing). The guard discovers each
driver's *DriverFactoryExtensions.Register by reflection and invokes it, then
asserts the registered type-name set matches the DriverTypeNames constants. It
built the argument array positionally — registry first, null for everything after
— on the assumption that the remaining parameters were optional. Galaxy's are
not: its Register takes a REQUIRED ISecretResolver and guards it with
ArgumentNullException.ThrowIfNull, so the reflective call threw and all three
parity facts failed. OpcUaClient's equivalent parameter is optional, which is why
only Galaxy tripped it.
Worth stating plainly: the guard was not partially broken, it was completely
inert. The throw happened while building the registered set, so NO driver's
parity was ever checked — a genuine drift in any of the nine would have been
invisible behind the same three red tests. Arguments are now supplied by
parameter type, and a parameter the helper cannot satisfy fails with a message
naming the factory and the parameter instead of an opaque ArgumentNullException.
The stand-in resolver reports every secret absent, so if the "factory func is
never invoked" assumption is ever broken, the driver fails closed.
Core.Abstractions.Tests: 138 passed, 0 failed.
MSBuild node reuse. MSBuild leaves worker nodes resident between builds so the
next one starts warm; across this session's repeated solution builds and test
runs that reached 55 idle processes holding ~6 GB. That is not just untidy — the
integration suites assert on timeouts, so memory pressure produces failures
indistinguishable from real regressions, the same failure mode the Workstation-GC
fix addressed from a different direction.
Directory.Build.rsp now passes -nodeReuse:false. Measured on the same incremental
solution build: 15 residual nodes without it, 2 with, and elapsed time 6.12s vs
6.11s — no cost worth the memory on this machine.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.
Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.
Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.
The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.
Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.
Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
I reported this gap as still open, citing a requirements doc line and the PR that
made the failover drill honest about it. Both were stale: ScadaBridge closed it
three hours earlier in cf3bd52f by switching to auto-down. My check missed it
because I compared only one direction (main..origin/main) and never asked whether
local main was AHEAD of origin, so two unpushed commits were invisible to me.
The substance matters more than the correction. Their commit message records a
live-proven finding: Akka.NET 1.5.62 KeepOldest.OldestDecision only lets
down-if-alone rescue a side with >= 2 members, so in a 1-vs-1 split the survivor
takes DownReachable and downs ITSELF. OtOpcUa runs precisely that configuration —
akka.conf active-strategy keep-oldest with down-if-alone on, plus the matching
typed KeepOldestOption — which means a two-node pair cannot fail over when the
oldest node crashes. That is a total-outage defect inside the redundancy feature,
independent of the mesh design, and it is now Phase 0a: most urgent, and needing a
crash-the-oldest live gate because the failure only appears 1-vs-1.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Completes 835fc08c, which fixed the harness's DisposeAsync but left StopNodeBAsync
calling NodeB.DisposeAsync() directly — despite that commit's own message naming
it as carrying the same bug. Caught by re-reading the code rather than trusting
the commit message.
This one is not just about memory. IHost.DisposeAsync never invokes
IHostedService.StopAsync, so CoordinatedShutdown never runs and node B never
performs a graceful Cluster.Leave. Node A instead notices only when the failure
detector times the member out — a slower and materially different path from the
graceful leave the failover tests are written to exercise. The doc comment
asserted the opposite ("shuts down node B via DisposeAsync, which runs
CoordinatedShutdown"); it now says what actually happens and why.
Verified: the three failover suites that call it pass (5/5), and the project is
unchanged at 194 passed with only the AbCip fixture failure, which needs a docker
fixture that is not running.
Also checked the sibling harness while here: LocalDbPairHarness already stops each
host before disposing it, so it needed nothing.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.
The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).
Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.
The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.
The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 moves the alarm store-and-forward buffer off its standalone
alarm-historian.db into the consolidated LocalDb as a replicated alarm_sf_events
table, so a node that dies holding undelivered alarm history no longer takes it to
the grave. SqliteStoreAndForwardSink becomes LocalDbStoreAndForwardSink; the
breaking AlarmHistorian:DatabasePath key is removed, surviving only as the path
AlarmSfLegacyMigrator reads to copy a pre-consolidation queue across on first boot.
Because the buffer now replicates, exactly-once delivery could no longer rest on
the enqueue-side gate alone — the Secondary holds a full copy — so the drain gate
had to land in the same commit that registered the table, not as a later
refinement. Delivery is at-least-once across a failover by design; row ids are a
hash of the payload, so an event both nodes accept in a boot window converges to
one row.
The live gate on the docker-dev rig found four production defects, three of which
crash-looped every driver node before its first check could run: an empty
ServerHistorian:ApiKey and a UseTls/scheme mismatch both kill the host (the
validator had classified them as degrading, but the gateway client validates its
own options at construction), and plaintext h2c was unreachable entirely because
the adapter forwarded TLS-only options unconditionally. The fourth was Phase 2's
own: the drain deferred to a cluster-wide elected Primary while the queue is
pair-local, so on a fleet rig every node — including unpaired ones — suspended its
drain and nothing drained anywhere.
Also carries the DoD sweep, which found eight live sites still naming the deleted
sink including user-visible AdminUI text, and the design work that followed: a
per-cluster Akka mesh design mirroring ScadaBridge, and a recorded limitation that
the redundancy election is scoped per Akka cluster rather than per application
Cluster.
Full suite verified twice against a pre-branch baseline worktree: failure sets
identical, zero regressions. The first run showed two extra deploy-test timeouts
that proved to be resource starvation from a leaked test-host process, not a
regression — they pass isolated and did not recur.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Section 5 still listed "site-local-primary storage" among the things deliberately
NOT copied from ScadaBridge — which §6.1 now reverses, since driver nodes stop
connecting to the ConfigDb and read their configuration from LocalDb. Replaced
with an explicit copied/not-copied split, and narrowed the not-copied item to what
it should have said all along: their transient-only central alarm cache, which
OtOpcUa has no reason to adopt because it persists alarm history through the
historian and Phase 2's replicated buffer already survives a failover.
Also corrected a renumbering artefact — the risk about rewriting the docker-dev
rig cited Phase 4, but the mesh partition moved to Phase 6 when the two data-path
phases were inserted.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
(1) One central DB, driver nodes disconnected from it. Verified in ScadaBridge
before adopting: AddConfigurationDatabase is called at Program.cs:264, inside the
Central branch (89-478); the Site branch starts at 479, so site nodes never
register the central database at all. There is still exactly one authoritative
configuration database — it is central-only, and sites fetch from central and
cache locally.
The consequence worth naming: this promotes LocalDb from an outage cache to the
driver node's steady-state configuration store. Boot-from-cache stops being the
exceptional path and becomes the normal one. Phase 1's chunking, SHA-256
verification, retention and pair replication all carry over — but a cache defect
that used to be a degraded-mode bug becomes a total-availability one, so the #485
unreadable-artifact class now has a larger blast radius.
Audited the five driver-side ConfigDb consumers that must be re-homed. Two are
more than mechanical: EfAlarmConditionStateStore holds pair-local Part 9 condition
state and should follow the alarm S&F buffer into LocalDb, and DbHealthProbeActor
feeds ServiceLevel — a driver node with no DB to probe changes what a
client-visible value means.
(2) The keep-oldest gap: corrected the status rather than the conclusion. What was
resolved is the documentation and the drill, not the hole. Gitea PR #12 (closed)
carried T1 badc97af, which made the drill "measure the registered keep-oldest
outage instead of pretending recovery", and T2 d5364506, which dropped "the
undeliverable ~25s active-crash promise". The gap itself is still the registered
deferred keep-oldest topology decision on the 2026-07-08 master tracker, with no
open issue. Accepted here with the same posture, recorded as a known risk.
(3) Auth matches ScadaBridge for now — unauthenticated inter-cluster transports on
a trusted network, recorded as an accepted risk with the fail-closed interceptor
already in our tree noted as the template for whenever it is revisited.
Sequencing grows from six phases to eight; the two that change a running system's
data path rather than its wiring each get their own live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Design for review, mirroring the ScadaBridge hub-and-spoke shape. Researched from
the ScadaBridge tree directly rather than from its CLAUDE.md, which understates
the boundary in five ways: there are THREE transports (ClusterClient, gRPC, plus
token-gated HTTP), the gRPC direction is inverted (central dials INTO each site),
active node is the OLDEST Up member and explicitly never the cluster leader, all
clusters share one ActorSystem name, and site nodes carry two roles.
Records a defect worth fixing regardless of the topology decision: OtOpcUa uses
two different node-selection rules at once. SBR is keep-oldest and singletons are
placed on the oldest member, but RedundancyStateActor elects Primary from
RoleLeader("driver") — lowest address. Those diverge permanently after a
restart-and-rejoin, so the node the SBR protects and hosts every singleton on need
not be the node the data-plane gates consider Primary. ScadaBridge forbids that
rule by name, for the reason it gives: both sides claim leadership during a
partition, which is the dual-primary shape archreview 03/S4 exists to prevent.
Three open questions are left explicitly undecided: whether to adopt their
autonomous-site data architecture or keep the shared ConfigDb; what to do about
the registered two-node keep-oldest total-outage gap they acknowledge but have not
closed; and whether to authenticate transports they left unauthenticated.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Corrects two claims made while writing the Phase 2 gate record.
The docker-dev rig is NOT misconfigured: central-1/central-2 carry admin,driver
deliberately — the compose says so — because they are themselves a pair that runs
the MAIN cluster's drivers.
And a redundant pair is not a missing concept. The application Cluster (ClusterId)
already is one: it owns drivers, devices and ClusterNode rows, and Phase 1's
deployment-artifact cache is keyed by it precisely so a pair shares one entry.
So the root cause is wider than the alarm drain. RedundancyStateActor elects one
Primary per AKKA cluster while every real unit of redundancy is an APPLICATION
cluster, and ServiceLevel, the alerts emit gate and the inbound device-write gate
all inherit that mis-scoping. On a fleet of pairs in one Akka cluster, exactly one
driver node cluster-wide services Primary-gated work.
Recommendation revised in the gate doc: fix the election scope (its own change,
its own gate, because it touches the safety-critical write gate), and explicitly
do NOT derive pair identity from LocalDb:Replication:PeerAddress — that would add
a second parallel notion of 'my partner' that the real fix obsoletes, and the
library has no initiator flag, so both halves would have to dial just to obtain a
config value.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.
Three defects crash-looped every driver node before check 1 could even run:
1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
Validator exists to turn exactly that class of failure into a named
OptionsValidationException, but its documented fail tier explicitly excluded
ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
calls". It does not: the client validates its own options at construction, so
the process dies during Akka startup and never makes a call.
2. UseTls disagreeing with the endpoint scheme kills the host too, in both
directions (both messages confirmed in the shipped client assembly). Moving an
endpoint from https to http without clearing UseTls is an ordinary migration
slip.
3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
to false, so it always sent RequireCertificateValidation=true — which the
client rejects outright when UseTls=false. Every http:// deployment crashed,
though the scheme is documented as the supported way to select h2c, and the
only workaround was to assert a certificate posture for a connection that has
no certificate.
The fourth was the blocker, and it is Phase 2's own:
4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
driver Primary is central-1 — it carries the driver Akka role, replicates
nobody's LocalDb and does not even run the alarm historian — so every driver
node logged "Historian drain suspended", including the two site-b nodes that
have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
fine. The cost is not a duplicate; it is the buffer growing to the capacity
wall and evicting the audit trail it exists to protect.
Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
drains; the two gates now deliberately disagree, and a test pins that); peer-
host matching in DriverHostActor so a node stands down only for a Primary
holding its rows; and AddAlarmHistorian short-circuiting the gate when
replication is unconfigured — testing BOTH Replication:PeerAddress and
SyncListenPort, since only the dialing half sets the former while both halves
share the queue.
Every one of these follows from the asymmetry: a false allow costs a duplicate
row, which at-least-once delivery already accepts and payload-hash ids
collapse; a false deny loses data silently.
A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.
Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.
Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.
Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline 2e46d054. The two failure SETS are identical -- all 13
tests, same names, zero new failures. Net +26 tests: +3 Core.AlarmHistorian
(drain gate), +6 Runtime (role view), +17 Host.IntegrationTests (migrator +
convergence). Set comparison rather than counts, because the suite carries
standing environment- and load-dependent failures a count would hide.
The greps found real drift Task 6 missed -- eight live sites still naming the
deleted SqliteStoreAndForwardSink, including the AdminUI /alarms/historian
panel text, which is user-visible, and a <see cref> in HistorianAdapterActor
that resolved to nothing without warning. All repointed at
LocalDbStoreAndForwardSink; CLAUDE.md's alarm-history paragraph now also
records the LocalDb buffer and the primary-gated drain, and drops DatabasePath
from the knob list. docs/AlarmTracking.md still promised an
AlarmHistorianOptions.Validate() startup warning for a relative DatabasePath
and an empty SharedSecret; both branches are gone, so it now says so.
Code references to AlarmHistorian:DatabasePath reduce to exactly two intentional
ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No `new SqliteConnection`
remains anywhere in Core.AlarmHistorian.
Recon doc gains the durable verification record: guard-deletion evidence for
both vacuous passes, the two exact-set replicated-table pins (both assert set
equality, so an added or a dropped registration fails), and the baseline test
comparison with a per-failure account of why each of the 13 is not this
branch's.
Stops here per the plan. Task 8's live gate needs explicit go-ahead; nothing on
this branch is to be merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
D-6 (payload-hash migrator ids over mig-{node}-{legacyId}) and D-7 (the
plan's Host.Tests project does not exist) were captured only in commit
messages and the tasks file. They belong with the other deviations, where
the next reader looks.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.
Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.
Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Moves the alarm store-and-forward buffer out of its own alarm-historian.db
and into the node's consolidated LocalDb, where it replicates to the
redundant pair peer. A node that dies holding undelivered alarm history no
longer takes it to the grave.
Tasks 2 and 3 land together, as the plan anticipated. They are not separable:
the gate is a constructor argument of the rewritten sink, and a commit that
replicated the queue without gating the drain would be a commit in which both
nodes of a pair deliver every alarm event, continuously.
That drain gate is the load-bearing part of the change, and the recon
explains why it is new work rather than a refinement. Exactly-once delivery
across a pair is enforced today on the ENQUEUE side, by
HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is
empty and its ungated drain has nothing to send. Replicating the table
destroys that invariant.
The gate is a Func<bool> the caller supplies, because the drain runs on a
timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot
reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new
IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate
verdict to -- the same verdict the inbound-write and native-ack gates use, so
there is no second notion of am-I-the-Primary to drift.
Two failure modes are deliberately closed:
- The view is seeded OPEN, matching the policy's own answer for an unknown
role with no driver peer. A deployment that runs no redundancy never
publishes to it, and defaulting closed would silently stop its alarm
history forever.
- A gate that throws is read as not-now, never as permission, and a closed
gate reports the new HistorianDrainState.NotPrimary rather than Idle. A
Secondary's rising queue is supposed to look different from a stalled
drain, and if BOTH nodes report NotPrimary the pair is misconfigured and
says so instead of quietly filling toward the capacity ceiling.
Row ids are a hash of the payload rather than fresh GUIDs. Both adapters
accept the same fanned transition in the window before the first redundancy
snapshot arrives, and under last-writer-wins an equal key converges those two
accepts into one row instead of duplicating them.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and
registers it for replication, so a node that dies with undelivered alarm
history no longer takes it to the grave.
The table keeps the legacy queue's shape rather than the status column the
plan sketched. An acknowledged row is deleted, not marked delivered: that
needs no second sweeper to keep the table bounded, leaves the capacity
semantics untouched, and the replication engine carries the delete as a
tombstone so the peer drops its copy anyway -- which is what the status
column was for. last_error is retained because it is the only operator-facing
record of why a row was dead-lettered.
The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot
replicate under last-writer-wins: two nodes would independently allocate
rowid 7 to different alarms and silently overwrite each other. The drain
index therefore orders by enqueued_at_utc rather than insertion order, with
id as a tiebreak so the ordering is total.
Tables are created unconditionally, independent of AlarmHistorian:Enabled.
An empty registered table costs three triggers; creating it lazily would mean
a node that enables the historian later writes rows before its capture
triggers exist, which is exactly the silent-loss shape OnReady's ordering
comment warns about.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Answers Task 0 with file:line citations against master 2e46d054.
The STOP condition does not fire: PayloadJson is TEXT, no BLOB column, so
alarm_sf_events can be registered for replication as-is.
Two findings reshape the plan's assumptions:
- The drain worker is not a hosted service or an actor. It is a
self-rescheduling Timer owned by the sink and started from the DI
factory, so the Primary gate has to live inside DrainOnceAsync as an
injected delegate -- Core.AlarmHistorian cannot reference
PrimaryGatePolicy, which lives in Runtime.
- A Primary gate already exists on the enqueue side
(HistorianAdapterActor.ShouldHistorize), using a different policy than
the one the plan specifies for the drain. That gate is why today's
ungated drain is safe: the Secondary never enqueues, so its queue is
empty. Replicating the table breaks that invariant, which makes the
drain gate mandatory in the same commit rather than a refinement.
Records deviations D-1 (deterministic content-hash ids so a boot-window
double-enqueue converges instead of duplicating), D-2 (gate denial must be
observable or a snapshot identity mismatch silently stops alarm history),
D-3 (Tasks 2+3 land as one commit) and D-4 (the rig will dead-letter in
~5 minutes at MaxAttempts 10 with no gateway).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2's recon carries a STOP condition: a BLOB payload column in the legacy
alarm store-and-forward table could not be registered for replication and would
have to become base64 TEXT, sized against the chunk guidance. Resolved ahead of
execution so the answer survives into the Phase 2 session:
- No BLOB. All 8 columns are TEXT/INTEGER; the payload is PayloadJson TEXT NOT
NULL. No base64 conversion, no chunk sizing needed.
- The executed DDL (SqliteStoreAndForwardSink.cs:657-667) matches the class
doc-comment at :17-26 exactly — checked because a doc comment is not evidence.
- Payload is bounded by shape: a serialized AlarmHistorianEvent is 10 scalar
fields, no collections or nesting, so realistic worst case is low single-digit
KB against a 171 KB guidance.
- The PK IS autoincrement, confirming the plan's prediction that the migrator
must mint deterministic mig-{node}-{legacyId} ids and the new table needs a
TEXT GUID PK.
- Also captured for reuse: the drain's covering index (DeadLettered, RowId), and
the exact legacy column list the migrator's pragma_table_info intersection
check compares against.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Deferred work was living in plan docs and a CLAUDE.md banner rather than the
issue tracker, so it was invisible unless you happened to read the right file.
Each item was verified against the current tree before filing — nothing
speculative, nothing already fixed:
- #487 scripted-alarm redeploy recovery (CONFIRMED pre-existing bug). Verified
still unfixed: ReassertValue exists only for VirtualTags, there is no
OnAlarmsLoaded equivalent. Carries double-alarm-history risk, hence the
careful proposed shape (node-only re-assert, never the alerts topic).
- #488 periodic desired-vs-actual subscription reconcile. Records what exists
today (Connected-entry re-assert + a health-poll tick that never touches
subscriptions) and why a true reconcile needs a new ISubscribable member
across 7 drivers, plus the cheap no-interface-change variant to try first.
- #489 raw-tag rename hot-rebind (today: deploy-THEN-recreate).
- #490 absolute-RawPath Monaco tag-path completion.
- #491 continuous-historization live end-to-end + restart-convergence gate.
CLAUDE.md's "KNOWN LIMITATION 1" was STALE and told the reader not to trust a
cutover that has in fact been live-validated (read/write-persist/alarm-send all
green; alarm-readback skipped as a confirmed protocol limitation, not a bug) —
evidence in docs/plans/2026-06-27-otopcua-historian-followups.md. Rewritten as
a passed gate, keeping the recipe for re-running it. Limitation 2 is genuinely
open and now points at #491.
The cross-repo ScadaBridge cutover was already tracked in that repo (its #14
open, #20 closed), so no duplicate was filed; the Batch 4 doc now says so, and
records that the umbrella-index half is done.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Regression check first, since this changes ACK semantics: a normal deploy still
records Applied on all six nodes. Then the same induction as #485 — the node
now records Failed with a reason while the five nodes that read the real
artifact record Applied (pre-fix all six claimed Applied), and the log shows it
staying on the PREVIOUS revision rather than the dispatched one, which is what
lets a retry land. Values kept flowing throughout (59603, current), and the
next real deploy applied normally.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ApplyAndAck advanced _currentRevision and recorded NodeDeploymentState=Applied
immediately after ReconcileDrivers, which returns null when the artifact could
not be read. So a node that applied NOTHING reported success, claimed a
revision whose configuration it never applied, and — because
HandleDispatchFromSteady short-circuits on a revision match — could never be
healed by re-dispatching that revision. Only a later, different revision
recovered it.
Now a null blob fails the apply: the revision is left where it was,
NodeDeploymentState records Failed with the reason, and the coordinator gets a
Failed ACK. Leaving the revision alone is what makes the retry actually land
instead of being waved through.
This is about telling the truth, not about tearing anything down — the node
keeps serving its last-known-good address space, drivers and subscriptions
throughout (#485).
Tests, RED-first (both verified failing with "should be Failed but was
Applied"): the ACK/revision assertion, plus the one that matters — after a
failed apply, re-dispatching the SAME revision now genuinely applies. Its
recovered artifact adds a second driver precisely so a short-circuited ACK
(which has no side effects) cannot satisfy it.
Four existing tests changed, and both changes are deliberate:
- DriverHostActorTests.SeedDeployment never set ArtifactBlob at all. Its three
consumers are about the apply/ACK/state machine, not the artifact, and now
need a genuine apply — so the helper seeds a well-formed artifact declaring
no drivers. That is what the composer emits for an empty configuration, and
is precisely what "bytes we could not read" is NOT.
- EmptyArtifact_IsNotCached asserted "the apply still succeeds — an empty
artifact is a legitimate no-op deployment". That premise is the bug. Flipped
to Failed; the test's actual subject (nothing is cached) is untouched and now
holds for two independent reasons.
Closes#486
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
All three guards observed firing in production, on two nodes, with the live
values proving the subscription half rather than just the log line: the Modbus
tag advanced 56278 -> 56336 -> 56354 straight across the induction.
The RED control was not staged. site-a-1 had hit #485 during the PREVIOUS
gate's SQL flapping and had been serving an empty address space for 34 minutes
(PureRemove, removed=17) on the pre-fix image — the bug in its natural habitat.
Rebuilding on the fixed image restored it, and the induction then failed to
reproduce the teardown.
The original trigger was a race, so the gate drives the empty-bytes branch
instead — byte-for-byte what the code sees when the row cannot be read, and the
branch the driver-side guards key on (their throw paths already returned early
before this work). Pausing the target node opens the window deterministically;
both runs measured <1s, well inside acceptable-heartbeat-pause, so no failover
was triggered. Of the six nodes exactly one took the guard path — the other
five read the real artifact and applied it, which is its own control.
Records one pre-existing defect this work does NOT fix: ApplyAndAck advances
_currentRevision and writes NodeDeploymentState=Applied before knowing whether
anything applied, so a node that skipped reports success for a revision it
never applied — and since dispatch short-circuits on a revision match it cannot
self-heal from a re-dispatch. Data plane is unaffected (the node keeps serving
its last good config); it is a reporting/convergence bug. Left out of these
fixes because correcting it changes deploy ACK semantics fleet-wide.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The driver-side half of the same mistake the address-space fix corrected.
ReconcileDrivers and PushDesiredSubscriptionsFromArtifact both read the
artifact as `...FirstOrDefault() ?? Array.Empty<byte>()`. Their THROW paths
already returned early, but empty bytes flowed onwards as a real answer: zero
driver specs, so DriverSpawnPlanner planned every running child for StopChild,
and then an empty desired set dropped each surviving driver's live
subscription handle. A node lost its entire field I/O and still ACKed Applied.
Same rule as the address space: no bytes is no answer, not "a configuration
with no drivers". Nothing legitimate produces a zero-length blob — deleting
the last driver still deploys a JSON document with empty arrays. Both sites
now skip and keep what is running. The two guards are independent because
PushDesiredSubscriptions does its OWN ConfigDb read, so the row can go missing
between them.
Also corrects the ReconcileDrivers doc comment, which claimed an empty blob
made it "effectively a no-op" — with children running it was the opposite.
Tests: DriverHostActorUnreadableArtifactTests, RED-first (verified failing —
after the empty dispatch the driver list was empty). A zero-length ArtifactBlob
reproduces byte-for-byte what the missing-row case delivers, so the race does
not have to be constructed.
Two controls, both load-bearing:
- a READABLE driverless artifact still stops the driver, so the guard keys on
"the read gave us nothing", not on "fewer drivers than before";
- dropping a driver's LAST TAG still clears its subscription. This one was
added after the absence assertion was caught passing on a race: the
unsubscribe is an async self-tell, so with the second guard deleted the test
still went green. The control observes that same unsubscribe arriving well
inside the settle window, which is what makes the absence meaningful — with
the guard deleted the suite now correctly goes RED.
SubscribableStubDriver gains an UnsubscribeCount for that observation.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A transient ConfigDb error while loading a deployment's artifact emptied the
served address space. LoadArtifact caught the exception, logged "rebuild
becomes no-op" and returned zero bytes; those parsed to an EMPTY composition,
which the planner diffed against the live one as a PureRemove and the applier
faithfully executed — 16 nodes gone, while the log claimed nothing happened.
An artifact we could not obtain is not an empty configuration. Nothing
legitimate produces a zero-length blob (an operator who really deletes
everything still deploys a JSON document with empty arrays), so "no bytes" can
only mean "no answer". HandleRebuild now abandons the rebuild on one, leaving
both the materialised nodes and _lastApplied intact so the retry diffs against
what is actually being served. The loader's own log line is now true.
The two cases are logged differently: warn when there is a live address space
being protected, debug when nothing has been deployed to this node yet.
Covered by a RED-first actor test (verified failing: the load error tore down
eq-2's subtree), plus a positive control proving a READABLE artifact that
genuinely drops an equipment still removes it — the guard suppresses teardown
only when the read failed.
Found on the LocalDb Phase 1 follow-up live gate, which induced it by flapping
SQL; that gate's log is the production evidence for the seam.
Closes#485
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
- Oplog growth on default-OFF nodes, fixed at the source: the artifact cache
now skips the store entirely when the pointer already names this
deployment/revision, the SHA matches the bytes, and the expected chunk count
is present. Both extra conditions are load-bearing — guard tests go RED
against a naive pointer-only skip.
- Wiped-node back-fill, fixed upstream in ZB.MOM.WW.LocalDb 0.1.2 + 0.1.3 and
pinned here.
Live-gated on docker-dev (docs/plans/2026-07-21-localdb-followups-live-gate.md).
The gate found a third defect that offline tests could not have found first,
because it lived behind the one being fixed: once back-fill worked, the rebuilt
node's own writes turned out to go nowhere. It also surfaced one unrelated
pre-existing bug, filed as issue #485.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The gate closed both follow-ups on the docker-dev rig and found a third defect
that offline tests could not have found first: it lived behind the one being
fixed. Once 0.1.2 made back-fill work, a rebuilt node could be observed writing
for the first time — and its writes went nowhere, because the healthy node kept
the old peer's seq watermark. Library 0.1.1 -> 0.1.2 -> 0.1.3 over the course
of the gate; both consumers now pinned to 0.1.3.
Check 4 (was PARTIAL, now PASS): with SQL stopped and site-a-2's LocalDb volume
destroyed, a-1 logged "Snapshot sent (as_of_seq 0, 4 rows)" — a line that could
not appear on 0.1.1 — and a-2 came back with a row_version dump byte-identical
to a-1's, carrying a-1's origin node ids and its ORIGINAL applied_at_utc. It
then booted from cache and served 17 ns=2 nodes, diff-identical to its peer,
from a configuration it never applied and could not have fetched.
Check 8 (PASS): two restarts with an unchanged artifact left the oplog at 7 and
the pointer timestamp frozen — the re-cache was skipped outright. Positive
control: a real config change still wrote, 7 -> 10 -> 13.
Also records one unrelated pre-existing finding the gate's SQL flapping
surfaced: a transient ConfigDb error during artifact load empties the served
address space while logging "rebuild becomes no-op". The cache layer correctly
refused to store the bad artifact and the node recovered on restart, but the
two logs disagree and it is the same class as the Phase 1 gate's check-3
defect. Untouched by this work; wants its own issue.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
1. Wiped-node back-fill (live gate check 4). Fixed upstream in
ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
off: the oplog cap was not the gate. Snapshot detection measured the peer's
gap against the oldest surviving oplog row and read an empty oplog as "no
gap possible" — and an empty oplog is the steady state of a converged pair,
since ack-pruning deletes everything the peer confirmed. The healthy state
was the one state that could not heal a wiped peer. Now measured against
last_acked_seq when the oplog is empty.
Pinned at this level too, not just the library's: the pair harness grows a
WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
id and a zero watermark), and the new scenario asserts the emptied oplog as
its precondition before wiping, then requires the deployment artifact to
come back byte-identical with no new deploy. Verified RED against the pinned
0.1.1 (times out waiting for back-fill) and green on 0.1.2.
2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
StoreAsync now skips entirely when the pointer already names this
deployment/revision, the SHA matches the bytes, and the expected chunk count
is present. Re-caching an artifact the node already holds — every restart's
boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
rows, where before it cost a delete plus an insert per chunk plus a pointer
update, all identical to what was already there.
Identity is over the bytes and the chunks are counted, both deliberately: a
re-composed artifact can carry the same ids with different bytes, and a
pointer can name a deployment whose chunks are missing, where skipping would
make an unreadable cache permanent. Both guards verified RED against a naive
pointer-only skip.
The gate's stated bound was also wrong and is corrected in the doc: growth
was never headed for the 1M row cap. AddZbLocalDbReplication is registered
unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
and the 7-day MaxOplogAge cap prunes.
Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Records the docker-dev live gate: 8/8 checks pass (check 4 with a documented
limitation). The gate caught four real defects every offline test missed — the
NU1101 packageSourceMapping gap, the ASPNETCORE_HTTP_PORTS re-bind, the empty
address space on boot-from-cache, and the cache not repopulating on
RestoreApplied — all fixed on-branch with regression tests. Two limitations
documented as follow-ups: no replication back-fill of a fully-wiped node, and
oplog growth on default-OFF nodes.
CacheAppliedArtifact ran only on a fresh apply (ApplyAndAck), never on the
bootstrap RestoreApplied path. So a node whose cache was lost — fresh/wiped
volume, disk failure — recovered its served state from central yet stayed
cache-less, unable to boot-from-cache on the NEXT central outage until some
future new deploy happened to land. Replication does not heal this: a peer's
already-acked cache rows are pruned from its oplog, so a fully-wiped, converged
node is never back-filled. RestoreApplied now writes the served artifact back to
the cache (invariant: the cache holds what the node serves). Found on the
docker-dev live gate wiping a node's LocalDb volume.
Boot-from-cache told OpcUaPublishActor to RebuildAddressSpace(deploymentId), which
re-loads the artifact from the ConfigDb by id — the very database that is
unreachable during the outage this path exists to survive. The rebuild no-op'd, so
a cache-booted node ran its drivers but served OPC UA clients an EMPTY address
space. RebuildAddressSpace now carries an optional in-hand Artifact blob;
ApplyCachedArtifact passes the cached bytes so materialisation happens from them
directly. Found on the docker-dev live gate: healthy peer served 16 nodes, the
cache-booted node served 1.
The Kestrel re-bind read only urls/ASPNETCORE_URLS. The aspnet:8.0+ base images
set ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the CONTAINER default in place
of ASPNETCORE_URLS, so a driver node that never sets URLS fell straight through to
Kestrel's localhost:5000 default — silently moving its health/metrics surface to
loopback when the sync listener took over Kestrel. Now falls through URLS →
HTTP_PORTS/HTTPS_PORTS (wildcard, all-interfaces) → localhost:5000, mirroring
Kestrel's own source precedence. Found on the docker-dev live gate (central nodes
were fine — they set ASPNETCORE_URLS=9000 explicitly).
Joseph Doherty