dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/ScadaBridge:feat/localdb-phase1
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0
...
compare: dohertj2/ScadaBridge:f1ad96708316eb508cf634cdbbfeaf96eae50abe
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0

50 Commits
feat/localdb-phase1 ... f1ad967083

Author SHA1 Message Date
Joseph Doherty f1ad967083 docs(plans): ClusterClient→gRPC-only migration plan (phases 0–5 + tasklist) 
Complete executable plan: PSK-from-Secrets auth (Phase 0), central_control +
site_command proto contracts behind transport seams inside the two
communication actors (1A ∥ 1B in worktrees), per-direction cutover flags,
ClusterClient/receptionist deletion, 8-check live gate. Design doc:
scadaproj/scadabridge_clusterclient_to_grpc.md (§7 = deep-dive corrections).
 2026-07-22 17:10:33 -04:00
Joseph Doherty 654df8abc2 docs(cluster): site-pair manual failover runbook + component spec 2026-07-22 07:49:51 -04:00
Joseph Doherty c8e2f4da02 feat(cluster): site-pair manual failover relayed from the central UI (Task 10) 
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
 2026-07-22 07:48:56 -04:00
Joseph Doherty d66e0d585f chore(plans): mark self-first ordering + manual failover tasks complete (live gate PASS) 2026-07-22 07:14:11 -04:00
Joseph Doherty caf14a3e03 feat(ui): admin manual-failover control on the health page 
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles /
SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without
standing up the whole dashboard's DI graph.

- Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is
  intentionally all-roles, so the gate belongs on the control, not the page.
- Disabled with an explanatory title when the pair has no online standby;
  the authoritative guard remains server-side against live cluster membership.
- Confirmation dialog (IDialogService, the page idiom) warns that singletons
  hand over, in-flight work on the active node is interrupted, and THIS PAGE
  will disconnect and reconnect against the new active node -- Traefik routes
  the UI to the node being restarted, so a working failover otherwise reads as
  a crash the admin caused.
- A refused failover (service returns null) surfaces the refusal; the UI never
  reports a failover that did not happen.

7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a
cascading AuthenticationState (the app supplies it from the layout), and
BunitContext pre-registers a placeholder IAuthorizationService that throws on
policy evaluation -- both handled the same way NavMenuTests documents.

Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual
Failover section) and docker/README.md.
 2026-07-22 07:00:10 -04:00
Joseph Doherty f679d5c749 feat(cluster): manual central failover service — graceful Leave of the oldest Up member 
Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.

- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
  the node acted on is exactly the one hosting the singletons (never the leader,
  whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
  node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
  the one that goes away, and an audit written after could be lost to the shutdown it
  describes. Best-effort — audit failure never blocks the failover.

New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.

alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.

ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
 2026-07-22 06:55:15 -04:00
Joseph Doherty eca69505bc docs(plans): record the self-form watchdog rejection and the self-first ordering that replaced it 2026-07-22 06:33:28 -04:00
Joseph Doherty 4a6341d871 feat(cluster): self-first seed ordering closes the boot-alone outage gap 
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.

- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
  self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
  fails silently when broken, so it is enforced loudly. NOTE: the gitignored
  deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
  that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
  failure-detection timings, incl. a falsifiability control proving the OLD
  peer-first ordering never forms.

Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
 2026-07-22 06:32:00 -04:00
Joseph Doherty 69b3ccfc37 docs(components): correct the three component docs against the code they describe 
CLAUDE.md was corrected in 34227991 but the component docs were not, so the
same understatements — plus several outright wrong statements — survived where
a reader is most likely to meet them.

ClusterInfrastructure.md carried the worst of it. It described active/standby as
cluster leadership and showed an IsActiveNode snippet doing
`cluster.State.Leader == self.Address`. No such code exists: ActiveNodeGate
returns ClusterActivityEvaluator.SelfIsOldest, and ActiveNodeEvaluator's own doc
comment says "never cluster.State.Leader". A second snippet (siteCallAuditShutdown
.AddTask) described a hand-rolled drain that has since been folded into
SingletonRegistrar.Start. Both snippets are replaced with what the code does. The
central singleton table listed 3 of the 7 registered singletons. The downing
section still described keep-oldest as the strategy and omitted
downing-provider-class entirely; it now shows the branching block cf3bd52f
introduced, with the accepted dual-active trade stated rather than the old
"impossible to boot" framing. Note the requirements-side spec,
docs/requirements/Component-ClusterInfrastructure.md, was already rewritten by
cf3bd52f — this is the components-side doc, which was untouched.

Communication.md described two transports; there are three — the deployment
config fetch over HTTP was missing. Each row now names which side dials, because
the gRPC entry gave a data direction (site to central) without saying who hosts
the server, which reads as the opposite of the truth: the only MapGrpcService in
the tree is in Program.cs's Site branch, and central dials in. The SiteEnvelope
snippet called DeployInstanceAsync, which is not a member of CommunicationService;
the heartbeat bullet claimed a cluster-leadership check; the proto summary listed
4 of 6 RPCs.

DeploymentManager.md still said the pipeline sends DeployInstanceCommand carrying
FlattenedConfigurationJson. It stages a PendingDeployment and sends a
RefreshDeploymentCommand; the config travels over the HTTP fetch. That is the
change the 128 KB frame-size issue forced, and the doc predated it.

Two of the five briefed drifts turned out not to be errors: nothing claimed the
transports were authenticated, and nothing asserted per-site ActorSystem names —
both were simply unstated. They are now stated, since silence about an
unauthenticated boundary is its own problem.

Docs only; no code changed.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 18:45:56 -04:00
Joseph Doherty cf3bd52f93 feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety) 
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.

SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.

Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
 2026-07-21 10:53:40 -04:00
Joseph Doherty dced0d2794 fix(deps): pin System.Security.Cryptography.Xml 10.0.10 — four new NU1903 advisories on the 10.0.7 DataProtection transitive broke fresh restores (docker image build) 
Same pattern as the SQLitePCLRaw pin: direct PackageReference at the chain's entry
project (ConfigurationDatabase). Bumping the DataProtection parent instead was tried
and rejected — 10.0.10 floors Microsoft.Extensions.*/EF at 10.0.10 (NU1605 cascade).
 2026-07-21 10:53:26 -04:00
Joseph Doherty 34227991ea docs(claude): correct five understatements about the inter-cluster boundary 
Found while OtOpcUa researched this repo as the model for its own per-cluster
mesh work. Each was verified against the code, not inferred from the doc.

Three transports cross the boundary, not two: ClusterClient, gRPC, and plain
token-gated HTTP for the deploy config fetch (DeploymentConfigEndpoints,
X-Deployment-Token, AllowAnonymous with the per-deployment token as the entire
security boundary).

The gRPC direction is inverted from the data flow. Data moves site to central,
but each SITE hosts the gRPC server and central dials in — MapGrpcService appears
only in the Site branch of Program.cs. There is no gRPC server on central, which
is why the two Ingest* RPCs are dead in practice. Also records the 6-RPC surface
(the doc implied 2), the (site, endpoint) factory key that fixed an arch-review
High, and the vendored-generated-code caveat.

Active/standby is ActiveNodeEvaluator.SelfIsOldestUp — the OLDEST Up member, and
explicitly never cluster.State.Leader, because leadership diverges from singleton
placement permanently after a restart-and-rejoin and both sides claim it during a
partition. The equivalence oldest-Up == singleton placement is the design, and it
was not stated anywhere in this file.

All clusters share one ActorSystem name ("scadabridge", hardcoded at
AkkaHostedService.cs:191); they are separate clusters only by seed partitioning.
Required, not incidental — Akka.Remote address matching means ClusterClient could
not reach a differently-named system. Site nodes also carry two roles, base plus
site-{SiteId}, with singletons scoped to the site-specific one.

Neither inter-cluster transport is authenticated or encrypted: no Akka TLS or
secure cookie, gRPC is h2c, and LocalDbSyncAuthInterceptor gates only the LocalDb
sync path — so the whole SiteStreamService surface, including the two Pull RPCs
returning audit rows, is reachable by anyone who can hit :8083. The boundary
assumes a trusted network; that assumption deserves to be explicit.

Also promotes two things from docs/ into CLAUDE.md because they change decisions:
the default 128 KB Akka frame size with log-frame-size-exceeding off and no custom
serializer (a silent single-message drop that leaves the association healthy), and
the registered two-node keep-oldest total-outage gap, whose own drill has a mode
existing "to make the registered gap observable — not to pretend it is covered."

Committed to a branch rather than main; merge at your discretion.
 2026-07-21 10:00:01 -04:00
Joseph Doherty 9d925c3347 Merge chore/localdb-0.1.3: pin ZB.MOM.WW.LocalDb 0.1.3 
Picks up two rebuilt-peer replication fixes found by the OtOpcUa LocalDb
Phase 1 live gate. They matter more here than in OtOpcUa, which replicates 2
tables to this repo's 10:

- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
  detection read that as 'no gap possible', so a node whose database was lost
  rejoined empty and stayed empty until the next deploy.
- 0.1.3 — with back-fill working, the rebuilt node's own writes were silently
  dropped until its restarted seq counter climbed past the peer's stale
  watermark.

Build clean; SiteRuntime 512 green.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:19:13 -04:00
Joseph Doherty 8c6fc2f886 chore(localdb): pin ZB.MOM.WW.LocalDb 0.1.3 (rebuilt-peer replication fixes) 
Two fixes, both about a node whose LocalDb file is lost. They matter more here
than in OtOpcUa, which replicates 2 tables to this repo's 10.

0.1.2 — a converged pair prunes every oplog row on ack, and snapshot detection
read an empty oplog as "no gap possible". The steady state of a healthy pair
was the one state from which a rebuilt node could never be healed: it rejoined
empty and stayed empty until the next deploy.

0.1.3 — with back-fill working, the rebuilt node's OWN writes turned out to be
silently dropped: last_applied_remote_seq is a watermark in the peer's seq
space, and a rebuilt peer numbers from 1, so the healthy node's stale watermark
made the sender skip its whole oplog.

Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig; the
second only became reachable once the first was fixed.

Build clean; SiteRuntime 512, SiteEventLogging 70 green (Host 330,
HealthMonitoring 97 green on 0.1.2, unchanged by the pin).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:18:02 -04:00
dohertj2 28ca04d7de LocalDb adoption Phase 1 + 2: consolidate the site database, delete the bespoke replicators (#23) 
10-check live gate PASS; 3,509 tests green; replication remains default-OFF.
 2026-07-20 06:06:04 -04:00
Joseph Doherty 7621b48925 docs(known-issues): cached-telemetry drain hot-loops on a missing tracking snapshot 
An audit row whose tracking snapshot cannot be resolved is skipped and left
Pending, on the reasoning that central reconciliation will pick it up. Nothing
removes it from the local drain queue, so the next tick re-reads it, fails
identically, and logs again — forever. Measured on the rig at ~2,800
warnings/minute, sustained across both a process restart and a container
restart, until the audit database itself was discarded.

The code comment already names the cause ("the tracking store was reset"), so
this is an anticipated input, not an exotic one. Two production triggers need
no operator error: the tracking retention window elapsing before the drain
catches up (long central outage, large backlog), or restoring one store
independently of the other. The two stores are easy to desync because they live
in different places — audit in auditlog.db, which on the docker rig is INSIDE
the container, and tracking in the bind-mounted LocalDb database.

Skipping the row is right; leaving it Pending with no other state change is
not. There is no attempt counter, no backoff, no terminal state, and one
Warning per row per pass. Suggested fixes ranked by effort, the cheapest being
to rate-limit the warning the way MaintenanceBackgroundService already does for
oplog caps.

No data loss — the audit rows are intact and still reach central by the
reconciliation path. Found while cleaning the rig after the Phase 2 live gate.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:59:35 -04:00
Joseph Doherty 7b5a5a6f34 docs(localdb): phase 2 truth pass across both repos 
Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.

Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.

Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.

Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.

Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.

Deletes the phase2 resume-state scratch doc, which said to delete it on landing.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:37:48 -04:00
Joseph Doherty 158e79bb50 docs(localdb): phase 2 live gate evidence — all 10 checks pass 
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.

Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
  __localdb_row_version HLC and origin node id, and the standby logged zero
  config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
  oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
  nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
  the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
  errors.

Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.

Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:20:54 -04:00
Joseph Doherty 3c87b11bcf docs(localdb): phase 2 live gate evidence (INCOMPLETE — 5 of 10) 
Checks 1, 2, 3, 4 and the oplog/dead-letter/metrics half of 8 captured and
passing. Check 5 blocked on rig state, checks 6, 7, 9, 10 not run. Task 21 must
NOT proceed on this evidence.

Strongest result is check 3+4 together: the deployed config row is byte-
identical on both site-a nodes with the SAME __localdb_row_version HLC and
originating node id, and the standby logged zero config fetches in the deploy
window. The standby holds the row node A wrote, obtained purely by CDC — which
is the whole point of deleting notify-and-fetch.

Check 5 is blocked by a rig-shaping problem, not a product fault: removing the
owed soakgen instances orphaned their 11,804 buffered sf_messages, and a
replacement probe's external system never reached the site nodes (site config
tables come from deployment artifacts, and an external system referenced only
by name inside script code is not carried). Same blocker stops check 10.

Two method corrections recorded in the doc:
- The plan says to run DB checks host-side against the bind mounts. That is
  WRONG — a host sqlite3 open poisons the container's WAL. All reads here copy
  the db/-wal/-shm triplet and query the copy.
- An apparent "localdb_* metrics missing" finding was an artifact of the
  aspnet:10.0 image having no curl, with the error swallowed by 2>/dev/null.
  Re-scraped via a network-sharing sidecar: the metrics are present.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:10:43 -04:00
Joseph Doherty 166f07fa68 chore(localdb): adopt LocalDb 0.1.1 and drop the directory shim 
LocalDb 0.1.1 creates the parent directory of LocalDb:Path itself, so the
SiteLocalDbDirectory shim this repo carried through Phase 2 is deleted along
with its call site. The gap was found here but was never ScadaBridge's alone —
every LocalDb consumer had it — so the fix moved to the library.

SiteLocalDbDirectoryTests is RETAINED and retargeted rather than deleted with
the shim. It was already written against the site registration path, not the
mechanism, so it needed only its Ensure() call removed: what a site node
requires is that resolving ILocalDb not fail on a fresh machine, regardless of
who provides that. Verified it still earns its place — pinned back to LocalDb
0.1.0 it fails inside SqliteLocalDb..ctor -> SqliteConnection.Open(), so it
genuinely depends on the library behaviour and not on a coincidence.

Also corrects the coverage-split note in StoreAndForwardStorageTests, which
asserted that directory creation is "NOT LocalDb's" — true when written, wrong
as of 0.1.1.

Build 0 warnings; Host 330, StoreAndForward 130, LocalDb integration 20 pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:58:20 -04:00
Joseph Doherty 9ec7966dac docs(localdb): task state for tasks 17-19 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:42:16 -04:00
Joseph Doherty 921edab454 chore(docker): size the site-a oplog caps from the phase 2 soak 
MaxBatchSize 500 -> 16. The default is a ROW count, not a byte budget, so the
batch size in bytes is set by the widest replicated column — config_json, which
Task 1 measured at up to ~60-70 KB in production. 70 KB x 500 is ~35 MB against
gRPC's 4 MB default receive limit; 16 keeps a worst-case batch near 1.1 MB.

MaxOplogRows 1,000,000 -> 250,000 and MaxOplogAge 7d -> 2d, sized from the
soak's 0.80 sf_messages rows/sec (~69k rows/day). Tighter than default is
correct here because exceeding a cap is not data loss: the oplog prunes to the
ceiling and sets needs_snapshot, so the peer catches up by snapshot resync
instead of incrementally. That trades a rare full resync for a bounded file.

site-b and site-c stay unreplicated, so the default-OFF posture is still proven
side by side on one rig.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:41:47 -04:00
Joseph Doherty 15013156bf test(localdb): two-node convergence for config tables + sf_messages 
Four scenarios over the real site pair, driven through the REAL
SiteStorageService rather than hand-written SQL. The sibling suites had to
hand-write SQL because they were specifications written BEFORE the cutover;
this suite runs after it, so it can drive the shipped writers — which is what
makes the cascade scenario meaningful.

- DeployedConfigRow_ConvergesToB_ColumnForColumn: the existing suite asserts
  config_json only, so a capture that dropped or defaulted any other column
  would still pass. deployed_at matters most — SiteReconciliationActor's
  guarded write compares it.
- RemovingAnInstance_ConvergesAllThreeCascadeTables: the plan's flagged
  highest-risk case. RemoveDeployedConfigAsync deletes from three tables in one
  transaction, the schema has no foreign keys, and CDC ships three independent
  per-table streams. A dropped delete leaves a permanently stale override or
  alarm row on the standby, invisible until the instance name is redeployed.
  Carries a never-removed control instance, without which "the cascade
  converged" is indistinguishable from "node B lost these tables entirely".
- ANativeAlarmBurst_Converges_AndTheOplogDrains: convergence alone would pass
  if entries replicated but were never acked, and an oplog that only grows
  trips the caps into a snapshot resync.
- RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin: the union-survives
  property is per-table, and an unregistered table is silently local-only
  rather than an error, so it is re-proved on the tables the N1 scenario does
  not touch.

Non-vacuity verified as the plan requires: with the eight Phase 2
RegisterReplicated calls commented out in SiteLocalDbSetup, all four go red
(4 failed / 0 passed); restored, 20/20 pass across the three LocalDb suites.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:41:01 -04:00
Joseph Doherty 605e56829e chore(config): retire ReplicationEnabled, make legacy db paths migration-only 
LocalDb Phase 2 deleted the bespoke replicators, so three config keys changed
meaning or died outright:

- ScadaBridge:StoreAndForward:ReplicationEnabled is fully dead. Deleted the
  property, its 10 config entries, and the 5 test references.
- SqliteDbPath / SiteDbPath are now migration-only: they name the legacy files
  SiteLocalDbLegacyMigrator drains at boot, not live databases. Both mandatory
  rules are relaxed accordingly (StartupValidator's Site-only Require, and the
  S&F validator's non-empty rule) — an absent value now means "nothing to
  migrate", so an already-migrated node can drop the key. DatabaseOptions-
  Validator still rejects a present-but-blank value.
- SiteRuntime:ConfigFetchRetryCount's only reader was SiteReplicationActor.
  Deleted with its validator rule.

The two path keys stay present in every config, now with a comment explaining
why: removing them would strand un-migrated data on a node that has not yet
started once.

Both relaxations are pinned by the inverse of the test they replace
(Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each
verified to fail with the old rule restored.

Note: deploy/wonder-app-vd03/appsettings.Site.json is under a gitignored
deploy/ tree, so its edit is local-only and must be repeated on the box.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:35:11 -04:00
Joseph Doherty 3364145d63 docs(localdb): resume state through the cutover (tasks 1-16 done, task 17 next) 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:22:30 -04:00
Joseph Doherty 0ad11d6b55 docs(localdb): task state for the 14/15/16 cutover 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:20:17 -04:00
Joseph Doherty 037798b367 feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators 
Tasks 14, 15 and 16, landed as ONE commit.

PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor
takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both);
DeploymentManagerActor Tells message types declared in ReplicationMessages.cs
(Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any
ordering leaves a broken intermediate. Combining them also strengthens the
invariant Task 14 already stated for itself — the two mechanisms never both
run, and never neither.

Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site
config tables. notification_lists and smtp_configurations are deliberately NOT
registered — permanently empty by design, so registering them would open a
standing replication channel whose only historical payload was plaintext SMTP
passwords. Migrate stays the LAST call in OnReady, after all registrations, so
migrated rows enter the oplog through live capture triggers.

Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService,
StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is
not merely unused but unsafe to keep: a mass DELETE on a now-replicated table
would be captured and shipped to the peer.

Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc
corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor).

The positional-argument hazard the plan flagged was real: removing
DeploymentManagerActor's optional IActorRef? replicationActor shifted 6
trailing optionals, and 4 test call sites bound the wrong arguments with no
compile error at some positions. Converted them to named arguments where
possible — Props.Create builds an expression tree, which rejects out-of-position
named args, so the rest are padded positionally with a comment saying why.

The Task 7 'not yet registered' test was INVERTED rather than deleted, and is
exact in both directions: too few means a table silently stops replicating, too
many means the SMTP tables leak. Added a separate security-named test for those
two, and a composite-PK test (LWW keys on the full PK, so a truncated key set
would collapse distinct rows). The convergence suites now get their
registrations from the real OnReady — their temporary harness registration is
deleted, so they prove the cutover rather than agreeing with themselves.

Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb
integration 16 — all pass, 0 failures.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:20:05 -04:00
Joseph Doherty df0c6031ba docs(localdb): task state for tasks 10-13 + deviations 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:03:48 -04:00
Joseph Doherty 79ce51612e test(site): pin the active-node notification-config purge; scope the guarded write 
Task 12 + Task 13. No production behaviour change in either.

Task 12: DeploymentManagerActor.HandleDeployArtifacts already purges
notification_lists and smtp_configurations on every artifact apply, but nothing
pinned the actor's CALL to it — ArtifactStorageTests covers the storage method
only. Task 15 deletes SiteReplicationActor's copy, making this the sole
remaining call site, and Task 16 edits this actor's wiring; dropping the call
would leave plaintext SMTP passwords on disk with every suite still green.
Verified red-first by commenting the call out: the pin fails with that message.

Task 13: StoreDeployedConfigIfNewerAsync STAYS. Re-verified both callers —
SiteReplicationActor:375 (dies at Task 15) and SiteReconciliationActor:166
(survives). Reconciliation is a per-node startup self-heal against central
whose fetch races real deploys, so the deployed_at guard still does real work
there. Doc comment rewritten to say so, and to warn against porting the guard
onto the replication path where it would fight the HLC rather than help it.
Also corrected a stale 'guarded standby write' section header in the tests.

Re-ran the Task 13 step 2 scope check: ConfigFetchRetryCount's only production
reader remains SiteReplicationActor:157, so its option + validator rule stay
until Task 17, after Task 15 deletes the actor.

Verified: build 0 warnings; SiteRuntime 533, Host 329, StoreAndForward 153,
LocalDb integration 16 — all pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:03:00 -04:00
Joseph Doherty c56bf4ae65 test(localdb): port resync + config replication intents as CDC convergence specs 
Task 11. Companion to Task 10, covering the intents held by
SiteReplicationActorTests and SfBufferResyncPredicateTests.

N1 Critical is re-expressed, not ported literally (D2). SfBufferResyncPredicate
existed because ReplaceAllAsync was a destructive DELETE-then-INSERT, so a
wrong-direction resync wiped a live buffer and the code needed an oldest-Up
predicate to decide authority. LocalDb's snapshot resync merges per row under
LWW and never deletes, so this asserts the property the guard protected — no
node loses rows to a peer's snapshot — with no directional-authority assertion,
because there is no active/standby asymmetry left to enforce. The setup is the
wipe scenario made concrete: each node writes rows the other never sees while
partitioned, plus a contended row, then they resync.

DEVIATION on the zero-fetch assertion. The plan asked for a fetcher test double
recording zero invocations; this harness has no actor system, no central and no
IDeploymentConfigFetcher in the graph, so the double would record zero calls
whether or not the fetch path still existed. An assertion that cannot fail is
worse than none, so the test proves the positive half — config reaches B over
replication alone — and the file records that the negative half is proved by
Task 15 deleting the code and the build still passing. It also records the D1
scope note: SiteReconciliationActor survives and legitimately fetches at
startup, so 'the standby never fetches, ever' would be false.

Non-vacuity verified by unregistering deployed_configurations: all 4 fail.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:54:42 -04:00
Joseph Doherty 2bbe66311d test(localdb): port store-and-forward replication intents as CDC convergence specs 
Task 10. Specifications first, deletion second: the bespoke ReplicationService
(explicit Add/Remove/Park/Requeue over Akka) dies at Task 14, so its behaviour
is restated here as outcomes the CDC replacement must still deliver. Written
in terms of ROWS, not operations — under CDC there is no Add or Park message to
observe, only a row that must end up right on both nodes.

Not ported: ReplicationOperations_AreDispatchedInIssueOrder. It asserts the
mechanism (inline fire-and-forget dispatch), and CDC capture is asynchronous
and batched by construction. Its portable content is the ordering OUTCOME —
add-then-remove must never converge to present — which is a test here, with
that reasoning recorded in the file so it does not read as an accidental drop.

DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness rather
than duplicating ~150 lines. Phase 1's tests now derive from it and still pass
unchanged. The harness registers the Phase 2 tables itself, since production
OnReady does not until Task 14; that method is marked for deletion at the
cutover, and the 8-table list is written literally so a cutover registering the
wrong set fails these tests instead of agreeing with itself.

Non-vacuity verified by unregistering sf_messages: 6 of 7 failed. The 7th —
the ordering test — PASSED, because an absent row is also what a pair that
replicates nothing looks like. Fixed with a control row that must converge in
the same window, so the absence is evidence rather than silence.

Also corrected two comments from Task 9 that claimed Task 14 makes
notification_lists/smtp_configurations replicated. It explicitly does not
register them, for the same reason the migrator skips them.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:50:48 -04:00
Joseph Doherty 0dbfefba62 docs(localdb): task state for tasks 7-9 + deviations 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:40:59 -04:00
Joseph Doherty 5ddc7eed6d feat(localdb): migrate legacy scadabridge.db config tables into the consolidated DB 
Task 9. Seven of the nine site configuration tables are copied out of the
legacy scadabridge.db in a single transaction, with one rename at the end: a
partial config migration would leave a site node running against half its old
configuration, which is worse than failing startup outright.

notification_lists and smtp_configurations are deliberately NOT migrated.
Both are purged on every deploy and permanently empty by design since the
site write paths were removed (2026-07-10), but a pre-fix legacy file can
still hold rows — and smtp_configurations.password is plaintext. Task 14
makes these tables replicated, so migrating them would push plaintext SMTP
passwords across a channel whose only historical payload was exactly that.
The tables are still created; only their historical contents stay behind.

Generalizes Task 8's MigrateTable into MigrateFile(many tables, one
transaction, one rename), keeping the legacy/current column intersection.

Five tests, including a column-parity test asserting each declared column
list equals the live SiteStorageSchema's. That mismatch is otherwise
invisible: a typo'd column is silently dropped by the intersection, and a
missing one silently leaves data behind. Non-vacuity verified twice — once by
removing the Migrate call (3 fail), once by wrongly adding notification_lists
and smtp_configurations to the table map (the skip test fails).

Verified: solution build 0 warnings; Host 329, SiteRuntime 532,
StoreAndForward 153 — all pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:40:31 -04:00
Joseph Doherty bdc0dffea2 feat(localdb): migrate legacy store-and-forward.db into the consolidated DB 
Task 8. Adds ResolveStoreAndForwardPath + MigrateStoreAndForward, wired as a
third call in Migrate alongside the two Phase 1 files.

Unlike the Phase 1 paths, the store-and-forward default sits INSIDE the data
volume, so a real deployment has a real file here holding undelivered
messages. This migration genuinely moves data; losing it would discard exactly
the buffered calls store-and-forward exists to protect.

No id synthesis: sf_messages.id is already a caller-assigned TEXT primary key,
so INSERT OR IGNORE is idempotent across a crash-then-rerun.

The copy intersects the legacy column set with the current one rather than
naming all 16 columns outright. A file from an older build predates
execution_id / parent_execution_id / last_attempt_at_ms, and naming a missing
column throws 'no such column' — which the existing reader treats as an
unrecognised shape and silently discards every row. A required-column (PK)
guard keeps that tolerance from degrading into copying NULL-keyed rows.

Four tests: copy+rename, crash-before-rename idempotence, the __localdb_oplog
assertion that catches migrate-before-register, and the older-column-set case.
All four verified to fail without the Migrate call.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:32:10 -04:00
Joseph Doherty f8aa02e2a9 feat(localdb): create config + sf_messages tables in the consolidated DB 
Task 7. SiteLocalDbSetup.OnReady now applies SiteStorageSchema and
StoreAndForwardSchema alongside the Phase 1 schemas, so the nine site
configuration tables and the store-and-forward buffer live in the
consolidated LocalDb file.

Deliberately NOT registered for replication. The bespoke SiteReplicationActor
and the StoreAndForward ReplicationService still own these tables until the
Task 14 cutover deletes both and registers them in one commit; registering
early would run two replicators over the same rows and let either one's
defects hide behind the other's writes.

Pinned by two tests through the real composition root: one asserting all
twelve tables exist, one asserting ReplicatedTables is EXACTLY the Phase 1
pair. Non-vacuity verified by removing the DDL and observing the failure.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:28:05 -04:00
Joseph Doherty fefbbb31da docs(localdb): resume state through wave 2 (tasks 1-6 done, wave 3 next) 
Records the latent Phase 1 directory defect and its open library-vs-app decision,
the CreateConnection contract change, the new TestSupport library, and the
verification numbers at the wave-2 boundary.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:14:18 -04:00
Joseph Doherty 19ab0ac913 chore(localdb): record tasks 5-6 completion, the latent directory defect, and the CreateConnection contract change 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:06:06 -04:00
Joseph Doherty f2efeb37b7 refactor(sf,site): both stores take ILocalDb instead of a connection string 
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.

StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.

DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.

Two things the plan did not anticipate, both worth reading:

1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
   directory creation simply moved to LocalDb along with file ownership. It did
   not: the LocalDb library never creates the parent directory, and
   SqliteLocalDb opens the file eagerly in its constructor — so a missing
   directory is a hard boot failure ("SQLite Error 14: unable to open database
   file"), not a degraded start. The default site config points at the RELATIVE
   path ./data/site-localdb.db, so any site node without a pre-existing data/
   directory fails to boot. The docker rig escapes only because its volume mount
   happens to create /app/data — a coincidence that would have hidden this until
   a bare-metal or fresh deployment. This has been latent since Phase 1 made
   LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
   would have widened it. Re-established the guarantee at the layer that now
   owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
   pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
   tests written against the wrong assumption failed with exactly this
   SQLite Error 14 before the fix existed.

2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
   project; the constructor change actually reaches 40 files across 7 test
   projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
   equivalent for, so every one had to move to a real temp file. Rather than
   copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
   tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
   WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.

Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.

Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:05:45 -04:00
Joseph Doherty 3dfb288b74 chore(localdb): record tasks 3-4 completion and their deviations in the task state 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:21:09 -04:00
Joseph Doherty ac5eb12cce refactor(site): extract SiteStorageSchema.Apply from SiteStorageService 
Task 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync
and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending
only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers.

Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns
the connection's pragmas, and the service still opens its own connections until
Task 6.

One deviation: TryAddColumnAsync's `catch (SqliteException) when
(ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe,
matching OperationTrackingSchema. The message-matching form depends on an error
string that is not part of SQLite's contract, and it swallowed every
SqliteException whose message happened to contain that substring. The probe also
drops the ILogger dependency, which is what let the class become static. Cost:
the per-column "Migrated: added column" info log is gone — it fired once per
column per legacy database and nothing consumes it.

Also added a test beyond the plan's specified one, for the same reason as Task 3:
the specified test asserts against a freshly-created database, where CREATE TABLE
already lists the migration columns, so it would pass with every ALTER deleted.
The added test starts from the pre-migration shapes with a row present and proves
the migration path runs and preserves data across a NOT NULL DEFAULT add.

SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:20:50 -04:00
Joseph Doherty 9e3239c5d9 refactor(sf): extract StoreAndForwardSchema.Apply from the storage class 
Task 3 of the Phase 2 plan. Mirrors OperationTrackingSchema: the sf_messages
DDL, the four additive ALTERs, the last_attempt_at_ms backfill and the due-index
move verbatim into a static sync Apply(SqliteConnection), depending only on
Microsoft.Data.Sqlite. The Host needs to apply this to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers; the store
still calls it so a directly-constructed store stays self-sufficient.

Two deviations from the plan as written, both deliberate:

1. The PRAGMA journal_mode=WAL in InitializeAsync STAYS. The plan's Step 4
   snippet drops it, but Task 4 explicitly says not to move the equivalent
   pragma ("LocalDb owns the connection's pragmas") — the two tasks contradict
   each other. Keeping it preserves behaviour for the intermediate commits and
   for directly-constructed stores; it becomes moot in Task 5 when the store
   stops opening its own connections. Dropping it now would quietly regress the
   concurrent-writer support the pragma's own comment documents.

2. Added a second test beyond the plan's. The specified test asserts only
   against a freshly-created table, where CREATE TABLE already lists all 16
   columns — it would pass with every ALTER deleted. The added test starts from
   the pre-upgrade 12-column shape with a row in it and proves the upgrade path
   runs and preserves data.

That second test also surfaced that the backfill lands 1 ms low: julianday()'s
double day-fraction cannot represent every millisecond. Pre-existing behaviour,
carried over verbatim, asserted with a 1 ms tolerance rather than pinning a
precision the implementation never had.

Suite: 154 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:18:05 -04:00
Joseph Doherty 12eb97e07a docs(localdb): close the phase 2 gate — questions answered, plan written 
Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.

Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:

- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
  needs_snapshot (graceful resync), so the caps need no defensive sizing; the
  real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
  sweeps). The honest cost is D2's semantic change: the standby is convergent,
  no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
  deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
  leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.

§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.

Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.

GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:15:53 -04:00
Joseph Doherty d512572dac docs(localdb): phase 2 resume state — where the plan stands and what carries forward 
Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its
STOP verdict superseded, what must happen before Task 3, the four plan-premise
corrections, the rig gotchas that are not discoverable from the repo, and the
rig cleanup the Task 20 live gate needs.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:01:42 -04:00
Joseph Doherty 8652eab98e fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups 
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.

Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.

The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.

Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
  LocalDb-adjacent catch sites (the missing extended code is what made the
  original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
  (NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
  mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
  volume hung the reseed forever; compose mounts annotated as informational.

Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.

Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:01:00 -04:00
Joseph Doherty e9e11d635e docs(known-issues): root-cause brief for the LocalDb disk-I/O-error-under-load defect 
Standalone handoff for a follow-up investigation. Records the symptom, a full
reproduction (including the two rig-tooling blockers and the CachedCall-vs-Call
detail needed to generate load at all), the isolation evidence, a code map of
the LocalDb connection model, four ranked hypotheses, and acceptance criteria.

Not root-caused. Highest-value next step identified: capture the SQLite EXTENDED
result code (logs only carry the generic primary code 10), and run the load with
LocalDb:Path off the bind mount to partition environmental vs library causes.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:11:08 -04:00
Joseph Doherty 82c869a1df docs(localdb): record the phase 2 task-1 gate verdict in the task record 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:07:18 -04:00
Joseph Doherty fc553bd9ba docs(localdb): phase 2 rig soak findings — GATE FAILS on a phase 1 defect 
Task 1 of the Phase 2 plan. The gate stops the plan, but not for the reason it
anticipated: oplog sizing is fine and D6 is resolved.

BLOCKER: the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite
Error 10 'disk I/O error' on essentially every write on the ACTIVE node under
sustained load — site_events, OperationTracking and the audit telemetry paths
all fail, and site event logging silently drops events. Isolated to the load
(not the node, not host-side observation) by failing over between nodes, and to
LocalDb specifically (legacy store-and-forward.db / scadabridge.db in the same
bind-mounted directory take zero errors under identical load).

Phase 2 would register 8 more tables into that database — including the two
highest-volume ones — while deleting the bespoke mechanisms that currently
carry them. Must be root-caused first.

Also recorded: D6's premise corrected (largest known production config_json is
~60-70 KB, not >128 KB — but MaxBatchSize 500 is still unsafe, use 16); D4's
premise corrected (alarm writes are bounded by per-SourceReference coalescing at
a 100 ms flush); sf_messages has a hard 50 rows/sec structural ceiling; and
exceeding the oplog caps is a graceful snapshot-resync, not a failure.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:06:59 -04:00
Joseph Doherty cf46e59680 fix(rig): seed LDAP group mappings with the canonical role names 
docker/seed-sites.sh inserted the pre-rename 'Design' / 'Deployment' role
strings, but the canonical vocabulary in Roles.cs is 'Designer' / 'Deployer'.
The mismatch authorized nothing: on a freshly reseeded rig every
Designer/Deployer-gated management command failed UNAUTHORIZED, including
seed-sites.sh's own trailing `deploy artifacts` and reseed.sh's stage 6d
encrypted-secret restore.

Found while standing up the Phase 2 soak rig.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 17:55:04 -04:00
Joseph Doherty 25463d522f docs(localdb): phase 2 plan review pass — corrections from code verification 
Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs
plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write
has a second surviving caller, SiteReconciliationActor), D3 (the active node
already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs
row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not
in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 17:27:11 -04:00
Joseph Doherty f6ca82a9e2 docs(localdb): phase 2 implementation plan (21 tasks, full scope) 
Moves scadabridge.db's 9 config tables + sf_messages into the consolidated
LocalDb file and deletes SiteReplicationActor + StoreAndForward's
ReplicationService.

Resolves the phase 2 gate's five open questions from code recon (D1-D5):

D1 notify-and-fetch is DELETED, not preserved. It exists only because the
   config blob exceeds Akka's 128KB frame; LocalDb sync is gRPC. The
   deployed_at version guard protected against a stale fetch racing, so it
   dies with the fetch. Do not reproduce it on LWW - different clocks.
D2 ReplaceAllAsync is deleted and the N1 directional guard becomes
   unnecessary: LocalDb's snapshot resync merges per-row LWW and never
   deletes (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards a
   lower-HLC incoming row). Semantic change - the standby is convergent,
   no longer byte-identical.
D3 The SMTP purge (plaintext passwords) rides the replication path and is
   re-homed to the active node BEFORE any deletion.
D4 native_alarm_state volume is measured by a rig soak, not assumed. Task 1
   gates the plan and stops it if the oplog cannot absorb the churn.
D5 No dual-mechanism period forecloses rolling site upgrades - both nodes
   must stop and start together.

Recon also found the gate doc's "two test files are the spec" undercounts:
the real specification is five files, including the N1 Critical regression
test and the only Requeue coverage.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 15:04:37 -04:00
164 changed files with 9713 additions and 4011 deletions
Expand all files Collapse all files
CLAUDE.md
+21 -7
View File
@@ -120,15 +120,27 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- DCL write failures returned synchronously to calling script.
- Tag path resolution retried periodically for devices still booting.
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1, 2026-07-19).** `OperationTracking` and `site_events` now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file, configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
- `ScadaBridge:OperationTracking:ConnectionString` and `ScadaBridge:SiteEventLog:DatabasePath` are **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started — see `docs/plans/2026-07-19-localdb-phase2-gate.md`.
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them:
- `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing.
- `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now.
- `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
- All timestamps are UTC throughout the system.
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
- Inter-cluster communication uses **three** transports, not two: **ClusterClient** for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots); **gRPC** server-streaming for real-time data (attribute values, alarm states); and **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist (**per node, not as a singleton** — contact rotation reaches whichever node answers). Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. **Discovery is asymmetric by design:** central discovers sites from the *database* (`Site.NodeAAddress`/`NodeBAddress`, refreshable at runtime), sites discover central from *appsettings* (`ScadaBridge:Communication:CentralContactPoints`, static — restart required). **Central never buffers for an unreachable site** — the send is dropped with a warning and the caller's Ask times out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded at `AkkaHostedService.cs:191`. Central and each site are separate clusters *only* by seed-node partitioning. This is required, not incidental: Akka.Remote address matching means a ClusterClient could not reach a differently-named system.
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`BuildRoles`, `AkkaHostedService.cs:386`). Singletons scope to the **site-specific** role.
- **Neither inter-cluster transport is authenticated or encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths`; the gRPC listener is **h2c** and `LocalDbSyncAuthInterceptor` gates *only* `/localdb_sync.v1.LocalDbSync/`, so the whole `SiteStreamService` surface — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — is callable by anyone who can reach :8083. The boundary assumes a trusted network. (The HTTP fetch token and the LocalDb sync interceptor are the only authenticated pieces.)
- **Akka frame size is the default 128 KB with `log-frame-size-exceeding` off**, and no custom serializer is configured — so payload carried over ClusterClient is JSON-escaped a second time by the default Newtonsoft serializer, roughly doubling it. Over the limit the transport drops **that one message** without tearing down the association (heartbeats keep flowing, the site still reports healthy) and the central Ask simply times out. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`; `DeployArtifactsCommand` still carries payload and remains exposed.
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. There is **no gRPC server on central at all**, which is why the two `Ingest*` unary RPCs — documented as a "central-side ingest surface" — are dead in practice (acknowledged in `AkkaHostedService.cs:510-519`); sites reach central over ClusterClient instead. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: `sitestream.proto`, **6 RPCs** (2 server-streaming: `SubscribeInstance`, `SubscribeSite` (site-wide, alarm-only); 4 unary: `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`), `SiteStreamEvent` (oneof: AttributeValueUpdate, AlarmStateUpdate). Field numbers are never reused; evolution is additive only (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. DebugStreamEvent message removed (no longer flows through ClusterClient).
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up).
@@ -201,11 +213,13 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
### Cluster & Failover
- Keep-oldest split-brain resolver with `down-if-alone = on`, 15s stable-after.
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- Both nodes are seed nodes. `min-nr-of-members = 1`.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
- CoordinatedShutdown for graceful singleton handover.
- Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
### UI & Monitoring
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
Directory.Packages.props
+24 -3
View File
@@ -108,9 +108,9 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" />
</ItemGroup>
<!--
@@ -145,6 +145,27 @@
<PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
</ItemGroup>
<!--
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h,
GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against
System.Security.Cryptography.Xml 10.0.7, pulled in TRANSITIVELY by
Microsoft.AspNetCore.DataProtection 10.0.7 (ConfigurationDatabase's DataProtection
key storage). With TreatWarningsAsErrors any FRESH restore — notably the docker
image build — went red (surfaced 2026-07-21; local builds had cached audit data).
Same pattern as SQLitePCLRaw above: pin the vulnerable transitive package to its
patched version (10.0.10) with an explicit <PackageReference> in the one project
where the chain enters (ConfigurationDatabase; every other resolver — AuditLog,
SiteCallAudit, Transport, PerformanceTests, tests — reaches it through that
ProjectReference). Bumping the DataProtection parent instead was tried and
rejected: 10.0.10 floors Microsoft.Extensions.* and (via the EFCore adapter)
Microsoft.EntityFrameworkCore at 10.0.10, forcing a family-wide servicing bump
(NU1605 downgrade errors) that belongs in its own reviewed commit.
-->
<ItemGroup>
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
</ItemGroup>
<!--
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached only transitively via bunit
in ZB.MOM.WW.ScadaBridge.CentralUI.Tests. With TreatWarningsAsErrors it made the WHOLE
ZB.MOM.WW.ScadaBridge.slnx
+1
View File
@@ -36,6 +36,7 @@
<Project Path="tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/ZB.MOM.WW.ScadaBridge.Communication.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests.csproj" />
alog.md
+9 -2
View File
@@ -108,7 +108,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `EventId` | `uniqueidentifier` PK | Generated where the event originates (site or central). Idempotency key. |
| `OccurredAtUtc` | `datetime2` | When the event happened (call returned, retry attempted, etc.). |
| `IngestedAtUtc` | `datetime2` | When central persisted the row (lags `OccurredAtUtc` for site-originated rows). |
| `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound`. |
| `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound` \| `SecuredWrite` \| `Cluster`. The last two are not script trust-boundary crossings: `SecuredWrite` records the two-person write lifecycle, and `Cluster` records operator-initiated topology actions (admin-triggered manual failover, decision 2026-07-22). |
| `Kind` | `varchar(32)` | Event kind discriminator (see kinds list below). |
| `CorrelationId` | `uniqueidentifier` NULL | Ties multi-event operations together. `TrackedOperationId` for cached calls, `NotificationId` for notifications, request-id for inbound API. NULL for sync one-shot calls. |
| `SourceSiteId` | `varchar(64)` NULL | NULL for central-originated events (inbound API, central notification dispatch). |
@@ -135,7 +135,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
- `IX_AuditLog_Target_Occurred (Target, OccurredAtUtc)` — "what did we send to system X."
- Partitioning by month on `OccurredAtUtc` from day one (purge becomes a partition switch instead of a delete storm).
**`Kind` values (flat — 10 discriminators across all channels):**
**`Kind` values (flat — 17 discriminators across all channels; pinned by `AuditEnumTests`):**
| Kind | Fires when |
|---|---|
@@ -149,6 +149,13 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `InboundAuthFailure` | An inbound API request was rejected at the auth boundary (bad/missing key). One row, `Status=Failed`, `HttpStatus=401`. |
| `CachedSubmit` | Script-side enqueue of a cached call (`ExternalSystem.CachedCall` / `Database.CachedWrite`); first row in the cached-call lifecycle, written to site SQLite before any forward attempt. |
| `CachedResolve` | Terminal row for a cached operation — `Status` = `Delivered` / `Failed` / `Parked` / `Discarded`. |
| `SecuredWriteSubmit` | An Operator submitted a two-person secured write; row written after the `PendingSecuredWrite` is persisted so it carries the store-assigned id as `CorrelationId`. |
| `SecuredWriteApprove` | A distinct Verifier approved a pending secured write (no self-approval; enforced server-side). |
| `SecuredWriteReject` | A Verifier rejected a pending secured write. |
| `SecuredWriteExecute` | An approved secured write was relayed to the site MxGateway connection. |
| `SecuredWriteExpire` | A `Pending` secured write aged past its server-side TTL and was transitioned to `Expired` without executing — emitted by the system (no verifier). |
| `ReconciliationAbandoned` | A reconciliation pull row failed to insert up to the permanent-abandon threshold and central advanced its cursor past it; one synthetic row so the loss is queryable in the Audit Log itself. |
| `ManualFailover` | An administrator triggered a manual failover of the central pair from the Health page; one row per invocation, written BEFORE the graceful `Cluster.Leave` is issued. `Target` = the leaving node's address. |
### Site: `AuditLog` (SQLite)
docker-env2/central-node-a/appsettings.Central.json
+1 -1
View File
@@ -11,7 +11,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker-env2/central-node-b/appsettings.Central.json
+3 -3
View File
@@ -8,10 +8,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker-env2/site-x-node-a/appsettings.Site.json
+10 -3
View File
@@ -13,13 +13,17 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -29,8 +33,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
docker-env2/site-x-node-b/appsettings.Site.json
+12 -5
View File
@@ -10,16 +10,20 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -29,8 +33,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
docker/README.md
+41 -12
View File
@@ -273,29 +273,43 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l
### Automated failover drill (`failover-drill.sh`)
```bash
DRILL_MODE=standby bash docker/failover-drill.sh # default — survivable younger-node crash
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — measures the registered outage gap
DRILL_MODE=standby bash docker/failover-drill.sh # default — younger-node crash, active untouched
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — survivor must TAKE OVER
```
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, because under the unified oldest-member semantics the *active* node IS the oldest, i.e. the one crash two-node keep-oldest cannot survive:
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The survivable direction: SBR downs the crashed member and the active node keeps its singletons. Expected result: **no routing outage at all** (the active node is never touched, so `/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive). PASS = the survivor logs the member removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** Expected result: a **total central outage** until the victim container is restarted — this is the registered deferred keep-oldest decision (master tracker 2026-07-08): keep-oldest downs the partition *without* the oldest, so the younger survivor downs itself, and it cannot re-form a cluster alone (see the seed-node constraint below). The drill confirms the dark window, then recovery within ~2 min of restarting the victim. The mode exists to make the registered gap *observable*, not to pretend it is covered.
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** The survivor must **take over while the victim is still down**: it auto-downs the dead oldest, becomes the oldest member itself, re-hosts all singletons, and its `/health/active` goes 200. PASS = survivor active within `TIMEOUT_S`, then Traefik routing to it. (Under the pre-2026-07-21 `keep-oldest` strategy this direction was a proven total outage — the younger survivor took `DownReachable` and downed itself, because Akka's `down-if-alone` only rescues a side with ≥ 2 members.)
The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
Both modes finish by restarting the victim and confirming it rejoins as a ready standby. The drill exercises downing-on-hard-crash, S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
**Seed-node bootstrap constraint.** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — it never reaches `Up`, and `/health/active` never returns 200. Operator recovery actions: **(1)** restart the dead first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (the cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
**Partition trade (accepted).** Auto-down is availability-first: in a *real network partition* (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
> **Observed results** (plan R2-01 T3):
**Seed-node ordering — every node lists ITSELF first (decision 2026-07-22).** Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node's own address; every other node runs `JoinSeedNodeProcess`, which retries `InitJoin` forever and can never form a cluster. Each shipped node config therefore lists itself first and its partner second (`docker/central-node-b/appsettings.Central.json` leads with `scadabridge-central-b`), and `StartupValidator` fails the boot if that ordering is ever broken. This closes the former **registered outage gap**, where a lone cold-starting `central-b` (with `central-a` down) never came `Up` and recovery was operator-driven.
Self-first ordering is safe, and the three interesting cases are covered by `SelfFirstSeedBootstrapTests` (real in-process clusters at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Forms alone in ~5s (`seed-node-timeout`) — operational, unattended |
| Restart into a **live** peer | `InitJoinAck` answers, node rejoins; never islands |
| Both cold-start simultaneously (mutually reachable) | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
> An earlier revision of this README claimed the repo deliberately avoided self-first ordering because simultaneous cold start would produce "two one-node clusters that never merge". That is **not** what happens while the nodes are mutually reachable — the handshake converges them (measured, row 3 above). Only a genuine boot-time *partition* splits them, which is the same class `auto-down` already accepts.
> **Rejected alternative — an external self-form timer.** A watchdog that waits N seconds for membership and then calls `Cluster.Join(SelfAddress)` was implemented and discarded: it cannot see Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address — a **permanent** split (measured: still split after 90s). Akka's own first-seed process has no such race because it *is* part of the handshake.
> **Observed results** (auto-down decision verification):
>
> **Run 2026-07-13** against a freshly-deployed cluster on `main` @ `99544985` (round-2 merged image; `active=central-a`). Both directions behaved exactly as the design predicts.
> **Run 2026-07-21** against a freshly-deployed cluster with `SplitBrainResolverStrategy: auto-down` (first drill: `active=central-a`). Both directions recovered.
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | **PASS** — SBR downed+removed the crashed `central-b`; active `central-a` kept all 7 singletons; recovered on restart. | Member removed in **27s** (budget ~25s: 10s detection + 15s stable-after); **0** `/health/active` routing blips (active node never touched); routable **0s** after victim restart. |
> | `active` (oldest-node crash) | **Outage as designed** — killing the oldest/active `central-a` made the younger `central-b` self-down (total central outage — the registered keep-oldest gap); recovered after restarting the victim, `central-b` then assuming Oldest and re-hosting all singletons. | Outage confirmed at **9s**; central routable again **4s** after restarting `central-a`. |
> | `active` (oldest-node crash) | **PASS — TAKEOVER** — `central-b` auto-downed the dead oldest, went `Younger -> Oldest` on all 7 singletons, and served `/health/active` **while the victim was still down**; restarted victim rejoined as standby. | Survivor active + Traefik routing in **28s** (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready **2s** after restart. |
> | `standby` (younger-node crash) | **PASS** — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby. | Member removed in **27s**; **0** `/health/active` routing blips; victim ready **2s** after restart. |
>
> Notes: the `standby` PASS shows the survivable direction is clean end-to-end (SBR `DownUnreachable` decision + per-singleton "Member removed" in the survivor log, zero routing interruption). The `active` result **empirically confirms the deferred keep-oldest topology gap** (master tracker 2026-07-08 / `docs/plans/2026-07-08-deferred-work-register.md`): a hard crash of the active/oldest central node is a total outage until that node (the first seed) is restarted — the remedy remains the pending topology/strategy decision. In-process envelope (`FailoverTimingTests`, plan R2-01 T4) independently measured full failover at **33.7s**.
> Historical baseline (keep-oldest, run 2026-07-13 on `99544985`): `standby` PASS with member removal in 27s / 0 routing blips; `active` was a **total outage** — `central-b` self-downed ~20s after the kill (live SBR log 2026-07-21: `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`) and could not re-bootstrap until `central-a` returned. That result is what motivated the auto-down decision. In-process envelope (`FailoverTimingTests`) measured full failover at **33.7s**.
### Central Failover
@@ -313,6 +327,14 @@ open http://localhost:9002
docker start scadabridge-central-a
```
**Manual failover from the UI (admin-only).** Instead of stopping a container, an Administrator can trigger a planned role swap from the **Trigger failover** button on the central-cluster card at `/monitoring/health` (via Traefik, `http://localhost:9000`). The active (oldest Up) node leaves the cluster **gracefully**, so singletons hand over rather than being killed; the node then restarts under `restart: unless-stopped` and rejoins as the standby.
- The button is disabled when the pair has no online standby — the same guard is re-enforced server-side, since failing over a lone node is an outage, not a failover.
- Triggering it **disconnects the page you clicked it on**: Traefik routes the UI to the active node, which is the node being restarted. The page reconnects against the new active node.
- Each invocation writes one `Cluster` / `ManualFailover` row to `dbo.AuditLog` naming the admin and the target address, written before the Leave is issued.
To verify on the rig: press the button, watch `central-a` restart and `central-b`'s badge flip to Primary, then confirm the audit row landed.
### Site Failover
```bash
@@ -329,3 +351,10 @@ docker start scadabridge-site-a-a
Same pattern applies for site-b (`scadabridge-site-b-a`/`scadabridge-site-b-b`) and site-c (`scadabridge-site-c-a`/`scadabridge-site-c-b`).
Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).
**Manual site failover from the UI (admin-only).** Each site card on `/monitoring/health` carries the same **Trigger failover** button as the central card. Central and each site are separate Akka clusters, so this is a *request* relayed over the ClusterClient command/control channel — the site's own communication actor performs the graceful `Leave` against its `site-{SiteId}` role and acks the result.
- Unlike central failover, this does **not** disconnect your page — a site is a different cluster.
- A refusal from the site (no standby, or a command addressed to a different site) reads differently from an unreachable site (Ask timeout); the UI shows the site's own reason. Only the timeout leaves any doubt about whether the failover took effect.
- A site running an older binary has no handler for the command, so it dead-letters and you see "site did not respond".
- Each invocation writes a `Cluster` / `ManualFailover` audit row stamped with the site id.
docker/central-node-a/appsettings.Central.json
+1 -1
View File
@@ -11,7 +11,7 @@
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker/central-node-b/appsettings.Central.json
+3 -3
View File
@@ -8,10 +8,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"akka.tcp://scadabridge@scadabridge-central-b:8081",
"akka.tcp://scadabridge@scadabridge-central-a:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker/failover-drill.sh
+52 -39
View File
@@ -1,33 +1,33 @@
#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
# ACTIVE central node — but under the unified oldest-member semantics the
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
# survive (registered deferred user decision, master tracker 2026-07-08;
# SbrFailoverTests.cs XML doc). Two modes:
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster now runs the 'auto-down'
# downing strategy (availability-first): the leader among the REACHABLE members
# downs the unreachable peer after StableAfter, so a hard crash of EITHER
# central node — the active/oldest included — fails over to the survivor. The
# accepted trade (made explicitly by the owner) is dual-active during a real
# network partition. Both drill directions therefore expect RECOVERY:
#
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
# The survivable direction: SBR downs the crashed member, the active node
# keeps its singletons, and Traefik routing never goes dark. PASS = the
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
# failure detection + 15s stable-after) while /health/active stays up.
# The active node is untouched: expect zero /health/active routing blips
# and member removal on the survivor within ~25s (10s failure detection +
# 15s auto-down-unreachable-after).
#
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
# cannot help — the alone-oldest is dead and cannot down itself), and the
# self-downed survivor cannot re-form a cluster alone unless it is the
# FIRST seed (both nodes list central-a first; only the first seed may
# self-join). This mode measures the dark window and PASSes only when
# central recovers AFTER the victim container is restarted. It exists to
# make the registered gap observable — not to pretend it is covered.
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE SURVIVOR
# MUST TAKE OVER: it downs the dead oldest, becomes oldest itself, hosts
# the singletons, and /health/active goes 200 on the survivor WHILE THE
# VICTIM IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe
# margin. (Under the pre-2026-07-21 keep-oldest strategy this direction
# was a total outage — the younger survivor downed ITSELF, verified live;
# Akka's down-if-alone only rescues a side with >= 2 members.)
#
# Both modes finish by restarting the victim and confirming it rejoins as a
# fresh incarnation (standby).
set -euo pipefail
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
active_container() {
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
@@ -35,6 +35,7 @@ active_container() {
else echo "ERROR: no active central node found" >&2; exit 1; fi
}
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
port_of() { [ "$1" = scadabridge-central-a ] && echo 9001 || echo 9002; }
case "$DRILL_MODE" in
standby|active) ;;
@@ -47,6 +48,7 @@ if [ "$DRILL_MODE" = standby ]; then
else
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi
SURVIVOR_PORT=$(port_of "$SURVIVOR")
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
@@ -54,60 +56,71 @@ docker kill "${VICTIM}" > /dev/null
START=$(date +%s)
if [ "$DRILL_MODE" = standby ]; then
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (budget ~25s)..."
BLIPS=0
while true; do
ELAPSED=$(( $(date +%s) - START ))
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "auto-downing|marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor downed/removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s auto-down)."
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
break
fi
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — auto-down did not act" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
else
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
DARK_STREAK=0
echo "Active crash: waiting for ${SURVIVOR} to take over as the active node (victim stays DOWN; budget ~25s + hand-over)..."
while true; do
ELAPSED=$(( $(date +%s) - START ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
if (( DARK_STREAK >= 10 )); then
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
if curl -sf -o /dev/null "http://localhost:${SURVIVOR_PORT}/health/active"; then
echo "PASS: ${SURVIVOR} took over as active in ${ELAPSED}s with the victim still down"
echo "(downed the dead oldest via auto-down, assumed Oldest, re-hosted the singletons)."
break
fi
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
break
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of killing the oldest — takeover did not happen." >&2
docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|oldest|shutting down|terminated" | tail -20 >&2 || true
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Confirming Traefik routes to the new active node..."
TR_START=$(date +%s)
while ! curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; do
if (( $(date +%s) - TR_START > 60 )); then
echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Traefik routing recovered $(( $(date +%s) - START ))s after the kill."
fi
echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
echo "Waiting for the restarted victim to rejoin as a ready standby (${VICTIM} /health/ready)..."
VICTIM_PORT=$(port_of "$VICTIM")
while true; do
ELAPSED=$(( $(date +%s) - RESTART_AT ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
if curl -sf -o /dev/null "http://localhost:${VICTIM_PORT}/health/ready"; then
echo "Recovered: ${VICTIM} is ready (rejoined as a fresh incarnation) ${ELAPSED}s after restart."
break
fi
if (( ELAPSED > 120 )); then
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
echo "FAIL: ${VICTIM} not ready 120s after restart" >&2
exit 1
fi
sleep 1
done
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
echo "Survivor downing/singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "auto-downing|singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
docker/seed-sites.sh
+9 -4
View File
@@ -96,7 +96,7 @@ for ident in site-a site-b site-c; do
done
echo ""
echo "Seeding LDAP group mappings (Design + Deployment)..."
echo "Seeding LDAP group mappings (Designer + Deployer)..."
# SecurityConfiguration.HasData declares 4 mappings but the InitialSchema
# migration only inserts the Admin row, so a fresh ScadaBridgeConfig starts
# with multi-role getting Admin only -- no Design and no Deployment access.
@@ -106,11 +106,16 @@ docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-d ScadaBridgeConfig -Q "
SET IDENTITY_INSERT LdapGroupMappings ON;
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 2)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Design');
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Designer');
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 3)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployment');
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployer');
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 4)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployment');
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployer');
-- Role strings MUST match the canonical vocabulary in
-- src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs ('Designer' / 'Deployer').
-- These rows previously carried the pre-rename 'Design' / 'Deployment', which
-- authorized nothing: every Designer/Deployer-gated management command failed
-- UNAUTHORIZED on a freshly reseeded rig.
SET IDENTITY_INSERT LdapGroupMappings OFF;
"
docker/site-a-node-a/appsettings.Site.json
+28 -4
View File
@@ -14,13 +14,17 @@
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
@@ -80,7 +87,24 @@
// pre-host secret expander.
"Replication": {
"PeerAddress": "http://scadabridge-site-a-b:8083",
"ApiKey": "dev-site-a-localdb-sync-key"
"ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
//
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
// size in bytes is set by the widest replicated column. That is
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
"MaxBatchSize": 16,
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That
// makes tighter-than-default correct here - it trades a rare full resync for a
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
// room for burst without approaching the 1,000,000 default.
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
}
}
}
docker/site-a-node-b/appsettings.Site.json
+30 -6
View File
@@ -11,16 +11,20 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
@@ -73,7 +80,24 @@
// fail-closed, so a typo here does not degrade to unauthenticated replication;
// it rejects every stream and the pair silently stops converging.
"Replication": {
"ApiKey": "dev-site-a-localdb-sync-key"
"ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
//
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
// size in bytes is set by the widest replicated column. That is
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
"MaxBatchSize": 16,
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That
// makes tighter-than-default correct here - it trades a rare full resync for a
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
// room for burst without approaching the 1,000,000 default.
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
}
}
}
docker/site-b-node-a/appsettings.Site.json
+10 -3
View File
@@ -14,13 +14,17 @@
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
docker/site-b-node-b/appsettings.Site.json
+12 -5
View File
@@ -11,16 +11,20 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
docker/site-c-node-a/appsettings.Site.json
+10 -3
View File
@@ -14,13 +14,17 @@
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
docker/site-c-node-b/appsettings.Site.json
+12 -5
View File
@@ -11,16 +11,20 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
docs/components/ClusterInfrastructure.md
+96 -61
View File
@@ -1,23 +1,33 @@
# Cluster Infrastructure
The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, split-brain resolution, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.
The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, the downing strategy for unreachable members, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.
## Overview
Cluster Infrastructure (#13) is a **design responsibility** spanning two projects rather than a single buildable project:
- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, split-brain settings), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, downing strategy), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.Host/`** owns the cluster bootstrap and runtime wiring: `AkkaHostedService` builds the Akka HOCON from `ClusterOptions` and `NodeOptions`, starts the `ActorSystem`, wires `CoordinatedShutdown`, and creates all role-specific actors including the cluster singletons.
This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. `ClusterInfrastructure` is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host.
Both central and site clusters run this same topology: two nodes, one active (cluster leader), one standby, with automatic failover and no manual intervention required for dual-node recovery.
Both central and site clusters run this same topology: two nodes, one active (the oldest `Up` member), one standby, with automatic failover and no manual intervention required for dual-node recovery.
## Key Concepts
### Active/standby via cluster leadership
### One `ActorSystem` name for every cluster
Akka.NET cluster leadership determines which node is "active". The cluster leader is the oldest node in the cluster, as tracked by the keep-oldest split-brain resolver. `ActiveNodeGate` (in the Host) exposes `IsActiveNode` by checking whether `cluster.SelfMember.Status == MemberStatus.Up` and `cluster.State.Leader == cluster.SelfAddress`. Cluster singletons — which run on the oldest `Up` member — automatically migrate to the surviving node on failover.
Every node in every cluster — central and all sites — joins an `ActorSystem` named **`"scadabridge"`**, hardcoded at `AkkaHostedService.cs:191` (`ActorSystem.Create("scadabridge", config)`). Central and each site are separate clusters *only* by seed-node partitioning, not by system name. This is required rather than incidental: Akka.Remote matches addresses including the system name, so a `ClusterClient` could not reach a differently-named system.
### Active/standby is the oldest `Up` member — never the cluster leader
A node is "active" when it is the **oldest `Up` member** of its role scope — the member `ClusterSingletonManager` places singletons on. Akka's *cluster leader* (lowest address) is a different, Akka-internal concept: it diverges from singleton placement permanently once the original first node restarts and rejoins. Every product-level active/standby decision therefore goes through one evaluator and never reads `cluster.State.Leader`:
- `ActiveNodeEvaluator.SelfIsOldestUp(Cluster, string? role)` (`Communication/ClusterState/ActiveNodeEvaluator.cs:35`) is the single implementation — self is `Up`, carries the role when one is given, and no other `Up` member in that scope is older (`self.IsOlderThan(m)`).
- `ClusterActivityEvaluator.SelfIsOldest` (`Host/Health/ClusterActivityEvaluator.cs:23`) delegates to it, and is what `ActiveNodeGate.IsActiveNode` (`Host/Health/ActiveNodeGate.cs:48`), `OldestNodeActiveHealthCheck`, and `AkkaClusterNodeProvider.SelfIsPrimary` all call.
- `SiteCommunicationActor` stamps its heartbeat's `IsActive` from the same evaluator (`Communication/Actors/SiteCommunicationActor.cs:517-518`).
Cluster singletons automatically migrate to the surviving node on failover, and because "active" is defined as the singleton-placement member, the health/routing view and the singleton view can never disagree.
### Configuration contract vs. bootstrap split
@@ -33,22 +43,26 @@ Cluster Infrastructure provides the hosting platform; each singleton is owned an
`AkkaHostedService.BuildHocon` constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through `QuoteHocon` (string escaping) and `DurationHocon` (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision.
The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits three additional stanzas: `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).
The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).
The downing block is **not** a fixed stanza — `BuildHocon` branches on `ClusterOptions.SplitBrainResolverStrategy` and emits one of two shapes (`AkkaHostedService.cs:275-286`):
```csharp
// Abbreviated — see AkkaHostedService.BuildHocon for the full method.
public static string BuildHocon(
NodeOptions nodeOptions,
ClusterOptions clusterOptions,
IEnumerable<string> roles,
TimeSpan transportHeartbeat,
TimeSpan transportFailure)
{
var seedNodesStr = string.Join(",",
clusterOptions.SeedNodes.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
var downingBlock = string.Equals(
clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
: $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}";
return $@"
return $@"
audit-telemetry-dispatcher {{
type = ForkJoinDispatcher
throughput = 100
@@ -66,13 +80,7 @@ akka {{
seed-nodes = [{seedNodesStr}]
roles = [{rolesStr}]
min-nr-of-members = {clusterOptions.MinNrOfMembers}
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}
{downingBlock}
failure-detector {{
heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
@@ -83,23 +91,35 @@ akka {{
run-by-clr-shutdown-hook = on
}}
}}";
}
```
A `downing-provider-class` is always named explicitly. Akka defaults to `NoDowning`, under which the downing configuration is inert and singletons never migrate on a hard crash or partition; naming the provider is what activates automatic downing.
The HOCON also defines the `audit-telemetry-dispatcher` (a two-thread `ForkJoinDispatcher`) so `SiteAuditTelemetryActor`'s SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors.
### Split-brain resolution
Nothing in the emitted document enables remoting TLS or an Akka secure cookie — there is no `enable-ssl`, no `require-cookie`, no `trusted-selection-paths`. Akka remoting between nodes and from a `ClusterClient` is plaintext and unauthenticated; the deployment is assumed to sit on a trusted network.
The keep-oldest strategy is the only strategy `ClusterOptionsValidator` permits for ScadaBridge's two-node clusters. Quorum strategies (`keep-majority`, `static-quorum`) cannot distinguish a crash from a partition with two nodes — both sides would be below quorum and both would shut down. Keep-oldest with `down-if-alone = on` ensures at most one node runs the cluster at any time:
### Downing strategy (auto-down — availability-first)
- On a network partition, the older node stays active; the younger node downs itself.
- If the oldest node finds itself alone (no reachable members), it downs itself rather than running in isolation. Without `down-if-alone`, the oldest node could run as a single-node cluster while the younger node forms its own — producing two live clusters with divergent singleton state.
**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **`StableAfter` is the debounce**, not a resolver phase: 15 s of sustained unreachability before downing, which absorbs startup, rolling restarts, and transient blips.
`keep-oldest` remains a supported value (`ClusterOptionsValidator` allows exactly `auto-down` and `keep-oldest`) for deployments that prefer partition-safety, but it **cannot survive a crash of the oldest node in a two-node cluster**: Akka's `down-if-alone` only rescues the survivor when its own side has ≥ 2 members, so a 1-vs-1 survivor takes `DownReachable` and downs *itself*. Quorum strategies are rejected outright — `static-quorum` with quorum 1 trips Akka's `IsTooManyMembers` guard and downs *all* members on any unreachability, and `keep-majority` merely moves the fatal crash from the oldest node to the lowest-address node.
### Downed-node recovery
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
### Failure detection and failover timeline
Detection uses two independent Akka heartbeat channels:
- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers `Unreachable` events that the split-brain resolver acts on.
- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers the `Unreachable` events the downing provider acts on.
- **Transport failure detector** (`akka.remote.transport-failure-detector`): monitors the underlying TCP transport between nodes; configured separately from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`.
With the defaults in `ClusterOptions`, the total failover budget is approximately 25 seconds:
@@ -107,28 +127,33 @@ With the defaults in `ClusterOptions`, the total failover budget is approximatel
| Phase | Duration | Source |
|-------|----------|--------|
| Failure detection (`acceptable-heartbeat-pause`) | 10 s | `ClusterOptions.FailureDetectionThreshold` |
| Split-brain stable-after | 15 s | `ClusterOptions.StableAfter` |
| Downing window (`auto-down-unreachable-after`) | 15 s | `ClusterOptions.StableAfter` |
| Singleton restart | < 1 s | Actor `PreStart` |
The docker failover drill (`docker/failover-drill.sh`) measures both directions — `standby` mode kills the younger node, `active` mode kills the active/oldest node and asserts the survivor takes over while the victim is still down.
### Graceful shutdown and singleton handover
When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout. `SiteCallAuditActor` has an explicit graceful-stop task registered on `PhaseClusterLeave` with a 10-second timeout to drain any in-flight EF Core upsert before handover opens:
When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout.
Every singleton is created through the shared `SingletonRegistrar.Start` helper (`Host/Actors/SingletonRegistrar.cs`), so the drain is uniform rather than per-singleton boilerplate. The registrar applies the canonical `{name}-singleton` / `{name}-proxy` naming, a `PoisonPill` termination message, an optional `.WithRole(role)` on both the manager and proxy settings, and a `PhaseClusterLeave` task that `GracefulStop`s the manager (10-second default) so in-flight EF Core (central) or SQLite (site) work completes before handover opens:
```csharp
siteCallAuditShutdown.AddTask(
// SingletonRegistrar.Start — the drain task registered for every singleton
Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
"drain-site-call-audit-singleton",
$"drain-{name}-singleton",
async () =>
{
try
{
await siteCallAuditSingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
await manager.GracefulStop(timeout);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"SiteCallAudit singleton did not drain within the graceful-stop "
+ "timeout; falling through to PoisonPill handover");
logger.LogWarning(ex,
"{Singleton} singleton did not drain within the graceful-stop timeout; "
+ "falling through to PoisonPill handover", name);
}
return Akka.Done.Instance;
});
@@ -136,25 +161,29 @@ siteCallAuditShutdown.AddTask(
### Cluster roles and singleton scoping
Each node carries one or more cluster roles set in the HOCON `roles` list. Site nodes carry both a base `"Site"` role and a site-specific role (`"site-{SiteId}"`, e.g. `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster, not on any other site's nodes. Central singletons use no role scope — all central nodes share the `"Central"` role.
Each node carries one or more cluster roles set in the HOCON `roles` list, built by `AkkaHostedService.BuildRoles` (`AkkaHostedService.cs:406-417`). Site nodes carry **two** roles: the base `"Site"` role plus a site-specific `"site-{SiteId}"` (a node with `SiteId: "site-a"` gets `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster. Central singletons pass no role to the registrar and so are unscoped — all central nodes share the `"Central"` role.
### Dual-node recovery
Because both nodes are configured as seed nodes, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. No startup ordering dependency exists, and no manual intervention is required. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with.
Because both nodes are configured as seed nodes **and each lists itself first**, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. Since 2026-07-22 there is no remaining ordering dependency: a node that must boot *alone* forms a cluster regardless of which node it is. Two nodes cold-starting at the same moment converge on one cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class `auto-down` already accepts.
### Cluster singletons hosted
The Host wires the following singletons. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern; each singleton's behaviour is documented in the owning component.
The Host wires the following singletons through `SingletonRegistrar.Start`. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern and the drain hook; each singleton's behaviour is documented in the owning component.
**Central singletons (active central node, no role scope):**
**Central singletons (oldest `Up` central node, no role scope):**
| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
| `notification-outbox` | `NotificationOutboxActor` | Notification Outbox (#21) |
| `audit-log-ingest` | `AuditLogIngestActor` | Audit Log (#23) |
| `site-call-audit` | `SiteCallAuditActor` | Site Call Audit (#22) |
| `audit-log-purge` | `AuditLogPurgeActor` | Audit Log (#23) |
| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Audit Log (#23) |
| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI History |
| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Deployment Manager (#2) |
**Site singletons (active site node, scoped to `"site-{SiteId}"` role):**
**Site singletons (oldest `Up` node of that site, scoped to the `"site-{SiteId}"` role):**
| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
@@ -173,29 +202,29 @@ Every host calls `AddClusterInfrastructure` to register `ClusterOptionsValidator
services.AddClusterInfrastructure();
```
This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section (wrong strategy, `MinNrOfMembers != 1`, `DownIfAlone = false`, fewer than two seed nodes) throws an `OptionsValidationException` at startup rather than booting into a broken cluster.
This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section throws an `OptionsValidationException` at startup rather than booting into a broken cluster. The validator rejects: a strategy other than `auto-down` or `keep-oldest`; `MinNrOfMembers != 1`; a non-positive `StableAfter`, `HeartbeatInterval` or `FailureDetectionThreshold`; a `HeartbeatInterval` not below `FailureDetectionThreshold`; fewer than two seed nodes unless `AllowSingleNodeCluster = true`; and `DownIfAlone = false` **only when the strategy is `keep-oldest`** (the flag is inert under `auto-down`, so any value passes there).
### Checking active-node status
Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root):
Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root). The gate is a thin wrapper over the oldest-`Up` evaluator — it never inspects cluster leadership:
```csharp
// Host/Health/ActiveNodeGate.cs
public bool IsActiveNode
{
get
{
var system = _akkaService.ActorSystem;
if (system == null) return false;
if (system == null)
return false;
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up) return false;
var leader = cluster.State.Leader;
return leader != null && leader == self.Address;
return ClusterActivityEvaluator.SelfIsOldest(cluster);
}
}
```
This returns `false` while the actor system is warming up — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes.
This returns `false` while the actor system is warming up, and `SelfIsOldest` returns `false` unless the node has reached `MemberStatus.Up` — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes, and `OldestNodeActiveHealthCheck` backs `/health/active` off the same evaluator, so the proxy's routing decision and the API's gating decision can never disagree.
## Configuration
@@ -205,13 +234,14 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries; both nodes list both themselves and their partner. |
| `SplitBrainResolverStrategy` | `string` | `"keep-oldest"` | Must be `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Cluster must be stable for this duration before the resolver acts to down unreachable nodes. |
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries (1 with `AllowSingleNodeCluster`); both nodes list both themselves and their partner. Only the **first** entry may self-form a new cluster. |
| `SplitBrainResolverStrategy` | `string` | `"auto-down"` | `"auto-down"` or `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. See downing strategy above. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Sustained unreachability before downing. Emitted as `auto-down-unreachable-after` under `auto-down`, as the SBR `stable-after` under `keep-oldest`. |
| `HeartbeatInterval` | `TimeSpan` | `00:00:02` | Cluster failure-detector heartbeat frequency. Must be less than `FailureDetectionThreshold`. |
| `FailureDetectionThreshold` | `TimeSpan` | `00:00:10` | `acceptable-heartbeat-pause` for the cluster failure detector. |
| `MinNrOfMembers` | `int` | `1` | Must be `1`. A value of `2` blocks the cluster singleton after failover. |
| `DownIfAlone` | `bool` | `true` | Must be `true`. See split-brain resolution above. |
| `DownIfAlone` | `bool` | `true` | `keep-oldest` only — inert under `auto-down`. Validated as `true` only when the strategy is `keep-oldest`. |
| `AllowSingleNodeCluster` | `bool` | `false` | Acknowledges a deliberate single-node install: permits exactly one seed node instead of the usual two. |
### `ScadaBridge:Node`
@@ -241,7 +271,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -251,7 +281,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
}
```
`DownIfAlone` is not present in the docker files because its default value of `true` is correct and `ClusterOptionsValidator` rejects `false`.
`DownIfAlone` is not present in the docker files because it is a `keep-oldest`-only knob and every shipped deployment runs `auto-down`, under which the flag is inert.
## Dependencies & Interactions
@@ -261,22 +291,26 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
- [Site Runtime (#3)](./SiteRuntime.md) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee.
- [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md) — each hosts one or more central singletons wired by `RegisterCentralActors`. Cluster Infrastructure provides the `ClusterSingletonManager`/`ClusterSingletonProxy` boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component.
- [Central–Site Communication (#5)](./Communication.md) — `CentralCommunicationActor` and `SiteCommunicationActor` are created and registered with `ClusterClientReceptionist` inside the same `AkkaHostedService` startup, making them addressable by remote `ClusterClient` instances. The transport-level heartbeat (`TransportHeartbeatInterval`, `TransportFailureThreshold`) is configured separately from the cluster failure-detector and comes from `CommunicationOptions`.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the cluster leader.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the oldest `Up` member.
- Design spec: [Component-ClusterInfrastructure.md](../requirements/Component-ClusterInfrastructure.md).
## Troubleshooting
### Node fails to join cluster on startup
`ClusterOptionsValidator` rejects fewer than two seed nodes, a non-`keep-oldest` strategy, `MinNrOfMembers != 1`, or `DownIfAlone = false` at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
`ClusterOptionsValidator` rejects fewer than two seed nodes (without `AllowSingleNodeCluster`), a strategy outside `auto-down` / `keep-oldest`, `MinNrOfMembers != 1`, or `DownIfAlone = false` under `keep-oldest`, at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
A node that boots, logs no validation error, but never reaches `Up` was — before 2026-07-22 — usually hitting the seed-node bootstrap constraint: it was not the first entry in `SeedNodes` and the first seed was down, so it looped on `InitJoin` waiting for a peer that could form the cluster. Self-first ordering plus the `StartupValidator` rule that enforces it should make this unreachable; if you still see it, check that `seed-nodes[0]` really resolves to this node's own `NodeHostname:RemotingPort` (the validator compares host *and* port, and Akka does no DNS canonicalisation — `node-a` and `node-a.example.com` are different seed identities).
### Singleton not starting after failover
If the surviving node is `Up` but singletons do not start, `MinNrOfMembers` is the first thing to check. A value of `2` keeps the surviving node waiting for a second member indefinitely. The validator enforces `1`, but a manually patched `appsettings.json` that bypasses the validator could produce this.
### Two live clusters (split-brain)
### Two live clusters (dual-active)
If `DownIfAlone = false` were accepted (the validator rejects it), the oldest node could run alone while the younger forms its own cluster, producing two live clusters with divergent singleton state and dual MS SQL writers on central. `ClusterOptionsValidator` makes this configuration impossible to boot.
Under `auto-down` this is the **accepted trade, not a misconfiguration**: during a real network partition each side downs the other and continues as a one-node cluster, so both nodes are oldest-`Up`, both host a full set of singletons, and both answer `/health/active` with 200 — including dual MS SQL writers on central. Monitoring surfaces it directly (both nodes stamp `IsActive` on their heartbeats; the Health dashboard shows two Primaries). The two sides do **not** merge on their own — the mutual downing quarantines the association. Recovery is operator-driven: once the link is restored, restart **one** side; it rejoins its peer as a fresh incarnation and comes back as standby.
Deployments that would rather lose availability than run dual-active should set `SplitBrainResolverStrategy: "keep-oldest"` (with `DownIfAlone = true`), accepting that a crash of the oldest node is then a total outage.
### Graceful shutdown takes longer than expected
@@ -285,6 +319,7 @@ If a clean node stop takes up to 25 seconds instead of seconds, `CoordinatedShut
## Related Documentation
- [Cluster Infrastructure design specification](../requirements/Component-ClusterInfrastructure.md)
- [Auto-down downing strategy — availability over partition-safety (decision, 2026-07-21)](../plans/2026-07-21-auto-down-availability-decision.md)
- [Host](./Host.md)
- [Site Runtime](./SiteRuntime.md)
- [Health Monitoring](./HealthMonitoring.md)
docs/components/Communication.md
+54 -24
View File
@@ -1,6 +1,6 @@
# Central–Site Communication
The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides two independent transports — Akka.NET `ClusterClient` for command/control and gRPC server-streaming for real-time data — wired together through a pair of actors that each cluster registers with the `ClusterClientReceptionist`.
The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports — Akka.NET `ClusterClient` for command/control, gRPC server-streaming for real-time data, and plain token-gated HTTP for the deployment-config fetch — anchored by a pair of actors that each cluster registers with the `ClusterClientReceptionist`.
## Overview
@@ -18,15 +18,33 @@ DI registration is called from the Host composition root via `AddCommunication`.
## Key Concepts
### Two transports, two concerns
### Three transports, three concerns
| Transport | Direction | Purpose |
|-----------|-----------|---------|
| Akka.NET `ClusterClient` | bidirectional (command/control) | Deployments, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications |
| gRPC server-streaming (`SiteStreamService`) | site → central | Real-time attribute value and alarm state changes |
| Transport | Who dials | Data direction | Purpose |
|-----------|-----------|----------------|---------|
| Akka.NET `ClusterClient` | both (central → site per site; site → central) | bidirectional | Deploy notifies, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications |
| gRPC (`SiteStreamService`) | **central dials the site** | mostly site → central | Real-time attribute value and alarm state changes (server-streaming), plus the audit ingest/pull unary RPCs |
| HTTP `GET` (`/api/internal/deployments/{id}/config`) | **site dials central** | central → site | The flattened deployment config itself (notify-and-fetch), gated by a per-deployment `X-Deployment-Token` |
The transports are independent. A gRPC stream interruption does not affect in-flight `ClusterClient` commands, and vice versa.
**The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService<SiteStreamGrpcServer>()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site.
**None of the three transports carries a mutual-auth or transport-encryption story.** `BuildHocon` emits no Akka `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting and `ClusterClient` are plaintext and unauthenticated. The site gRPC listener is **h2c** — `ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` (`Host/Program.cs:501-506`) — and `SiteStreamGrpcClient` opens a bare `GrpcChannel.ForAddress(endpoint, …)` with no credentials. `LocalDbSyncAuthInterceptor` shares that listener but gates **only** methods under `/localdb_sync.v1.LocalDbSync/`, explicitly letting every `SiteStreamService` call through untouched, so the whole surface — including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows — is callable by anything that can reach the site's gRPC port. The one authenticated piece is the HTTP config fetch, and its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`). The design assumes a trusted network between central and sites.
### Notify-and-fetch: the deployment-config HTTP path
An instance deployment does not carry its flattened configuration inside the Akka message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over `ClusterClient`, carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
```csharp
// SiteRuntime/Deployment/HttpDeploymentConfigFetcher.cs
var url = $"{centralFetchBaseUrl.TrimEnd('/')}/api/internal/deployments/{Uri.EscapeDataString(deploymentId)}/config";
using var req = new HttpRequestMessage(HttpMethod.Get, url);
req.Headers.Add("X-Deployment-Token", token);
```
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This exists because a flattened config can exceed the default 128 KB Akka frame size, which drops the single oversized message without tearing down the association — heartbeats keep flowing, the site still reports healthy, and the deploy just hangs to its Ask timeout. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline.
### Hub-and-spoke topology
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `ClusterClient` per site; each site maintains a single `ClusterClient` pointed at both central nodes.
@@ -36,9 +54,9 @@ Sites do not communicate with each other. All inter-cluster traffic flows throug
Central-side callers wrap outbound messages in a `SiteEnvelope(SiteId, Message)`. `CentralCommunicationActor` resolves the site's `ClusterClient` by `SiteId` and forwards the inner message to `/user/site-communication` on the site:
```csharp
// CommunicationService.cs — deployment pattern
public async Task<DeploymentStatusResponse> DeployInstanceAsync(
string siteId, DeployInstanceCommand command, CancellationToken cancellationToken = default)
// CommunicationService.cs — deployment pattern (notify-and-fetch)
public async Task<DeploymentStatusResponse> RefreshDeploymentAsync(
string siteId, RefreshDeploymentCommand command, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<DeploymentStatusResponse>(
@@ -86,7 +104,7 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
`SiteCommunicationActor` is a `ReceiveActor` created at `/user/site-communication` and registered with `ClusterClientReceptionist`. It owns:
- An `IActorRef? _centralClient` — the site's outbound `ClusterClient` to central. Injected post-construction via `RegisterCentralClient`.
- A `Timers`-based heartbeat (default 5-second interval, first tick after 1 second). Each tick sends a `HeartbeatMessage` with `IsActive` stamped from the Akka `Cluster` leader check — the node is active when its `MemberStatus` is `Up` and it holds cluster leadership.
- A `Timers`-based heartbeat on `CommunicationOptions.ApplicationHeartbeatInterval` (default 5 s; deliberately distinct from the Akka.Remote `TransportHeartbeatInterval`, so retuning the transport failure detector cannot silently retune the health heartbeat). Each tick sends a `HeartbeatMessage` whose `IsActive` is stamped from `ActiveNodeEvaluator.SelfIsOldestUp` — the node is active when it is the **oldest `Up` member**, *not* when it holds cluster leadership (`SiteCommunicationActor.cs:517-518`). A throwing active-check is caught and reported as `IsActive = false`.
- Dispatch to local handlers for every inbound command pattern. Handlers for event-log, parked-message, integration, and artifact patterns are registered post-construction via `RegisterLocalHandler`; unregistered patterns receive an inline error reply so the central Ask does not stall.
Site-to-central messages (health reports, audit batches, notification submissions) are sent via:
@@ -108,9 +126,9 @@ A malformed address for one site does not abort the refresh loop — the actor c
### gRPC real-time data transport
Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, a gRPC server-streaming service defined in `sitestream.proto`.
Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, defined in `sitestream.proto`. The **server runs on every site node and the client runs on central** — central dials in to receive the stream (see the transport table above).
**Site-side** — `SiteStreamGrpcServer` (Kestrel HTTP/2, port 8083):
**Site-side** — `SiteStreamGrpcServer` (Kestrel h2c, HTTP/2 only, port 8083):
- Implements `SiteStreamService.SiteStreamServiceBase`.
- For each `SubscribeInstance` call, creates a `StreamRelayActor` (named `stream-relay-{correlationId}-{seq}`) and subscribes it to `ISiteStreamSubscriber` (implemented by `SiteStreamManager` in the Site Runtime project — `SiteStreamGrpcServer` holds only the interface so it does not reference `SiteRuntime` directly).
@@ -144,7 +162,7 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
**Central-side** — `SiteStreamGrpcClient` / `SiteStreamGrpcClientFactory`:
- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per site identifier. On `GetOrCreate`, it compares the cached client's `Endpoint` to the requested endpoint and atomically replaces a stale client (different endpoint — NodeA→NodeB failover flip, or an edited address) with a fresh one.
- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per **`(site, endpoint)` pair** — a `ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient>`. The key was widened from site-only to fix an arch-review High: with a site-only key, one debug session's NodeA→NodeB failover flip disposed a channel another session was still using. `GetOrCreate` therefore no longer disposes on endpoint mismatch; both of a site's node channels coexist, and site *removal* (`RemoveSiteAsync`) is the only shared-disposal path. The trade-off is that an edited gRPC address leaves the old endpoint's idle channel cached until site removal or process shutdown — bounded at a handful of entries per site.
- `SiteStreamGrpcClient` opens a `GrpcChannel` with HTTP/2 keepalive (`KeepAlivePingDelay` default 15 s, `KeepAlivePingTimeout` default 10 s, `KeepAlivePingPolicy.Always`). `SubscribeAsync` is a plain `async Task` that calls `SubscribeInstance` and reads the response stream with `await foreach`, invoking `onEvent` for each received event and `onError` on any non-cancellation exception. The caller (`DebugStreamBridgeActor.OpenGrpcStream`) launches it inside a `Task.Run` so the long-running stream loop runs off the actor thread.
### Debug stream session lifecycle
@@ -162,18 +180,22 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
### Proto definition summary
```proto
// Protos/sitestream.proto
// Protos/sitestream.proto — six RPCs, all served by the SITE
service SiteStreamService {
rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent);
rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent);
rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck);
rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck);
rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse);
rpc PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse);
}
```
`SubscribeInstance` carries the real-time data stream. The other three RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`) serve the Audit Log component's gRPC telemetry push and reconciliation pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there.
Two are server-streaming: `SubscribeInstance` carries the per-instance real-time stream; `SubscribeSite` is the **site-wide, alarm-only** stream (no instance filter, attribute updates never carried) that feeds the aggregated central live alarm cache. The four unary RPCs serve the Audit Log and Site Call Audit push/pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there. As noted above, the two `Ingest*` RPCs are dead in the shipped topology (no central gRPC server exists for a site to dial); the two `Pull*` RPCs are live, with central as the caller.
`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–21) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work.
`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–23) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work. Field numbers are never reused and evolution is additive only.
The generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out, so editing `sitestream.proto` does not regenerate on build — regeneration is a manual toggle-build-copy-untoggle.
## Usage
@@ -181,7 +203,7 @@ Central callers interact through `CommunicationService`, which wraps each comman
| Pattern | Method | Timeout |
|---------|--------|---------|
| Instance deployment | `DeployInstanceAsync` | 120 s |
| Instance deployment (notify-and-fetch) | `RefreshDeploymentAsync` | 120 s |
| Instance lifecycle | `DisableInstanceAsync`, `EnableInstanceAsync`, `DeleteInstanceAsync` | 30 s |
| Artifact deployment | `DeployArtifactsAsync` | 60 s |
| Integration routing | `RouteIntegrationCallAsync` | 30 s |
@@ -197,11 +219,11 @@ For real-time streaming, callers use `DebugStreamService.StartStreamAsync`, whic
## Configuration
All options are bound from the `Communication` section via `CommunicationOptions`:
All options are bound from the `ScadaBridge:Communication` section via `CommunicationOptions`:
| Key | Default | Description |
|-----|---------|-------------|
| `DeploymentTimeout` | `00:02:00` | Ask timeout for instance deployment commands. |
| `DeploymentTimeout` | `00:02:00` | Ask timeout for the `RefreshDeploymentCommand` round-trip (covers the site's HTTP config fetch and apply). |
| `LifecycleTimeout` | `00:00:30` | Ask timeout for lifecycle commands (disable, enable, delete). |
| `ArtifactDeploymentTimeout` | `00:01:00` | Ask timeout for system-wide artifact deployment. |
| `QueryTimeout` | `00:00:30` | Ask timeout for remote queries and management commands. |
@@ -213,8 +235,12 @@ All options are bound from the `Communication` section via `CommunicationOptions
| `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. |
| `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. |
| `GrpcMaxConcurrentStreams` | `100` | Max concurrent `SubscribeInstance` streams per site node. |
| `TransportHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` heartbeat cadence. |
| `ApplicationHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` site→central heartbeat cadence. |
| `TransportHeartbeatInterval` | `00:00:05` | Akka.Remote transport failure-detector heartbeat interval (emitted into the HOCON by the Host). Distinct from the application heartbeat above. |
| `TransportFailureThreshold` | `00:00:15` | Akka remoting failure-detection threshold. |
| `CentralFetchBaseUrl` | `""` | Base URL (Traefik/LB) the site uses to fetch deploy configs from central. Carried in `RefreshDeploymentCommand` so sites need no standing config; **empty makes a deploy impossible** — `DeploymentService` fails fast. |
| `PendingDeploymentTtl` | `00:05:00` | How long a staged `PendingDeployment` row and its fetch token stay valid. Must comfortably cover both site nodes' fetches within one deploy window. |
| `PendingDeploymentPurgeInterval` | `01:00:00` | Cadence of the central `pending-deployment-purge` singleton that sweeps TTL-expired staging rows. Hygiene only — the fetch endpoint already enforces the TTL. |
Three layers of dead-client detection protect the gRPC stream path:
@@ -227,16 +253,16 @@ Three layers of dead-client detection protect the gRPC stream path:
## Dependencies & Interactions
- [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the active/standby leader model that `SiteCommunicationActor`'s `IsActive` check and `CentralCommunicationActor`'s `DistributedPubSub` fanout both depend on.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name that makes cross-cluster `ClusterClient` addressing possible at all. `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deployments, lifecycle commands, and artifact deployments to sites.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deploy notifies, lifecycle commands, and artifact deployments to sites. It also owns the staging half of the notify-and-fetch HTTP path (`PendingDeployment` rows + fetch tokens); the endpoint itself is served by the Management Service.
- [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly.
- [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts `IngestAuditEvents`, `IngestCachedTelemetry`, and `PullAuditEvents` RPCs. `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` ClusterClient messages to the `AuditLogIngestActor` proxy.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. Because there is no central gRPC server, the `Ingest*` pair is unused in the shipped topology: sites push audit telemetry over ClusterClient, and `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site.
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting
@@ -257,6 +283,10 @@ A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means
After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect to the other node endpoint (`_useNodeA` flips on each error). If both nodes are unreachable, the actor exhausts its 3-retry budget and calls `onTerminated`. The engineer must restart the debug session.
### Deployments fail immediately with a config-fetch error
The site received the `RefreshDeploymentCommand` over ClusterClient but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
### Heartbeats arrive but health reports do not
`SiteCommunicationActor` sends heartbeats and health reports via separate paths. Health reports are sent only when the site's `HealthReportSender` publishes them (every 30 s by default). If heartbeats arrive but reports do not, the health-report sender on the site may have faulted — check site-side logs for errors in `HealthReportSender`.
docs/components/DeploymentManager.md
+20 -6
View File
@@ -26,10 +26,20 @@ Every instance deployment carries two correlated identifiers:
- **`DeploymentId`** — a new `Guid` (formatted `"N"`) minted by `DeploymentService` at the start of each `DeployInstanceAsync` call.
- **`RevisionHash`** — computed by the Template Engine's `RevisionHashService` over the fully resolved `FlattenedConfiguration`. The hash captures the template state at the moment of flattening, so concurrent last-write-wins template edits do not affect an in-flight deployment.
The pair travels inside `DeployInstanceCommand` to the site. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
The pair travels to the site inside the `RefreshDeploymentCommand` notify and is echoed back on the fetched config. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
Central stores the `RevisionHash` on `DeploymentRecord` and, after a confirmed success, on `DeployedConfigSnapshot`. Comparing the snapshot hash against the current-template hash determines whether an instance is stale without a site round-trip.
### Notify-and-fetch: the config does not travel in the Akka message
A deployment crosses the central↔site boundary over **two** transports, not one. Central stages the flattened configuration in a `PendingDeployment` row (config JSON, a generated `DeploymentFetchToken`, and an expiry of `CommunicationOptions.PendingDeploymentTtl`, default 5 minutes) and then sends only a small `RefreshDeploymentCommand` over ClusterClient carrying the deployment id, instance name, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton fetches the config back over plain HTTP — `GET {CentralFetchBaseUrl}/api/internal/deployments/{deploymentId}/config` with an `X-Deployment-Token` header — and only then runs its normal apply path.
This exists because a flattened configuration can exceed the default 128 KB Akka frame size, and an over-limit message is dropped silently without tearing down the association — the deploy then simply hangs to its Ask timeout. `CentralFetchBaseUrl` is therefore mandatory: `DeployInstanceAsync` fails fast with "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)" rather than attempting a deploy that cannot complete. Note that `DeployArtifactsCommand` was **not** moved to this path — artifact deployment still carries its payload inline and remains exposed to the frame limit.
Staged rows are cleaned up by **TTL only** — they are deliberately not deleted on success or in the failure path. Three things keep that safe: `AddPendingDeploymentAsync` supersedes (deletes) any prior pending row for the same instance before inserting, so at most one row exists per instance; the fetch endpoint enforces the TTL itself, so an un-purged row is not a usable one; and the central `pending-deployment-purge` singleton sweeps expired rows on `PendingDeploymentPurgeInterval` (default 1 hour).
The site's **startup reconciliation** path uses the same endpoint but stages its own rows: a site node reports its local instance→revision-hash map on boot, and central's `ReconcileService` diffs it against the expected deployed set, stages a fresh `PendingDeployment` (with a new token) for each missing or stale instance, and returns the gap plus `CentralFetchBaseUrl` for the node to fetch. Intra-site replication to the standby node does **not** use this path — `deployed_configurations` is a replicated LocalDb table, so the active node's write reaches the peer as an ordinary row change.
### Per-instance operation lock
`OperationLockManager` holds a `Dictionary<string, LockEntry>` keyed by instance `UniqueName`. Each `LockEntry` wraps a `SemaphoreSlim(1,1)` with a reference count so the semaphore is created on first contention and disposed when the last waiter clears. The lock covers all four mutating operations — deploy, disable, enable, delete — so they can never interleave on a single instance. Operations on different instances proceed in parallel.
@@ -67,7 +77,7 @@ The operation lock is in-memory. If the active central node fails mid-deployment
3. **Flatten and validate** — `IFlatteningPipeline.FlattenAndValidateAsync` runs the Template Engine pipeline and returns a `FlatteningPipelineResult` containing the `FlattenedConfiguration`, `RevisionHash`, and a `ValidationResult`. Semantic validation failures (call targets, argument types, trigger operand types, connection binding completeness) are returned to the caller before any record is written.
4. **Pre-deploy site reconciliation** — when the prior `DeploymentRecord` for the instance is `InProgress` or `Failed` with a timeout marker (`"Communication failure:"`), the service queries the site via `CommunicationService.QueryDeploymentStateAsync`. If the site already holds the target revision hash, the prior record is updated to `Success` and no new deployment is sent.
5. **Write `InProgress` record** — a single `DeploymentRecord` insert directly at `InProgress` status (no transient `Pending` hop). `IDeploymentStatusNotifier.NotifyStatusChanged` fires to push the status to the UI.
6. **Send `DeployInstanceCommand`** — the command carries `DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `FlattenedConfigurationJson`, `DeployedBy`, and `Timestamp`.
6. **Stage and notify** — insert a `PendingDeployment` row holding the flattened config JSON and a fresh fetch token, then send `RefreshDeploymentCommand` (`DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `DeployedBy`, staging timestamp, `CentralFetchBaseUrl`, `FetchToken`) via `CommunicationService.RefreshDeploymentAsync`. The site fetches the config over HTTP and replies with the same `DeploymentStatusResponse` as before.
7. **Commit terminal status** — the `DeploymentRecord` is updated to `Success` or `Failed` and saved before any post-success side effects run. This ordering ensures the recorded outcome can never be lost if a post-success write fails.
8. **Post-success side effects** — `ApplyPostSuccessSideEffectsAsync` sets `Instance.State = Enabled` (or preserves `Disabled` on the reconciliation path) and upserts the `DeployedConfigSnapshot`. These writes are best-effort: a failure here is logged at `Error` but does not flip the already-committed `Success` record back to `Failed`.
9. **Audit log** — `IAuditService.LogAsync` records `Deploy` / `DeployFailed` / `DeployReconciled` with the `DeploymentId`, status, and user.
@@ -76,7 +86,7 @@ Any exception in the site round-trip (steps 6–7) writes `DeploymentStatus.Fail
```csharp
// DeploymentService.DeployInstanceAsync — exception handler
var isTimeout = ex is TimeoutException or OperationCanceledException;
var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException;
record.Status = DeploymentStatus.Failed;
record.ErrorMessage = isTimeout
@@ -171,11 +181,11 @@ Options are registered via `AddDeploymentManager` and bound from `ScadaBridge:De
- [Template Engine (#1)](./TemplateEngine.md) — `FlatteningPipeline` delegates to `FlatteningService`, `ValidationService`, and `RevisionHashService`. Template state is captured at flatten time; last-write-wins edits made after flatten do not affect the in-flight deployment. `DiffService.ComputeDiff` powers the deployment diff view.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — owns the EF Core implementation of `IDeploymentManagerRepository`, which stores `DeploymentRecord`, `DeployedConfigSnapshot`, and `SystemArtifactDeploymentRecord`. `IAuditService` (also registered by the Configuration Database component) writes all deployment audit rows.
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `DeployInstanceAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `DeploymentStatus`, `InstanceState`, `DeployInstanceCommand`, `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `RefreshDeploymentAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`, all over the ClusterClient command/control transport. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. `CommunicationOptions.CentralFetchBaseUrl` / `PendingDeploymentTtl` (also owned by that component) parameterise the notify-and-fetch HTTP leg.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `PendingDeployment`, `DeploymentFetchToken`, `DeploymentStatus`, `InstanceState`, `RefreshDeploymentCommand`, `DeployInstanceCommand` (retained as the site-side in-process apply DTO), `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Site Runtime (#3)](./SiteRuntime.md) — receives `DeployInstanceCommand` and `DeployArtifactsCommand` via the Communication Layer. Site-side apply is all-or-nothing per instance: the Deployment Manager singleton at the site stores the config, compiles all scripts, and creates or replaces the Instance Actor as a unit. A failure at any step is reported back with the specific error message and the previous configuration remains active.
- [Central UI (#9)](./CentralUI.md) — engineers trigger deployments, view diffs, manage instance lifecycle, and deploy system-wide artifacts through the UI. The deployment status page subscribes to `IDeploymentStatusNotifier.StatusChanged` for real-time push updates via Blazor Server SignalR.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. It also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route a site calls to fetch a staged config. That endpoint is `AllowAnonymous`; the per-deployment token, compared in constant time, is the entire security boundary, and existence/TTL are checked before the token so unknown, superseded and expired ids are indistinguishable `404`s.
- [Security & Auth (#10)](./Security.md) — the Deployment role is required for all deploy and artifact operations; site-scoped permissions are enforced by the Central UI and Management Service before commands reach `DeploymentService`.
## Troubleshooting
@@ -188,6 +198,10 @@ The operation lock is in-memory. On failover the new active node has no lock ent
The site round-trip timed out or was cancelled before a response arrived. The site may or may not have applied the config. On the next deploy attempt the reconciliation query determines the ground truth. If the query also fails (site unreachable), a new `DeployInstanceCommand` is sent; the site rejects it with "already applied" if it ran the previous one.
### A deployment fails with a config-fetch error
The notify reached the site but the HTTP leg did not complete. `CentralFetchBaseUrl` must be resolvable and reachable **from the site** — a value that only works inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl` (existence is hidden, so those are indistinguishable); a `401` means the row is live but the presented token did not match. A fetch failure applies nothing, and the site replies `Failed` rather than letting central's Ask hang to timeout.
### DeleteOrphaned audit entry
The site destroyed the Instance Actor but the central DB removal failed. The instance record exists in the central DB but has no corresponding site actor. It cannot be deleted through the normal UI path (the site will reject the delete command because the instance does not exist). Reconcile by removing the central record directly via the Management API or database, referencing the `CommandId` in the audit entry.
docs/components/Host.md
-1
View File
@@ -104,7 +104,6 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon
`SiteServiceRegistration.Configure` registers the site-only components. `AkkaHostedService.RegisterSiteActorsAsync` creates:
- `DeploymentManagerActor` — cluster singleton scoped to `"site-{SiteId}"`.
- `SiteCommunicationActor` — registered with `ClusterClientReceptionist`; creates a `ClusterClient` to configured central contact points.
- `SiteReplicationActor` — one per node (not a singleton); handles best-effort S&F replication to the standby.
- `EventLogHandlerActor` — cluster singleton scoped to `"site-{SiteId}"`.
- `ParkedMessageHandlerActor` — bridges Akka to `StoreAndForwardService`.
- `SiteAuditTelemetryActor` — created on a dedicated `audit-telemetry-dispatcher` (2-thread `ForkJoinDispatcher`) so SQLite reads and gRPC pushes never contend with hot-path actors.
docs/components/SiteRuntime.md
+3 -3
View File
@@ -8,7 +8,7 @@ Site Runtime (#3) operates exclusively on site clusters. Its entry point is the
The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`, `SiteReplicationActor`.
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`.
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
- `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream).
- `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`.
@@ -79,7 +79,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped).
2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`.
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync`, clears static overrides and native alarm state, and — if `_replicationActor` is non-null (it is optional and null in isolated deployments/tests) — tells `SiteReplicationActor` to push to the peer node.
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
For redeployment (instance already running), the existing actor is stopped and watched:
@@ -216,7 +216,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
### Standby replication
`SiteReplicationActor` runs on every site node (not a singleton). The active node's `DeploymentManagerActor` tells it `ReplicateConfigDeploy`, `ReplicateConfigRemove`, `ReplicateConfigSetEnabled`, `ReplicateArtifacts`, or `ReplicateStoreAndForward`. The replication actor tracks the peer node via Akka cluster membership events and forwards each command to `/user/site-replication` on the peer via `ActorSelection`. Replication is fire-and-forget (no ack wait per design), so a failed write to the standby is logged but does not fail the primary operation.
Config replication has no actor. `SiteReplicationActor` — which received `ReplicateConfigDeploy` / `ReplicateConfigRemove` / `ReplicateConfigSetEnabled` / `ReplicateArtifacts` / `ReplicateStoreAndForward` from the active node's `DeploymentManagerActor`, tracked the peer through cluster membership events, and forwarded each command to `/user/site-replication` via `ActorSelection` — was deleted in LocalDb Phase 2, together with its notify-and-fetch exchange (the standby was told a deploy had happened and then HTTP-fetched the config itself). The site's config tables are now replicated by LocalDb CDC, so a deploy on either node reaches the other as an ordinary row change. `SiteReconciliationActor` still fetches over HTTP at node startup when central reports gaps; that is a different path and it survives.
## Usage
docs/components/StoreAndForward.md
+4 -3
View File
@@ -4,13 +4,12 @@ The Store-and-Forward Engine buffers site-originated outbound messages when a ta
## Overview
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` SQLite store and an optional `ReplicationService` that fans each buffer mutation to the standby.
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` store. As of LocalDb Phase 2 that store writes to the consolidated LocalDb database, and the buffer reaches the peer as the replicated `sf_messages` table — the `ReplicationService` that used to fan each mutation to the standby by hand was deleted.
The component code lives in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`:
- `StoreAndForwardService` — the core buffer: enqueue, retry sweep, park/retry/discard, and the `ICachedCallLifecycleObserver` audit hook.
- `StoreAndForwardStorage` — the SQLite layer; all reads and writes against `sf_messages`.
- `ReplicationService` — fire-and-forget buffer replication to the standby.
- `ParkedMessageHandlerActor` — Akka actor bridge that exposes parked-message query/retry/discard to the `SiteCommunicationActor`.
- `NotificationForwarder` — the delivery handler for the `Notification` category; forwards buffered notifications to central via the ClusterClient transport and interprets the ack.
- `StoreAndForwardOptions` — options class bound from the `StoreAndForward` configuration section.
@@ -129,7 +128,9 @@ else
### Async replication to standby
`ReplicationService` wraps each buffer mutation — add, remove, park, requeue — in a `Task.Run` fire-and-forget. The active node does not wait for standby acknowledgment. The standby applies each `ReplicationOperation` via `ApplyReplicatedOperationAsync`, which calls the same `StoreAndForwardStorage` methods. Replication failures are logged at Debug and discarded; the standby may be slightly behind the active at any moment, producing at-most a few duplicate deliveries or missed retries after a failover — an accepted trade-off for zero added latency on the enqueue path.
Replication is no longer the buffer's own concern. `sf_messages` is registered with LocalDb (`SiteLocalDbSetup.OnReady`), so every insert and status change is captured by a CDC trigger and shipped to the peer on the shared sync stream. The four hand-written operations — add, remove, park, requeue — and the `Task.Run` fan-out that carried them are gone, along with `ApplyReplicatedOperationAsync` and `ReplaceAllAsync`.
The trade-off is unchanged in shape: replication is still asynchronous, so the peer may be slightly behind at any instant. What changed is the bound. Convergence is now per row under last-writer-wins with HLC-ordered tombstones, so a lagging peer converges rather than diverging, and duplicate delivery after a failover is limited to messages the old primary delivered whose status change had not yet replicated. See `Component-StoreAndForward.md` for the normative statement of that bound.
The four `ReplicationOperationType` values are `Add`, `Remove`, `Park`, and `Requeue` (requeue was added to cover the operator-initiated `Parked→Pending` transition so the standby preserves retry intent after failover).
docs/deployment/installation-guide.md
+8
View File
@@ -187,6 +187,14 @@ ALTER ROLE db_owner ADD MEMBER scadabridge_svc;
Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions.
## Upgrading a Site Pair
**Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
`TombstoneRetention` bound on how long one node may stay offline.
## Post-Installation Verification
1. Start the service: `sc.exe start ScadaBridge-Central`
docs/deployment/topology-guide.md
+28 -4
View File
@@ -88,17 +88,19 @@ Both central nodes must be configured as seed nodes for each other:
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@central-01.example.com:8081",
"akka.tcp://scadabridge@central-02.example.com:8081"
"akka.tcp://scadabridge@central-02.example.com:8081",
"akka.tcp://scadabridge@central-01.example.com:8081"
]
}
}
}
```
> **Seed order is load-bearing — each node lists ITSELF first** (decision 2026-07-22). Note Node B's list is the reverse of Node A's. Akka only lets `seed-nodes[0]` form a *new* cluster, so a node listing its partner first can never boot while that partner is down. `StartupValidator` rejects the boot if the ordering is wrong, comparing host **and** port; use the same spelling of the hostname in `NodeHostname` and in the seed URI, since Akka does no DNS canonicalisation (`central-02` and `central-02.example.com` are different seed identities). See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
### Cluster Behavior
- **Split-brain resolver**: Keep-oldest with `down-if-alone = on`, 15-second stable-after.
- **Split-brain resolver**: `auto-down` (`AutoDowning` provider, `auto-down-unreachable-after` = 15s) since the 2026-07-21 availability-over-partition-safety decision — the leader among the *reachable* members downs the unreachable peer, so a hard crash of **either** node fails over. Accepted trade: a real partition leaves both sides active until an operator restarts one. `keep-oldest` (with `down-if-alone = on`) remains a supported `SplitBrainResolverStrategy` value, but in a two-node cluster it cannot survive a crash of the oldest node. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- **Minimum members**: `min-nr-of-members = 1` — a single node can form a cluster.
- **Failure detection**: 2-second heartbeat interval, 10-second threshold.
- **Total failover time**: ~25 seconds from node failure to singleton migration.
@@ -145,12 +147,34 @@ Each site has its own two-node cluster:
}
```
> **Site Node B reverses this list** — `site-01-b` first, `site-01-a` second — per the self-first seed rule above. It applies to site pairs exactly as it does to the central pair: without it, `site-01-b` cannot boot while `site-01-a` is down.
### Site Cluster Behavior
- Same split-brain resolver as central (keep-oldest).
- Singleton actors: Site Deployment Manager migrates on failover.
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
- SQLite persistence: Both nodes access the same SQLite files (or each has its own copy with async replication).
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
share a SQLite file.
### Site Pair Upgrades — stop and start BOTH nodes together
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby
could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler
along with the replicator, so a mixed-version pair has no common replication path: the two nodes
will run, but they will not converge, and the divergence is silent.
Stop both nodes of a site pair, upgrade both, then start both.
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
nothing left to suppress the stale row the returning node still holds. Within the retention window
a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
database from its peer rather than letting it rejoin.
### Central-Site Communication
docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md
+21
View File
@@ -10,6 +10,27 @@ Fixed via the **notify-and-fetch** rework (the primary recommendation below), no
- **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md)
- **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps.
## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely
The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb
Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's
`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an
ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any
more, so the 128 000-byte frame constraint does not apply to it in any form.
The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site
still fetches over HTTP, so that half of the original fix stands.
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
Note the failure mode differs from the one documented below: an oversized gRPC message is
**rejected**, not silently dropped.
The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
## Summary
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md
+92
View File
@@ -0,0 +1,92 @@
# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone
**Date:** 2026-07-20 · **Status:** OPEN · **Severity:** Medium (log flood + wasted I/O; no data loss)
· **Area:** AuditLog / Site Telemetry
## Summary
`SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's
tracking snapshot by `CorrelationId`, and pushes the combined packet to central. When the lookup
returns `null` the row is **skipped and deliberately left Pending**
(`SiteAuditTelemetryActor.cs:307`), on the reasoning that "central reconciliation will pick it up".
Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the
same lookup, logs the same warning, and leaves it Pending again — **forever**. With a batch of
unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass.
Measured on the docker rig: **~2 800 warnings/minute, sustained**, surviving both a process restart
and a container restart, until the audit database itself was discarded.
```
[09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for
a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping.
```
## Why the rows became unresolvable
Two independent stores must agree:
- the **audit** rows live in `auditlog.db` (site-local, and on the docker rig **inside the
container at `/app/auditlog.db`**, not on the bind-mounted data volume);
- the **tracking** rows live in `OperationTracking`, which LocalDb Phase 1 moved into the
consolidated `LocalDb:Path` database (bind-mounted).
Anything that resets one without the other strands every audit row that referenced it. The code
comment already anticipates the cause — *"possible if the audit row is older than the tracking
retention window, or the tracking store was reset"* — so this is a known-and-accepted input, not an
exotic one.
**Two realistic production triggers, neither requiring operator error:**
1. **Tracking retention expiry.** If the tracking retention window elapses before the audit drain
catches up — a long central outage, a large backlog — the snapshots are pruned out from under
still-Pending audit rows and every one of them becomes a permanent hot-loop entry.
2. **Restoring or resetting one store independently of the other**, e.g. rebuilding a node's
LocalDb file from its peer while its container-local `auditlog.db` survives.
It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while
`auditlog.db` — being inside the container — survived.
## Why the current handling is not enough
Skipping the row is correct; **leaving it Pending with no other state change is not**. The row is
now in a state it can never leave:
- no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one;
- no backoff, so the actor runs at full non-idle rate against a queue that can never shrink;
- no terminal state, so it is retried for the life of the database;
- one Warning per row per pass, which buries every other log line on the node.
The "central reconciliation will pick it up" comment is about the **audit half** reaching central by
another path. That may well be true — but it does not release the row from the local drain queue,
which is what actually loops.
## Suggested fix
Give an unresolvable row somewhere to go. Roughly, in increasing order of effort:
1. **Bound the retries.** Add an attempt count; past a threshold mark the row terminal
(`TrackingUnavailable`) and stop re-reading it. Emit a single summary Warning with the count
rather than one per row per pass.
2. **Rate-limit the warning** to one per drain episode regardless of row count — the same pattern
`MaintenanceBackgroundService` already uses for the oplog caps-exceeded warning (`_snapshotFlagWarned`).
3. **Push the audit half alone** when the tracking snapshot is missing, rather than skipping the row
entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether
central accepts a packet with no tracking half.
(2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O.
## Reproduction
1. Run a site node until it has cached-call audit rows with tracking correlations.
2. Stop the node; delete its consolidated LocalDb database (which holds `OperationTracking`);
leave `auditlog.db` in place.
3. Start the node. The drain warning repeats indefinitely; the rate does not decay.
## Notes
- **No data loss.** The audit rows are intact and still reach central by the reconciliation path;
what is broken is the local drain's ability to ever finish.
- Discovered while cleaning the rig after the LocalDb Phase 2 live gate
(`docs/plans/2026-07-19-localdb-phase2-live-gate.md`), which is also where the related
"deleting an instance orphans its buffered messages" observation is recorded.
docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md
+475
View File
@@ -0,0 +1,475 @@
# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load
**Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening
**Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes
**Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md)
**Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code)
> **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a
> minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped.
> Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved
> as written, with corrections annotated where its conclusions did not survive
> (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8).
---
## 0. ROOT CAUSE (verified)
**A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and
resets the WAL out from under the container process, permanently poisoning that process's
connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved —
the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all.
Mechanism, step by step:
1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount
boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice
versa), so it believes it is the **only** connection.
2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to
0 bytes**. Because the read happened while the container was idle (a standby node, or a gap
between write bursts), nothing blocks the checkpoint. This is exactly the file signature
found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute.
3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping,
kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite
connection pool). That index says the WAL contains N frames; the file now has none. Every
subsequent statement — reads and writes both consult the WAL index — fails with
**`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10
`'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is
**permanent until the process reopens the database** (restart).
### Why the original brief's conclusions were wrong
- **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by
the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56
mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only
because a standby issues ~zero LocalDb statements; the errors "followed the load" because
the load is what generates statements against an already-poisoned handle. Node-b's very
first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`)
failed — it had been poisoned for 3 minutes with nothing to say about it.
- **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in
the same directory were healthy only because no host process ever read *them*. The minimal
repro poisons an arbitrary WAL database the same way.
- **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby
was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic.
### Verification (2026-07-20, all on the live rig + minimal repro)
1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened
site-a-a took the full soak load for **10+ minutes with zero errors** (the original model
predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB.
2. **One host read is sufficient and immediate:** a single
`sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"`
against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the
first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next
40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident.
3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a
WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op,
`synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`:
- against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) +
`SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL);
- during an **idle window** (connections held open, WAL populated) → WAL reset
1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run
(200/200)** — the persistent variant, matching the rig.
### Consequences
1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is
the root cause, not a hygiene note.** One violation silently destroys the node's local
persistence until restart. This applies to *every* WAL SQLite file on the bind mount
(`scadabridge.db`, `store-and-forward.db` included), not just LocalDb.
2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the
copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g.
`docker run --rm -v <dir>:/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the
macOS host against live files.
3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write
load indefinitely once nothing external touched its file.
4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):**
- **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`,
`CachedCallTelemetryForwarder`, `SiteEventLogger`) now log
`SqliteException` primary/extended codes (`sqlite 10/522`-style) via
`SqliteErrorCodes.Describe` / `DescribeSqliteError`.
- **DONE — §8 async-context bug** (see §8).
- **DONE — §9.5 load regression test** (see §9).
- **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for
persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change
(pool clear + master reopen + in-flight coordination) protecting against *external
interference only* — the trigger is operator/tooling action, now prevented at the
source, and on same-kernel production deployments external readers see the locks and
are safe. File as its own issue if production ever runs where a foreign-kernel reader
can touch the files.
---
> **Original brief follows, preserved as written on 2026-07-20 before root-causing.**
---
## 1. Summary
On a ScadaBridge site node, once the node is **active** and under sustained concurrent write
load, effectively every write to the consolidated LocalDb database (`site-localdb.db`) fails
with:
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
```
Observed rate: **~1 000–1 500 failures per minute**, sustained, not transient. The node stays up
and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same
process, under the same load, are completely unaffected.
## 2. Why it matters
1. **Silent data loss today.** `SiteEventLogger` fails its inserts and logs
`[ERR] Failed to record event: script from ScriptActor:…`. Site event logging is dropping
events on the floor on the active node whenever the site is busy. `OperationTracking` writes
fail too, which breaks cached-call status tracking (`Cached-telemetry drain: no tracking
snapshot for …; skipping`).
2. **It blocks LocalDb Phase 2.** Phase 2 registers eight further tables into this same
database — including `native_alarm_state` (highest-volume table on the node) and
`sf_messages` — **and deletes the bespoke mechanisms that currently carry that data**
(`SiteReplicationActor`, `StoreAndForward.ReplicationService`) in the same commit. Cutting
over onto this store while removing the fallback would convert a logging defect into config
and buffer loss.
3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence,
**not sustained write load** — which is why this was not caught.
## 3. Exact symptom
Two representative stacks, both from `docker logs scadabridge-site-a-b` while that node was
active and under load:
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.<ProcessWriteQueueAsync>b__0(SqliteConnection connection)
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221
```
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137
at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...)
in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148
```
Note both fail inside `SqliteDataReader.NextResult()` — i.e. at statement execution, not at
`Open()`. Connections are being acquired successfully; the failure is on the write itself.
### Error-source distribution
Error-stack frames counted over one 3-minute window on the loaded node:
| Store | Backing file | Frames |
|---|---|---|
| `OperationTrackingStore` | `site-localdb.db` (**LocalDb**) | 13 044 |
| `SiteAuditTelemetryActor` | `site-localdb.db` (**LocalDb**) | 4 350 |
| `SiteEventLogger` | `site-localdb.db` (**LocalDb**) | 900 |
| `CachedCallTelemetryForwarder` | `site-localdb.db` (**LocalDb**) | 162 |
| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** |
| `SiteStorageService` | `scadabridge.db` (legacy) | **0** |
## 4. Evidence — what has been established
### 4.1 It tracks the load, not the node
The load was moved between the two site-a nodes by restarting the active one (the surviving node
becomes oldest-up and takes over):
| Node | Role | Under load | `disk I/O error` / 4 min |
|---|---|---|---|
| site-a-a | active | yes | 2 175 |
| site-a-a | standby (after restart) | no | **0** |
| site-a-b | standby | no | **0** |
| site-a-b | active (after failover) | yes | **4 391** |
### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0**
> **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read
> from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven).
This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same
bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the
**same process**, are also **WAL-mode**, and are being written **concurrently under the same
load** — and they log zero errors. Only the LocalDb-managed file fails.
### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause**
> **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an
> unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at
> 0 bytes); it was poisoned then and merely silent until failover gave it write traffic.
Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted
database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is
excluded:
- After node-a was restarted (fresh open, `-shm` recovered) and load failed over to node-b,
**node-b** — whose files no host process had touched since a single baseline read, and which
had been error-free for the entire preceding period — began erroring immediately at a *higher*
rate.
- **node-a**, whose files *had* been sampled, dropped to zero once it stopped carrying load.
The variable that tracks the errors is load. (Host-side `sqlite3` against a live WAL database
over a bind mount is still unsafe and should be avoided — it is just not the cause here.)
### 4.4 Not disk pressure
Host had 215 GiB free throughout (`df -h`: 76 % used on the data volume). Files are small:
`site-localdb.db` 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0.
## 5. Reproduction
Fully reproducible in ~10 minutes on the local docker rig.
### 5.1 Rig prerequisites
Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings:
- `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`).
- **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now
applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the
`/docker-entrypoint-initdb.d/` compose mounts are informational only — the official
`mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose
file). The manual workaround below is retained for historical context / older checkouts.
Original problem: after `infra/reseed.sh` dropped the volume, nothing created
`ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its
setup.sql poll. The by-hand equivalent:
```bash
cd ~/Desktop/ScadaBridge/infra
for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f"
done
```
Then restart the app containers so EF migrations run, and restart central again after
`seed-sites.sh` writes `LdapGroupMappings` (they are cached at startup).
### 5.2 Build the load generator
**The seeded `Motor Controller` template (id 4) cannot be used** — it fails pre-deployment
validation with 34 errors (30 `ConnectionBinding`, 4 `ScriptCompilation`). Build a minimal one.
**Critical:** `ExternalSystem.Call` does **not** buffer to store-and-forward in practice.
`ExternalSystem.CachedCall` is the buffering surface. Using `Call` produces HTTP traffic and no
S&F rows, and will not reproduce this.
```bash
cd ~/Desktop/ScadaBridge
SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge # dotnet build src/...CLI first
AUTH="--url http://localhost:9000 --username multi-role --password password"
# 1. Point the seeded external system at a refusing address (discard port).
$SB $AUTH external-system update --id 1 --name "Test REST API" \
--endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1"
# 2. Minimal template: no attributes, no compositions, no connection bindings.
$SB $AUTH --format json template create --name "SoakGenerator" # -> note the id
$SB $AUTH --format json template script add --template-id <TID> --name "SoakCall" \
--trigger-type Interval --trigger-config '{"intervalMs":5000}' \
--code 'var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);'
# 3. Four instances on site-a (site id 1), then deploy each.
for i in 1 2 3 4; do
$SB $AUTH --format json instance create --name "soakgen-$i" --template-id <TID> --site-id 1
done
$SB $AUTH instance deploy --id <each instance id>
```
Note the CLI's `template script update` requires `--name` and `--trigger-type` even when only
changing `--code`. In zsh, do not put the auth flags in an unquoted variable — zsh does not
word-split, so pass them literally or use `${=AUTH}`.
### 5.3 Observe
```bash
# Identify the ACTIVE node — it is the one running the ScriptActors.
docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused"
docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused"
# Errors appear on that node within ~2 minutes of load starting.
docker logs --since 4m scadabridge-site-a-<active> 2>&1 | grep -c "disk I/O error"
```
Metrics (port 8084 is **not** published, and the `aspnet:10.0` image has **no `curl`**) — use a
sidecar in the container's network namespace:
```bash
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \
-s localhost:8084/metrics | grep '^localdb_'
```
Do **not** query the databases with host-side `sqlite3` while containers are writing them.
## 6. Code map
### Library — `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs`
Facts relevant to the failure:
- **A `_master` connection is held open for the object's entire lifetime** (`:31`), explicitly to
"anchor the WAL journal". It is guarded by a `Lock _masterLock` because `SqliteConnection` is
not thread-safe.
- **`CreateConnection()` (`:83`) opens a brand-new `SqliteConnection` per call** — one per
operation, from many concurrent actors. Every call then runs
`PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON;` and registers a UDF:
```csharp
conn.CreateFunction("zb_hlc_next", () => _clock.Next());
```
- The connection string is **only** `DataSource=<path>` (`:57`) — **connection pooling is left at
the Microsoft.Data.Sqlite default (enabled)**, and no `Cache=` or `Mode=` is set.
- Effective options on the rig are the defaults: `BusyTimeoutMs = 5000`, `Synchronous = NORMAL`.
ScadaBridge's rig config (`docker/site-a-node-*/appsettings.Site.json`, `LocalDb` section) sets
only `Path` and the replication block.
- `zb_hlc_next()` is invoked **from inside the capture triggers**, i.e. on the SQLite thread
during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared
`HybridLogicalClock` from arbitrary threads.
### Failing call sites (ScadaBridge)
- `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236` — a channel-drained
single-writer loop (`ProcessWriteQueueAsync`) using a `WithConnection(...)` helper.
- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137` (`RecordEnqueueAsync`),
`:260,266` (`GetStatusAsync`).
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148`.
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs` — also see §8.
## 7. Hypotheses, ranked
> **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant
> of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side
> reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The
> extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522).
None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as
healthy WAL databases".
1. **Connection churn × pooling × per-connection UDF registration.** LocalDb opens a fresh
`SqliteConnection` per operation with pooling enabled, and calls `CreateFunction` on every
acquisition. Under high concurrency this drives far more open/close and `-shm` mapping churn
than the legacy stores (which reuse a small number of connections), and is the clearest
structural difference between the failing and healthy databases. Suspect the interaction of
the pool with the long-lived `_master` connection and WAL index growth.
2. **`-shm` / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency.**
Would explain why the same mount is fine for lower-concurrency databases. `mmap` of the shared
WAL index across virtiofs is a known-fragile area. **Distinguishing test: run the same load
with `LocalDb:Path` pointed at a container-local path (a `tmpfs` or a plain volume rather than
the bind mount).** If the errors vanish, this is confirmed and the fix is environmental /
deployment-shaped rather than a library bug. **Run this test first — it is cheap and it
partitions the hypothesis space.**
3. **`zb_hlc_next` UDF failing inside a trigger.** An exception thrown out of the managed UDF
callback during trigger execution can surface as a generic SQLite error at the statement
level. Check `HybridLogicalClock.Next()` for thread-safety and for anything that can throw
under contention (e.g. a spin/overflow path when many callers request stamps in the same
millisecond).
4. **Busy-timeout exhaustion misreported.** `BusyTimeoutMs = 5000` with heavy multi-connection
write contention on one file. This would normally surface as `SQLITE_BUSY` (5), not
`SQLITE_IOERR` (10), so it is a weaker fit — but worth excluding.
### The single highest-value next step
**Capture the extended result code.** The logs only show the primary code (`10` = `SQLITE_IOERR`),
which is generic. `SqliteException.SqliteExtendedErrorCode` names the failing syscall and would
likely settle this outright:
| Extended code | Meaning | Points at |
|---|---|---|
| `SQLITE_IOERR_SHMMAP` (6154) / `SQLITE_IOERR_SHMSIZE` (4874) | WAL index mmap/resize failed | hypothesis 2 |
| `SQLITE_IOERR_WRITE` (778) / `SQLITE_IOERR_FSYNC` (1034) | plain write/fsync failed | filesystem |
| `SQLITE_IOERR_LOCK` (3850) | file locking failed | bind mount locking |
Add the extended code to the exception logging (or attach a debugger / run the repro against a
local non-container build) before pursuing any fix.
## 8. Secondary defect in the same path — **FIXED 2026-07-20**
```
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
this is most likely due to use of async operations from within this actor.
Cause: System.NotSupportedException
```
`SiteAuditTelemetryActor` is touching `Context` (or `Self`/`Sender`) after an `await`. This is a
real bug independent of the I/O errors, though it sits in the same write path and may be
contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context`
after an `await` inside an actor.
> **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so
> their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with
> no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is
> worse than the logged one: depending on what the pool thread's thread-static cell slot holds,
> `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes
> and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on
> that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed
> under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and
> `Self` into fields at construction (both are thread-safe immutable handles) and use only those
> from the re-arm path. Regression test
> `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing`
> forces the mock awaits to complete off the actor thread — which every pre-existing test
> avoided by returning already-completed tasks — and catches **both** variants (EventFilter for
> the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green.
## 9. What a fix must satisfy
> **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no
> host process touches the live files — verified 10+ min of soak load with zero errors, events
> durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load
> test) would **not** have caught this — the trigger is an external reader, not load — but is
> now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent
> writers × 250 inserts through fresh pooled connections against a registered/triggered table on
> a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite
> 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8.
1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the
active node.
2. No `Failed to record event` errors — site events are durably written under load.
3. `localdb_oplog_depth` rises under load and **drains** between bursts; zero dead letters.
4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and
live gate still pass).
5. A regression test that would have caught this — i.e. a **concurrent-write load test** against
a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's
gate passed precisely because no test applied sustained concurrent write pressure.
## 10. Rig state as left
- Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped
(`reseed.sh` stage 2 does `rm -rf docker/site-*/data/*`).
- `ExternalSystemDefinitions` id 1 is **still repointed to `http://127.0.0.1:9`** — restore to
`http://scadabridge-restapi:5200` when done.
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed
and still generating load** on site-a.
- `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names.
## 11a. Fix pass (2026-07-20, same day — all verified)
Everything actionable that this incident identified is now fixed (uncommitted on each repo's
current branch; ScadaBridge full solution builds clean, 0 warnings):
| # | Issue | Fix | Verification |
|---|---|---|---|
| 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 |
| 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite <primary>/<extended>` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) |
| 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) |
| 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 |
| 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 |
Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0
consequence 4 for the rationale and the condition under which to file it).
## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC)
- Both site-a nodes restarted during verification, curing both poisonings. End state:
**site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on
both** under sustained load.
- The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at
`http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and
generating load — the Phase 2 soak can now proceed on a clean baseline.
- Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in
the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild.
docs/plans/2026-07-08-deferred-work-register.md
+1 -1
View File
@@ -48,5 +48,5 @@ Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|------|-------------|------------------------|-----------------|
| SBR | **SBR oldest-crash total-outage gap** — 2-node `keep-oldest` downs the partition *without* the oldest, so a hard crash of the ACTIVE (oldest) central node makes the standby self-down (~10s) → total central outage until the crashed node restarts; only a younger-node crash fails over. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` | Remedy is a production SBR topology/strategy decision (keep-majority + a 3rd/lighthouse seed node, static-quorum, or an accepted-risk note) — **owner: user decision**, not silently changeable. | Before the next production deployment that adds a central node, or the first real active-node crash. |
| SBR | ~~**SBR oldest-crash total-outage gap**~~ **RESOLVED 2026-07-21 (owner decision — availability over partition-safety).** All clusters switched from `keep-oldest` to the `auto-down` downing strategy (Akka `AutoDowning`, `auto-down-unreachable-after` = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest `DownReachable … including myself` log, Akka.NET 1.5.62 `KeepOldest.OldestDecision` source, rejected alternatives incl. the static-quorum-1 `DownAll` trap): `docs/plans/2026-07-21-auto-down-availability-decision.md`. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` (both now historical) | — | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in `Component-ClusterInfrastructure.md`); dual-active recovery is operator-driven. |
| vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** |
docs/plans/2026-07-19-localdb-adoption-phase2.md
+1207
View File
File diff suppressed because it is too large Load Diff
docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json
+106
View File
@@ -0,0 +1,106 @@
{
"planPath": "docs/plans/2026-07-19-localdb-adoption-phase2.md",
"execution": {
"mode": "parallel-waves",
"implementerModel": "opus",
"isolation": "worktree",
"branch": "feat/localdb-phase2",
"baseBranch": "feat/localdb-phase1",
"note": "Phase 1's branch is NOT merged/pushed, so phase 2 branches from it. Dispatch every unblocked task concurrently per the wave table in the plan. Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively."
},
"scopeDecision": {
"date": "2026-07-19",
"by": "user",
"choice": "Full scope as designed",
"note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D6 rather than deferred."
},
"reviewPass": {
"date": "2026-07-19",
"note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - sample via throwaway copies of the DB triplet, NEVER host-side sqlite3 against the live files [2026-07-20: that poisons the container's WAL state - see docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md]; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)."
},
"decisions": [
{
"id": "D1",
"subject": "Config moves to CDC; notify-and-fetch is DELETED - but the guarded write STAYS",
"evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md, already marked RESOLVED by the notify-and-fetch rework). LocalDb sync is gRPC - no such limit (but see D6). CORRECTED: StoreDeployedConfigIfNewerAsync (SiteStorageService.cs:301-336, guard at :325) has TWO production callers - SiteReplicationActor.cs:375 (dies in Task 15) AND SiteReconciliationActor.cs:166 (per-node startup self-heal vs central, SURVIVES Phase 2, stale-fetch race still real). Method + guard stay; reconcile becomes a benign second CDC writer. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent."
},
{
"id": "D2",
"subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary",
"evidence": "LocalDb's snapshot resync MERGES per-row LWW and never WIPES: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. Row-level deletes DO replicate (delete-trigger tombstones, streamed by SnapshotStreamer, applied as real DELETEs by LwwApplier) - only the destructive whole-table replace is gone. Caveat: tombstones pruned after TombstoneRetention (default 7d); a node offline longer can resurrect deleted rows (runbook, Task 21). SEMANTIC CHANGE: the standby is convergent, no longer byte-identical."
},
{
"id": "D3",
"subject": "CORRECTED: the SMTP purge already runs on the active node - pin it, don't move it",
"evidence": "PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821) has TWO callers: DeploymentManagerActor.cs:1921 (ACTIVE node's HandleDeployArtifacts, :1864-1963) and SiteReplicationActor.cs:456 (standby copy, dies with the actor). The purge never lapses; the original 're-home before any deletion' premise was false. Task 12 = pin test only (ArtifactStorageTests covers the storage method, not the actor call site Task 16 edits). No site writer to notification_lists/smtp_configurations since 2026-07-10 (verified: only test seeding inserts exist) + migrator skips them => permanently empty in the consolidated DB => Task 14 does NOT register them."
},
{
"id": "D4",
"subject": "native_alarm_state volume is MEASURED, not assumed",
"evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 measures both AT THE LEGACY-DB SOURCE (they are not in the Phase 1 oplog - see reviewPass) and sets MaxOplogRows/MaxOplogAge arithmetically; Task 20 evidence 10 does the empirical post-cutover drain check. If growth is monotonic, STOP: keyed-instances escape hatch = ~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141 (adoption design doc, NOT the 07-17 lib doc), a scadaproj library effort that would suspend this plan."
},
{
"id": "D5",
"subject": "Cutover forecloses rolling site upgrades",
"evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment docs (installation-guide.md/topology-guide.md - no file literally named runbook)."
},
{
"id": "D6",
"subject": "NEW (review pass): 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling",
"evidence": "Neither side configures gRPC message sizes (ScadaBridge AddGrpc at Program.cs:519-521 sets only the auth interceptor; the lib's initiator channel is bare GrpcChannel.ForAddress) => 4 MB default receive limit both directions. Batching is row-count-only (MaxBatchSize default 500; SyncSession.cs:227, SnapshotStreamer.cs:55; no byte-aware chunking). deployed_configurations.config_json documented >128 KB/row; a few dozen such rows in one batch exceeds 4 MB and wedges the stream on a poison batch. Task 1 measures max/avg config_json bytes; Task 19 sets LocalDb:Replication:MaxBatchSize so max-row-bytes x MaxBatchSize << 4 MB; any single row near 4 MB = STOP (needs byte-aware batching or size knobs in the LocalDb lib - scadaproj effort)."
}
],
"reconFindings": [
"The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).",
"sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.",
"No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. LocalDb RegisterReplicated SUPPORTS composite PKs (ordered pk ordinals) and rejects BLOB columns - no Phase 2 table has one (verified). Phase 1's site_events GUID conversion has no Phase 2 analogue.",
"SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).",
"DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter at :169 (ctor :161-175; :184 is the field ASSIGNMENT). The same-typed IActorRef? optional dclManager sits immediately BEFORE it at :168 - that's the real silent-shift hazard. Props.Create passes it positionally at AkkaHostedService.cs:810. Check every call site by hand (Task 16).",
"ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it (AkkaHostedService.cs:866 SetDeliveryGate -> SelfIsPrimary -> SelfIsOldestUp). Doc comment :14/:16 mentions replication; SiteReplicationActor.cs:288 calls it directly (dies with the actor).",
"ConfigFetchRetryCount's ONLY production reader is SiteReplicationActor.cs:157 (verified) - dead after Task 15; removed in Task 17 (removing it in Task 13, before the actor deletion, would not compile). IDeploymentConfigFetcher is KEPT: DeploymentManagerActor refresh path + SiteReconciliationActor + DI at ServiceCollectionExtensions.cs:84.",
"notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) AND NOT registered (Task 14, corrected) - migrating or replicating them would resurrect/ship plaintext SMTP passwords; they are permanently empty by design (writers removed 2026-07-10; verified only test-seeding inserts exist).",
"REVIEW-PASS ADDITIONS: SiteReconciliationActor (runs on EVERY node at startup, best-effort self-heal vs central) is the second caller of both StoreDeployedConfigIfNewerAsync and IDeploymentConfigFetcher - it survives Phase 2 and constrains Tasks 11/13/20 (a startup fetch on the standby is legitimate; zero-fetch assertions must scope to the deploy window).",
"SiteStorageService has 21 (not 22) inline connection+OpenAsync pairs: 60,206,248,311,345,386,414,443,470,498,538,583,608,640,663,692,730,769,813,837,868. CreateConnection():51's only repository consumer is SiteExternalSystemRepository.",
"StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.",
"ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.",
"10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).",
"Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts but NEVER run sqlite3 host-side against the live files (poisons the container's WAL state, root cause of the 2026-07-20 disk-I/O-error incident): cp the .db/-wal/-shm triplet and query the copy (see the snap() helper in the plan); ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).",
"Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.",
"MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
],
"tasks": [
{"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."},
{"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."},
{"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: kept PRAGMA journal_mode=WAL in InitializeAsync - the plan Step 4 snippet drops it but Task 4 says not to move the equivalent pragma; contradictory, and dropping it would regress documented concurrent-writer support before Task 5 makes it moot. Added a legacy-upgrade test beyond the plan's (the specified test asserts against a FRESH table where CREATE TABLE already lists all 16 columns - it would pass with every ALTER deleted). That test found the last_attempt_at_ms backfill lands 1 ms low: julianday() double day-fraction rounding, pre-existing, carried verbatim, asserted with 1 ms tolerance. Suite 154/154."},
{"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."},
{"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [3], "note": "Test fallout was ~7x the plan estimate: 40 files across 7 test projects, not \"fixtures\" in one. Most used Mode=Memory;Cache=Shared and LocalDb has NO in-memory mode, so all moved to real temp files. Added shared tests/ZB.MOM.WW.ScadaBridge.TestSupport lib (TestLocalDb) instead of copying the Phase 1 fixture into 7 projects. WAL test retargeted to the LocalDb-backed store; directory-creation test moved to Host.Tests (different owner - see Task 6 note)."},
{"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [4], "note": "LATENT PHASE-1 DEFECT FOUND+FIXED: LocalDb does NOT create the parent directory and SqliteLocalDb opens the file eagerly, so a missing dir is a HARD BOOT FAILURE (SQLite Error 14). Default site config is the RELATIVE ./data/site-localdb.db; docker escapes only because the volume mount creates /app/data. The plan wrongly assumed dir-creation moved to LocalDb with file ownership. Fixed via SiteLocalDbDirectory.Ensure(config) before AddZbLocalDb + Host.Tests/SiteLocalDbDirectoryTests (non-vacuity observed: 2 tests failed with exactly Error 14 pre-fix). ALSO a contract change: SiteStorageService.CreateConnection() used to return an UNOPENED connection; it now returns an ALREADY-OPEN one - SiteExternalSystemRepository dropped 5 OpenAsync calls. AddSiteRuntime(string) overload deleted."},
{"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "completed", "classification": "standard", "blockedBy": [5, 6], "commit": "f8aa02e2", "note": "As planned. Two pins in SiteLocalDbWiringTests through the REAL composition root: all 12 tables exist, and ReplicatedTables is EXACTLY the Phase 1 pair (an equality check, not Contains - the 'not yet' is the assertion that matters). Task 14 should INVERT that second test, not delete it. Non-vacuity verified by removing the DDL."},
{"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "bdc0dffe", "note": "DEVIATION: the copy INTERSECTS the legacy column set with the current one instead of naming all 16 columns. A legacy file from an older build lacks execution_id/parent_execution_id/last_attempt_at_ms, and naming a missing column throws 'no such column' - which the existing ReadAll treats as an unrecognised shape and SILENTLY DISCARDS EVERY ROW. For undelivered messages that is real data loss. A required-column (PK) guard stops the tolerance degrading into NULL-keyed copies. Tests register sf_messages on their own harness (production OnReady does not until Task 14), so Task 14 MUST keep Migrate as the LAST call in OnReady, after all registrations. Non-vacuity verified: all 4 fail without the Migrate call."},
{"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "5ddc7eed", "note": "7 of 9 tables migrated; notification_lists + smtp_configurations skipped as planned (plaintext passwords must not enter a table Task 14 makes replicated) - the skip is pinned by a test that also greps __localdb_oplog.row_json for the secret. Task 8 MigrateTable generalized to MigrateFile(many tables, one transaction, one rename). ADDED beyond the plan: MigratorColumnLists_MatchTheLiveSchema, a column-parity test vs SiteStorageSchema - the plan called for a manual column check, but a mismatch is invisible at runtime in BOTH directions (a typo is silently dropped by the intersection; a missing column silently leaves data behind). SiteStorageTables + LegacyTable are internal so the test can read them. Non-vacuity verified twice: removing the call (3 fail) and wrongly adding notification/smtp to the map (skip test fails)."},
{"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "completed", "classification": "standard", "blockedBy": [8], "commit": "2bbe6631", "note": "DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness (abstract base) instead of duplicating ~150 lines; Phase 1 tests derive from it and pass unchanged. The harness registers the 8 Phase 2 tables itself (production OnReady does not until Task 14) - DELETE RegisterPhase2TablesUntilCutover at Task 14. The 8-table list is literal, not derived from production code, so a cutover registering the wrong set fails these tests. Non-vacuity: unregistering sf_messages failed 6 of 7 - the 7th (add-then-remove ordering) PASSED because an absent row is also what a non-replicating pair looks like; fixed with a control row that must converge in the same window."},
{"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "completed", "classification": "standard", "blockedBy": [9], "note": "N1 re-expressed per D2 as a no-rows-lost property (LWW merge, never deletes), not directional authority. DEVIATION on the zero-fetch assertion: the plan wanted a fetcher double recording zero calls, but this harness has no actor system / no central / no IDeploymentConfigFetcher in the graph, so the double could not fail either way. Replaced with the positive half (config reaches B over replication alone) plus an in-file comment recording that the negative half is proved by Task 15 deleting the code and the build passing. D1 scope note recorded in-file. Non-vacuity: unregistering deployed_configurations fails all 4.", "commit": "c56bf4ae"},
{"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "completed", "classification": "standard", "blockedBy": [10, 11], "note": "As planned - NO production change; D3 confirmed correct, the call already exists at DeploymentManagerActor.cs:1921. Pin verified RED-FIRST by commenting out the call. Test needed a using for Commons.Messages.Artifacts and polls (the apply runs on a Task.Run inside the actor).", "commit": "79ce5161"},
{"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "completed", "classification": "standard", "blockedBy": [12], "note": "Doc-comment only, as planned. Re-verified both callers: SiteReplicationActor:375 (dies Task 15) + SiteReconciliationActor:166 (survives). Step 2 scope check re-run: ConfigFetchRetryCount's only production reader is still SiteReplicationActor:157, so option + validator rule stay until Task 17. Also fixed a stale 'guarded standby write' header in SiteStorageServiceTests.", "commit": "79ce5161"},
{"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "completed", "classification": "high-risk", "blockedBy": [13], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
{"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "completed", "classification": "high-risk", "blockedBy": [14], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
{"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
{"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"},
{"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"},
{"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"},
{"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "completed", "classification": "high-risk", "blockedBy": [18, 19], "note": "ALL 10 CHECKS PASS. Evidence: docs/plans/2026-07-19-localdb-phase2-live-gate.md. Key blocker found and fixed mid-run: external systems reach a site ONLY via ArtifactDeploymentService, which `instance deploy` never invokes - `deploy artifacts` was needed both to deliver the probe harness and to propagate the owed ExternalSystemDefinitions restore. THREE METHOD CORRECTIONS to the plan: (1) its instruction to run DB checks host-side against the bind mounts is UNSAFE - host sqlite3 poisons the container WAL; copy the db/-wal/-shm triplet and query the copy. (2) `docker exec ... curl` cannot scrape metrics (no curl in aspnet:10.0) and with 2>/dev/null the failure is silent - it nearly became a false 'metrics missing' finding; use a network-sharing curl sidecar. (3) checks needing S&F load need CachedCall, not Call. CAVEATS recorded not glossed: check 2's zero-count is vacuous alone (legacy source was also empty) and rests on the ABSENCE of CDC triggers; check 7's native_alarm_state leg was empty live and is covered only offline; check 10's sampling is coarse and the real rise/drain evidence comes from check 6's 0->4->0.", "commit": "158e79bb"},
{"id": 21, "subject": "Task 21: Documentation truth pass", "status": "completed", "classification": "standard", "blockedBy": [20], "note": "Both CLAUDE.md files + Component-StoreAndForward.md:83 (normative resync paragraph rewritten for CDC, stating the duplicate-delivery bound explicitly: limited to messages the OLD primary delivered whose status change had not yet replicated when the gate flipped - one flush interval plus in-flight ack, and it does NOT grow with backlog depth or absence duration) + components/{StoreAndForward,SiteRuntime,Host}.md + the frame-size known-issue (amended: Phase 2 deleted notify-and-fetch itself, so the 128KB Akka frame constraint is gone from the intra-site hop entirely; successor ceiling is the 4MB gRPC cap via MaxBatchSize, and note the failure mode differs - oversized gRPC is REJECTED, not silently dropped) + deployment topology-guide.md and installation-guide.md (D5 stop-both-together, D2 TombstoneRetention resurrection bound). DoD closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures). DoD grep nuance recorded in the plan: 4 matches remain in src/ and are all deliberate COMMENT prose explaining what was deleted - a literal 'no matches' would delete the explanations that stop someone re-introducing the old design.", "commit": null}
],
"knownFlakes": [
{
"test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary",
"note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation. Pre-existing, carried over from Phase 1."
},
{
"test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId",
"note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating."
}
],
"lastUpdated": "2026-07-20",
"phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load."
}
docs/plans/2026-07-19-localdb-phase2-gate.md
+120 -3
View File
@@ -1,8 +1,11 @@
# LocalDb Phase 2 — Gate Document
> **Status: NOT STARTED.** This is the gate, not the plan. Phase 2 work must not begin
> until an implementation plan is written against the facts below and the open questions
> in §5 are answered.
> **Status: CLOSED (2026-07-20).** The implementation plan is
> [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md); its
> decisions D1–D6 answer this gate and its Task 1 supplied the measurements §5 demanded
> (soak record: [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md)).
> Each §5 question is answered inline below. **The questions are deliberately not deleted —
> the reasoning is the value, and several of the answers overturn a premise stated above.**
**Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence.
@@ -50,6 +53,11 @@ Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-d
race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC
inherits the same bound, a tighter one, or a different failure shape.
> **Routed (2026-07-20).** The failure shape changes, so N5's bound does not simply carry
> over: CDC has no chunked anti-entropy hop to duplicate across, but LWW admits re-delivery
> of a row whose park loses to a later write (see the LWW answer in §5). **Task 21 states the
> new duplicate-delivery bound explicitly** and rewrites the N5 note rather than deleting it.
## 3. Substantially harder than Phase 1
Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no
@@ -84,16 +92,125 @@ class:
thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that
has not been run.* The live gate proved correctness, not steady-state behaviour over
time.
**ANSWERED — the soak ran (Task 1); the caps are not the binding constraint, batch
*bytes* are.** Two of this question's own premises turned out to be wrong, and both
corrections relax it:
1. **The stop condition was much weaker than written.** Exceeding `MaxOplogRows` /
`MaxOplogAge` does not wedge or lose data — `OplogStore` prunes and flags the peer
`needs_snapshot` (`OplogStore.cs:109-138`, `MaintenanceBackgroundService.cs:57`), which
degrades to a **graceful snapshot resync**. Overrun is a performance event, not a
correctness event, so these caps do not need to be sized defensively.
2. **`sf_messages` has a hard ceiling, not an estimate.** `SweepBatchLimit` (500) ÷
`RetryTimerInterval` (10 s) = **≤50 row-writes/sec**, structurally. Measured rig rate
under the purpose-built `SoakGenerator` load was far below that (see the soak record).
Rows are small — max `payload_json` **76 bytes** on the rig.
`native_alarm_state` is answered under the last question in this section, not here (and
the plan's D4, whose premise it corrects). The **real** ceiling this
question was groping toward is D6's 4 MB gRPC message cap, and it binds on
`MaxBatchSize × max-row-bytes`, not on oplog depth — **Task 19 sets
`LocalDb:Replication:MaxBatchSize = 16`**, the one firmly evidence-backed number the soak
produced. The keyed-instances escape hatch is **not** needed; this plan proceeds.
Independently, the soak retired a scare: a `disk I/O error` storm initially read as a
Phase 1 library defect and STOP-gated this plan was root-caused as **observer-induced**
(host-side `sqlite3` against live bind-mounted WAL files) — see
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md)
§0. The unmodified library sustains the full soak load with **zero** SQLite errors.
- **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do
when one node parks a message the other is mid-delivery on?
**ANSWERED — the scenario cannot arise on a correctly-behaving pair, and D2 covers what
happens if it does.** Only the **active** node sweeps and delivers; the standby holds a
convergent copy and sends nothing. So "one node mid-delivery while the other parks" is
a split-brain symptom, not steady-state behaviour, and store-and-forward is not where it
should be defended against.
If it does happen, LWW resolves per row by HLC: the later write wins, and the losing
node's view converges to it. The concrete risk is **at-least-once delivery** — a park
that loses to a stale in-flight update can be re-swept and re-sent. That is not new;
store-and-forward is already at-least-once by construction (a send that succeeds but
whose ack is lost is retried). What **is** new is D2's semantic change, recorded here
because it is the honest cost of this gate: the standby is no longer guaranteed
**byte-identical** to the active buffer, only **convergent**. The bespoke replicator's
`ReplaceAllAsync` bought identity by wiping — and the N1 directional guard existed
precisely because that wipe was dangerous. The library's snapshot resync **merges per
row and never wipes** (`SnapshotApplier`, `LwwApplier.cs:69-78`), so the failure the
guard prevented is structurally impossible rather than merely tested against. Task 10
ports `SfBufferResyncPredicateTests` as a **convergence** assertion, not a
directional-authority one.
- **Migration-under-load.** How does the one-time copy of an actively-replicating
`scadabridge.db` interact with the bespoke replicator still running during cutover?
**ANSWERED — the interaction is designed out, not managed.** The two mechanisms never run
concurrently. `SiteReplicationActor` and StoreAndForward's `ReplicationService` are
**deleted in the same commit** that registers the Phase 2 tables (Tasks 14/15), and D5
forecloses rolling upgrades: **both nodes of a site stop and start together**. A process
that boots with the new code has no bespoke replicator to race, and one running the old
code has no CDC triggers. There is no window in which a row is written by one mechanism
and read by the other.
Within a single booting process the ordering is the load-bearing part, and it is the same
invariant Phase 1 established: **DDL → `RegisterReplicated` → migrate → writes**
(`SiteLocalDbSetup.OnReady`). Rows written *before* registration are invisible to the peer
forever, silently — which is why Tasks 8/9 run the migrator strictly **after**
registration, so every migrated row is captured by the CDC triggers and replicates
normally. Both nodes migrating independently is fine: they migrate the same source rows,
and LWW converges them.
- **Rollback story.** With no dual-mechanism period, what is the recovery path if the
cutover fails in production — beyond "revert the commit"?
**ANSWERED — "revert the commit" is genuinely the path, and it is safe because the
migration is additive and the legacy files are left intact.** Tasks 8/9 **copy** rows out
of `scadabridge.db` and `store-and-forward.db` into the consolidated LocalDb file; they do
not drop, truncate, or delete the source databases. Rolling back is therefore: stop both
nodes of the site, deploy the previous build, start both together (D5). The old code
reopens the legacy files and finds them exactly as it left them.
The bounded, honest cost of a rollback is **the delta** — rows written into the
consolidated DB after cutover do not flow back into the legacy files. For config tables
that self-heals: `SiteReconciliationActor` reports local inventory to central at startup
and fetches whatever it lacks, so a rolled-back node re-converges to central's truth
without operator action. For `sf_messages` the delta is **lost undelivered buffer** — the
practical mitigation is to drain the buffer before cutting over, which Task 20's live gate
and Task 21's runbook both call for.
What has **no** rollback is a site pair split across versions — hence D5. That is a
deployment-procedure constraint, and Task 21 puts it in the runbook rather than leaving it
as tribal knowledge.
- **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly
because Phase 1's tables were small. Adding config + S&F changes the size and write
profile of that single file.
**ANSWERED — yes, and the soak is the evidence.** The write profile the consolidated file
must absorb is bounded on every axis:
| Table | Rate bound | Row size | Basis |
|---|---|---|---|
| `sf_messages` | **≤50 writes/sec** (hard) | 76 B max on the rig | `SweepBatchLimit` ÷ `RetryTimerInterval` |
| `native_alarm_state` | `distinct_source_refs × 10/sec` | small | per-`SourceReference` coalescing on a 100 ms flush (`NativeAlarmActor.cs:473-502`) |
| config tables | deploy-driven, effectively idle | ~721 B max on the rig | Task 1 measurement |
These are unremarkable for SQLite in WAL mode, and the soak ran the full generator load
against the Phase 1 consolidated file for 30 minutes with **zero** SQLite errors and no
oplog growth pathology. **One DB per process stays.**
Two caveats carried into the plan rather than hidden here. First, the rig's config rows
are tiny (max 721 B) and **cannot** be treated as representative — the size ceiling that
matters comes from the documented ~60–70 KB production `config_json`, which is what
motivates D6 and `MaxBatchSize = 16`. Second, **D4's premise was wrong**:
`native_alarm_state` is *not* "unbounded by design" and *not* "by a wide margin the
highest-volume table" — the coalescing flush bounds it, and this rig has no alarm
generator at all (measured 0 rows), so its bound is analytic rather than observed. If a
production site ever shows alarm churn that swamps the shared oplog, the keyed-instances
hatch named in D4 remains the escape — it is simply not needed to start.
## 6. Not in scope (unchanged from the design)
`auditlog.db` (diverges per node by design — central pulls the union; replicating it would
docs/plans/2026-07-19-localdb-phase2-live-gate.md
+209
View File
@@ -0,0 +1,209 @@
# LocalDb Phase 2 — live gate evidence (2026-07-20)
Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: all 10 checks captured and
passing**, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's
third table was empty).
Build under test: `feat/localdb-phase2` @ `166f07fa`, LocalDb `0.1.1` — confirmed live, not
assumed: the metrics scrape reports `otel_scope_name="ZB.MOM.WW.LocalDb.Replication"
otel_scope_version="0.1.1"`. Rig redeployed with `docker/deploy.sh`, exit 0, all 8 nodes up.
## Method notes — two corrections to the plan
**1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs.** macOS↔container
locks do not cross virtiofs, so a host `sqlite3` open recovers the WAL out from under the container
and triggers a permanent `disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of
the 2026-07-20 incident. Every read below used a helper that copies the `db`/`-wal`/`-shm` triplet
and queries the **copy**; `cp` is a pure byte read with no SQLite involvement.
**2. `docker exec … curl` cannot scrape the metrics** — the `aspnet:10.0` image has no `curl`, and
with `2>/dev/null` the failure is silent and looks exactly like "no metrics exported". An earlier
pass nearly recorded that as a finding. Use a network-sharing sidecar:
```bash
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
```
**3. The probe harness must be delivered by `deploy artifacts`, not `instance deploy`.** External
systems reach a site only through `ArtifactDeploymentService` (`FetchGlobalArtifactsAsync` →
`GetAllExternalSystemsAsync`), which the per-instance deploy path does not invoke. A script that
names an external system that never arrived fails silently — no rows, no error.
---
## 1. Migration ran — PASS
| Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` |
|---|---|---|---|
| site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 |
| site-a-node-b | both | 11804 | 11804 |
Both nodes migrated their own legacy files independently and the identical-content rows then
LWW-converged — expected per the plan, not an anomaly.
**Caveat:** the migration itself ran on an earlier deploy of this build (markers dated 05:04), so
this is after-the-fact evidence. `deployed_configurations` was 0 in both legacy files, so that
table's migration path is untested here; check 3 exercises it fresh instead.
## 2. No SMTP/notification rows migrated — PASS
`smtp_configurations` and `notification_lists` are 0 on both nodes.
**That count alone is vacuous** — the legacy source also held 0 SMTP rows, so a broken exclusion
would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for
exactly these ten tables on both nodes —
```
OperationTracking, data_connection_definitions, database_connections,
deployed_configurations, external_systems, native_alarm_state,
sf_messages, shared_scripts, site_events, static_attribute_overrides
```
— and `smtp_configurations` / `notification_lists` have **no triggers at all**, so no replication
channel exists for them regardless of content. That is the property the security decision rests on.
## 3. Config converges byte-identical — PASS
Deployed `gate-sensor-1` (template 3) to site-a. Both nodes:
- whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f`
- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`,
`deployed_at` `2026-07-20T09:06:11.2087221+00:00`
- `__localdb_row_version`: HLC `116951506695487488`, origin node
`cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both**
Identical HLC *and* origin id is the strong form: node B holds the row A wrote, not an
independently-derived copy of it.
## 4. The standby made no config fetch — PASS
Deploy window from `2026-07-20T09:05:52Z`:
- `site-a-a` (active, deploy target): **1** fetch — `Fetching config for deployment 20b81607…
(notify-and-fetch)`. Legitimate: the active node pulling its own artifact.
- `site-a-b` (standby): **0** fetches.
Under the old architecture `SiteReplicationActor` would have notified the standby and it would have
fetched independently. It now receives the config purely by CDC — corroborated from the other
direction by check 3's shared origin id.
## 5. Store-and-forward converges — PASS
Harness: external system `GateDeadTarget` → `http://127.0.0.1:9` with method `Ping`, plus an
interval script calling `ExternalSystem.CachedCall` (only `CachedCall` buffers; plain
`ExternalSystem.Call` does not). Delivered via `deploy artifacts`.
- Buffered rows appear on **both** nodes with an identical rowset md5 `a484e369f6e0f00e7debff9b64464566`.
- `__localdb_row_version` for `sf_messages`: 2509 rows from **1 origin** on both nodes.
- The Pending→Parked transition of the pre-existing backlog also replicated identically
(10280 Pending / 1524 Parked on both), so **UPDATE** replicates, not just INSERT.
## 6. Site failover — PASS
Stopped the active node `site-a-a` at 09:18:37. Ten seconds later `site-a-b` logged
`InstanceActor started for gate-churn-1` and `ScriptActor GateBufferProbe started` — the standby
picked up the workload and kept buffering. While partitioned, node B's oplog rose to **4** unacked
entries.
Restarted `site-a-a` at 09:18:57. After rejoin:
| | node-a | node-b |
|---|---|---|
| probe rows | 48 | 48 |
| total `sf_messages` | 11852 | 11852 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
| duplicate ids | 0 | 0 |
| rowset md5 | `54ded633c28134222bf034f3d0cec680` | `54ded633c28134222bf034f3d0cec680` |
Buffered messages survived the flip, the backlog drained on rejoin, and **zero duplicate ids** on
either node is the exactly-once evidence.
## 7. Cascade delete — PASS
Seeded `gate-sensor-1` with a `static_attribute_overrides` row (via an `Instance.SetAttribute`
script — the attribute-override deploy path writes into the config JSON, not this table), confirmed
it replicated, then deleted the instance.
| | pre-delete | post-delete |
|---|---|---|
| `deployed_configurations` | 1 / 1 | 0 / 0 |
| `static_attribute_overrides` | 1 / 1 | 0 / 0 |
| `native_alarm_state` | 0 / 0 | 0 / 0 |
Crucially, **both nodes hold explicit tombstones** — `deployed_configurations=1`,
`static_attribute_overrides=1` in `__localdb_row_version WHERE is_tombstone=1`. The rows are not
merely absent on B; B applied the deletes.
**Caveat:** `native_alarm_state` was empty for this instance, so the third leg of the cascade is
untested live. It is covered offline by
`LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables`, which was
verified non-vacuous.
## 8. Dead letters, oplog, metrics — PASS
- `__localdb_oplog` = 0 and `__localdb_dead_letter` = 0 on both nodes at every sampling point.
- `localdb_oplog_depth` 0; `localdb_sync_applied_total` 11808 (a) / 224 (b);
`localdb_sync_reconnects_total` 1.
## 9. Both nodes stopped and started together (D5) — PASS
Stopped `site-a-a` and `site-a-b` together, started them together.
| | node-a | node-b |
|---|---|---|
| `sf_messages` | 11845 | 11845 |
| probe rows | 41 | 41 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
Zero occurrences of `disk I/O error` / `database disk image` / `SQLITE_IOERR` / `corrupt` in either
node's log after restart. Clean rejoin.
## 10. Drain under churn — PASS
Sampled every 8 s under sustained probe load:
```
t=8s oplog_a=0 oplog_b=0 probe_a=23 probe_b=23
t=16s oplog_a=0 oplog_b=0 probe_a=25 probe_b=25
t=24s oplog_a=0 oplog_b=0 probe_a=26 probe_b=26
t=32s oplog_a=0 oplog_b=0 probe_a=28 probe_b=28
t=40s oplog_a=0 oplog_b=0 probe_a=30 probe_b=30
t=48s oplog_a=0 oplog_b=0 probe_a=31 probe_b=31
t=56s oplog_a=0 oplog_b=0 probe_a=33 probe_b=33
t=64s oplog_a=0 oplog_b=0 probe_a=35 probe_b=35
```
Writes climb steadily and the two nodes stay in lockstep at **every** sample, with no oplog
accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the
plan cared about (monotonic growth that never drains) does not occur.
**On its own this sampling is weak** — an 8 s interval is far coarser than the flush interval, so a
transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The
rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer
was down and back to 0 after rejoin.
---
## Observations outside the gate
- **A central external-system delete does not remove the row from sites.** After deleting
`GateDeadTarget` centrally and re-running `deploy artifacts`, both site nodes still carry it.
Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in
the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans.
- **Deleting an instance orphans its buffered messages.** Removing the `soakgen-*` instances left
their 11,804 `sf_messages` with no tracking snapshot, producing a continuous
`Cached-telemetry drain: no tracking snapshot for …` warning flood. Also pre-existing, and worth
a cleanup path.
## Rig state as left
- Owed cleanup **done**: `SoakGenerator` (2021) and `soakgen-1..4` deleted;
`ExternalSystemDefinitions` id 1 restored to `http://scadabridge-restapi:5200` **and confirmed
propagated to both site nodes**.
- Gate harness removed: `gate-sensor-1`, `gate-churn-1`, template-3 scripts `GateBufferProbe` /
`GateSetStatic`, and the `GateDeadTarget` external system are deleted centrally. The site-side
`external_systems` row for `GateDeadTarget` remains, per the observation above.
- Remaining instances: `soak-motor-1..4` (template 4, not deployable) — untouched, as found.
- The ~11.8k orphaned `sf_messages` remain on both site-a nodes.
docs/plans/2026-07-19-localdb-phase2-soak.md
+348
View File
@@ -0,0 +1,348 @@
# LocalDb Phase 2 — rig soak findings
**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair
**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
> ## GATE VERDICT: **STOP — do not proceed to Task 3.** *(superseded — see update)*
>
> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak
> instead surfaced what looked like a **pre-existing Phase 1 defect**: the consolidated LocalDb
> database (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every
> write on the **active** node under sustained concurrent load. See
> [Finding 1](#finding-1-blocker).
>
> **Update 2026-07-20: Finding 1 is root-caused and is NOT a product defect.** The soak's own
> host-side `sqlite3` sampling poisoned both nodes (WAL reset across the virtiofs boundary);
> the unmodified code sustains the full soak load indefinitely with zero errors —
> reproduced/refuted on demand, see
> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
>
> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.**
>
> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based
> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages`
> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop
> condition from any of D1–D6 is met. The one binding number Phase 2 must honour is
> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets.
---
## 1. Method as actually executed
The plan's method needed four corrections before it would run. Recorded here so the next run
does not rediscover them.
| Plan said | Reality |
|---|---|
| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` |
| Sample with host-side `sqlite3` against the bind mounts | **This IS the cause of Finding 1** (root-caused 2026-07-20 — the "ruled out" verdict below did not survive; see the known-issue doc §0). Host↔container POSIX locks don't propagate over virtiofs; a host-side read checkpoints + resets the WAL under the container and permanently poisons its connections. Copy the file triplet and query the copy, or read counters from `/metrics`. |
| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). |
| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. |
### The generator that worked
`ExternalSystem.Call` does **not** buffer to store-and-forward in practice; `ExternalSystem.CachedCall`
is the buffering surface. This is the single most important operational detail for reproducing
S&F load.
- Template `SoakGenerator` (id 2021), one `Interval` script at `{"intervalMs":5000}`:
```csharp
var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 };
await ExternalSystem.CachedCall("Test REST API", "Add", parms);
```
- No attributes, no compositions, no connection bindings — deliberately, so it deploys cleanly.
- `ExternalSystemDefinitions` id 1 repointed to `http://127.0.0.1:9` (discard port → connection
refused → classified transient → buffered).
- 4 instances (`soakgen-1..4`) deployed to site-a.
Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refused per 4 min).
### Two rig-tooling bugs found and fixed en route
1. **`docker/seed-sites.sh` seeded stale role names** — `Design`/`Deployment` instead of the
canonical `Designer`/`Deployer` (`src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs:46-47`). Every
Designer/Deployer-gated management command failed `UNAUTHORIZED` on a freshly reseeded rig,
including `seed-sites.sh`'s own trailing `deploy artifacts` and `reseed.sh` stage 6d.
**Fixed** (commit `cf46e596`).
2. **`infra/mssql/setup.sql` never executes.** It is mounted into
`/docker-entrypoint-initdb.d/`, a convention the official `mcr.microsoft.com/mssql/server`
image does not implement. After `reseed.sh` drops the volume (`docker compose down -v`),
nothing recreates `ScadaBridgeConfig` or the `scadabridge_app` login, so `reseed.sh` hangs
forever on its "Waiting for setup.sql to create ScadaBridgeConfig" poll. Worked around by
applying the three init scripts by hand. **NOT yet fixed in the repo.**
---
## Finding 1 (BLOCKER) — *root-caused 2026-07-20: observer-induced, not a product defect; see the gate-verdict update and the known-issue doc §0*
### The Phase 1 consolidated LocalDb fails under sustained write load on the active node
`site-localdb.db` throws `SQLite Error 10: 'disk I/O error'` on essentially every write once the
active node is under concurrent load. Both Phase 1 tables and the audit telemetry paths are
affected.
Representative stacks (`docker logs scadabridge-site-a-b`):
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
SiteEventLogger.cs:line 221/236
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
OperationTrackingStore.cs:line 137
at ...CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) line 148
```
User-visible symptom: `[ERR] Failed to record event: script from ScriptActor:SoakCall` — **site
event logging is silently dropping events on the floor under load.**
#### It follows the load, not the node, not the observer
The failure was isolated by moving the load between nodes:
| Node | Role | Under load | `disk I/O error` in 4 min |
|---|---|---|---|
| site-a-a | active | yes | 2 175 |
| site-a-a | standby (after restart) | no | **0** |
| site-a-b | standby | no | **0** |
| site-a-b | active (after failover) | yes | **4 391** |
#### It is LocalDb-specific, not the filesystem — *wrong: sampling-selection bias; only the LocalDb file was ever host-read*
The decisive control. Under identical load, on the same node, in the same bind-mounted
directory, counting error-stack frames over 3 minutes:
| Store | Backing file | Errors |
|---|---|---|
| `OperationTrackingStore` | `site-localdb.db` (LocalDb) | 13 044 |
| `SiteAuditTelemetryActor` | `site-localdb.db` (LocalDb) | 4 350 |
| `SiteEventLogger` | `site-localdb.db` (LocalDb) | 900 |
| `CachedCallTelemetryForwarder` | `site-localdb.db` (LocalDb) | 162 |
| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** |
| `SiteStorageService` | `scadabridge.db` (legacy) | **0** |
Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed
database fails.
#### Ruling out the observer — **RETRACTED 2026-07-20: the observer was the cause**
Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made
observer-induced `-shm` corruption the leading hypothesis. The original run "excluded" it:
after node-a was restarted and the load failed over to node-b, node-b began erroring while no
host process touched its files, and sampled node-a went to zero once idle.
**That exclusion was wrong.** The 04:56 sampling had poisoned *both* nodes' files (both carry
the 04:56 main-DB mtime; node-b's WAL was left at 0 bytes) — node-b was silent only because a
standby issues ~no LocalDb statements, and erupted on its first post-failover write. Verified
2026-07-20: 10+ min of full soak load on a freshly-reopened node with zero errors, then a
single host `sqlite3` read reset its 4.6 MiB WAL to 0 bytes and started the error storm one
second later (`SQLITE_IOERR_SHORT_READ`, 522). Full mechanism + minimal repro:
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
#### Secondary defect, same area
```
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
this is most likely due to use of async operations from within this actor.
Cause: System.NotSupportedException
```
`SiteAuditTelemetryActor` is closing over `ActorContext` across an `await`. Likely a
contributing cause rather than a separate issue — it is in the same write path — but it is a
real bug on its own terms.
#### Why this blocks Phase 2
Phase 2 registers **eight further tables** into this database, including `native_alarm_state`
(the highest-volume table in either DB) and `sf_messages`. It also **deletes** the bespoke
mechanisms (`SiteReplicationActor`, `ReplicationService`) that currently carry that data
independently of LocalDb. Cutting over onto a store that cannot absorb the *current* write
load — and doing so in the same commit that removes the fallback — would convert a logging
defect into site-wide config and buffer loss.
~~This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.~~
**Root-caused 2026-07-20: not a Phase 1 defect** — observer-induced WAL reset across the
virtiofs bind-mount boundary; see the gate-verdict update at the top of this document.
---
## Finding 2 — D6 resolved: no stop condition, but `MaxBatchSize` must be lowered
The plan asserts `deployed_configurations.config_json` is "documented to exceed 128 KB per row."
That misreads the source. `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`
says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default serializer
double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the
128 KB frame."**
So the largest known real production `config_json` is on the order of **60–70 KB**.
Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan
predicted, hence the production figure above is the one to size against. (A direct measurement
against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the
servecli SSH service on :2222 refuses connections, so the box was unreachable.)
**Verdict: no stop condition.** Nothing approaches the 4 MB single-row ceiling.
**But the batching risk is real.** Batching is row-count-only (`MaxBatchSize` default **500**),
and neither side configures gRPC message limits, so the 4 MB default receive cap applies:
```
500 rows × 70 KB ≈ 35 MB ≫ 4 MB → poison batch, stream wedged
```
Recommended for Task 19:
```
LocalDb:Replication:MaxBatchSize = 16
```
`16 × 128 KB = 2 MB` — 2× headroom on row size over the known worst case, and 2× headroom
against the 4 MB cap.
---
## Finding 3 — D4 corrected: alarm write rate is bounded, not unbounded
The plan calls `native_alarm_state` "unbounded by design." It is not.
`NativeAlarmActor.MarkDirtyUpsert` (`NativeAlarmActor.cs:473-502`) coalesces into a dictionary
**keyed by `SourceReference`** and flushes on a timer (`_persistFlushInterval`, default
**100 ms**). One flush writes at most one row per distinct source reference, regardless of how
many transitions occurred in that window.
```
worst-case rows/sec = distinct_source_refs × 10
```
An alarm storm on N sources costs N rows per 100 ms flush, not N × transition-rate. This makes
the table analytically sizeable without a generator — which matters, because this rig cannot
produce alarm load at all (see §1).
**Not empirically validated.** `native_alarm_state` held 0 rows for the entire run.
---
## Finding 4 — `sf_messages` has a hard structural ceiling of 50 rows/sec
Confirmed by code rather than measurement, which is stronger here. The retry sweep takes at most
`SweepBatchLimit` messages every `RetryTimerInterval`, and each failed attempt is one `UPDATE`
incrementing `retry_count`:
```
SweepBatchLimit (500) ÷ RetryTimerInterval (10 s) = 50 row-writes/sec, hard ceiling
```
This confirms the plan's "~50 row-writes/sec worst case" — and it is a ceiling, not an estimate.
`DefaultMaxRetries = 50` at `DefaultRetryInterval = 30 s` bounds each message to 50 updates over
25 minutes.
Observed during the run: ~2.9 attempts/sec, far below the ceiling. Row counts could not be
sampled reliably (see §1) and the run was cut short by Finding 1.
---
## Finding 5 — exceeding the oplog caps is a graceful degradation, not a failure
The plan's Task 1 step 7 treats "the shared oplog cannot absorb this write profile" as a hard
stop requiring the keyed-instances escape hatch. It is much weaker than that.
`OplogStore.EnforceCapsAsync` (`OplogStore.cs:109-138`) prunes to the ceiling and sets
`needs_snapshot`; `MaintenanceBackgroundService.cs:57` logs *"Oplog backlog/age caps exceeded:
pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync."*
`SyncSession.ComputeSnapshotRequiredAsync` (`:413-415`) then forces a snapshot resync.
So overrunning the caps costs a **full snapshot resync**, not data loss and not a wedged stream.
The caps therefore express *"how long may a peer be absent before it needs a full resync"*, and
should be sized to the longest tolerable peer outage rather than treated as a correctness
boundary. In healthy two-node operation the oplog drains continuously — `localdb_oplog_depth`
read **0** throughout.
### Provisional cap recommendation (Task 19)
Sized for a ~3-hour peer outage at a conservative 100 rows/sec aggregate, pending re-measurement
after Finding 1 is fixed:
```
LocalDb:Replication:MaxOplogRows = 1000000 # default; ≈2.8 h at 100 rows/s
LocalDb:Replication:MaxOplogAge = 3.00:00:00
LocalDb:Replication:MaxBatchSize = 16 # Finding 2 — this one is NOT optional
```
Only `MaxBatchSize` is firmly evidence-backed. The other two rest on an assumed aggregate write
rate that this run could not measure.
---
## 1b. Clean re-run (2026-07-20, post-root-cause)
The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after
both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side
`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load
unchanged (`SoakGenerator` ×4 against a refusing endpoint):
| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` |
|---|---|---|---|---|---|
| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 |
| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 |
| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 |
| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 |
| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 |
| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 |
| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 |
**Measured:**
- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six
intervals (+288 rows over 360 s, zero variance).
- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached
`retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`,
status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only.
- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json`
max **721 B** over 4 rows (rig config is trivially small — see the caveat below).
- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded.
- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load,
which is the expected result and not a measurement failure (Phase 2's tables are not yet
registered, so this load cannot reach the oplog by construction).
- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window
(`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that
Finding 1 was observer-induced: identical load, identical library, nothing host-reading the
files, no errors.
- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under
the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC.
**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural
ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never
exercised at all. This re-run therefore confirms *steady-state health and the absence of the
Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is
**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so
sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not**
representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on
this number.
## 2. What still owes measurement
Carry forward (items 1–2 partially discharged by §1b above):
1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still
unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s
ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap.
2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an
A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does*
answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions
and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator
"does not reliably expose A&C" is stale.)
3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference.
4. The empirical oplog drain-under-churn check — Task 20 evidence item 10.
## 3. Rig state left behind
- Fully reseeded (central config volume dropped and replayed; site SQLite state wiped by
`reseed.sh` stage 2).
- `ExternalSystemDefinitions` id 1 is **repointed to `http://127.0.0.1:9`** — restore to
`http://scadabridge-restapi:5200` before using the rig for anything else.
- Template `SoakGenerator` (2021) and instances `soakgen-1..4` (ids 5–8) remain deployed on
site-a and are **still generating load**. Undeploy or delete them before the Task 20 live gate.
- `LdapGroupMappings` corrected in the live DB to the canonical role names.
docs/plans/2026-07-21-auto-down-availability-decision.md
+108
View File
@@ -0,0 +1,108 @@
# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)
**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).
## The decision
All two-node ScadaBridge clusters (central and every site pair) switch their downing
strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):
- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)
The leader among the **reachable** members downs the unreachable peer after the
stability window. Consequence: a hard crash of **either** node — the active/oldest
included — fails over to the survivor in ~25s (10s failure detection + 15s window),
with no operator action and no victim restart required.
**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
each site with no Kubernetes and no SQL available site-side, and "network partitions are
less of a risk than if this stops working." Availability wins.
## The accepted trade (read this before debugging a dual-active)
In a **real network partition** (both nodes alive, link cut) each side downs the other
and continues as a one-node cluster: **both run active** — two oldest-Up members, two
sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
resolver would instead have sacrificed the younger side. Recovery from dual-active is
operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
their own — the mutual downing quarantines the association.)
## Why the crashed-oldest direction was unsurvivable before (evidence)
Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
(config verified live): killing the active/oldest `central-a` produced, on `central-b`:
```
SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
[1] unreachable of [2] members
```
The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):
```csharp
// oldest is on the OTHER (unreachable) side:
if (DownIfAlone && otherSide == 1 && thisSide >= 2) // survivor side must be >= 2
return DownUnreachable.Instance;
return DownReachable.Instance; // 1-vs-1 → down MYSELF
```
`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
oldest side ("the node on the other side is no better" — upstream comment). So two-node
keep-oldest can never survive an oldest crash. This corrected an earlier
mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
## Alternatives rejected
| Option | Why not |
|---|---|
| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |
## What changed (implementation slice, same session)
- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
`DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
scoped to keep-oldest.
- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
provider + `auto-down-unreachable-after` vs the SBR block).
- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
`deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
**Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
together.**
- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
the victim is down (previously it asserted the outage).
- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
`AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
a `strategy` parameter (default `auto-down`).
- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
`docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
resolved.
## Residual operational notes
- **Seed-node bootstrap constraint still applies to boot-alone**: only the first seed
may self-form a cluster. Auto-down removes the active-crash outage (the survivor never
restarts), but a node that must BOOT alone while its peer is dead (cold start of only
the non-first-seed VM, or the survivor crashing while the peer is still down) still
waits in `InitJoin` for its peer. Operator recovery unchanged (restart first seed, or
self-first seed override).
- Monitoring already surfaces dual-active if it ever happens: both nodes report
`IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
two Primaries.
docs/plans/2026-07-22-clusterclient-to-grpc-plan.md
+355
View File
@@ -0,0 +1,355 @@
# ClusterClient → gRPC-only cross-cluster transport — implementation plan
**Date:** 2026-07-22. **Design:** `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md`
(read it first, §7 especially — it contains the deep-dive corrections this plan builds on).
**Goal:** all site↔central traffic rides gRPC with PSK auth from `ZB.MOM.WW.Secrets`;
`ClusterClient`/`ClusterClientReceptionist` are deleted; Akka remoting never crosses the
site↔central boundary.
Everything below was code-verified 2026-07-22. If a cited line has drifted, re-locate by the
quoted identifier, never by line number.
## How to execute this plan (read before starting)
- **Phase order:** 0 → (1A ∥ 1B) → (2 ∥ 3) → 4 → 5. Phases marked ∥ are parallel-safe **only in
separate git worktrees** (`git worktree add ../ScadaBridge-1B feat/grpc-sitecommand`) — never
run two agents against one working tree (destructive git races are a known family incident).
Designated merge order when tracks meet: 1A lands first, 1B rebases (expected conflicts are
confined to `ZB.MOM.WW.ScadaBridge.Communication.csproj` proto ItemGroups and `Program.cs`
service/Map blocks — resolve as union).
- **Branches:** `feat/grpc-phase0-psk`, `feat/grpc-central-control` (1A),
`feat/grpc-sitecommand` (1B), then per-phase branches; PR each phase to `main` with its DoD met.
- **Build/test:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx` (0 warnings — TreatWarningsAsErrors),
`dotnet test ZB.MOM.WW.ScadaBridge.slnx`. Tests are xunit **v2** (2.9.3) + `Akka.TestKit.Xunit2`
1.5.62 + NSubstitute. TestKit tests inherit `TestKit` — never hand-roll
`ActorSystem.Create`+join.
- **Proto codegen is CHECKED-IN, not build-time** (protoc segfaults in the linux_arm64 image).
For every new/changed proto follow the sitestream recipe documented in
`src/ZB.MOM.WW.ScadaBridge.Communication/ZB.MOM.WW.ScadaBridge.Communication.csproj`:
temporarily uncomment/add the `<Protobuf Include=... GrpcServices="Both" />` item, delete stale
generated files, `dotnet build` on macOS, copy `obj/**/Protos/*.cs` into a committed folder
(mirror `SiteStreamGrpc/`), re-comment the item. `docker/regen-proto.sh` exists.
- **Rig:** `docker/deploy.sh` rebuilds the 2-central + 3×2-site cluster (+ traefik). Central UI
`9001/9002:5000`, site gRPC `90x3/90x4:8083`. **External sites↔central rules:** never
host-`sqlite3` a live WAL DB; `aspnet:10.0` has no `curl`; seeding via `docker/seed-sites.sh`.
- **Coexistence rule:** every migrated path sits behind a config flag with the Akka
implementation as default until Phase 4. Rollback at any point = flip the flag.
## The two choke points (where all transport code changes)
- **Site→central:** `SiteCommunicationActor`
(`src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs`) — 7 message types,
all `ClusterClient.Send` to `/user/central-communication` (Ask, except `HeartbeatMessage` Tell).
- **Central→site:** `CentralCommunicationActor` (same dir) — the `SiteEnvelope` handler
(`:452-467`) unwraps and routes via per-site ClusterClient. ALL producers
(`CommunicationService`'s 27 + `SiteCallAuditActor`'s 2 relays) go through it.
**`CommunicationService` gets NO interface extraction; its ~20 consumers are untouched.**
---
## Phase 0 — PSK auth + dead-code removal (standalone hardening; parallel-safe with 1A/1B prep)
### T0.1 Delete the vestigial `/user/management` receptionist registration
`AkkaHostedService.cs:459` (`RegisterService` of `ManagementActor`) — delete the registration
only (the actor stays; `ManagementEndpoints.cs:117` asks it in-process via
`ManagementActorHolder`). Grep-verify nothing sends to `/user/management`. Update
`docs/requirements/Component-Host.md` REQ-HOST-6a to record the removal.
### T0.2 File the dead-integration issue
`IntegrationCallRequest` (`CommunicationService.cs:238`) is unwired in production —
`RegisterLocalHandler(Integration, …)` exists only in `SiteCommunicationActorTests.cs:102`, so
production always replies "Integration handler not available" (`SiteCommunicationActor.cs:134-136`).
File a Gitea issue (decide: wire or delete); **exclude it from the gRPC contract** (28 of 29
commands migrate). Do not change its behavior in this program.
### T0.3 PSK interceptor + options
New `src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs`, copied from
`LocalDbSyncAuthInterceptor.cs` (same file layout: seal, 4 server handlers → shared
`Authorize`, `authorization: Bearer` metadata extraction, `CryptographicOperations.FixedTimeEquals`
over UTF-8, **fail-closed when the expected key is unset**, reject with
`StatusCode.PermissionDenied`). Differences from the template:
- **Path scope:** gates a *set* of service prefixes (constructor-provided), initially the
sitestream service prefix (read the real package/service name from `sitestream.proto` —
`/…SiteStreamService/`); later phases add the new services. LocalDb sync keeps its own
interceptor + key untouched.
- **Key source, site side (server on `:8083`):** new option
`CommunicationOptions.GrpcPsk` (`ScadaBridge:Communication:GrpcPsk`), production value
`${secret:SB-GRPC-PSK-<siteId>}` resolved by the existing pre-host `SecretReferenceExpander`
(`Program.cs:56-65`) — zero new resolution code.
- **Key source, central side (clients today; server in 1A):** central's site set is dynamic, so
central resolves secret **`SB-GRPC-PSK-{siteId}`** at channel-build time via the runtime
`ISecretResolver` (copy the fail-closed lazy pattern from
`DataConnectionLayer/Adapters/MxGatewayDataConnection.cs:82-98`), cached per site, cache
invalidated on site remove. New helper `SitePskProvider` in Communication (interface) +
Host (implementation over `ISecretResolver`).
Wire: add the interceptor to the site's existing `AddGrpc` (`Program.cs:526-527`, alongside the
LocalDb one). Attach the PSK on central's existing site-dialing clients as call-level
`Metadata` `Authorization: Bearer <psk>` (the LocalDb sync client pattern,
`SyncBackgroundService.cs:82-86`): `SiteStreamGrpcClient` (subscribe calls),
`GrpcPullAuditEventsClient`, `GrpcPullSiteCallsClient` — each already flows through a channel/
invoker creation point where the site id is known.
### T0.4 Rig + tests
- Rig: mirror the LocalDb dev-key pattern — literal `ScadaBridge:Communication:GrpcPsk:
"dev-grpc-psk-site-a"` in BOTH `docker/site-a-node-*/appsettings.Site.json` (and site-b/c with
their own keys, since unlike LocalDb this is not optional), plus central-side dev secrets: seed
`SB-GRPC-PSK-site-{a,b,c}` into both centrals' secret stores (secret CLI seed flow, or dev-KEK
env as the rig's Secrets setup already does).
- Tests: interceptor unit tests (wrong key / missing header / unset expected key ⇒
`PermissionDenied`; non-gated service path passes; constant-time compare exercised);
`SitePskProvider` fail-closed test; one Host wiring test asserting the interceptor is
registered.
**Phase 0 DoD:** suite green; on the rig, an unauthenticated `grpcurl`/test client gets
`PermissionDenied` on SiteStream, authenticated streaming + pull still work end-to-end. PR merged.
---
## Phase 1A — Central control plane (site→central) — worktree A
### T1A.1 `central_control.proto`
New `Protos/central_control.proto` in the Communication project (checked-in codegen per recipe),
package `scadabridge.centralcontrol.v1`, service `CentralControlService`:
| RPC | Wraps DTO (source file under `Commons/Messages/`) | Notes |
|---|---|---|
| `SubmitNotification` | `NotificationSubmit`/`NotificationSubmitAck` (`Notification/NotificationMessages.cs:30,47`) | 11 fields incl. `Guid?` execution ids → string |
| `QueryNotificationStatus` | `NotificationStatusQuery`/`Response` (`:55,62`) | |
| `IngestAuditEvents` | reuse existing `AuditEventBatch`/`IngestAck` from sitestream.proto | import, don't duplicate; `ForwardState`/`IngestedAtUtc` stay off-wire (`AuditEventDtoMapper.cs:23-27`) |
| `IngestCachedTelemetry` | reuse `CachedTelemetryBatch`/`IngestAck` | same |
| `ReconcileSite` | `ReconcileSiteRequest`/`Response` (`Deployment/ReconcileSiteRequest.cs`, `ReconcileSiteResponse.cs` incl. `ReconcileGapItem`) | map<string,string> for name→hash |
| `ReportSiteHealth` | `SiteHealthReport`/`SiteHealthReportAck` (`Health/SiteHealthReport.cs`) | the big one: ~30 fields incl. maps of `ConnectionHealth` enum, `TagResolutionStatus`, `TagQualityCounts`, `NodeStatus` list, `SiteAuditBacklogSnapshot`; model nullable ints/doubles with wrappers; keep `SequenceNumber` |
| `Heartbeat` | `HeartbeatMessage` (`Health/HeartbeatMessage.cs`) → `google.protobuf.Empty` reply | fire-and-forget semantics preserved client-side (don't await failure into caller) |
Mappers in `Communication/Grpc/CentralControlDtoMapper.cs` with **round-trip golden tests**
(construct DTO → proto → DTO, assert deep-equal; include null-optional cases).
### T1A.2 Central hosting
Central branch of `Program.cs` (~`:262` services, `:449-469` Map block):
`builder.Services.AddGrpc(o => o.Interceptors.Add<ControlPlaneAuthInterceptor>())` — central's
interceptor variant verifies `Bearer` against the per-site PSK looked up by the **required
`x-scadabridge-site` metadata header** via `SitePskProvider` (fail-closed: missing header ⇒
`PermissionDenied`). Add an **explicit Kestrel listener** for h2c gRPC: new option
`ScadaBridge:Node:CentralGrpcPort` (default **8083**, symmetric with sites), configured like the
site branch does (`Program.cs:507-512`, `HttpProtocols.Http2`); central's `:5000` stays as-is
(Traefik is HTTP/1 — gRPC does NOT go through traefik). `MapGrpcService<CentralControlGrpcService>()`.
`CentralControlGrpcService` (Host or Communication): decode proto → the SAME message types →
`Ask` the existing `CentralCommunicationActor` handlers (they already handle all 7 — zero
handler logic changes) → encode reply. Reuse the readiness convention: reject `Unavailable`
until the central actor system is up (mirror `SiteStreamGrpcServer.SetReady`).
### T1A.3 Site-side client + transport seam
- New `ICentralTransport` (Communication): one method per the 7 sends. Implementations:
`AkkaCentralTransport` (extracted verbatim from today's `SiteCommunicationActor` send blocks)
and `GrpcCentralTransport` (new).
- `GrpcCentralTransport`: **channel pair with sticky failover + failback** per design §3.5 — new
shared `CentralChannelProvider`: endpoints from new option
`ScadaBridge:Communication:CentralGrpcEndpoints` (List<string>, e.g.
`["http://scadabridge-central-a:8083","http://scadabridge-central-b:8083"]`; validator: required
when transport=Grpc); sticky-until-failure; flip on connect-fail/`Unavailable`; background
failback probe every 30–60 s (gRPC health or a `Heartbeat` ping); reconnect backoff copied from
`SyncBackgroundService.cs:151` (1 s doubling, cap 60 s). Attach PSK (`GrpcPsk` option) +
`x-scadabridge-site` on every call; per-call deadlines from the matching `CommunicationOptions`
timeout (`NotificationForwardTimeout`, `HealthReportTimeout`, etc. — today's Ask timeouts,
unchanged values). **Cross-node auto-retry only on connect-fail/`Unavailable`** — never on
`DeadlineExceeded`.
- `SiteCommunicationActor` selects the implementation from new option
`ScadaBridge:Communication:CentralTransport` (`Akka` | `Grpc`, **default `Akka`**). The 7
handler bodies delegate to the injected transport; reply/fault semantics identical (timeout or
non-OK status ⇒ same `Status.Failure` the S&F/audit layers already treat as transient).
### T1A.4 Tests
Extend `Communication.Tests` (TestKit): `SiteCommunicationActor` with an NSubstitute
`ICentralTransport` — all 7 paths, fault propagation (transport throw ⇒ same failure the S&F
tests expect). `GrpcCentralTransport` unit tests with an in-process `TestServer` gRPC host:
failover flip, sticky behavior, failback probe, PSK attached, deadline set, no-retry-on-deadline.
Reuse/extend `DirectActorSiteStreamAuditClient` for the ingest integration harness.
`NotificationForwarderTests`/`SiteAuditTelemetryActorTests`/`HealthReportSenderTests` must pass
unmodified (they sit above the seam — if they need edits, the seam is wrong).
**1A DoD:** suite green; on the rig with site-a flipped to `CentralTransport=Grpc`
(central gRPC port published, e.g. `9013/9014:8083`): notification e2e, audit rows land, health
page live, heartbeat drives active flag, reconcile works after site restart — while site-b/c
still run Akka (coexistence proven).
---
## Phase 1B — Site command plane (central→site) — worktree B
### T1B.1 `site_command.proto`
Package `scadabridge.sitecommand.v1`, service `SiteCommandService` — **28 commands** (29 minus
dead `IntegrationCallRequest`), grouped into domain RPCs with `oneof` request/response
envelopes (full command list + reply types + `CommunicationService.cs` line refs in the design
doc §7 / recon inventory):
| RPC | Commands (count) | Deadline source |
|---|---|---|
| `ExecuteLifecycle` | RefreshDeployment, Enable/Disable/DeleteInstance, DeploymentStateQuery, DeployArtifacts (6) | `DeploymentTimeout`/`LifecycleTimeout`/`ArtifactDeploymentTimeout` |
| `ExecuteOpcUa` | BrowseNode, SearchAddressSpace, ReadTagValues, VerifyEndpoint, Trust/List/RemoveServerCert, WriteTag (8) | `QueryTimeout` (browse/search per existing Ask usage) |
| `ExecuteQuery` | EventLogQuery, DebugSnapshot, Subscribe/UnsubscribeDebugView (4) | `QueryTimeout`/`DebugViewTimeout` |
| `ExecuteParked` | ParkedMessageQuery/Retry/Discard, RetryParkedOperation, DiscardParkedOperation (5) | `QueryTimeout`; relay callers keep `RelayTimeout`(10s) < `QueryTimeout`(30s) ordering |
| `ExecuteRoute` | RouteToCall/GetAttributes/SetAttributes/WaitForAttribute (4) | `IntegrationTimeout`; WaitForAttribute uses its dynamic timeout |
| `TriggerFailover` | TriggerSiteFailover (1) | `LifecycleTimeout` |
Mappers `SiteCommandDtoMapper.cs` + round-trip golden tests for every command/reply (the bulk of
this track — budget accordingly; enums, `TrackedOperationId` struct → string guid, nullable
wrappers).
### T1B.2 Site server: shared dispatcher
Refactor `SiteCommunicationActor`'s receive table into `SiteCommandDispatcher` (pure routing:
message → `_deploymentManagerProxy` / `_artifactHandler` / `_eventLogHandler` /
`_parkedMessageHandler` / failover handler — preserving EXACTLY today's targets, including the
node-local parked-message handler; see design §7.3, the replicated-store semantics are
deliberate). The actor and a new `SiteCommandGrpcService` (mapped in the site branch next to
`SiteStreamGrpcServer`, gated by `ControlPlaneAuthInterceptor` + the readiness flag) both call
the dispatcher — one routing truth, both transports.
### T1B.3 Central client + transport seam
`ISiteCommandTransport` (send `SiteEnvelope`-equivalent, Ask or Tell) injected into
**`CentralCommunicationActor`**; implementations `AkkaSiteTransport` (today's per-site
ClusterClient path, extracted) and `GrpcSiteTransport`. `GrpcSiteTransport` uses a new shared
**`SitePairChannelProvider`**: addresses from the `Site` entity's existing
`GrpcNodeAAddress`/`GrpcNodeBAddress` (the streaming path's columns — do NOT invent
`LoadSiteAddressesFromDb`, it doesn't exist; reuse the `ISiteRepository` reads +
`CentralCommunicationActor`'s existing DB-driven cache-refresh loop `:532-598` to build/refresh
channels instead of ClusterClients), sticky failover/failback per §3.5, PSK from
`SitePskProvider` + deadlines per the table above. Config flag
`ScadaBridge:Communication:SiteTransport` (`Akka` | `Grpc`, default `Akka`) selected inside
`CentralCommunicationActor` — `CommunicationService` and `SiteCallAuditActor` unchanged.
### T1B.4 Tests
TestKit: `CentralCommunicationActor` with substitute `ISiteCommandTransport` (envelope routing,
Ask-sender reply plumbing, per-site transport lifecycle on site add/remove/change);
`SiteCommandDispatcher` unit tests (every command → correct target, incl. parked→local handler
and failover→local); `SiteCommandGrpcService` via TestServer (auth, readiness, one command per
oneof group); existing `CommunicationServiceTests`/`CentralCommunicationActor*Tests` pass with
the Akka implementation as default.
**1B DoD:** suite green; rig central flipped to `SiteTransport=Grpc` for site-a only: from
CentralUI — deploy refresh, enable/disable instance, browse, read tag, write tag, event-log
query, parked query/retry/discard (run retry against the STANDBY site node explicitly —
replicated-store semantics), `TriggerSiteFailover` — all work; site-b/c untouched on Akka.
---
## Phase 2 (after 1A) ∥ Phase 3 (after 1B) — full cutover on the rig + hardening
- **Phase 2:** flip all sites to `CentralTransport=Grpc`. Soak: S&F drain under central-a kill
(rows stay Pending, resume without loss or duplicates — sequence/dedup layers unchanged),
failback observed when central-a returns, health/heartbeat cadence unchanged in
`CentralHealthAggregator` (no sequence regressions logged).
- **Phase 3:** flip central to `SiteTransport=Grpc` for all sites. Soak: full CentralUI command
matrix against each site; site-node kill mid-command returns a clean error (no hang beyond
deadline); site pair failover mid-stream of commands.
- Both phases: watch for `PermissionDenied` noise (would indicate PSK drift), and confirm zero
ClusterClient log activity on flipped paths.
## Phase 4 — deletion + config cutover (sequential, after 2+3)
1. Flip both flag defaults to `Grpc`; rig + docs updated; one soak cycle.
2. Delete: `AkkaCentralTransport`/`AkkaSiteTransport`, ClusterClient creation
(`AkkaHostedService.cs:942-953`), `DefaultSiteClientFactory` (+ its tests), per-site
ClusterClient cache in `CentralCommunicationActor` (keep the DB refresh loop — it now feeds
`SitePairChannelProvider`), receptionist registrations `:436` and `:935` (T0.1 already removed
`:459`), `CommunicationOptions.CentralContactPoints` (+ validator + rig configs), then the
flags themselves.
3. Grep-gates: `rg -i "clusterclient|receptionist" src tests docker docs` → only historical docs;
`rg "CentralContactPoints"` → empty.
4. Docs: update `grpc_streams.md` (its "ClusterClient keeps command/control" split is
superseded; also fix its `LoadSiteAddressesFromDb` doc-vs-code gap), `Component-Host.md`,
`Component-StoreAndForward.md:137`, and add a `docs/known-issues` cross-ref note that the
frame-size class is retired. Keep `Akka.Cluster.Tools` (ClusterSingleton still used).
## Phase 5 — live gate (rig; sequential; every check PASS required)
1. **PSK negative:** no key / wrong key / missing `x-scadabridge-site` ⇒ `PermissionDenied`;
unset server key ⇒ all rejected (fail-closed); LocalDb sync key unaffected.
2. **Site→central matrix:** notification e2e (delivered + status query), audit + cached-telemetry
rows in `dbo.AuditLog`/site-calls, `/monitoring/health` live per site, heartbeat → active
flag, reconcile self-heal after site-node restart.
3. **Central→site matrix:** all 6 RPC groups exercised from CentralUI, parked retry/discard on
the standby node, failover command drains cleanly.
4. **Failover/failback:** kill central-a → sites flip to central-b sticky (S&F uninterrupted);
restart central-a → failback within probe cadence; same for a site node from central's side;
booting node rejects `Unavailable` until ready and the client fails over.
5. **Mid-drain kill:** kill central during an S&F drain burst — zero loss, zero duplicates.
6. **Frame-class retirement:** issue a command/reply > 128 KB (large browse/event-log result) —
succeeds over gRPC (impossible before).
7. **Boundary check:** with everything on gRPC, verify NO Akka association exists between any
site container and central (`netstat`/Akka logs) — remoting is pair-internal only.
8. **Restart discipline:** full-rig restart (pairs together) comes up clean; no receptionist/
ClusterClient log lines anywhere.
Record results in `docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md` (check-by-check,
the family's live-gate format).
## Effort & parallelization summary
| Track | Est. | Parallel with |
|---|---|---|
| Phase 0 | 2–3 d | 1A/1B proto authoring |
| 1A | 1.5–2 wk | 1B (separate worktrees; 1A merges first) |
| 1B | 2–3 wk (mapper-heavy) | 1A |
| 2, 3 | 2–4 d each | each other (independent flags/paths) |
| 4 | 2–3 d | — |
| 5 | 2–3 d | — |
Critical path ≈ 1B: **~4–6 weeks total**, matching the design estimate.
## Task checklist (tick as you go; IDs reference the sections above)
**Phase 0 — PSK + dead code** (branch `feat/grpc-phase0-psk`)
- [ ] T0.1 Delete `/user/management` receptionist registration (`AkkaHostedService.cs:459`) + Component-Host.md update
- [ ] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate)
- [ ] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients
- [ ] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests
- [ ] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged
**Phase 1A — central control plane** (worktree, `feat/grpc-central-control`)
- [ ] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
- [ ] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate
- [ ] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator
- [ ] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified
- [ ] 1A DoD: rig site-a on `Grpc` proves all 5 site→central paths while site-b/c stay Akka; PR merged (before 1B)
**Phase 1B — site command plane** (worktree, `feat/grpc-sitecommand`)
- [ ] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 + replies)
- [ ] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local)
- [ ] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka`
- [ ] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green
- [ ] 1B DoD: rig central on `Grpc` for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged
**Phase 2 ∥ 3 — cutover + soak**
- [ ] P2 All sites `CentralTransport=Grpc`; central-kill S&F soak (no loss/dupes), failback observed, health sequences clean
- [ ] P3 Central `SiteTransport=Grpc` all sites; full UI command matrix per site; site-kill mid-command clean; no PSK noise; zero ClusterClient log activity on flipped paths
**Phase 4 — deletion**
- [ ] Defaults flip to `Grpc` + soak; then delete Akka transports, ClusterClient creation, `DefaultSiteClientFactory`, receptionist registrations, `CentralContactPoints`, then the flags
- [ ] Grep-gates pass (`clusterclient|receptionist` → historical docs only; `CentralContactPoints` → empty)
- [ ] Docs updated: `grpc_streams.md`, `Component-Host.md`, `Component-StoreAndForward.md:137`, known-issues cross-ref
**Phase 5 — live gate** (record in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
- [ ] 1 PSK negatives · [ ] 2 site→central matrix · [ ] 3 central→site matrix · [ ] 4 failover/failback both directions · [ ] 5 mid-drain kill · [ ] 6 >128 KB frame-class proof · [ ] 7 no cross-boundary Akka association · [ ] 8 full-rig restart clean
## Gotchas for the executor (will bite; read twice)
- Generated proto C# is committed; never add an active `<Protobuf>` item to the csproj in a
final commit (linux_arm64 protoc segfault breaks the Docker build).
- `HeartbeatMessage` must stay fire-and-forget end-to-end — don't let a gRPC failure surface as
a fault to the heartbeat timer path.
- Deadline ≠ retry: no automatic cross-node retry on `DeadlineExceeded` for WriteTag/Deploy/
Failover; only on provably-unsent failures.
- The parked-message handler is node-local **on purpose** (replicated store); do not "fix" it
onto the singleton proxy.
- Ack-before-Leave on `TriggerSiteFailover` (`SiteCommunicationActor.cs:569`): the gRPC reply
must be written before the node leaves — verify the response completes under failover.
- Inner-before-outer timeouts: `RelayTimeout`(10 s) < `QueryTimeout`(30 s) must survive the
deadline mapping (`CommunicationService.cs:779-786` documents why).
- `SiteStreamGrpcServer.AuditIngestAskTimeout` (30 s) is "one source of truth" shared with
`CentralCommunicationActor` — keep the new central service on the same constant.
- Rig sites reach central by container name (`scadabridge-central-{a,b}:8083`), NOT via traefik
(HTTP/1 only).
- xunit v2: use `Xunit.SkippableFact` for env-gated tests, not `Assert.Skip`.
docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md
+661
View File
@@ -0,0 +1,661 @@
# ScadaBridge: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The OtOpcUa half lives in `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
**Goal:** (1) Either node of a 2-node ScadaBridge cluster can cold-start alone and become operational, unattended. (2) An admin-only "Trigger failover" control on the Health page performs a graceful, audited role swap of the central pair.
> ## ⚠️ ARCHITECTURE REVISED DURING EXECUTION (2026-07-22)
>
> **Part 1 shipped as self-first seed ordering, NOT the `SelfFormAfter` watchdog described below.** Tasks 1–7 as originally written are superseded; what was actually built is in "Part 1 as executed". Tasks 8–10 (manual failover) are unaffected and still apply as written.
>
> **Why.** A code review of Task 2 raised, and a written test then confirmed, that the watchdog's success signal ("am I `Up` within the window?") cannot distinguish *no seed answered InitJoin* from *a seed answered and the join is in flight* — it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. **Measured: a permanent split, still unhealed after 90 s** — converting a routine restart into an outage of the previously-healthy node. That is strictly worse than the gap being closed, and it is not the boot-partition trade the design accepted.
>
> Akka's own `FirstSeedNodeProcess` already implements exactly the intended semantics — InitJoin the other seeds, self-join only if nobody answers — and, being part of the handshake, has no such race. It runs only when `seed-nodes[0]` is the node's own address. So the fix is seed **ordering**, not new runtime code.
>
> The plan's stated safety property — "a booting node only self-forms when NO seed answers InitJoin" — is true of Akka's native first-seed rule and **false** of the watchdog. That claim also appears in the shared cross-repo design doc (`scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`) and in the OtOpcUa half; **both still need correcting** (owner deferred, 2026-07-22).
## Part 1 as executed — self-first seed ordering
**Architecture:** Every node lists ITSELF as `seed-nodes[0]` and its partner second. No new runtime code, no timer, no new option. `StartupValidator` enforces the ordering at boot (host **and** port comparison) because a broken ordering fails silently. Manual failover (Part 2) is unchanged: graceful `Cluster.Leave(oldest Up member)` via a new `IManualFailoverService` (CentralUI seam, Host implementation) — singleton drain, watchdog process-exit, supervisor restart, rejoin as youngest.
**Behavior** (all rows covered by `SelfFirstSeedBootstrapTests` — real in-process clusters from production `BuildHocon` at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5 s) — operational, unattended |
| Restart into a **live** peer | Peer answers `InitJoinAck`; node rejoins, never islands |
| Both cold-start simultaneously, mutually reachable | `InitJoin` handshake converges them → **one** 2-member cluster |
| Both cold-start during a genuine boot **partition** | Each forms its own cluster — same dual-active class `auto-down` already accepts |
| Peer-first ordering (the old config) | Never forms — retained as a falsifiability control in the test suite |
**Shipped:**
- 6 node appsettings reordered (the `*-node-b` configs; `-a` nodes were already self-first). All 14 satisfy the invariant.
- `StartupValidator` self-first rule + 3 tests; `SelfFirstSeedBootstrapTests` (4 tests).
- Docs corrected: `docker/README.md`, `docs/requirements/Component-ClusterInfrastructure.md` (new **Seed Node Ordering** section), `docs/components/ClusterInfrastructure.md` (3 passages), `docs/deployment/topology-guide.md` (incl. the stale keep-oldest claim Task 6 flagged), `CLAUDE.md`.
- Several docs had asserted self-first ordering was *unsafe* because a simultaneous cold start would produce two clusters that never merge. Disproved by test (row 3) and corrected.
**⚠️ Ops action:** the gitignored `deploy/wonder-app-vd03/` overlay must have its `SeedNodes` reordered self-first before its next deploy, or the node will now **refuse to boot**. The validator rule is a hard gate deliberately — the alternative is the silent wedge it replaces.
---
<details>
<summary>Original Part 1 architecture (SUPERSEDED — kept for the decision record)</summary>
**Architecture:** New `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, appsettings-bound) arms `ClusterBootstrapFallback` right after ActorSystem creation: wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)`. Safety gate: fires only when this node's own address is in its own seed list.
</details>
**Tech Stack:** .NET 10, Akka.NET 1.5.62, Blazor Server (CentralUI), bUnit, xunit. No new packages.
**Branch:** `feat/selfform-fallback` off `main`.
---
## Design essentials (from the shared design doc)
**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever. Both docker central nodes list `central-a` first, so a lone cold-starting `central-b` never comes Up (the "registered outage gap" — `docker/README.md:289`). `ClusterOptions.SeedNodes`' doc comment claims "either can start first", which the deployed configs do not deliver — this plan makes it true and fixes the comment.
**Behavior spec:**
| Scenario | Behavior with fallback |
|---|---|
| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
| Lone cold-start, self IS in own seed list | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
| Lone cold-start, self NOT in own seed list | Fallback inert (info log) — self-forming would island the node from the real seeds |
| Peer boots after survivor self-formed | Peer's InitJoin is answered → joins as youngest/standby. No island. |
| Both cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined |
**Manual failover rules:** graceful `Leave`, never `Down`; admin-only (`AuthorizationPolicies.RequireAdmin`); peer guard (disabled when <2 Up `Central` members); confirmation dialog warning the Blazor circuit will drop (Traefik routes the UI to the active node — triggering failover disconnects your own page, which reconnects against the new active); audited via the app's **central** audit writer (`ICentralAuditWriter` — NOT the shared seam, see the dual-seam gotcha) before the Leave is issued. No interplay with `SelfFormAfter` (the peer is alive on this path, so the restarted node rejoins normally).
**Multi-node TestKit:** assessed and NOT used — in-process real clusters via `TwoNodeClusterFixture` (production `BuildHocon`) cover every deterministic scenario; MNTR would need a dedicated no-parallelization test project for no added coverage. See the shared design doc for the full verdict.
---
### Task 1 (A1): `SelfFormAfter` option + validator
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (first task)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs`
**Step 1: Create the branch**
```bash
cd ~/Desktop/ScadaBridge && git checkout main && git checkout -b feat/selfform-fallback
```
**Step 2: Write the failing tests** (append to the existing test classes, matching their assertion style — read them first)
```csharp
// ClusterOptionsTests.cs
[Fact]
public void SelfFormAfter_defaults_to_ten_seconds()
{
new ClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
}
// ClusterOptionsValidatorTests.cs
[Fact] public void SelfFormAfter_null_passes_validation() { /* valid options + null → Succeeded */ }
[Fact] public void SelfFormAfter_zero_passes_validation() { /* zero = explicit disable → Succeeded */ }
[Fact] public void SelfFormAfter_negative_fails_validation() { /* -1s → Failed, message mentions SelfFormAfter */ }
```
**Step 3: Run to verify failure**
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SelfFormAfter"
```
Expected: FAIL (compile error — property doesn't exist).
**Step 4: Implement.** `ClusterOptions.cs` — add after `AllowSingleNodeCluster` (line 104):
```csharp
/// <summary>
/// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
/// while its peer is down loops on InitJoin forever. When this node has waited longer than
/// this window without becoming a cluster member, it forms a cluster on itself
/// (<c>Cluster.Join(SelfAddress)</c>) — but ONLY if its own address is in its seed list
/// (a non-seed node self-forming would create a permanent island). Default 10s: the pair
/// shares a datacenter, so a live peer answers InitJoin in milliseconds and waiting longer
/// buys nothing. <c>null</c> or a non-positive value disables the fallback (wait-forever).
/// Accepted trade: both nodes cold-starting within the window while mutually unreachable
/// form two clusters — the same partition class the auto-down strategy already accepts.
/// </summary>
public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
```
`ClusterOptionsValidator.cs` — inside `Validate`, after the `FailureDetectionThreshold` rules:
```csharp
builder.RequireThat(options.SelfFormAfter is null || options.SelfFormAfter.Value >= TimeSpan.Zero,
"ClusterOptions.SelfFormAfter must be null (disabled), zero (disabled) or a positive duration; "
+ "a negative value is always a configuration mistake.");
```
**Step 5: Run tests → PASS**
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
```
**Step 6: Commit**
```bash
git add -A && git commit -m "feat(cluster): SelfFormAfter option — bootstrap self-form fallback window"
```
---
### Task 2 (A2): `ClusterBootstrapFallback` + first integration test
**Classification:** high-risk (cluster formation behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/ClusterBootstrapFallback.cs`
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`
**Step 1: Write the failing test**
```csharp
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// Guards the InitJoin self-form fallback (decision 2026-07-22): Akka only lets the FIRST
/// listed seed form a new cluster, so without the fallback a non-first seed cold-starting
/// alone waits on InitJoin forever — the "registered outage gap". These tests build REAL
/// single/dual-node clusters from the production BuildHocon output, exactly like
/// TwoNodeClusterFixture, and arm the production fallback.
/// </summary>
public sealed class SelfFormBootstrapTests : IAsyncLifetime
{
private readonly List<ActorSystem> _systems = new();
/// <summary>Starts a node whose seed list puts the PEER first (self second, or absent),
/// so Akka's own first-seed rule can never self-form it — only the fallback can.</summary>
private ActorSystem StartNode(int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
{
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
var clusterOptions = new ClusterOptions
{
SeedNodes = selfInSeeds
? new List<string>
{
$"akka.tcp://scadabridge@127.0.0.1:{peerPort}",
$"akka.tcp://scadabridge@127.0.0.1:{selfPort}",
}
: new List<string> { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}" },
SelfFormAfter = selfFormAfter,
StableAfter = TimeSpan.FromSeconds(3),
HeartbeatInterval = TimeSpan.FromMilliseconds(500),
FailureDetectionThreshold = TimeSpan.FromSeconds(2),
MinNrOfMembers = 1,
AllowSingleNodeCluster = !selfInSeeds,
};
var hocon = AkkaHostedService.BuildHocon(
nodeOptions, clusterOptions, new[] { "Central" },
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
_systems.Add(system);
ClusterBootstrapFallback.Arm(system, clusterOptions, NullLogger.Instance);
return system;
}
[Fact]
public async Task Lone_non_first_seed_self_forms_after_the_window()
{
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening
var node = StartNode(selfPort, deadPeerPort, selfFormAfter: TimeSpan.FromSeconds(2));
// Without the fallback this waits forever (Akka first-seed rule). With it, the node
// must be a 1-member Up cluster shortly after the 2s window.
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
Cluster.Get(node).SelfMember.Status.ShouldBe(MemberStatus.Up);
}
public Task InitializeAsync() => Task.CompletedTask;
public async Task DisposeAsync()
{
foreach (var s in _systems)
{
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
}
}
}
```
(Match the assertion library to `SbrFailoverTests.cs` — Shouldly vs xunit `Assert`.)
**Step 2: Run to verify failure** — compile error (`ClusterBootstrapFallback` doesn't exist):
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SelfFormBootstrapTests"
```
**Step 3: Implement `ClusterBootstrapFallback.cs`**
```csharp
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
/// <summary>
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
/// forever. So "both nodes are seed nodes" (ClusterOptions.SeedNodes) does NOT mean either can
/// cold-start alone — a non-first seed booting while its peer is down waits indefinitely (the
/// "registered outage gap", docker/README.md). This watchdog waits <see cref="ClusterOptions.SelfFormAfter"/>
/// for membership; on expiry it forms a cluster on itself.
///
/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed,
/// a later-booting real seed would form a second cluster and the two can never merge. For nodes
/// that ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing
/// cluster (a booting node only self-forms when NO seed answers InitJoin), so a peer booting
/// after this node self-formed simply joins it.</para>
///
/// <para><b>Races are benign.</b> If the join completes between window expiry and
/// <c>Cluster.Join(SelfAddress)</c>, Akka ignores the join — a node joins a cluster at most once
/// per incarnation. The residual risk is both nodes cold-starting inside the window while
/// mutually unreachable (a boot-time partition): both self-form, the same dual-active class the
/// auto-down downing strategy already accepts, with the same recovery (restart one side).</para>
/// </summary>
public static class ClusterBootstrapFallback
{
public static void Arm(ActorSystem system, ClusterOptions options, ILogger logger)
{
if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
{
logger.LogInformation(
"Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+ "while its peer is down will wait on InitJoin indefinitely.");
return;
}
var cluster = Akka.Cluster.Cluster.Get(system);
var self = cluster.SelfAddress;
var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
if (!isSeed)
{
logger.LogInformation(
"Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+ "[{Seeds}] — self-forming here would island it from the real seeds.",
self, string.Join(", ", options.SeedNodes));
return;
}
var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
cluster.RegisterOnMemberUp(() => joined.TrySetResult());
_ = Task.Run(async () =>
{
var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
if (winner == joined.Task || system.WhenTerminated.IsCompleted)
return;
logger.LogWarning(
"No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
+ "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
+ "merely partitioned (not dead), the pair is now dual-active — restart one side after "
+ "the partition heals (accepted availability-first trade, decision 2026-07-22).",
window, self);
cluster.Join(self);
});
}
private static bool TryParseAddress(string seed, out Address address)
{
try { address = Address.Parse(seed); return true; }
catch { address = default!; return false; }
}
}
```
(Add the `Microsoft.Extensions.Logging` using. If non-generic `TaskCompletionSource` is unavailable, use `TaskCompletionSource<bool>` + `TrySetResult(true)`.)
**Step 4: Run → PASS** (~5–10 s). **Step 5: Commit**
```bash
git add -A && git commit -m "feat(cluster): InitJoin self-form fallback — lone non-first seed becomes Up"
```
---
### Task 3 (A3): Remaining fallback tests — disabled / late-peer merge / non-seed
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4
**Files:**
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`
**Step 1: Append three tests** (each absence assertion carries an in-test positive control — repo convention):
```csharp
[Fact]
public async Task Disabled_fallback_keeps_waiting_and_the_node_was_otherwise_formable()
{
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort();
var node = StartNode(selfPort, deadPeerPort, selfFormAfter: null);
await Task.Delay(TimeSpan.FromSeconds(6)); // 3x the window used in the enabled test
var cluster = Cluster.Get(node);
cluster.State.Members.ShouldBeEmpty(); // still InitJoin-looping — today's behavior
// POSITIVE CONTROL: prove the node COULD have formed; only the fallback was missing.
cluster.Join(cluster.SelfAddress);
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
}
[Fact]
public async Task Peer_booting_after_self_form_joins_the_existing_cluster_no_island()
{
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
// B cold-starts alone (A dead), self-forms after 2s.
var nodeB = StartNode(portB, peerPort: portA, selfFormAfter: TimeSpan.FromSeconds(2));
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 1, TimeSpan.FromSeconds(20));
// A boots later with the pair seed list. B answers InitJoin, so A must JOIN B's
// cluster instead of islanding. Generous window on A so its fallback can't race.
var nodeA = StartNode(portA, peerPort: portB, selfFormAfter: TimeSpan.FromSeconds(30));
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(20));
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(20));
}
[Fact]
public async Task Non_seed_node_never_self_forms()
{
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadSeedPort = TwoNodeClusterFixture.GetFreeTcpPort();
var node = StartNode(selfPort, deadSeedPort, selfFormAfter: TimeSpan.FromSeconds(1), selfInSeeds: false);
await Task.Delay(TimeSpan.FromSeconds(5)); // 5x the window
var cluster = Cluster.Get(node);
cluster.State.Members.ShouldBeEmpty(); // guard refused to island a non-seed
// Positive control: the guard (not the environment) prevented formation.
cluster.Join(cluster.SelfAddress);
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
}
```
**Step 2: Run** — all 4 PASS. **Step 3: Commit** `test(cluster): self-form fallback — disabled, late-peer merge, non-seed guard`.
---
### Task 4 (A4): Production wiring + fix the misleading `SeedNodes` doc comment
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 3
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (inside `GetOrCreateActorSystem`, after the `WhenTerminated` continuation ending ~line 218, before `_actorSystem = system;`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs:33-38`
**Step 1: Wire the fallback:**
```csharp
// InitJoin self-form fallback (decision 2026-07-22): without it a non-first seed
// cold-starting while its peer is down loops on InitJoin forever — auto-down closed
// the crash-failover gap, this closes the cold-start-alone gap. Guarded inside Arm:
// disabled when SelfFormAfter is unset, inert when this node is not its own seed.
ClusterBootstrapFallback.Arm(system, _clusterOptions, _logger);
```
**Step 2: Fix the `SeedNodes` doc comment** (replace the `<summary>`):
```csharp
/// <summary>
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
/// partner. NOTE: listing both is NOT sufficient for "either can start first": Akka only lets
/// the FIRST listed seed form a new cluster, so a lone non-first seed waits on InitJoin until
/// <see cref="SelfFormAfter"/> expires and the self-form fallback fires (decision 2026-07-22).
/// Must contain at least one entry.
/// </summary>
```
**Step 3: Build + targeted tests**
```bash
dotnet build ZB.MOM.WW.ScadaBridge.slnx # 0 warnings (TreatWarningsAsErrors)
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
```
**Step 4: Commit** `feat(cluster): arm self-form fallback at ActorSystem creation; honest SeedNodes doc`.
---
### Task 5 (A5): appsettings sweep
**Classification:** trivial
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 6
Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the `ScadaBridge:Cluster` section of each (explicit for operator visibility; matches the code default):
- `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json`, `appsettings.Site.json`
- `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`
- `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json`
- `docker/site-b-node-a/appsettings.Site.json`, `docker/site-b-node-b/appsettings.Site.json`
- `docker/site-c-node-a/appsettings.Site.json`, `docker/site-c-node-b/appsettings.Site.json`
- `docker-env2/` — the 4 node appsettings files
Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`.
> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy.
---
### Task 6 (A6): Docs
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 5
- `docs/requirements/Component-ClusterInfrastructure.md` — replace the "registered outage gap" recovery text (env-var override) with `SelfFormAfter` semantics + island guard + boot-partition trade.
- `docker/README.md` (~line 289) — same replacement; keep the partition-trade note.
- `docs/deployment/topology-guide.md:101` — **fix the stale keep-oldest claim** (still says "Keep-oldest with `down-if-alone = on`"): rewrite to auto-down default + `SelfFormAfter`, pointing at the decision records.
- `CLAUDE.md` (~line 222) — update the boot-order note: pairs no longer require the first seed for cold start; note the 10 s window.
Commit: `docs(cluster): SelfFormAfter fallback; fix stale keep-oldest note in topology guide`.
---
### Task 7 (A7): Full verification + optional docker live gate
**Classification:** standard
**Estimated implement time:** ~5 min (suite runtime dominates)
**Parallelizable with:** none
**Step 1:**
```bash
cd ~/Desktop/ScadaBridge
dotnet build ZB.MOM.WW.ScadaBridge.slnx # expect 0 warnings
dotnet test ZB.MOM.WW.ScadaBridge.slnx # expect green vs pre-existing baseline
```
**Step 2 (LIVE GATE — run if the docker rig is available; else record deferred-live):**
```bash
cd docker && bash deploy.sh # rebuild with the fallback
docker compose stop central-a central-b # verify service names in docker-compose.yml first
docker compose start central-b # cold-start ONLY the non-first seed
docker compose logs -f central-b | grep -m1 "Self-forming a cluster" # ≈10s after start
curl -fsS http://localhost:9002/health/active # expect 200
docker compose start central-a # first seed returns → must JOIN, not island
docker compose logs central-a | grep -i "Welcome"
```
Expected: `central-b` self-forms ≈10 s and serves (previously a permanent wedge); `central-a` rejoins as youngest/standby.
---
### Task 8 (D1): `IManualFailoverService` + cluster-level test
**Classification:** high-risk (cluster behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs`
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (register in the Central branch, next to `IActiveNodeGate` ~line 330)
- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ManualFailoverTests.cs`
**Step 1: Failing tests** on `TwoNodeClusterFixture`:
```csharp
[Fact]
public async Task Failover_makes_the_oldest_leave_and_the_survivor_take_over()
{
await using var f = await TwoNodeClusterFixture.StartAsync();
var oldest = Akka.Cluster.Cluster.Get(f.NodeA); // NodeA started first = oldest
var target = AkkaManualFailoverService.FailOverCore(f.NodeB, "Central"); // issued from the OTHER node
target.ShouldBe(oldest.SelfAddress);
// Graceful exit path: the left node's own ActorSystem terminates…
await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
// …and the survivor becomes a 1-member cluster and the oldest-Up active node.
await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30));
ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB)).ShouldBeTrue();
}
[Fact]
public void Failover_refuses_when_no_peer_exists()
{ /* 1-node cluster → FailOverCore returns null, node still Up afterwards (positive assert) */ }
```
**Step 2:** Run → FAIL (service missing).
**Step 3: Implement.** Interface in CentralUI (CentralUI stays Akka-free):
```csharp
public interface IManualFailoverService
{
/// <summary>Gracefully fails over the central cluster: the current active (oldest Up)
/// member leaves, restarts via its supervisor, and rejoins as standby. Returns the
/// address string acted on, or null when there is no peer to fail over to.</summary>
Task<string?> FailOverCentralAsync(string actor);
}
```
Host implementation — static testable core + thin DI wrapper:
```csharp
public sealed class AkkaManualFailoverService : IManualFailoverService
{
// ctor: (AkkaHostedService akka, ICentralAuditWriter audit, ILogger<AkkaManualFailoverService> logger)
public async Task<string?> FailOverCentralAsync(string actor)
{
var system = _akka.GetOrCreateActorSystem();
var target = FailOverCore(system, role: "Central", dryRun: true);
if (target is null) return null; // peer guard
await _audit.WriteAsync(/* canonical AuditEvent: Action=cluster.manual-failover,
Actor=actor, DetailsJson={"target": target}, Outcome=Success — copy the exact
call shape from an existing audited admin action (e.g. the Sites admin service);
use the CENTRAL audit writer, not the shared seam */);
FailOverCore(system, role: "Central"); // Cluster.Leave(target)
_logger.LogWarning("Manual failover triggered by {Actor}: {Target} is leaving the cluster.", actor, target);
return target.ToString();
}
/// <summary>Oldest Up member with the role leaves — mirrors ActiveNodeEvaluator's oldest-Up
/// rule so the node acted on is exactly the one hosting the singletons. Returns null when
/// fewer than 2 Up members carry the role (no peer = failover would be an outage).</summary>
public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
{
var cluster = Akka.Cluster.Cluster.Get(system);
var withRole = cluster.State.Members
.Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
.OrderBy(m => m, Member.AgeOrdering)
.ToList();
if (withRole.Count < 2) return null;
var oldest = withRole[0];
if (!dryRun) cluster.Leave(oldest.Address);
return oldest.Address;
}
}
```
**Step 4:** Register in the Central branch of `Program.cs`. **Step 5:** Tests PASS → commit `feat(ui): manual central failover service — graceful Leave of the oldest Up member`.
---
### Task 9 (D2): Health page button + bUnit tests + runbook
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` (central-cluster card, near the Nodes column ~line 233)
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/HealthFailoverButtonTests.cs` (follow the project's existing bUnit page-test pattern)
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` + `docker/README.md` — manual-failover runbook paragraph
**Step 1: Failing bUnit tests:** (a) button absent without the admin policy; (b) present + enabled for admin with ≥2 online central nodes; (c) disabled with tooltip at 1 node; (d) confirm flow calls `IManualFailoverService.FailOverCentralAsync` exactly once (fake service).
**Step 2: Implement:**
```razor
<AuthorizeView Policy="@AuthorizationPolicies.RequireAdmin">
<button class="btn btn-outline-warning btn-sm"
disabled="@(!CentralHasPeer)"
title="@(CentralHasPeer ? "Gracefully restart the active node; the standby takes over."
: "No standby available — failover would be an outage.")"
@onclick="() => _showFailoverConfirm = true">
Trigger failover
</button>
</AuthorizeView>
```
plus a confirmation modal (copy the page's existing dialog idiom). The warning text MUST state: the active node restarts, roles swap, and this page will briefly disconnect and reconnect against the new active node (Traefik routes to the active). On confirm: call the service with the authenticated user name; surface the returned target address.
**Step 3:** Tests PASS → commit `feat(ui): admin manual-failover control on the health page`.
**Live check (fold into the Task 7 gate when the rig is up):** press the button, watch `central-a` restart and `central-b`'s badge flip to Primary; audit row lands in `dbo.AuditLog`.
---
### Task 10 (D3, OPTIONAL — confirm with the user before executing): site-pair failover from the central UI
**Classification:** high-risk (new cross-cluster message contract)
**Estimated implement time:** exploration first; likely 2–3 tasks if approved
The Health page also shows per-site node cards (Primary/Standby from heartbeats), but central and sites are **separate Akka clusters** — a site failover needs a `TriggerSiteFailover(siteId)` command over the existing central→site transport (same channel as the Retry/Discard relay), handled on the site's active node by `Cluster.Leave(SelfAddress)`. That adds a versioned message contract (rolling-upgrade surface). If approved: explore `ZB.MOM.WW.ScadaBridge.Communication` for the command path, mirror an existing command end-to-end, per-site button with the same guard/confirm/audit rules. Otherwise: file a follow-up issue and skip.
---
## Completion
- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `main` + push to gitea, or PR — ask the user).
- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked as the family-docs task in the scadaproj index plan).
- Verification-before-completion applies throughout: no task is done without its command output; live gates may be recorded deferred-live if the rig is down.
docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json
+61
View File
@@ -0,0 +1,61 @@
{
"planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
"tasks": [
{
"id": 0,
"subject": "Part 1 (SUPERSEDED Tasks 1-4): SelfFormAfter watchdog - implemented, review+test rejected it, reverted",
"status": "completed"
},
{
"id": 1,
"subject": "Part 1 as executed: self-first seed ordering + StartupValidator rule + SelfFirstSeedBootstrapTests + 14-config sweep + docs",
"status": "completed",
"blockedBy": [
0
]
},
{
"id": 2,
"subject": "Task 7: full solution verification + optional docker live gate",
"status": "completed",
"blockedBy": [
1
]
},
{
"id": 3,
"subject": "Task 8: IManualFailoverService + cluster-level test",
"status": "completed",
"blockedBy": [
2
]
},
{
"id": 4,
"subject": "Task 9: Health page failover button + bUnit tests + runbook",
"status": "completed",
"blockedBy": [
3
]
},
{
"id": 5,
"subject": "Task 10: site-pair failover via central\u2192site transport (BUILT \u2014 user approved 2026-07-22)",
"status": "completed",
"blockedBy": [
4
]
},
{
"id": 6,
"subject": "FOLLOW-UP: OtOpcUa converged on self-first seed ordering (2 local commits, unpushed); scadaproj CLAUDE.md index rows still owner-held",
"status": "completed"
},
{
"id": 7,
"subject": "OPS: reorder SeedNodes self-first in gitignored deploy/wonder-app-vd03/ before next deploy (hard boot gate)",
"status": "pending"
}
],
"lastUpdated": "2026-07-22T00:00:00Z"
}
docs/requirements/Component-ClusterInfrastructure.md
+68 -12
View File
@@ -92,27 +92,83 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
- Health reporting resumes from the new active node.
- Alarm states are re-evaluated from incoming values (alarm state is in-memory only).
## Split-Brain Resolution
## Downing Strategy (auto-down — availability-first)
The system uses the Akka.NET **keep-oldest** split-brain resolver strategy:
**Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question):** the clusters run the **`auto-down`** downing strategy — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15s). The leader among the *reachable* members downs the unreachable peer after the stability window:
- On a network partition, the node that has been in the cluster longest remains active. The younger node downs itself.
- **Stable-after duration**: 15 seconds. The cluster membership must remain stable (no changes) for 15 seconds before the resolver acts to down unreachable nodes. This prevents premature downing during startup or rolling restarts.
- **`down-if-alone = on`**: The keep-oldest resolver is configured with `down-if-alone` enabled. If the oldest node finds itself alone (no other reachable members), it downs itself rather than continuing as a single-node cluster. This prevents the oldest node from running in isolation during a network partition while the younger node also forms its own cluster.
- **Why keep-oldest**: With only two nodes, quorum-based strategies (static-quorum, keep-majority) cannot distinguish "one node crashed" from "network partition" — both sides see fewer than quorum and both would down themselves, resulting in total cluster shutdown. Keep-oldest with `down-if-alone` provides safe singleton ownership — at most one node runs the cluster singleton at any time.
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues (as before). If the **active/oldest** node crashes, the younger survivor becomes leader among the reachable members, downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — **no operator action and no victim restart required**. This closes the keep-oldest total-outage gap.
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
### Down-if-alone recovery
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
When a node downs itself (via `down-if-alone`, or any other SBR decision), the resolver is configured with `run-coordinated-shutdown-when-down = on`, so the self-down runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
- **static-quorum (quorum-size 1)**: Akka's `IsTooManyMembers` guard (`members > quorum*2-1`, i.e. `2 > 1`) returns **DownAll** on any unreachability — total shutdown, strictly worse.
- **keep-majority**: a 1-vs-1 split keeps the side with the lowest address, which just moves the fatal crash from "the oldest" to "the lowest-address node".
- **lease-majority**: needs a shared lease store (Kubernetes API, SQL, …) reachable from both nodes — not available at sites (one node per VM, no shared infrastructure).
1. Self-down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
### Downed-node recovery
When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
4. The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly; there is no stale membership to reconcile) — **but only while a peer still holding cluster state is reachable**. A lone restarted node that is *not* the first seed cannot re-form a cluster on its own (see the seed-node bootstrap constraint below); it waits for its peer.
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
**Seed-node bootstrap constraint.** Only the FIRST seed listed in `Cluster:SeedNodes` may self-join to form a *new* cluster. All nodes list the same first seed (e.g. `scadabridge-central-a`), so a lone restarted non-first-seed node (with the first seed still down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven — either restart the dead first-seed node (preferred) or restart the survivor with a self-first seed override (`ScadaBridge__Cluster__SeedNodes__0` = self, `__1` = peer). The repo does not ship self-first ordering per node: with both nodes self-first a simultaneous cold start risks two independent one-node clusters that never merge. Removing the gap itself is the **registered deferred keep-oldest topology/strategy decision** (master tracker 2026-07-08, owner: user).
### Seed Node Ordering
The docker failover drill (`docker/failover-drill.sh`) exercises this per direction: `standby` mode proves SBR downing + singleton continuity on the oldest; `active` mode measures the registered total-outage gap and the recovery-on-restart path.
**Every node lists ITSELF as `seed-nodes[0]` and its partner second (decision 2026-07-22).**
Akka runs two different bootstrap processes depending on that first entry. When `seed-nodes[0]` is the node's own address it runs `FirstSeedNodeProcess`: it `InitJoin`s the *other* seeds and self-joins only after `seed-node-timeout` elapses with nobody answering. When it is not, the node runs `JoinSeedNodeProcess`, which can never form a new cluster — it retries `InitJoin` indefinitely.
Until 2026-07-22 every node listed the same first seed, so a node that had to **boot alone** — a cold start of only the non-first-seed VM, or the survivor crashing while its peer was still dead — never reached `Up` and was never routable. That was the **registered outage gap**, and recovery was operator-driven. Self-first ordering closes it using Akka's own protocol, and `StartupValidator` fails the boot if a node config ever breaks the ordering (the invariant is silent when violated, so it is enforced loudly).
Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters built from `BuildHocon` at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5s) — operational, unattended |
| Restart into a **live** peer | The peer answers `InitJoinAck`; the node joins the existing cluster and never islands |
| Both cold-start simultaneously, mutually reachable | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
| Both cold-start during a genuine boot-time **partition** | Each forms its own cluster — the same dual-active class `auto-down` already accepts, same recovery (restart one side) |
**Rejected alternative — an external self-form timer.** A watchdog that waited a configurable window for membership and then called `Cluster.Join(SelfAddress)` was implemented and discarded. Its success signal ("am I `Up` yet?") cannot distinguish *no seed answered* from *a seed answered and the join is in flight*, because it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation (the peer must down it, then wait for the failure detector and a leader action); a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. Measured: a permanent split that had not healed after 90s, converting a routine restart into an outage of the previously-healthy node. Akka's first-seed process has no such race because it is part of the handshake, which is why the ordering — not a timer — is the mechanism.
The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
### Manual Failover (admin-triggered)
An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).
Semantics:
- **Graceful `Leave`, never `Down`.** The active node leaves the cluster so `ClusterSingletonManager` hands its singletons to the standby before the member is removed. A `Down` would skip the hand-off.
- **Target = the oldest Up member with the `Central` role**, the same rule `ActiveNodeEvaluator` uses, so the node acted on is exactly the one hosting the singletons — never Akka's cluster *leader*, whose address-ordered definition diverges from singleton placement after a restart.
- **Peer guard.** Refused when fewer than two Up `Central` members exist: failing over a lone node is an outage, not a failover. Enforced server-side in `AkkaManualFailoverService`; the button is also disabled client-side when the pair has no online standby.
- **Admin-only** (`AuthorizationPolicies.RequireAdmin`). The Health page itself is all-roles, so the gate lives on the control.
- **Audited before acting.** One `AuditChannel.Cluster` / `AuditKind.ManualFailover` row is written through `ICentralAuditWriter` *before* the Leave is issued, naming the actor and the target address — the acting node can be the one that goes away, and an audit written afterwards could be lost to the shutdown it describes. Audit failure never blocks the failover.
- **Your own page will disconnect.** Traefik routes the UI to the active node, so triggering a failover drops the admin's Blazor circuit; it reconnects against the new active node. The confirmation dialog says so.
After the Leave the node's `ActorSystem` terminates, the `WhenTerminated` watchdog exits the process, the service supervisor restarts it, and it rejoins as the youngest member (the new standby). Recovery is the normal restart contract above — no interaction with seed ordering, since the peer is alive on this path.
#### Site-pair failover
The same control appears on each **site** card. Central and each site are separate Akka clusters, so central cannot act on a site's membership — it *asks*, over the existing ClusterClient command/control channel:
1. `CommunicationService.TriggerSiteFailoverAsync` sends a `TriggerSiteFailover` inside a `SiteEnvelope`.
2. The site's `SiteCommunicationActor` (registered per node, so contact rotation reaches whichever answers) resolves the target from cluster state and issues the graceful `Leave` locally.
3. It replies `SiteFailoverAck` — sent **before** the Leave takes effect, so the ack still arrives when the acking node is the one leaving.
Differences from central failover, all deliberate:
- **Role scope is `site-{SiteId}`, not the base `Site` role.** Site singletons (the Deployment Manager) are placed on the site-specific role; using the base role would move the wrong node. Pinned by both a unit test asserting the role string and a real-cluster test.
- **Misroute is refused.** A command whose `SiteId` does not match the receiving node's is rejected rather than acted on — a misroute must never fail over a site the operator did not select.
- **A fault is acked, not thrown.** Reporting through the ack keeps the reason; letting it reach supervision would restart the communication actor and reduce central's Ask to a bare timeout.
- **The operator's own session is unaffected** — a different cluster entirely. The confirmation dialog therefore does *not* carry the "this page will disconnect" warning that the central one does.
- **Refusal vs unreachable are distinct.** A `false` ack is a definitive answer from the site (peer guard, misroute); a timeout means the site never answered. Both surface to the operator with their own wording, because only one of them means "nothing happened".
**Rolling upgrade.** A site running a binary older than this contract has no handler for `TriggerSiteFailover`; the message dead-letters and central's Ask times out, reported as "site did not respond". That is the correct user-facing outcome — an old site genuinely cannot honour the request. Message evolution stays additive-only.
## Single-Node Operation
docs/requirements/Component-StoreAndForward.md
+1 -1
View File
@@ -80,7 +80,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
- **Peer-join anti-entropy resync (chunked, ack-confirmed).** Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was **down for an extended period** (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node **(re)tracks its peer**, a **standby** requests a full-buffer snapshot (`RequestSfBufferResync`); the **active** node loads up to `MaxResyncRows` (10 000) of its oldest rows and answers with a **sequence of byte-budgeted chunks** (`SfBufferSnapshotChunk`), each carrying a shared `ResyncId`, a 1-based `Sequence`, and the `TotalChunks` count. Chunking is mandatory because the monolithic snapshot exceeds Akka remoting's **default 128 000-byte frame** for any realistic backlog and `BuildHocon` sets no override — a single oversized message is silently undeliverable (review 02 round 2, **N2 High**). Rows accumulate into a chunk until the estimated payload budget (`MaxResyncChunkBytes` = 64 000, ≈50% frame headroom) or the row cap (`MaxResyncChunkRows` = 200) is hit; a single row whose payload alone exceeds the budget ships solo with a Warning. The standby **assembles all chunks of one `ResyncId`** (a new `ResyncId` discards any stale partial assembly; a partial that never completes is dropped after `resyncAssemblyTimeout`, default 30 s, and counted a replication failure), then **replaces its entire local buffer** with the assembled snapshot (`ReplaceAllAsync`, one transaction) and returns a delivery confirmation (`SfBufferResyncAck`). The active node arms an ack window (`resyncAckTimeout`, default 60 s): an acknowledged resync increments `scadabridge.store_and_forward.resync.completed`; an unacknowledged one (lost chunks / dead peer) logs a Warning and increments `scadabridge.store_and_forward.resync.ack_missing` — closing N2's silent-loss mode (previously nothing counted a lost snapshot and nothing retried until the next peer-track). Only the active node answers; only a standby applies, and both sides re-check at apply time (a mid-flight active-flip aborts the wipe) — each side checks the repo-standard **oldest-Up member** active-node predicate (singleton-host semantics via the shared `ActiveNodeEvaluator`, the **same predicate as the S&F delivery gate**; review 02 round 2, **N1 Critical** — using cluster *leadership* here let a rolling restart of the lower-address node make the delivering node wipe its own live buffer). Because replicated applies are **upserts** (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta; the one accepted exception is the rare **N5** orphan-row race (a replicated `Remove` ordered before the snapshot chunks can re-add the removed row, re-delivered once and self-corrected at the next resync — inherent to no-ack replication). If the buffer exceeds the 10 000-row cap the snapshot is flagged `Truncated` and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers. The legacy monolithic `SfBufferSnapshot` message + standby handler are **retained** for rolling-upgrade compatibility (an old active node's monolithic snapshot is still applied by a new standby).
- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed by bounding `LocalDb:Replication:MaxBatchSize` (set to 16 on the rig, sized against a ~70 KB worst-case `config_json`; see the Phase 2 plan, D6). Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
### Operation Tracking Table (lives in Site Runtime, not here)
infra/docker-compose.yml
+3
View File
@@ -68,6 +68,9 @@ services:
MSSQL_PID: "Developer"
volumes:
- scadabridge-mssql-data:/var/opt/mssql
# NOTE: the official mssql/server image does NOT run
# /docker-entrypoint-initdb.d — these mounts are informational only;
# infra/reseed.sh applies the scripts explicitly via sqlcmd.
- ./mssql/setup.sql:/docker-entrypoint-initdb.d/setup.sql:ro
- ./mssql/machinedata_seed.sql:/docker-entrypoint-initdb.d/machinedata_seed.sql:ro
- ./mssql/setup-env2.sql:/docker-entrypoint-initdb.d/setup-env2.sql:ro
infra/reseed.sh
+9 -6
View File
@@ -82,12 +82,15 @@ if ! $SKIP_TEARDOWN; then
done
echo " MSSQL ready."
echo " Waiting for setup.sql to create ScadaBridgeConfig..."
until docker exec scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
-Q "IF DB_ID('ScadaBridgeConfig') IS NULL THROW 50000, 'not ready', 1;" \
>/dev/null 2>&1; do
sleep 2
# The official mcr.microsoft.com/mssql/server image does NOT implement
# /docker-entrypoint-initdb.d, so the compose-mounted init scripts never run
# on their own — waiting for them here hangs forever on a fresh volume.
# Apply them explicitly instead (all are idempotent).
echo " Applying MSSQL init scripts (the mssql/server image has no initdb hook)..."
for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
echo " $f"
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$SCRIPT_DIR/$f"
done
echo " ScadaBridgeConfig present."
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs
+4 -4
View File
@@ -114,8 +114,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder
// Kind/Status are domain fields carried in DetailsJson — decompose to log them.
var d = AuditRowProjection.Decompose(telemetry.Audit);
_logger.LogWarning(ex,
"CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status})",
d.EventId, d.Kind, d.Status);
"CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status}, sqlite {SqliteError})",
d.EventId, d.Kind, d.Status, SqliteErrorCodes.Describe(ex));
}
}
@@ -192,8 +192,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder
catch (Exception ex)
{
_logger.LogWarning(ex,
"CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status})",
telemetry.Operational.TrackedOperationId, telemetry.Operational.Status);
"CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status}, sqlite {SqliteError})",
telemetry.Operational.TrackedOperationId, telemetry.Operational.Status, SqliteErrorCodes.Describe(ex));
}
}
}
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs
+27 -10
View File
@@ -68,6 +68,15 @@ public class SiteAuditTelemetryActor : ReceiveActor
private readonly IOperationTrackingStore? _trackingStore;
private readonly SiteAuditTelemetryOptions _options;
private readonly ILogger<SiteAuditTelemetryActor> _logger;
// Captured at construction (both are thread-safe immutable handles) because
// ScheduleNext/ScheduleNextCached run from the drains' finally blocks, whose
// ConfigureAwait(false) continuations complete on pool threads with no
// active ActorContext — reading Context/Self there either throws
// NotSupportedException or, worse, silently resolves a STALE cell left in
// the thread-static slot and re-arms the tick at the wrong actor
// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8).
private readonly IScheduler _scheduler;
private readonly IActorRef _self;
private ICancelable? _pendingTick;
private ICancelable? _pendingCachedTick;
// Per-actor lifecycle CTS so an in-flight drain (queue read,
@@ -108,6 +117,8 @@ public class SiteAuditTelemetryActor : ReceiveActor
_options = options.Value;
_logger = logger;
_trackingStore = trackingStore;
_scheduler = Context.System.Scheduler;
_self = Self;
ReceiveAsync<Drain>(_ => OnDrainAsync());
ReceiveAsync<CachedDrain>(_ => OnCachedDrainAsync());
@@ -197,7 +208,9 @@ public class SiteAuditTelemetryActor : ReceiveActor
{
// Catch-all so a SQLite hiccup or mapper bug never crashes the
// actor. The next tick is still scheduled in the finally block.
_logger.LogError(ex, "Unexpected error during audit-log telemetry drain.");
_logger.LogError(ex,
"Unexpected error during audit-log telemetry drain (sqlite {SqliteError}).",
SqliteErrorCodes.Describe(ex));
}
finally
{
@@ -278,8 +291,8 @@ public class SiteAuditTelemetryActor : ReceiveActor
// batch — the audit half is best-effort. Log and skip
// this row; it stays Pending for the next drain.
_logger.LogWarning(ex,
"Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}); skipping.",
auditRow.EventId, auditRow.CorrelationId);
"Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}, sqlite {SqliteError}); skipping.",
auditRow.EventId, auditRow.CorrelationId, SqliteErrorCodes.Describe(ex));
continue;
}
@@ -332,7 +345,9 @@ public class SiteAuditTelemetryActor : ReceiveActor
}
catch (Exception ex)
{
_logger.LogError(ex, "Unexpected error during cached-telemetry drain.");
_logger.LogError(ex,
"Unexpected error during cached-telemetry drain (sqlite {SqliteError}).",
SqliteErrorCodes.Describe(ex));
}
finally
{
@@ -428,24 +443,26 @@ public class SiteAuditTelemetryActor : ReceiveActor
return list;
}
// Must stay off Context/Self: called from off-context continuations — see
// the _scheduler/_self field comment.
private void ScheduleNext(TimeSpan delay)
{
_pendingTick?.Cancel();
_pendingTick = Context.System.Scheduler.ScheduleTellOnceCancelable(
_pendingTick = _scheduler.ScheduleTellOnceCancelable(
delay,
Self,
_self,
Drain.Instance,
Self);
_self);
}
private void ScheduleNextCached(TimeSpan delay)
{
_pendingCachedTick?.Cancel();
_pendingCachedTick = Context.System.Scheduler.ScheduleTellOnceCancelable(
_pendingCachedTick = _scheduler.ScheduleTellOnceCancelable(
delay,
Self,
_self,
CachedDrain.Instance,
Self);
_self);
}
/// <summary>Self-tick message that triggers an audit-only drain cycle.</summary>
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SqliteErrorCodes.cs
+31
View File
@@ -0,0 +1,31 @@
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
/// <summary>
/// Renders the primary/extended SQLite result codes of a
/// <see cref="SqliteException"/> for log messages. The exception's own message
/// carries only the primary code ("SQLite Error 10: 'disk I/O error'"), which
/// is too generic to act on — the 2026-07-20 disk-I/O incident
/// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md) had to be
/// reproduced from scratch to learn the extended code (522 =
/// SQLITE_IOERR_SHORT_READ) that names the failing operation.
/// </summary>
internal static class SqliteErrorCodes
{
/// <summary>
/// "primary/extended" (e.g. "10/522") for a <see cref="SqliteException"/>
/// anywhere in the exception chain; "n/a" for non-SQLite failures.
/// </summary>
public static string Describe(Exception ex)
{
for (Exception? e = ex; e is not null; e = e.InnerException)
{
if (e is SqliteException se)
{
return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}";
}
}
return "n/a";
}
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Health/CentralFailoverControl.razor
+149
View File
@@ -0,0 +1,149 @@
@using Microsoft.AspNetCore.Components.Authorization
@using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared
@using ZB.MOM.WW.ScadaBridge.CentralUI.Services
@using ZB.MOM.WW.ScadaBridge.Security
@inject IManualFailoverService Failover
@inject IDialogService Dialog
@* Admin-only manual failover of the central pair (decision 2026-07-22). Rendered on the
Health dashboard's central-cluster card. The page itself is all-roles, so the gate lives
here rather than on the page's [Authorize] attribute. *@
<AuthorizeView Policy="@AuthorizationPolicies.RequireAdmin">
<span class="d-inline-flex align-items-center gap-2">
<button class="btn btn-outline-warning btn-sm"
disabled="@(!HasPeer || _busy)"
title="@(HasPeer
? (IsSite
? $"Gracefully restart site {SiteId}'s active node; its standby takes over."
: "Gracefully restart the active node; the standby takes over.")
: "No standby available — failing over a lone node would be an outage, not a failover.")"
@onclick="TriggerAsync">
@(_busy ? "Failing over…" : "Trigger failover")
</button>
@if (_message is not null)
{
<small class="@(_failed ? "text-danger" : "text-muted")" role="status">@_message</small>
}
</span>
</AuthorizeView>
@code {
/// <summary>
/// Number of central nodes currently reporting online. Fewer than two means there is no
/// standby to take over, so the control is disabled — the guard is enforced again
/// server-side in the failover service, which is the authoritative check.
/// </summary>
[Parameter]
public int OnlineCentralNodeCount { get; set; }
/// <summary>
/// When set, this control fails over the named SITE pair instead of the central pair.
/// Sites are separate Akka clusters reached over the command/control relay, so the site
/// path cannot disturb the admin's own session — which is why the confirmation text
/// differs. <c>null</c> (the default) means the central pair.
/// </summary>
[Parameter]
public string? SiteId { get; set; }
private bool IsSite => !string.IsNullOrWhiteSpace(SiteId);
[CascadingParameter]
private Task<AuthenticationState>? AuthState { get; set; }
private bool HasPeer => OnlineCentralNodeCount >= 2;
private bool _busy;
private bool _failed;
private string? _message;
private async Task TriggerAsync()
{
// Central: the admin is almost certainly connected THROUGH the node about to restart
// (Traefik routes to the active node), so the dialog must set that expectation or a
// working failover reads as a crash they caused.
// Site: a different cluster entirely — this page is unaffected, and claiming otherwise
// would train operators to ignore the warning that does matter.
var confirmed = IsSite
? await Dialog.ConfirmAsync(
$"Trigger failover for site {SiteId}?",
$"The active node of site {SiteId} will leave its cluster and restart; the site's "
+ "standby takes over and its Deployment Manager singleton hands over gracefully. "
+ "In-flight work on that site node is interrupted, and the site is briefly "
+ "unavailable while the handover completes.",
danger: true)
: await Dialog.ConfirmAsync(
"Trigger central failover?",
"The active central node will leave the cluster and restart; the standby takes over "
+ "and becomes active. Cluster singletons hand over gracefully, but in-flight work on "
+ "the active node is interrupted. This page is served by the active node, so it will "
+ "briefly disconnect and reconnect against the new active node.",
danger: true);
if (!confirmed)
{
return;
}
_busy = true;
_failed = false;
_message = null;
try
{
var actor = await ResolveActorAsync();
if (IsSite)
{
var outcome = await Failover.FailOverSiteAsync(SiteId!, actor);
if (outcome.Accepted)
{
_message = $"Failover triggered — {outcome.TargetAddress} is leaving the site cluster.";
}
else
{
// Carry the site's own words through: a peer-guard refusal and an
// unreachable site are different situations for the operator.
_failed = true;
_message = outcome.ErrorMessage ?? "Refused by the site.";
}
}
else
{
var target = await Failover.FailOverCentralAsync(actor);
if (target is null)
{
// The server-side peer guard refused. Never report a failover that did not happen.
_failed = true;
_message = "Refused: no standby available to take over.";
}
else
{
_message = $"Failover triggered — {target} is leaving the cluster.";
}
}
}
catch (Exception ex)
{
_failed = true;
_message = $"Failover failed: {ex.Message}";
}
finally
{
_busy = false;
}
}
/// <summary>Authenticated user name recorded on the audit row.</summary>
private async Task<string> ResolveActorAsync()
{
if (AuthState is null)
{
return "unknown";
}
var state = await AuthState;
return state.User.Identity?.Name
?? state.User.FindFirst(JwtTokenService.UsernameClaimType)?.Value
?? "unknown";
}
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor
+21 -5
View File
@@ -218,11 +218,18 @@
<small class="text-muted ms-2">offline since @changedAt.ToString("u")</small>
}
</div>
<small class="text-muted">
Last report: <TimestampDisplay Value="@state.LastReportReceivedAt" Format="HH:mm:ss" NullText="awaiting first report" />
| Last heartbeat: <TimestampDisplay Value="@state.LastHeartbeatAt" Format="HH:mm:ss" />
| Seq: @state.LastSequenceNumber
</small>
<div class="d-flex align-items-center gap-3">
@* Admin-only; the control gates itself, and disables when the pair has
no online standby to take over. Central acts on the local cluster;
a site is asked over the command/control relay. *@
<CentralFailoverControl OnlineCentralNodeCount="@OnlineNodeCount(state)"
SiteId="@(isCentral ? null : siteId)" />
<small class="text-muted">
Last report: <TimestampDisplay Value="@state.LastReportReceivedAt" Format="HH:mm:ss" NullText="awaiting first report" />
| Last heartbeat: <TimestampDisplay Value="@state.LastHeartbeatAt" Format="HH:mm:ss" />
| Seq: @state.LastSequenceNumber
</small>
</div>
</div>
<div class="card-body p-3">
@if (state.LatestReport != null)
@@ -442,6 +449,15 @@
private string StaleTimeoutDisplay =>
FormatDuration(HealthOptions.Value.MetricsStaleTimeout);
// Online central nodes, from the same ClusterNodes list the Nodes column renders.
// Fewer than two means no standby, which disables the manual-failover control. This is
// a display-side guard only — AkkaManualFailoverService re-checks against live cluster
// membership, which is the authoritative answer.
private static int OnlineNodeCount(SiteHealthState state) =>
state.LatestReport?.ClusterNodes is { Count: > 0 } nodes
? nodes.Count(n => n.IsOnline)
: (state.IsOnline ? 1 : 0);
private static string FormatDuration(TimeSpan span) =>
span.TotalMinutes >= 1 && span == TimeSpan.FromMinutes(Math.Round(span.TotalMinutes))
? $"{span.TotalMinutes:0} minute{(span.TotalMinutes == 1 ? "" : "s")}"
src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs
+48
View File
@@ -0,0 +1,48 @@
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
/// <summary>
/// Admin-triggered manual failover of the central pair. Declared here — and in terms of
/// plain strings — so CentralUI stays Akka-free; the Akka implementation lives in the Host
/// (<c>AkkaManualFailoverService</c>) and is registered only in the Central branch.
/// </summary>
public interface IManualFailoverService
{
/// <summary>
/// Gracefully fails over the central cluster: the current active (oldest Up) member
/// leaves, restarts via its supervisor, and rejoins as standby. The Leave is graceful,
/// never a Down, so cluster singletons hand over instead of being killed.
/// <para>
/// The caller is usually connected THROUGH the node being failed over (Traefik routes
/// the UI to the active node), so the calling Blazor circuit should expect to drop and
/// reconnect against the new active node.
/// </para>
/// </summary>
/// <param name="actor">Authenticated user name, recorded on the audit row.</param>
/// <returns>The address acted on, or <c>null</c> when there is no peer to fail over to
/// (failing over a lone node would be an outage, not a failover).</returns>
Task<string?> FailOverCentralAsync(string actor);
/// <summary>
/// Asks a SITE to gracefully fail over its own two-node pair. Central and each site are
/// separate Akka clusters, so this is a request relayed over the command/control channel —
/// the site performs the Leave itself and reports the outcome. Unlike central failover,
/// this does NOT disturb the caller's own UI session.
/// </summary>
/// <param name="siteId">The site whose pair should fail over.</param>
/// <param name="actor">Authenticated user name, recorded on the audit row.</param>
/// <returns>The site's outcome — accepted with a target, or refused with a reason.</returns>
Task<SiteFailoverOutcome> FailOverSiteAsync(string siteId, string actor);
}
/// <summary>
/// Outcome of a site-pair failover request, in UI terms. Deliberately distinct from the wire
/// ack so CentralUI stays free of the Akka message contract.
/// </summary>
/// <param name="Accepted"><c>true</c> when the site issued the graceful Leave.</param>
/// <param name="TargetAddress">Address of the leaving site node; <c>null</c> when refused.</param>
/// <param name="ErrorMessage">
/// Why the request was refused or failed. A refusal (peer guard, misroute) is a definitive
/// answer FROM the site; an unreachable site surfaces here as a timeout message. Both are
/// reported to the operator rather than being flattened into a generic failure.
/// </param>
public sealed record SiteFailoverOutcome(bool Accepted, string? TargetAddress, string? ErrorMessage);
src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs
+40 -10
View File
@@ -31,19 +31,46 @@ public class ClusterOptions
// when the binding sites can be updated in the same commit.
/// <summary>
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists
/// itself and its partner — so either can start first and form the cluster.
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
/// partner.
/// <para>
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): every node must list ITSELF first.</b>
/// Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW
/// cluster when no peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is
/// this node's own address; any other node runs <c>JoinSeedNodeProcess</c> and retries
/// <c>InitJoin</c> forever. So merely listing both nodes does NOT mean either can start
/// first: a node that lists its partner first can never cold-start while that partner is
/// down (the "registered outage gap", <c>docker/README.md</c>). Self-first ordering closes
/// it using Akka's own protocol, which — unlike an external self-form timer — is part of
/// the join handshake and so cannot mistake an in-flight join for an absent peer.
/// Enforced at boot by <c>StartupValidator</c>.
/// </para>
/// Must contain at least one entry.
/// </summary>
public List<string> SeedNodes { get; set; } = new();
/// <summary>
/// Split-brain resolver strategy. Must be <c>keep-oldest</c> for the two-node
/// clusters ScadaBridge uses: quorum strategies (<c>keep-majority</c>,
/// <c>static-quorum</c>) cannot distinguish a crash from a partition with only
/// two nodes and would shut down the whole cluster.
/// Downing strategy for unreachable members. Two supported values:
/// <list type="bullet">
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
/// of EITHER node (oldest included) fails over to the survivor. The accepted trade:
/// a true network partition produces two live one-node clusters (dual-active) until
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
/// than a rare partition.</item>
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
/// side has ≥2 members (verified against Akka.NET 1.5.62 <c>KeepOldest.Decide</c>
/// and live on the docker rig, 2026-07-21).</item>
/// </list>
/// Other SBR strategies are rejected: <c>static-quorum</c> with quorum 1 hits Akka's
/// <c>IsTooManyMembers</c> guard (2 &gt; 2*1-1) and downs ALL on any unreachability;
/// <c>keep-majority</c> just moves the fatal crash from the oldest to the
/// lowest-address node.
/// </summary>
public string SplitBrainResolverStrategy { get; set; } = "keep-oldest";
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
/// <summary>
/// Time the cluster membership must remain stable before the split-brain
@@ -71,9 +98,12 @@ public class ClusterOptions
public int MinNrOfMembers { get; set; } = 1;
/// <summary>
/// The keep-oldest resolver's <c>down-if-alone</c> flag. When <c>true</c> (the
/// design-doc requirement), the oldest node downs itself if it finds it has no
/// other reachable members, rather than running as an isolated single-node cluster.
/// The keep-oldest resolver's <c>down-if-alone</c> flag; only consulted when
/// <see cref="SplitBrainResolverStrategy"/> is <c>keep-oldest</c>. When <c>true</c>,
/// the oldest node downs itself if it finds it has no other reachable members,
/// rather than running as an isolated single-node cluster. Note that in a two-node
/// cluster this does NOT let the younger survivor take over from a crashed oldest —
/// Akka's alone-check requires the surviving side to have ≥2 members.
/// </summary>
public bool DownIfAlone { get; set; } = true;
src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs
+18 -4
View File
@@ -12,9 +12,18 @@ namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
/// </summary>
public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOptions>
{
/// <summary>Split-brain resolver strategies safe for ScadaBridge's two-node clusters.</summary>
/// <summary>
/// Downing strategies supported for ScadaBridge's two-node clusters.
/// <c>auto-down</c> (default) survives a crash of either node at the accepted cost
/// of dual-active during a real partition; <c>keep-oldest</c> is partition-safe but
/// cannot survive a crash of the oldest node. Quorum strategies are rejected:
/// <c>static-quorum</c> quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll
/// on any unreachability in a 2-node cluster) and <c>keep-majority</c> keys the
/// fatal crash to the lowest-address node instead of the oldest.
/// </summary>
private static readonly HashSet<string> AllowedStrategies = new(StringComparer.OrdinalIgnoreCase)
{
"auto-down",
"keep-oldest"
};
@@ -37,8 +46,9 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
builder.RequireThat(
!string.IsNullOrWhiteSpace(options.SplitBrainResolverStrategy)
&& AllowedStrategies.Contains(options.SplitBrainResolverStrategy),
$"ClusterOptions.SplitBrainResolverStrategy must be 'keep-oldest' for a two-node cluster; " +
$"'{options.SplitBrainResolverStrategy}' would risk a total cluster shutdown on a partition.");
$"ClusterOptions.SplitBrainResolverStrategy must be 'auto-down' or 'keep-oldest' for a " +
$"two-node cluster; '{options.SplitBrainResolverStrategy}' would risk a total cluster " +
"shutdown on a partition or an unreachability event.");
builder.RequireThat(options.MinNrOfMembers == 1,
$"ClusterOptions.MinNrOfMembers must be 1 (was {options.MinNrOfMembers}); " +
@@ -58,7 +68,11 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
$"FailureDetectionThreshold ({options.FailureDetectionThreshold}); otherwise nodes are " +
"declared unreachable before a heartbeat can arrive.");
builder.RequireThat(options.DownIfAlone,
// DownIfAlone is a keep-oldest knob; under auto-down each side downs the
// unreachable peer regardless, so the flag is inert and any value is fine.
var isKeepOldest = string.Equals(
options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
builder.RequireThat(!isKeepOldest || options.DownIfAlone,
"ClusterOptions.DownIfAlone must be true for the keep-oldest resolver "
+ "(Component-ClusterInfrastructure.md → Split-Brain Resolution); with it false the "
+ "oldest node can run as an isolated single-node cluster during a partition while the "
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/SiteFailoverRelayMessages.cs
+53
View File
@@ -0,0 +1,53 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
/// <summary>
/// Central → site relay command: gracefully fail over the owning site's two-node cluster.
/// Sent over the command/control channel when an Administrator clicks "Trigger failover" on a
/// site card in the Central UI Health dashboard.
/// </summary>
/// <remarks>
/// <para>
/// Central and each site are SEPARATE Akka clusters, so central cannot act on a site's
/// membership directly — it can only ask. The site's own <c>SiteCommunicationActor</c> performs
/// the <c>Cluster.Leave</c> against its site-specific <c>site-{SiteId}</c> role, which is the
/// role its singletons (the Deployment Manager) are scoped to.
/// </para>
/// <para>
/// Either site node may receive this: <c>SiteCommunicationActor</c> is registered per node
/// (not as a singleton), and ClusterClient contact rotation reaches whichever answers. That is
/// fine — <c>Cluster.Leave(address)</c> is valid from any member, and the target is resolved
/// from cluster state rather than from who received the message.
/// </para>
/// <para>
/// <b>Rolling upgrade:</b> a site running a binary older than this contract has no handler for
/// this message; it becomes an unhandled message / dead letter and central's Ask times out.
/// The timeout is reported to the operator as "site did not respond", which is the correct
/// user-facing outcome — an old site genuinely cannot honour the request. Message evolution
/// stays additive-only.
/// </para>
/// </remarks>
/// <param name="CorrelationId">Correlation id echoed on the ack.</param>
/// <param name="SiteId">The site whose pair should fail over. Carried explicitly so a
/// misrouted command is detectable at the site rather than silently acted on.</param>
public sealed record TriggerSiteFailover(
string CorrelationId,
string SiteId);
/// <summary>
/// Site → central ack for a <see cref="TriggerSiteFailover"/> relay command.
/// </summary>
/// <param name="CorrelationId">Correlation id of the originating relay command.</param>
/// <param name="Accepted">
/// <c>true</c> when the site resolved a target and issued the graceful Leave.
/// <c>false</c> is a definitive refusal from the site — most often the peer guard (fewer than
/// two Up members in the site role, so a failover would be an outage) or a site-id mismatch.
/// A <c>false</c> ack is NOT a transport failure and must be distinguished from an
/// unreachable-site timeout.
/// </param>
/// <param name="TargetAddress">Address of the node that is leaving; <c>null</c> when refused.</param>
/// <param name="ErrorMessage">Reason for a refusal, or a fault message; <c>null</c> on success.</param>
public sealed record SiteFailoverAck(
string CorrelationId,
bool Accepted,
string? TargetAddress,
string? ErrorMessage);
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditChannel.cs
+12 -2
View File
@@ -3,7 +3,8 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
/// <summary>
/// Top-level Audit Log channel — the trust boundary the audited action crosses.
/// One of: outbound API call, outbound DB write, notification send/deliver, inbound API request,
/// or a two-person ("secured") write through its submit/approve/reject/execute lifecycle.
/// a two-person ("secured") write through its submit/approve/reject/execute lifecycle, or an
/// operator-initiated cluster topology action.
/// </summary>
public enum AuditChannel
{
@@ -11,5 +12,14 @@ public enum AuditChannel
DbOutbound,
Notification,
ApiInbound,
SecuredWrite
SecuredWrite,
/// <summary>
/// An operator-initiated change to cluster topology — currently only the admin-triggered
/// manual failover of the central pair. Distinct from the script trust boundary the other
/// channels describe: nothing here crosses into user script, but a human deliberately
/// restarted the active node, which is exactly the kind of act an audit log exists to
/// attribute. (decision 2026-07-22)
/// </summary>
Cluster
}
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs
+10 -1
View File
@@ -40,5 +40,14 @@ public enum AuditKind
/// <c>EventId</c>, source site, and final error) so the loss is queryable in
/// the Audit Log itself, not only in a rotating log file.
/// </summary>
ReconciliationAbandoned
ReconciliationAbandoned,
/// <summary>
/// An administrator triggered a manual failover of the central pair from the Health page:
/// the active (oldest Up) node was asked to leave the cluster gracefully so its singletons
/// hand over and the standby takes over. One row per invocation, written BEFORE the Leave
/// is issued (the acting node is usually not the one that goes away, but the audit must
/// survive either outcome). (decision 2026-07-22)
/// </summary>
ManualFailover
}
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs
+75 -1
View File
@@ -37,6 +37,7 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
/// do not need a real cluster.
/// </summary>
private readonly Func<bool> _isActiveCheck;
private readonly Func<string, string?> _failOverRole;
/// <summary>
/// Reference to the local Deployment Manager singleton proxy.
@@ -76,12 +77,14 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
string siteId,
CommunicationOptions options,
IActorRef deploymentManagerProxy,
Func<bool>? isActiveCheck = null)
Func<bool>? isActiveCheck = null,
Func<string, string?>? failOverRole = null)
{
_siteId = siteId;
_options = options;
_deploymentManagerProxy = deploymentManagerProxy;
_isActiveCheck = isActiveCheck ?? DefaultIsActiveCheck;
_failOverRole = failOverRole ?? DefaultFailOverRole;
// Registration
Receive<RegisterCentralClient>(msg =>
@@ -263,6 +266,14 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
}
});
// Central→site manual failover relay. Central and the site are separate clusters,
// so central can only ask — this node performs the graceful Leave locally, scoped to
// the SITE-SPECIFIC role, because that is what site singletons (the Deployment
// Manager) are placed on. Either node may receive this (the actor is per-node, not a
// singleton, and contact rotation picks whichever answers); the target is resolved
// from cluster state, not from who received the message.
Receive<TriggerSiteFailover>(HandleTriggerSiteFailover);
// Notification Outbox: forward a buffered notification submitted by the site
// Store-and-Forward Engine to the central cluster. The original Sender (the
// S&F forwarder's Ask) is forwarded as the ClusterClient.Send sender so the
@@ -517,6 +528,69 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
private bool DefaultIsActiveCheck() =>
ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(Cluster.Get(Context.System));
/// <summary>
/// Handles a central-initiated site failover. Refuses a command addressed to a different
/// site (a misroute must never silently fail over a site the operator did not select) and
/// refuses when the site pair has no peer to take over. The ack is sent BEFORE the Leave
/// takes effect on the wire, so it still reaches central even when this node is the one
/// leaving.
/// </summary>
private void HandleTriggerSiteFailover(TriggerSiteFailover msg)
{
if (!string.Equals(msg.SiteId, _siteId, StringComparison.Ordinal))
{
_log.Warning(
"Refusing TriggerSiteFailover addressed to site {Requested}; this node serves {Actual}",
msg.SiteId, _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: $"Command addressed to site '{msg.SiteId}' but this node serves '{_siteId}'."));
return;
}
var role = $"site-{_siteId}";
try
{
var target = _failOverRole(role);
if (target is null)
{
_log.Warning(
"Refusing TriggerSiteFailover for {SiteId}: fewer than 2 Up members in role {Role}, "
+ "so there is no standby to take over", _siteId, role);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null,
ErrorMessage: "No standby available — failing over a lone node would be an outage."));
return;
}
_log.Warning(
"Manual failover requested by central for site {SiteId}: {Target} is leaving the "
+ "site cluster gracefully; its singletons hand over to the standby.", _siteId, target);
Sender.Tell(new SiteFailoverAck(msg.CorrelationId, Accepted: true, target, ErrorMessage: null));
}
catch (Exception ex)
{
// A fault here must be reported to the operator, not thrown into supervision —
// restarting the communication actor would drop central's Ask into a timeout and
// lose the reason.
_log.Error(ex, "TriggerSiteFailover for {SiteId} faulted", _siteId);
Sender.Tell(new SiteFailoverAck(
msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message));
}
}
/// <summary>
/// Production failover action: gracefully Leave the oldest Up member carrying
/// <paramref name="role"/>, via the shared <see cref="ClusterState.ClusterFailoverCoordinator"/>
/// so the central and site paths cannot drift. Injected in tests for the same reason
/// <see cref="DefaultIsActiveCheck"/> is — a real Leave needs Akka.Cluster in the
/// ActorSystem, which the TestKit system does not load.
/// </summary>
/// <param name="role">Site-specific role scope.</param>
/// <returns>Address of the leaving node, or null when there is no peer.</returns>
private string? DefaultFailOverRole(string role) =>
ClusterState.ClusterFailoverCoordinator.FailOverOldest(Context.System, role)?.ToString();
// ── Internal messages ──
internal record SendHeartbeat;
src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs
+14 -5
View File
@@ -10,11 +10,20 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
/// once the original first node restarts and rejoins; every product-level active/standby
/// decision must use this evaluator, never <c>cluster.State.Leader</c>.
/// <para>
/// Lives in Communication (not Host) so BOTH <c>SiteCommunicationActor</c> and
/// <c>SiteReplicationActor</c> can default to it — Host cannot be referenced from either.
/// The Host's <c>ClusterActivityEvaluator.SelfIsOldest</c> delegates here, so the S&amp;F
/// delivery gate (<c>IClusterNodeProvider.SelfIsPrimary</c>), the resync authority checks,
/// and the heartbeat IsActive stamp all share one implementation.
/// Lives in Communication (not Host) so <c>SiteCommunicationActor</c> can default to it —
/// Host cannot be referenced from there. The Host's
/// <c>ClusterActivityEvaluator.SelfIsOldest</c> delegates here, so the S&amp;F delivery gate
/// (<c>IClusterNodeProvider.SelfIsPrimary</c>) and the heartbeat IsActive stamp share one
/// implementation.
/// <para>
/// It also backed <c>SiteReplicationActor</c>'s resync authority checks until LocalDb
/// Phase 2 deleted that actor. Those checks existed because the bespoke resync applied a
/// destructive delete-all-then-insert-all, so running it in the wrong direction wiped a
/// live store-and-forward buffer. LocalDb's snapshot resync merges per row under
/// last-writer-wins and never deletes, so there is no destructive apply left to gate — the
/// evaluator survives for the delivery gate and the heartbeat, which still genuinely need
/// a single active node.
/// </para>
/// </para>
/// </summary>
public static class ActiveNodeEvaluator
src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ClusterFailoverCoordinator.cs
+56
View File
@@ -0,0 +1,56 @@
using Akka.Actor;
using Akka.Cluster;
namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
/// <summary>
/// Deliberate, operator-initiated failover of a two-node cluster: the current active member
/// leaves GRACEFULLY so <c>ClusterSingletonManager</c> hands its singletons to the survivor.
///
/// <para>Lives beside <see cref="ActiveNodeEvaluator"/> — the two share one definition of
/// "active" (oldest Up member in a role scope), and they must not drift: the node asked to
/// leave has to be exactly the node hosting the singletons. Placed in Communication rather
/// than Host because BOTH sides need it — the central pair fails over via the Host's
/// <c>AkkaManualFailoverService</c>, and a site pair fails over inside
/// <c>SiteCommunicationActor</c>, which cannot reference Host.</para>
///
/// <para><b>Leave, never Down.</b> A Down skips singleton hand-off and hands the outcome to
/// the downing strategy — the wrong tool for a planned role swap.</para>
/// </summary>
public static class ClusterFailoverCoordinator
{
/// <summary>
/// Asks the oldest Up member carrying <paramref name="role"/> to leave the cluster.
/// Mirrors <see cref="ActiveNodeEvaluator.SelfIsOldestUp"/>'s rule, so the node acted on is
/// the singleton host — never Akka's cluster <i>leader</i>, whose address-ordered definition
/// diverges from singleton placement once the original first node restarts and rejoins.
/// </summary>
/// <param name="system">Actor system whose cluster is acted on.</param>
/// <param name="role">Role scope. Central uses <c>Central</c>; a site pair uses its
/// site-specific <c>site-{SiteId}</c> role, since site singletons are scoped to that role.</param>
/// <param name="dryRun">Resolve and return the target WITHOUT issuing the Leave. Used to
/// name the target in an audit row before acting, and to answer "would this work?".</param>
/// <returns>The address that leaves (or would leave), or <c>null</c> when fewer than two Up
/// members carry the role — failing over a lone node is an outage, not a failover.</returns>
public static Address? FailOverOldest(ActorSystem system, string role, bool dryRun = false)
{
var cluster = Akka.Cluster.Cluster.Get(system);
var withRole = cluster.State.Members
.Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
.OrderBy(m => m, Member.AgeOrdering)
.ToList();
if (withRole.Count < 2)
{
return null;
}
var oldest = withRole[0];
if (!dryRun)
{
cluster.Leave(oldest.Address);
}
return oldest.Address;
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationService.cs
+28
View File
@@ -809,6 +809,34 @@ public class CommunicationService
return await GetSiteCallAudit().Ask<DiscardSiteCallResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Asks a site to gracefully fail over its own two-node pair (Task 10). Central and each
/// site are SEPARATE Akka clusters, so central cannot act on site membership directly —
/// the site's own <c>SiteCommunicationActor</c> performs the <c>Cluster.Leave</c> against
/// its site-specific role and acks the outcome.
/// <para>
/// A site running a binary older than the <see cref="TriggerSiteFailover"/> contract has no
/// handler for it, so the message dead-letters and this Ask times out. That surfaces to the
/// operator as "site did not respond", which is the honest outcome — an old site genuinely
/// cannot honour the request.
/// </para>
/// </summary>
/// <param name="siteId">Target site.</param>
/// <param name="correlationId">Correlation id echoed on the ack.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The site's ack.</returns>
public async Task<SiteFailoverAck> TriggerSiteFailoverAsync(
string siteId, string correlationId, CancellationToken cancellationToken = default)
{
_logger.LogWarning(
"Relaying TriggerSiteFailover to site {SiteId}, correlationId={CorrelationId}",
siteId, correlationId);
var envelope = new SiteEnvelope(siteId, new TriggerSiteFailover(correlationId, siteId));
return await GetActor().Ask<SiteFailoverAck>(
envelope, _options.QueryTimeout, cancellationToken);
}
}
/// <summary>
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.csproj
+4
View File
@@ -20,6 +20,10 @@
<PackageReference Include="Microsoft.Extensions.Options" />
<PackageReference Include="Microsoft.AspNetCore.DataProtection" />
<PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore" />
<!-- Direct pin of a transitive dependency of Microsoft.AspNetCore.DataProtection:
10.0.7 carries four NU1903 high-severity advisories that break fresh restores
under TreatWarningsAsErrors. See Directory.Packages.props for the rationale. -->
<PackageReference Include="System.Security.Cryptography.Xml" />
</ItemGroup>
<ItemGroup>
src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs
+44 -43
View File
@@ -200,8 +200,9 @@ public class AkkaHostedService : IHostedService
_communicationOptions.TransportHeartbeatInterval.TotalSeconds,
_communicationOptions.TransportFailureThreshold.TotalSeconds);
// Down-if-alone recovery watchdog: SBR's keep-oldest down-if-alone plus
// run-coordinated-shutdown-when-down means a self-downed node terminates
// Downed-node recovery watchdog: any downing decision against this node
// (auto-down by the peer, or an SBR self-down under keep-oldest) plus
// run-coordinated-shutdown-when-down means a downed node terminates
// its own ActorSystem. If that happens outside our StopAsync, the Host
// process must exit so the service supervisor (docker
// `restart: unless-stopped` / Windows service recovery) restarts it and
@@ -228,14 +229,21 @@ public class AkkaHostedService : IHostedService
/// seed-node URI, role or split-brain strategy containing a quote, backslash or
/// whitespace cannot corrupt the document or be silently misparsed.
///
/// The <c>keep-oldest down-if-alone</c> flag is emitted from
/// The downing block branches on <see cref="ClusterOptions.SplitBrainResolverStrategy"/>:
/// <c>auto-down</c> (default; decision 2026-07-21) installs Akka's
/// <c>AutoDowning</c> provider with <c>auto-down-unreachable-after</c> =
/// <see cref="ClusterOptions.StableAfter"/> — the leader among the REACHABLE members
/// downs the unreachable peer, so a crash of either node (oldest included) fails
/// over to the survivor; the accepted trade is dual-active during a real network
/// partition. Any other value takes the SBR path, where the
/// <c>keep-oldest down-if-alone</c> flag is emitted from
/// <see cref="ClusterOptions.DownIfAlone"/> rather than hard-coded, so the bound
/// configuration value is actually consumed.
///
/// The split-brain-resolver <c>downing-provider-class</c> is installed
/// explicitly: Akka defaults to <c>NoDowning</c>, under which the entire
/// split-brain-resolver section is inert and singletons never migrate on a hard
/// crash or partition. Naming the SBR provider is what activates automatic downing.
/// A <c>downing-provider-class</c> is always installed explicitly: Akka defaults
/// to <c>NoDowning</c>, under which the downing configuration is inert and
/// singletons never migrate on a hard crash or partition. Naming the provider is
/// what activates automatic downing.
///
/// Every duration is rendered via <see cref="DurationHocon"/> in
/// milliseconds, so sub-second cluster timing values (e.g. a 750ms heartbeat) are
@@ -258,6 +266,25 @@ public class AkkaHostedService : IHostedService
clusterOptions.SeedNodes.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
// auto-down (default): AutoDowning provider — the leader among the reachable
// members downs the unreachable peer after StableAfter, so a crash of EITHER
// node fails over to the survivor (dual-active during a real partition is the
// accepted trade — decision 2026-07-21). Anything else: the SBR provider with
// the configured active-strategy (keep-oldest), which is partition-safe but
// cannot survive a crash of the oldest node in a two-node cluster.
var downingBlock = string.Equals(
clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
: $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}";
return $@"
audit-telemetry-dispatcher {{
type = ForkJoinDispatcher
@@ -287,14 +314,7 @@ akka {{
seed-nodes = [{seedNodesStr}]
roles = [{rolesStr}]
min-nr-of-members = {clusterOptions.MinNrOfMembers}
downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}
{downingBlock}
failure-detector {{
heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
@@ -766,37 +786,18 @@ akka {{
var deploymentConfigFetcher =
_serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment.IDeploymentConfigFetcher>();
// Create SiteReplicationActor on every node (not a singleton)
var sfStorage = _serviceProvider.GetRequiredService<StoreAndForwardStorage>();
var replicationService = _serviceProvider.GetRequiredService<ReplicationService>();
var replicationLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
.CreateLogger<SiteReplicationActor>();
// ONE active-node predicate instance governs the S&F delivery gate, the resync
// authority checks (SiteReplicationActor), and the heartbeat IsActive stamp
// (SiteCommunicationActor, wired below) — review 02 round 2, N1. Null in
// non-clustered test hosts: the actors fall back to the shared oldest-Up
// evaluator, never to a leader check.
// ONE active-node predicate instance governs the S&F delivery gate and the
// heartbeat IsActive stamp (SiteCommunicationActor, wired below) — review 02
// round 2, N1. It also governed SiteReplicationActor's resync authority until
// LocalDb Phase 2 deleted that actor: the library's snapshot resync merges per row
// under last-writer-wins and never deletes, so there is no destructive apply left
// to need an authority check. Null in non-clustered test hosts: the consumers fall
// back to the shared oldest-Up evaluator, never to a leader check.
var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>();
Func<bool>? activeNodeCheck = clusterNodeProvider != null
? () => clusterNodeProvider.SelfIsPrimary
: null;
var replicationActor = _actorSystem!.ActorOf(
Props.Create(() => new SiteReplicationActor(
storage, sfStorage, replicationService, siteRole, replicationLogger,
deploymentConfigFetcher, activeNodeCheck, siteRuntimeOptionsValue, null)),
"site-replication");
// Wire S&F replication handler to forward operations via the replication actor
replicationService.SetReplicationHandler(op =>
{
replicationActor.Tell(new ReplicateStoreAndForward(op));
return Task.CompletedTask;
});
_logger.LogInformation("SiteReplicationActor created and S&F replication handler wired");
// Deployment Manager — role-scoped singleton via SingletonRegistrar
// (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
// termination and NO PhaseClusterLeave drain, so in-flight SQLite
@@ -807,7 +808,7 @@ akka {{
_actorSystem!, "deployment-manager",
Props.Create(() => new DeploymentManagerActor(
storage, compilationService, sharedScriptLibrary, streamManager,
siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
siteRuntimeOptionsValue, dmLogger, dclManager,
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
_logger, role: siteRole);
var dmProxy = dm.Proxy;
@@ -1053,7 +1054,7 @@ akka {{
// SetReady asserts a deliberately narrow contract. By this point the
// actor system exists, SiteStreamManager.Initialize has run, and every
// role actor (SiteCommunicationActor, deployment-manager singleton,
// SiteReplicationActor, the ClusterClient) has been created with ActorOf —
// the ClusterClient) has been created with ActorOf —
// creation and the registration Tells are synchronous and strictly ordered.
// What is NOT guaranteed is completion of each actor's PreStart or the
// ClusterClient's initial-contact handshake with central: those are
src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs
+170
View File
@@ -0,0 +1,170 @@
using System.Text.Json;
using Akka.Actor;
using Akka.Cluster;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
namespace ZB.MOM.WW.ScadaBridge.Host.Health;
/// <summary>
/// Production <see cref="IManualFailoverService"/> backed by the running Akka.NET cluster.
/// Registered only in the Central-role branch of <c>Program.cs</c>.
///
/// <para><b>Leave, never Down.</b> The target is asked to leave gracefully, so
/// <c>ClusterSingletonManager</c> hands its singletons to the survivor before the member is
/// removed. A <c>Down</c> would skip that hand-off and leave the pair to the downing strategy
/// — the wrong tool for a deliberate, planned role swap.</para>
///
/// <para><b>The target is the oldest Up member, not the leader.</b> That mirrors
/// <c>ActiveNodeEvaluator</c>'s rule, which is where the singletons actually live; Akka's
/// cluster leadership is address-ordered and diverges from singleton placement after a
/// restart (review 01 [High]).</para>
///
/// <para><b>Audit before acting.</b> The row is written before the Leave is issued. The node
/// serving this call is usually NOT the one leaving, but it can be (an admin routed to the
/// active node fails that node over), and an audit written afterwards could be lost to the
/// very shutdown it describes.</para>
/// </summary>
public sealed class AkkaManualFailoverService : IManualFailoverService
{
private readonly AkkaHostedService _akka;
private readonly ICentralAuditWriter _audit;
private readonly CommunicationService _communication;
private readonly ILogger<AkkaManualFailoverService> _logger;
/// <summary>Initializes a new <see cref="AkkaManualFailoverService"/>.</summary>
/// <param name="akka">The Akka hosted service exposing the cluster's actor system.</param>
/// <param name="audit">Central direct-write audit writer.</param>
/// <param name="communication">Central→site command/control transport, used for site failover.</param>
/// <param name="logger">Logger.</param>
public AkkaManualFailoverService(
AkkaHostedService akka,
ICentralAuditWriter audit,
CommunicationService communication,
ILogger<AkkaManualFailoverService> logger)
{
_akka = akka;
_audit = audit;
_communication = communication;
_logger = logger;
}
/// <inheritdoc />
public async Task<string?> FailOverCentralAsync(string actor)
{
var system = _akka.GetOrCreateActorSystem();
// Resolve first so the audit row can name the target, and so the peer guard rejects
// before anything observable happens.
var target = FailOverCore(system, role: CentralRole, dryRun: true);
if (target is null)
{
_logger.LogWarning(
"Manual failover requested by {Actor} but refused: fewer than 2 Up '{Role}' members, "
+ "so there is no standby to take over.", actor, CentralRole);
return null;
}
await WriteAuditAsync(actor, target);
_logger.LogWarning(
"Manual failover triggered by {Actor}: {Target} is leaving the cluster gracefully; "
+ "its singletons hand over to the standby, it restarts via its supervisor and rejoins "
+ "as the youngest member.", actor, target);
FailOverCore(system, role: CentralRole);
return target.ToString();
}
/// <inheritdoc />
public async Task<SiteFailoverOutcome> FailOverSiteAsync(string siteId, string actor)
{
// Audit BEFORE relaying, for the same reason as central: the row must exist even if
// the outcome is never observed. Unlike central, the acting node is never the one
// leaving — but a relay can still time out, and an un-acked request that DID take
// effect at the site would otherwise be unattributed.
await WriteAuditAsync(actor, target: siteId, sourceSiteId: siteId);
_logger.LogWarning(
"Manual site failover triggered by {Actor} for site {SiteId}; relaying to the site cluster.",
actor, siteId);
try
{
var ack = await _communication.TriggerSiteFailoverAsync(siteId, Guid.NewGuid().ToString());
return new SiteFailoverOutcome(ack.Accepted, ack.TargetAddress, ack.ErrorMessage);
}
catch (Exception ex)
{
// Central never buffers for an unreachable site — the Ask simply times out. Report
// that distinctly from a refusal, which is a definitive answer FROM the site.
_logger.LogWarning(ex,
"Manual site failover for {SiteId} did not get an ack from the site.", siteId);
return new SiteFailoverOutcome(
Accepted: false,
TargetAddress: null,
ErrorMessage: $"Site did not respond: {ex.Message}");
}
}
/// <summary>The Akka role scoping central-cluster membership.</summary>
private const string CentralRole = "Central";
/// <summary>
/// Oldest Up member with the role leaves. Delegates to
/// <see cref="ClusterFailoverCoordinator.FailOverOldest"/>, which lives in Communication so
/// the site-pair failover path (inside <c>SiteCommunicationActor</c>, which cannot reference
/// Host) shares one implementation of the rule.
/// </summary>
/// <param name="system">The actor system whose cluster is acted on.</param>
/// <param name="role">Role scope for membership.</param>
/// <param name="dryRun">When true, resolve and return the target without issuing the Leave.</param>
/// <returns>The address that leaves (or would leave), or null when there is no peer.</returns>
public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
=> ClusterFailoverCoordinator.FailOverOldest(system, role, dryRun);
/// <summary>
/// Best-effort audit row. Audit failure must never block the failover the admin asked for
/// — the same rule the rest of the Audit Log follows (audit is best-effort; the
/// user-facing action's own success path is authoritative).
/// </summary>
private Task WriteAuditAsync(string actor, Address target)
=> WriteAuditAsync(actor, target.ToString(), sourceSiteId: null);
/// <inheritdoc cref="WriteAuditAsync(string, Address)"/>
private async Task WriteAuditAsync(string actor, string target, string? sourceSiteId)
{
try
{
var evt = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.Cluster,
kind: AuditKind.ManualFailover,
status: AuditStatus.Submitted,
actor: actor,
target: target,
sourceSiteId: sourceSiteId,
extra: JsonSerializer.Serialize(new
{
target,
// Central rows name the Central role; site rows name the site, so a query
// can tell which pair an operator moved.
scope = sourceSiteId is null ? CentralRole : $"site-{sourceSiteId}"
}));
await _audit.WriteAsync(evt);
}
catch (Exception ex)
{
_logger.LogWarning(
ex,
"Best-effort manual-failover audit emission failed (actor={Actor}, target={Target}); "
+ "the failover itself proceeds.", actor, target);
}
}
}
src/ZB.MOM.WW.ScadaBridge.Host/Program.cs
+6
View File
@@ -329,6 +329,12 @@ try
// which node is active.
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate, ActiveNodeGate>();
// Admin-triggered manual failover of the central pair (Health page control,
// decision 2026-07-22). Central-only: the seam is declared in CentralUI so that
// project stays Akka-free, and only this branch has a cluster to act on.
builder.Services.AddSingleton<
ZB.MOM.WW.ScadaBridge.CentralUI.Services.IManualFailoverService, AkkaManualFailoverService>();
// Cluster node status provider scoped to the Central role — feeds the
// CentralHealthReportLoop so the central cluster appears on /monitoring/health.
builder.Services.AddSingleton<IClusterNodeProvider>(sp =>
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs
+259 -3
View File
@@ -5,8 +5,9 @@ using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// One-time copy of the pre-Phase-1 site databases (<c>site-tracking.db</c> and
/// <c>site_events.db</c>) into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
/// One-time copy of the pre-consolidation site databases — Phase 1's
/// <c>site-tracking.db</c> and <c>site_events.db</c>, and Phase 2's
/// <c>store-and-forward.db</c> — into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
/// </summary>
/// <remarks>
/// <para>
@@ -31,13 +32,20 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// half-migrated state to reason about. A second boot sees the renamed file and no-ops.
/// </para>
/// <para>
/// <b>Finding nothing is the expected case on the docker rig.</b> Neither legacy config key
/// <b>Finding nothing is the expected case on the docker rig for the two Phase 1 files.</b>
/// Neither legacy config key
/// is set in any rig appsettings, so both fall back to CWD-relative code defaults
/// (<c>/app/site-tracking.db</c>, <c>/app/site_events.db</c>) that sit OUTSIDE the mounted
/// data volume — meaning they were already being discarded on every container recreate.
/// Phase 1 incidentally fixes that data-loss bug by consolidating into
/// <c>/app/data/site-localdb.db</c>. A no-op here is a legitimate result, not a failure.
/// </para>
/// <para>
/// <b>Store-and-forward is the exception.</b> Its default path <i>is</i> inside the data
/// volume (<c>./data/store-and-forward.db</c>), so a real deployment has a real file there
/// holding undelivered messages. That migration genuinely moves data, and dropping it would
/// silently discard exactly the buffered calls store-and-forward exists to protect.
/// </para>
/// </remarks>
public static class SiteLocalDbLegacyMigrator
{
@@ -49,6 +57,80 @@ public static class SiteLocalDbLegacyMigrator
/// <summary>Default legacy event-log path (<c>SiteEventLogOptions.DatabasePath</c>).</summary>
private const string DefaultEventLogPath = "site_events.db";
/// <summary>Default legacy store-and-forward path (<c>StoreAndForwardOptions.SqliteDbPath</c>).</summary>
private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db";
/// <summary>Default legacy site configuration path (<c>appsettings.Site.json</c>).</summary>
private const string DefaultSiteStoragePath = "./data/scadabridge.db";
/// <summary>
/// Every column of the current <c>sf_messages</c> schema, in a fixed order. Columns
/// absent from an older legacy file are dropped from the copy rather than failing it —
/// see <see cref="PresentColumns"/>.
/// </summary>
private static readonly string[] StoreAndForwardColumns =
[
"id", "category", "target", "payload_json",
"retry_count", "max_retries", "retry_interval_ms",
"created_at", "last_attempt_at", "status", "last_error", "origin_instance",
"execution_id", "source_script", "parent_execution_id", "last_attempt_at_ms",
];
/// <summary>
/// The site configuration tables copied out of the legacy <c>scadabridge.db</c>, with
/// the current schema's columns for each.
/// </summary>
/// <remarks>
/// <b><c>notification_lists</c> and <c>smtp_configurations</c> are deliberately absent.</b>
/// Both are purged on every deploy and are permanently empty by design — the site-side
/// write paths were removed on 2026-07-10. A pre-fix legacy file can still hold rows, and
/// <c>smtp_configurations.password</c> is plaintext.
/// <para>
/// Skipping them here is one half of a pair: the cutover also declines to register them
/// for replication, for the same reason. Migrating them would leave plaintext SMTP
/// passwords sitting in the consolidated database — one future <c>RegisterReplicated</c>
/// away from being shipped to a peer — in exchange for resurrecting config that nothing
/// reads. Keeping the tables permanently empty is what makes both decisions safe.
/// </para>
/// The tables themselves are still created (see <c>SiteStorageSchema</c>); only their
/// historical contents are left behind.
/// </remarks>
internal static readonly LegacyTable[] SiteStorageTables =
[
new("deployed_configurations", "instance_unique_name",
[
"instance_unique_name", "config_json", "deployment_id", "revision_hash",
"is_enabled", "deployed_at",
]),
new("static_attribute_overrides", "instance_unique_name",
[
"instance_unique_name", "attribute_name", "override_value", "updated_at",
]),
new("shared_scripts", "name",
[
"name", "code", "parameter_definitions", "return_definition", "updated_at",
]),
new("external_systems", "name",
[
"name", "endpoint_url", "auth_type", "auth_configuration", "method_definitions",
"updated_at", "timeout_seconds",
]),
new("database_connections", "name",
[
"name", "connection_string", "max_retries", "retry_delay_ms", "updated_at",
]),
new("data_connection_definitions", "name",
[
"name", "protocol", "configuration", "backup_configuration",
"failover_retry_count", "updated_at",
]),
new("native_alarm_state", "instance_unique_name",
[
"instance_unique_name", "source_canonical_name", "source_reference",
"condition_json", "last_transition_at", "metadata_json",
]),
];
/// <summary>
/// Copies any legacy site databases into <paramref name="db"/>, then renames them.
/// </summary>
@@ -63,6 +145,8 @@ public static class SiteLocalDbLegacyMigrator
MigrateTracking(db, ResolveTrackingPath(config));
MigrateEvents(db, ResolveEventLogPath(config), nodeName);
MigrateStoreAndForward(db, ResolveStoreAndForwardPath(config));
MigrateSiteStorage(db, ResolveSiteStoragePath(config));
}
/// <summary>
@@ -113,6 +197,121 @@ public static class SiteLocalDbLegacyMigrator
return Path.GetFullPath(path);
}
/// <summary>
/// Resolves the legacy store-and-forward database path from the OLD key, falling back
/// to the code default. Unlike the two Phase 1 paths, this default sits INSIDE the
/// mounted data volume (<c>./data/</c>), so on the docker rig there is a real file here
/// with real buffered messages — this migration is not the usual no-op.
/// </summary>
internal static string ResolveStoreAndForwardPath(IConfiguration config)
{
var path = config["ScadaBridge:StoreAndForward:SqliteDbPath"] ?? DefaultStoreAndForwardPath;
if (string.IsNullOrWhiteSpace(path) ||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
{
return string.Empty;
}
return Path.GetFullPath(path);
}
/// <summary>
/// Resolves the legacy site configuration database path from the OLD key, falling back
/// to the code default. Like store-and-forward — and unlike the two Phase 1 paths — this
/// default is inside the mounted data volume, so a real deployment has real config here.
/// </summary>
internal static string ResolveSiteStoragePath(IConfiguration config)
{
var path = config["ScadaBridge:Database:SiteDbPath"] ?? DefaultSiteStoragePath;
if (string.IsNullOrWhiteSpace(path) ||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
{
return string.Empty;
}
return Path.GetFullPath(path);
}
/// <summary>
/// Copies buffered store-and-forward messages out of the legacy file.
/// </summary>
/// <remarks>
/// No id synthesis, unlike <see cref="MigrateEvents"/>: <c>sf_messages.id</c> is already
/// a caller-assigned TEXT primary key, so <c>INSERT OR IGNORE</c> is naturally idempotent
/// across a crash-then-rerun.
/// <para>
/// These are undelivered messages, so dropping them is real data loss — a buffered call
/// that never reaches its external system is exactly what store-and-forward exists to
/// prevent. That is why the copy tolerates an older column set rather than bailing.
/// </para>
/// </remarks>
private static void MigrateStoreAndForward(ILocalDb db, string legacyPath)
=> MigrateFile(db, legacyPath, [new LegacyTable("sf_messages", "id", StoreAndForwardColumns)]);
/// <summary>
/// Copies the site's configuration tables out of the legacy <c>scadabridge.db</c>.
/// </summary>
/// <remarks>
/// All seven migrated tables live in one file, so they are copied inside a single
/// transaction and the file is renamed once: a partial config migration would leave a
/// site node running against half its old configuration, which is worse than failing
/// startup outright.
/// </remarks>
private static void MigrateSiteStorage(ILocalDb db, string legacyPath)
=> MigrateFile(db, legacyPath, SiteStorageTables);
/// <summary>
/// Copies one table from a legacy file into the consolidated database, then renames the
/// legacy file.
/// </summary>
/// <remarks>
/// The copy is restricted to the columns the legacy table <i>actually has</i>. A file
/// written by an older build predates some columns, and naming a missing column in the
/// SELECT would throw "no such column" — which the reader treats as an unrecognised
/// shape, silently discarding every row in the table. Intersecting first means an old
/// file migrates its data and simply leaves the newer columns at their schema defaults.
/// </remarks>
/// <param name="db">The consolidated database.</param>
/// <param name="legacyPath">The legacy file, which may not exist.</param>
/// <param name="tables">Every table to copy out of this file, in order.</param>
private static void MigrateFile(ILocalDb db, string legacyPath, IReadOnlyList<LegacyTable> tables)
{
if (!ShouldMigrate(legacyPath)) return;
using (var legacy = OpenLegacyReadOnly(legacyPath))
{
using var connection = db.CreateConnection();
using var transaction = connection.BeginTransaction();
foreach (var table in tables)
{
var present = PresentColumns(legacy, table.Table, table.Columns);
// An absent table probes as zero columns, so this one guard covers both
// "old file predating the table" and "file we do not recognise".
if (present.Contains(table.RequiredColumn))
CopyRows(legacy, connection, transaction, table.Table, present);
}
transaction.Commit();
}
MarkMigrated(legacyPath);
}
/// <summary>One table to copy out of a legacy file.</summary>
/// <param name="Table">Table name, identical on both sides.</param>
/// <param name="RequiredColumn">
/// A column without which the table is not the one we mean — normally the primary key.
/// Copying rows with a NULL PK would be worse than copying nothing.
/// </param>
/// <param name="Columns">The current schema's full column list, in a fixed order.</param>
internal sealed record LegacyTable(string Table, string RequiredColumn, string[] Columns);
private static void MigrateTracking(ILocalDb db, string legacyPath)
{
if (!ShouldMigrate(legacyPath)) return;
@@ -249,6 +448,63 @@ public static class SiteLocalDbLegacyMigrator
return rows;
}
private static SqliteConnection OpenLegacyReadOnly(string legacyPath)
{
var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly");
connection.Open();
return connection;
}
/// <summary>
/// Returns the subset of <paramref name="wanted"/> that the legacy table actually has,
/// in the caller's order. An absent table yields an empty list rather than throwing.
/// </summary>
private static List<string> PresentColumns(
SqliteConnection legacy, string table, IReadOnlyList<string> wanted)
{
var present = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
using (var probe = legacy.CreateCommand())
{
// Table name is a caller-controlled constant, never user input — safe to
// interpolate (parameters are not permitted as a pragma-function argument).
probe.CommandText = $"SELECT name FROM pragma_table_info('{table}')";
using var reader = probe.ExecuteReader();
while (reader.Read()) present.Add(reader.GetString(0));
}
return [.. wanted.Where(present.Contains)];
}
/// <summary>Streams every row of <paramref name="columns"/> from the legacy table into the target.</summary>
private static void CopyRows(
SqliteConnection legacy,
SqliteConnection target,
SqliteTransaction transaction,
string table,
IReadOnlyList<string> columns)
{
var columnList = string.Join(", ", columns);
var parameterList = string.Join(", ", columns.Select((_, i) => $"$p{i}"));
using var read = legacy.CreateCommand();
read.CommandText = $"SELECT {columnList} FROM {table};";
using var reader = read.ExecuteReader();
while (reader.Read())
{
using var write = target.CreateCommand();
write.Transaction = transaction;
write.CommandText =
$"INSERT OR IGNORE INTO {table} ({columnList}) VALUES ({parameterList});";
for (var i = 0; i < columns.Count; i++)
write.Parameters.AddWithValue($"$p{i}", reader.IsDBNull(i) ? DBNull.Value : reader.GetValue(i));
write.ExecuteNonQuery();
}
}
private static void Bind(SqliteCommand cmd, object?[] row)
{
for (var i = 0; i < row.Length; i++)
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs
+32
View File
@@ -1,7 +1,9 @@
using Microsoft.Extensions.Configuration;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
namespace ZB.MOM.WW.ScadaBridge.Host;
@@ -44,6 +46,11 @@ public static class SiteLocalDbSetup
{
OperationTrackingSchema.Apply(connection);
SiteEventLogSchema.Apply(connection);
// Phase 2: the site's configuration tables and the store-and-forward buffer
// now live in this file too.
SiteStorageSchema.Apply(connection);
StoreAndForwardSchema.Apply(connection);
}
// Both tables qualify: each has an explicit primary key (RegisterReplicated
@@ -52,6 +59,31 @@ public static class SiteLocalDbSetup
db.RegisterReplicated("OperationTracking");
db.RegisterReplicated("site_events");
// Phase 2: the store-and-forward buffer and the seven site configuration tables.
// These replaced the bespoke SiteReplicationActor and StoreAndForward
// ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations
// over Akka; both were deleted in the same commit that added these lines, so the
// two mechanisms never ran at once.
//
// Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by
// ordinal. No Phase 2 table has a BLOB column, which it would reject.
db.RegisterReplicated("sf_messages");
db.RegisterReplicated("deployed_configurations");
db.RegisterReplicated("static_attribute_overrides");
db.RegisterReplicated("shared_scripts");
db.RegisterReplicated("external_systems");
db.RegisterReplicated("database_connections");
db.RegisterReplicated("data_connection_definitions");
db.RegisterReplicated("native_alarm_state");
// notification_lists and smtp_configurations are created but deliberately NOT
// registered. They are permanently empty by design — the site-side write paths were
// removed on 2026-07-10, the legacy migrator skips them, and the active node's
// artifact apply purges them on every deploy. Registering them would open a standing
// replication channel whose only historical payload was plaintext SMTP passwords, in
// exchange for replicating nothing. Anyone adding them here should first establish
// that a site has a legitimate reason to hold SMTP credentials at all.
// AFTER registration, so migrated rows enter the oplog and reach the peer like
// any other write. Before it, they would be invisible to replication forever.
SiteLocalDbLegacyMigrator.Migrate(db, config);
src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs
+15 -4
View File
@@ -65,10 +65,13 @@ public static class SiteServiceRegistration
services.AddSingleton<ISiteIdentityProvider, SiteIdentityProvider>();
services.AddSingleton<IHealthReportTransport, AkkaHealthReportTransport>();
// Site-only components — AddSiteRuntime registers SiteStorageService with SQLite path
// and site-local repository implementations (IExternalSystemRepository, INotificationRepository)
var siteDbPath = config["ScadaBridge:Database:SiteDbPath"] ?? "site.db";
services.AddSiteRuntime($"Data Source={siteDbPath}");
// Site-only components — AddSiteRuntime registers SiteStorageService and the
// site-local repository implementations (IExternalSystemRepository,
// INotificationRepository). It takes no connection string any more:
// SiteStorageService persists to the consolidated LocalDb database registered
// just below (LocalDb:Path). ScadaBridge:Database:SiteDbPath survives only as
// the legacy migrator's source location.
services.AddSiteRuntime();
// Consolidated site database (LocalDb Phase 1). Holds OperationTracking and
// site_events as replicated tables so the pair stops losing them on failover.
@@ -78,6 +81,14 @@ public static class SiteServiceRegistration
// initiator idles.
//
// Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md
//
// The parent directory of LocalDb:Path is created by the library as of 0.1.1.
// ScadaBridge carried a SiteLocalDbDirectory shim for it during Phase 2, because
// SQLite creates the database file on demand but not its directory and
// SqliteLocalDb opens the file eagerly — a missing directory was a hard boot
// failure, and the default site path is the relative "./data/site-localdb.db".
// The shim is gone; SiteLocalDbDirectoryTests still pins the outcome here, since
// what this host needs is the guarantee, not any particular owner of it.
services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config));
// The replication engine, likewise unconditional but INERT by default: with no
src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs
+52 -4
View File
@@ -93,8 +93,24 @@ public static class StartupValidator
.Require("ScadaBridge:Cluster:SeedNodes",
_ => seedNodes != null && seedNodes.Count >= 2,
"must have at least 2 entries")
// Self-first seed ordering (decision 2026-07-22). Akka runs FirstSeedNodeProcess —
// the ONLY bootstrap path that can form a new cluster when no peer answers InitJoin —
// exclusively when seed-nodes[0] is this node's own address. Every other node runs
// JoinSeedNodeProcess and retries InitJoin forever, so a node listing its partner
// first cannot cold-start alone: that is the "registered outage gap"
// (docker/README.md), and it is a silent failure at boot rather than a loud one.
// Enforced here rather than in ClusterOptionsValidator because only this validator
// sees both the node identity and the seed list.
.Require("ScadaBridge:Cluster:SeedNodes",
_ => seedNodes is not { Count: >= 1 }
|| SeedNodeIsSelf(seedNodes[0], nodeSection["NodeHostname"], port),
"must list this node itself first: seed-nodes[0] has to be this node's own "
+ $"'akka.tcp://scadabridge@{nodeSection["NodeHostname"]}:{port}'. Akka only lets "
+ "seed-nodes[0] form a new cluster, so with the partner listed first this node "
+ "can never start while its peer is down (Component-ClusterInfrastructure.md → "
+ "Seed Node Ordering)")
// The big Site-only block: GrpcPort/MetricsPort validity + cross-field
// collisions + SiteDbPath + seed-node-port loop, in the original order.
// collisions + seed-node-port loop, in the original order.
.When(role == "Site", p =>
{
// GrpcPort range, then GrpcPort vs RemotingPort.
@@ -110,9 +126,12 @@ public static class StartupValidator
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort");
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort");
p.Require("ScadaBridge:Database:SiteDbPath",
_ => !string.IsNullOrEmpty(configuration.GetSection("ScadaBridge:Database")["SiteDbPath"]),
"required for Site nodes");
// ScadaBridge:Database:SiteDbPath was required here until LocalDb
// Phase 2. The site's tables now live in the consolidated LocalDb
// database (LocalDb:Path, which SiteServiceRegistration requires),
// and SiteDbPath survives only as the legacy migration source — so
// its absence means "nothing to migrate", not a misconfiguration.
// DatabaseOptionsValidator still rejects a present-but-blank value.
// A seed node must reference an Akka.Remote endpoint, never the
// Kestrel HTTP/2 gRPC port. A seed entry whose port equals this node's
@@ -152,4 +171,33 @@ public static class StartupValidator
return int.TryParse(seedNode[(lastColon + 1)..], out var port) ? port : -1;
}
/// <summary>
/// Extracts the host from an Akka seed-node address of the form
/// <c>akka.tcp://system@host:port</c>. Returns an empty string when no host can be parsed.
/// </summary>
private static string SeedNodeHost(string seedNode)
{
if (string.IsNullOrWhiteSpace(seedNode))
return string.Empty;
var at = seedNode.LastIndexOf('@');
var lastColon = seedNode.LastIndexOf(':');
if (at < 0 || lastColon <= at)
return string.Empty;
return seedNode[(at + 1)..lastColon];
}
/// <summary>
/// True when <paramref name="seedNode"/> addresses this node itself (host AND port).
/// Host comparison is case-insensitive because DNS names are; it is otherwise exact —
/// Akka does no DNS canonicalisation either, so <c>node-a</c> and
/// <c>node-a.example.com</c> are genuinely different seed identities to the cluster.
/// </summary>
private static bool SeedNodeIsSelf(string seedNode, string? nodeHostname, int remotingPort)
{
return SeedNodePort(seedNode) == remotingPort
&& string.Equals(SeedNodeHost(seedNode), nodeHostname, StringComparison.OrdinalIgnoreCase);
}
}
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json
+1 -1
View File
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@localhost:8081",
"akka.tcp://scadabridge@localhost:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json
+10 -3
View File
@@ -16,13 +16,17 @@
"akka.tcp://scadabridge@localhost:8082",
"akka.tcp://scadabridge@localhost:8085"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "./data/scadabridge.db"
},
"DataConnection": {
@@ -31,8 +35,11 @@
"WriteTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "./data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "./data/store-and-forward.db"
},
"Communication": {
"_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs
+21 -2
View File
@@ -257,8 +257,8 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
// (Health Monitoring reads FailedWriteCount) and fault the caller's
// Task instead of silently discarding the exception.
Interlocked.Increment(ref _failedWriteCount);
_logger.LogError(ex, "Failed to record event: {EventType} from {Source}",
pending.EventType, pending.Source);
_logger.LogError(ex, "Failed to record event: {EventType} from {Source} (sqlite {SqliteError})",
pending.EventType, pending.Source, DescribeSqliteError(ex));
pending.Completion.TrySetException(ex);
}
}
@@ -297,6 +297,25 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
}
}
/// <summary>
/// "primary/extended" SQLite result codes (e.g. "10/522") for a
/// <see cref="SqliteException"/> anywhere in the chain; "n/a" otherwise.
/// The exception message alone carries only the primary code, which proved
/// too generic to diagnose the 2026-07-20 disk-I/O incident
/// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md).
/// </summary>
private static string DescribeSqliteError(Exception ex)
{
for (Exception? e = ex; e is not null; e = e.InnerException)
{
if (e is SqliteException se)
{
return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}";
}
}
return "n/a";
}
/// <summary>An event awaiting persistence by the background writer.</summary>
private sealed record PendingEvent(
string Id,
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+8 -16
View File
@@ -52,7 +52,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private readonly ILoggerFactory _loggerFactory;
private readonly IActorRef? _dclManager;
private readonly IActorRef? _replicationActor;
private readonly ISiteHealthCollector? _healthCollector;
private readonly IServiceProvider? _serviceProvider;
/// <summary>
@@ -166,7 +165,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
SiteRuntimeOptions options,
ILogger<DeploymentManagerActor> logger,
IActorRef? dclManager = null,
IActorRef? replicationActor = null,
ISiteHealthCollector? healthCollector = null,
IServiceProvider? serviceProvider = null,
ILoggerFactory? loggerFactory = null,
@@ -181,7 +179,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_streamManager = streamManager;
_options = options;
_dclManager = dclManager;
_replicationActor = replicationActor;
_healthCollector = healthCollector;
_serviceProvider = serviceProvider;
_configFetcher = configFetcher;
@@ -785,15 +782,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
await _storage.ClearStaticOverridesAsync(instanceName);
await _storage.ClearNativeAlarmsForInstanceAsync(instanceName);
// Replicate to standby node — notify-and-fetch: send only the deployment id +
// central fetch coordinates (NOT the config JSON). The standby fetches the
// config over HTTP itself, so a large config never crosses the intra-site Akka
// hop (which would silently drop on the 128 KB frame trap). When the coords are
// absent (deploy paths other than RefreshDeployment), the standby fetch is a
// no-op miss and reconciliation is the durable backstop.
_replicationActor?.Tell(new ReplicateConfigDeploy(
instanceName, command.DeploymentId, command.RevisionHash, true,
command.CentralFetchBaseUrl ?? "", command.FetchToken ?? ""));
// No explicit replication step: deployed_configurations is a replicated table,
// so the write above is captured and shipped to the peer like any other row.
// This used to be a notify-and-fetch Tell — the standby was sent the deployment
// id plus central fetch coordinates and pulled the config over HTTP itself,
// because a large config would silently drop on the intra-site Akka hop's
// 128 KB frame trap. Replication carries the row directly and has no such limit,
// so the standby no longer fetches on deploy. (SiteReconciliationActor still
// fetches at node startup when central reports gaps — a different path.)
return new DeployPersistenceResult(
command.DeploymentId, instanceName, true, null, sender, isRedeploy);
@@ -967,7 +963,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
{
if (t.IsCompletedSuccessfully)
{
_replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, false));
// Operational `deployment` event — disable succeeded.
LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} disabled");
}
@@ -1001,7 +996,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
Task.Run(async () =>
{
await _storage.SetInstanceEnabledAsync(instanceName, true);
_replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, true));
var configs = await _storage.GetAllDeployedConfigsAsync();
var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
return new EnableResult(command, config, null, sender);
@@ -1099,7 +1093,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
{
if (t.IsCompletedSuccessfully)
{
_replicationActor?.Tell(new ReplicateConfigRemove(instanceName));
// Operational `deployment` event — delete succeeded.
LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} deleted");
}
@@ -1949,7 +1942,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// central-only and is never stored on a site (see the purge above).
// Replicate artifacts to standby node
_replicationActor?.Tell(new ReplicateArtifacts(command));
return new ArtifactDeploymentResponse(
command.DeploymentId, "", true, null, DateTimeOffset.UtcNow);
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs
-707
View File
@@ -1,707 +0,0 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Event;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
/// <summary>
/// Runs on every site node (not a singleton). Handles both config and S&amp;F replication
/// between site cluster peers.
///
/// Outbound: receives local replication requests and forwards to peer via ActorSelection.
/// Inbound: receives replicated operations from peer and applies to local SQLite.
/// Uses fire-and-forget (Tell) — no ack wait per design.
/// </summary>
public class SiteReplicationActor : ReceiveActor, IWithTimers
{
private readonly SiteStorageService _storage;
private readonly StoreAndForwardStorage _sfStorage;
private readonly ReplicationService _replicationService;
private readonly IDeploymentConfigFetcher? _configFetcher;
private readonly string _siteRole;
private readonly ILogger<SiteReplicationActor> _logger;
private readonly Cluster _cluster;
private readonly Func<bool> _isActive;
private readonly int _configFetchRetryCount;
private readonly TimeSpan _configFetchRetryDelay;
private Address? _peerAddress;
/// <summary>Akka timer scheduler injected by the framework via <see cref="IWithTimers"/>.</summary>
public ITimerScheduler Timers { get; set; } = null!;
// ── Chunked-resync assembly (standby side; actor-thread only) ──
private string? _assemblingResyncId;
private int _assemblingTotalChunks;
private bool _assemblingTruncated;
private readonly Dictionary<int, List<StoreAndForwardMessage>> _assemblingChunks = new();
private const string ResyncAssemblyTimerKey = "sf-resync-assembly-timeout";
/// <summary>How long a partial chunk assembly may wait for its missing chunks before
/// being discarded (a lost chunk = lost resync; the next peer-track retries). Ctor
/// test seam; production default 30 s.</summary>
private readonly TimeSpan _resyncAssemblyTimeout;
// ── Resync delivery confirmation (active side; actor-thread only) ──
private string? _pendingAckResyncId;
private const string ResyncAckTimerKey = "sf-resync-ack-timeout";
/// <summary>How long the active node waits for the standby's <see cref="SfBufferResyncAck"/>
/// before warning + counting the resync as unacknowledged (lost chunks / dead peer). Ctor
/// test seam; production default 60 s.</summary>
private readonly TimeSpan _resyncAckTimeout;
/// <summary>
/// Maximum rows an active node returns in a single anti-entropy resync snapshot.
/// A standby whose buffer exceeded this (Truncated snapshot) resyncs the oldest
/// 10 000 rows; further divergence drains naturally as the active node delivers.
/// </summary>
private const int MaxResyncRows = 10_000;
/// <summary>
/// Estimated per-chunk payload budget for a resync snapshot. Akka remoting's default
/// <c>maximum-frame-size</c> is 128 000 bytes and <c>BuildHocon</c> sets no override,
/// so the monolithic <see cref="SfBufferSnapshot"/> is silently undeliverable for any
/// realistic backlog (review 02 round 2, N2). 64 000 bytes leaves ≈50% headroom for
/// the JSON envelope, CLR type manifests, and the non-payload columns.
/// </summary>
internal const int MaxResyncChunkBytes = 64_000;
/// <summary>Row cap per resync chunk (bounds a chunk even when every row is tiny).</summary>
internal const int MaxResyncChunkRows = 200;
/// <summary>
/// Splits a resync snapshot into chunks that fit Akka remoting's default
/// 128 000-byte frame (review 02 round 2, N2): rows accumulate until the estimated
/// payload budget or the row cap is hit. Estimation is payload-dominated
/// (payload_json length + 512 bytes fixed overhead per row); a single row whose
/// payload exceeds the budget ships alone (Warning at the call site). Order is
/// preserved (oldest-first, matching GetAllMessagesAsync).
/// </summary>
internal static List<List<StoreAndForwardMessage>> ChunkForRemoting(
IReadOnlyList<StoreAndForwardMessage> rows, int maxChunkBytes, int maxChunkRows)
{
var chunks = new List<List<StoreAndForwardMessage>>();
var current = new List<StoreAndForwardMessage>();
var currentBytes = 0;
foreach (var row in rows)
{
var estimate = (row.PayloadJson?.Length ?? 0) + 512;
if (current.Count > 0 && (currentBytes + estimate > maxChunkBytes || current.Count >= maxChunkRows))
{
chunks.Add(current);
current = new List<StoreAndForwardMessage>();
currentBytes = 0;
}
current.Add(row);
currentBytes += estimate;
}
if (current.Count > 0) chunks.Add(current);
return chunks;
}
/// <summary>
/// Initializes a new <see cref="SiteReplicationActor"/> and registers Akka message handlers.
/// </summary>
/// <param name="storage">Service for accessing local site storage.</param>
/// <param name="sfStorage">Store-and-forward SQLite storage for replication of buffered messages.</param>
/// <param name="replicationService">Service providing replication transport logic.</param>
/// <param name="siteRole">Akka cluster role used to identify peer nodes to replicate to.</param>
/// <param name="logger">Logger instance.</param>
/// <param name="configFetcher">
/// Fetches a deployed instance's config JSON from central over HTTP. Used by the
/// notify-and-fetch standby apply path (<see cref="HandleApplyConfigDeploy"/>): the peer
/// replicates only the deployment id, and the standby fetches the config itself so a large
/// config never crosses the intra-site Akka hop. Null on nodes/tests without a fetcher.
/// </param>
/// <param name="isActiveOverride">
/// Active-node check that gates the buffer-resync roles (a standby requests a
/// resync, the active node answers). Production wiring passes the Host's
/// <c>IClusterNodeProvider.SelfIsPrimary</c> delegate (the same instance gating the
/// S&amp;F delivery sweep); null falls back to the shared oldest-Up evaluator
/// (<see cref="Communication.ClusterState.ActiveNodeEvaluator"/>).
/// </param>
/// <param name="options">Site runtime options, including the config-fetch retry count; production defaults apply when null.</param>
/// <param name="configFetchRetryDelay">Delay between config-fetch retry attempts; defaults to 2 seconds when null.</param>
public SiteReplicationActor(
SiteStorageService storage,
StoreAndForwardStorage sfStorage,
ReplicationService replicationService,
string siteRole,
ILogger<SiteReplicationActor> logger,
IDeploymentConfigFetcher? configFetcher = null,
Func<bool>? isActiveOverride = null,
SiteRuntimeOptions? options = null,
TimeSpan? configFetchRetryDelay = null,
TimeSpan? resyncAssemblyTimeout = null,
TimeSpan? resyncAckTimeout = null)
{
_storage = storage;
_sfStorage = sfStorage;
_replicationService = replicationService;
_configFetcher = configFetcher;
_siteRole = siteRole;
_logger = logger;
_cluster = Cluster.Get(Context.System);
_isActive = isActiveOverride ?? DefaultIsActive;
_resyncAssemblyTimeout = resyncAssemblyTimeout ?? TimeSpan.FromSeconds(30);
_resyncAckTimeout = resyncAckTimeout ?? TimeSpan.FromSeconds(60);
// UA2: bound the standby's replicated-config fetch retries. At least one
// attempt always runs; the fixed inter-attempt delay is a test seam
// (production default 2 s).
_configFetchRetryCount = Math.Max(1, options?.ConfigFetchRetryCount ?? 1);
_configFetchRetryDelay = configFetchRetryDelay ?? TimeSpan.FromSeconds(2);
// Cluster member events
Receive<ClusterEvent.MemberUp>(HandleMemberUp);
Receive<ClusterEvent.MemberRemoved>(HandleMemberRemoved);
Receive<ClusterEvent.CurrentClusterState>(HandleCurrentClusterState);
// Outbound — forward to peer
Receive<ReplicateConfigDeploy>(msg => SendToPeer(new ApplyConfigDeploy(
msg.InstanceName, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled,
msg.CentralFetchBaseUrl, msg.FetchToken)));
Receive<ReplicateConfigRemove>(msg => SendToPeer(new ApplyConfigRemove(msg.InstanceName)));
Receive<ReplicateConfigSetEnabled>(msg => SendToPeer(new ApplyConfigSetEnabled(
msg.InstanceName, msg.IsEnabled)));
Receive<ReplicateArtifacts>(msg => SendToPeer(new ApplyArtifacts(msg.Command)));
Receive<ReplicateStoreAndForward>(msg => SendToPeer(new ApplyStoreAndForward(msg.Operation)));
// Inbound — apply from peer
Receive<ApplyConfigDeploy>(HandleApplyConfigDeploy);
Receive<ApplyConfigRemove>(HandleApplyConfigRemove);
Receive<ApplyConfigSetEnabled>(HandleApplyConfigSetEnabled);
Receive<ApplyArtifacts>(HandleApplyArtifacts);
Receive<ApplyStoreAndForward>(HandleApplyStoreAndForward);
// Anti-entropy — full S&F buffer resync on peer (re)join
Receive<RequestSfBufferResync>(HandleRequestSfBufferResync);
Receive<SfResyncSnapshotLoaded>(HandleSfResyncSnapshotLoaded);
Receive<SfBufferSnapshotChunk>(HandleSfBufferSnapshotChunk);
Receive<ResyncAssemblyTimedOut>(HandleResyncAssemblyTimedOut);
Receive<SfBufferResyncAck>(msg =>
{
if (msg.ResyncId == _pendingAckResyncId)
{
_pendingAckResyncId = null;
Timers.Cancel(ResyncAckTimerKey);
}
ScadaBridgeTelemetry.RecordSfResyncCompleted();
_logger.LogInformation("S&F resync {ResyncId} acknowledged by standby: {Rows} row(s) applied",
msg.ResyncId, msg.RowCount);
});
Receive<ResyncAckTimedOut>(msg =>
{
if (msg.ResyncId != _pendingAckResyncId) return;
_pendingAckResyncId = null;
ScadaBridgeTelemetry.RecordSfResyncAckMissing();
_logger.LogWarning(
"S&F resync {ResyncId} was never acknowledged within {Window} — snapshot chunks may have been lost (frame drop / dead peer); the next peer-track retries",
msg.ResyncId, _resyncAckTimeout);
});
Receive<SfBufferSnapshot>(HandleSfBufferSnapshot); // legacy monolithic handler — retained for rolling compat
}
/// <inheritdoc />
protected override void PreStart()
{
base.PreStart();
_cluster.Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot,
typeof(ClusterEvent.MemberUp),
typeof(ClusterEvent.MemberRemoved));
_logger.LogInformation("SiteReplicationActor started, subscribing to cluster events for role {Role}", _siteRole);
}
/// <inheritdoc />
protected override void PostStop()
{
_cluster.Unsubscribe(Self);
base.PostStop();
}
private void HandleCurrentClusterState(ClusterEvent.CurrentClusterState state)
{
foreach (var member in state.Members)
{
if (member.Status == MemberStatus.Up)
TryTrackPeer(member);
}
}
private void HandleMemberUp(ClusterEvent.MemberUp evt)
{
TryTrackPeer(evt.Member);
}
private void HandleMemberRemoved(ClusterEvent.MemberRemoved evt)
{
if (evt.Member.Address.Equals(_peerAddress))
{
_logger.LogInformation("Peer node removed: {Address}", _peerAddress);
_peerAddress = null;
}
}
private void TryTrackPeer(Member member)
{
// Must have our site role, and must not be self
if (member.HasRole(_siteRole) && !member.Address.Equals(_cluster.SelfAddress))
{
_peerAddress = member.Address;
_logger.LogInformation("Peer node tracked: {Address}", _peerAddress);
OnPeerTracked();
}
}
/// <summary>
/// Side-effect run whenever a peer is (re)tracked. A <b>standby</b> requests a
/// full S&amp;F buffer snapshot for anti-entropy resync — this closes the "a
/// standby down for an hour rejoins and diverges forever" gap: it may have missed
/// replicated Add/Remove/Park ops while it was gone. The active node never
/// requests. <see langword="protected virtual"/> so tests can drive it without a
/// real two-node cluster.
/// </summary>
protected virtual void OnPeerTracked()
{
if (!SafeIsActive())
{
SendToPeer(new RequestSfBufferResync());
}
}
/// <summary>
/// Repo-standard active-node check: this node is active when it is the OLDEST Up
/// member carrying the site role — the same oldest-Up semantics as the S&F delivery
/// gate (IClusterNodeProvider.SelfIsPrimary → ClusterActivityEvaluator → shared
/// ActiveNodeEvaluator). NEVER the cluster leader: leadership is lowest-address and
/// diverges from singleton/delivery placement permanently after the lower-address
/// node restarts — the divergence that made the delivering node wipe its own live
/// buffer via a wrong-direction resync (review 02 round 2, N1 Critical). Any other
/// state reports standby — safe-by-default.
/// </summary>
private bool DefaultIsActive() =>
Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(_cluster, _siteRole);
/// <summary>
/// Evaluates the active-node check, treating a throwing check as standby
/// (safe-by-default: a standby never delivers or answers resyncs).
/// </summary>
private bool SafeIsActive()
{
try
{
return _isActive();
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Active-node check threw; treating node as standby");
return false;
}
}
/// <summary>
/// Forwards a replication message to the tracked peer node's <c>site-replication</c> actor
/// (fire-and-forget, dropped when no peer is tracked). <see langword="protected virtual"/>
/// so tests can intercept the peer send without standing up a real two-node cluster.
/// </summary>
/// <param name="message">The replication message to forward to the peer.</param>
protected virtual void SendToPeer(object message)
{
if (_peerAddress == null)
{
// A dropped op is a lost delta — surface it at Warning + a metric so a
// never-tracked peer (a dead standby) is visible, not silent (arch
// review 02). In single-node dev the peer is legitimately absent; the
// per-op warning is rate-tolerable (accepted per review).
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning("No peer available, dropping replication message {Type}", message.GetType().Name);
return;
}
var path = new RootActorPath(_peerAddress) / "user" / "site-replication";
Context.ActorSelection(path).Tell(message);
}
// ── Inbound handlers ──
private void HandleApplyConfigDeploy(ApplyConfigDeploy msg)
{
if (string.IsNullOrEmpty(msg.CentralFetchBaseUrl))
{
// The direct DeployInstanceCommand cross-cluster wire path was retired.
// This guard is a defensive fallback: skip quietly rather than calling FetchAsync("")
// and logging a spurious error. Reconciliation backstops any missed writes.
_logger.LogDebug(
"No fetch coords for {Instance} (deployment {DeploymentId}) — skipping replicated fetch; T18 reconciliation is the backstop",
msg.InstanceName, msg.DeploymentId);
return;
}
if (_configFetcher is null)
{
_logger.LogWarning(
"No config fetcher available; cannot apply replicated config for {Instance} (deployment {DeploymentId}) — reconciliation will backstop",
msg.InstanceName, msg.DeploymentId);
return;
}
_logger.LogInformation(
"Replicating config for {Instance} (deployment {DeploymentId}) — fetching from central",
msg.InstanceName, msg.DeploymentId);
// Notify-and-fetch: the peer sent only the id, so the standby fetches the config
// itself (off-thread; best-effort fire-and-forget, matching the no-ack replication
// model). The guarded write only overwrites a strictly-older local row. The fetch
// is retried up to ConfigFetchRetryCount times with a fixed delay (UA2) — a transient
// central hiccup no longer defers to the slower reconciliation backstop, which still
// covers a total failure after the last attempt.
_ = FetchWithRetryAsync();
return;
async Task FetchWithRetryAsync()
{
for (var attempt = 1; attempt <= _configFetchRetryCount; attempt++)
{
try
{
// Non-null: the outer method returns early when _configFetcher is null.
var json = await _configFetcher!.FetchAsync(
msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None);
await _storage.StoreDeployedConfigIfNewerAsync(
msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled);
return;
}
catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded)
{
// A superseded/expired fetch never heals by retrying — a newer deploy
// will replicate its own id.
_logger.LogInformation(
"Skip replicated config for {Instance}: superseded/expired (a newer deploy will replicate)",
msg.InstanceName);
return;
}
catch (Exception ex)
{
if (attempt < _configFetchRetryCount)
{
_logger.LogWarning(ex,
"Replicated config fetch attempt {Attempt}/{Max} failed for {Instance} (deployment {DeploymentId}) — retrying in {Delay}",
attempt, _configFetchRetryCount, msg.InstanceName, msg.DeploymentId, _configFetchRetryDelay);
await Task.Delay(_configFetchRetryDelay);
}
else
{
_logger.LogError(ex,
"Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop",
_configFetchRetryCount, msg.InstanceName, msg.DeploymentId);
}
}
}
}
}
private void HandleApplyConfigRemove(ApplyConfigRemove msg)
{
_logger.LogInformation("Applying replicated config remove for {Instance}", msg.InstanceName);
_storage.RemoveDeployedConfigAsync(msg.InstanceName)
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply replicated remove for {Instance}", msg.InstanceName);
});
}
private void HandleApplyConfigSetEnabled(ApplyConfigSetEnabled msg)
{
_logger.LogInformation("Applying replicated set-enabled={Enabled} for {Instance}", msg.IsEnabled, msg.InstanceName);
_storage.SetInstanceEnabledAsync(msg.InstanceName, msg.IsEnabled)
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply replicated set-enabled for {Instance}", msg.InstanceName);
});
}
private void HandleApplyArtifacts(ApplyArtifacts msg)
{
var command = msg.Command;
_logger.LogInformation("Applying replicated artifacts, deploymentId={DeploymentId}", command.DeploymentId);
Task.Run(async () =>
{
try
{
if (command.SharedScripts != null)
foreach (var s in command.SharedScripts)
await _storage.StoreSharedScriptAsync(s.Name, s.Code, s.ParameterDefinitions, s.ReturnDefinition);
if (command.ExternalSystems != null)
foreach (var es in command.ExternalSystems)
await _storage.StoreExternalSystemAsync(es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, es.MethodDefinitionsJson, es.TimeoutSeconds);
if (command.DatabaseConnections != null)
foreach (var db in command.DatabaseConnections)
await _storage.StoreDatabaseConnectionAsync(db.Name, db.ConnectionString, db.MaxRetries, db.RetryDelay);
// Notification lists and SMTP
// configuration are central-only and are never persisted on a site.
// Mirror the primary apply path: purge any pre-fix rows (including the
// plaintext SMTP password) instead of writing the command's
// (now-always-null) NotificationLists/SmtpConfigurations.
await _storage.PurgeCentralOnlyNotificationConfigAsync();
if (command.DataConnections != null)
foreach (var dc in command.DataConnections)
await _storage.StoreDataConnectionDefinitionAsync(dc.Name, dc.Protocol, dc.PrimaryConfigurationJson, dc.BackupConfigurationJson, dc.FailoverRetryCount);
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to apply replicated artifacts");
}
});
}
private void HandleApplyStoreAndForward(ApplyStoreAndForward msg)
{
_logger.LogDebug("Applying replicated S&F operation {OpType} for message {Id}",
msg.Operation.OperationType, msg.Operation.MessageId);
_replicationService.ApplyReplicatedOperationAsync(msg.Operation, _sfStorage)
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply replicated S&F operation {Id}", msg.Operation.MessageId);
});
}
/// <summary>
/// Active-node side of the anti-entropy resync: answers a standby's
/// <see cref="RequestSfBufferResync"/> with a sequence of byte-budgeted
/// <see cref="SfBufferSnapshotChunk"/>s (up to <see cref="MaxResyncRows"/> oldest rows).
/// A non-active node ignores the request — only the authoritative node may answer.
/// The snapshot is piped back to Self so chunking + ack bookkeeping stays actor-safe.
/// </summary>
private void HandleRequestSfBufferResync(RequestSfBufferResync msg)
{
if (!SafeIsActive())
{
_logger.LogDebug("Ignoring S&F buffer resync request — this node is not active");
return;
}
var replyTo = Sender;
_sfStorage.GetAllMessagesAsync(MaxResyncRows).PipeTo(
Self,
failure: ex => new Status.Failure(ex),
success: result => new SfResyncSnapshotLoaded(replyTo, result.Messages, result.Truncated));
}
/// <summary>
/// Active-node continuation: the resync snapshot finished loading; chunk it to fit the
/// remoting frame and send the sequenced chunks to the requester (all sharing one
/// resyncId). Task 7 arms the ack-timeout here.
/// </summary>
private void HandleSfResyncSnapshotLoaded(SfResyncSnapshotLoaded msg)
{
var chunks = ChunkForRemoting(msg.Messages, MaxResyncChunkBytes, MaxResyncChunkRows);
if (chunks.Count == 0) chunks.Add(new List<StoreAndForwardMessage>()); // empty buffer still resyncs (clears the standby)
var resyncId = Guid.NewGuid().ToString("N");
for (var i = 0; i < chunks.Count; i++)
{
if (chunks[i].Count == 1 && (chunks[i][0].PayloadJson?.Length ?? 0) + 512 > MaxResyncChunkBytes)
_logger.LogWarning(
"Resync row {Id} alone exceeds the chunk budget ({Bytes}B payload); sending solo — it may exceed the remoting frame",
chunks[i][0].Id, chunks[i][0].PayloadJson?.Length ?? 0);
msg.ReplyTo.Tell(new SfBufferSnapshotChunk(resyncId, i + 1, chunks.Count, chunks[i], msg.Truncated), Self);
}
_logger.LogInformation(
"Answered S&F resync request with {Rows} row(s) in {Chunks} chunk(s), resyncId={ResyncId}",
msg.Messages.Count, chunks.Count, resyncId);
// Arm the ack window: absence of an SfBufferResyncAck within it surfaces as a
// Warning + counter (the silent-loss mode N2 flagged). Single-outstanding-resync
// bookkeeping: a new request supersedes by overwriting the id and restarting the timer.
_pendingAckResyncId = resyncId;
Timers.StartSingleTimer(ResyncAckTimerKey, new ResyncAckTimedOut(resyncId), _resyncAckTimeout);
}
/// <summary>
/// Standby-node side of the anti-entropy resync: replaces the local buffer
/// wholesale with the active node's snapshot. Combined with the upsert-based
/// replicated applies (arch review 02), any replicated op that lands
/// after this resync merges cleanly onto the resynced state. An active node
/// ignores a snapshot — it is the source of truth, never a resync target.
/// </summary>
private void HandleSfBufferSnapshot(SfBufferSnapshot msg)
{
if (SafeIsActive())
{
_logger.LogDebug("Ignoring S&F buffer snapshot — this node is active");
return;
}
if (msg.Truncated)
{
_logger.LogWarning(
"S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally",
MaxResyncRows);
}
_logger.LogInformation(
"Applying S&F buffer resync snapshot ({Count} rows), replacing local buffer", msg.Messages.Count);
Task.Run(async () =>
{
// Belt-and-braces (N1): re-check at apply time. ReplaceAllAsync discards
// every in-flight row (StoreAndForwardStorage.cs "Never call on an active
// node"); a flip between message receipt and this point must abort.
if (SafeIsActive())
{
_logger.LogWarning(
"Discarding S&F buffer resync snapshot: this node became active before apply");
return;
}
await _sfStorage.ReplaceAllAsync(msg.Messages);
})
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply S&F buffer resync snapshot");
});
}
/// <summary>
/// Standby-node side of the chunked anti-entropy resync: accumulates the chunks of one
/// <c>ResyncId</c>, and once all have arrived, assembles them in sequence order and
/// replaces the local buffer atomically, then acks. A new <c>ResyncId</c> discards any
/// stale partial assembly (review 02 round 2, N2). An active node ignores chunks.
/// </summary>
private void HandleSfBufferSnapshotChunk(SfBufferSnapshotChunk msg)
{
if (SafeIsActive())
{
_logger.LogDebug("Ignoring S&F resync chunk — this node is active");
return;
}
if (_assemblingResyncId != msg.ResyncId)
{
if (_assemblingResyncId != null)
_logger.LogWarning(
"Discarding partial S&F resync assembly {Old} ({Have}/{Want} chunks): a new resync {New} superseded it",
_assemblingResyncId, _assemblingChunks.Count, _assemblingTotalChunks, msg.ResyncId);
_assemblingResyncId = msg.ResyncId;
_assemblingTotalChunks = msg.TotalChunks;
_assemblingTruncated = msg.Truncated;
_assemblingChunks.Clear();
}
_assemblingChunks[msg.Sequence] = msg.Messages;
Timers.StartSingleTimer(ResyncAssemblyTimerKey, new ResyncAssemblyTimedOut(msg.ResyncId), _resyncAssemblyTimeout);
if (_assemblingChunks.Count < _assemblingTotalChunks)
return;
// Complete: assemble in sequence order and apply atomically.
var assembled = Enumerable.Range(1, _assemblingTotalChunks)
.SelectMany(seq => _assemblingChunks[seq])
.ToList();
var resyncId = _assemblingResyncId!;
var truncated = _assemblingTruncated;
_assemblingResyncId = null;
_assemblingChunks.Clear();
Timers.Cancel(ResyncAssemblyTimerKey);
if (truncated)
_logger.LogWarning(
"S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally",
MaxResyncRows);
_logger.LogInformation(
"Applying chunked S&F resync {ResyncId} ({Count} rows), replacing local buffer", resyncId, assembled.Count);
// KNOWN, ACCEPTED race (review 02 round 2, N5 — do NOT "fix" this into something
// worse): a replicated Remove sent after the active node read its snapshot but
// before the snapshot's chunks is ordered BEFORE them on the wire (same
// sender/receiver pair), so this apply can re-add the removed row → an orphan
// Pending row that a later failover re-delivers ONCE. Bounded, self-correcting at
// the next resync, and inherent to no-ack replication; a delivered-side dedup or
// op-sequencing scheme would cost far more than one rare duplicate.
var replyTo = Sender;
Task.Run(async () =>
{
if (SafeIsActive()) // belt-and-braces, mirrors the monolithic path (T3)
{
_logger.LogWarning("Discarding chunked S&F resync {ResyncId}: node became active before apply", resyncId);
return;
}
await _sfStorage.ReplaceAllAsync(assembled);
replyTo.Tell(new SfBufferResyncAck(resyncId, assembled.Count));
})
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply chunked S&F resync {ResyncId}", resyncId);
});
}
private void HandleResyncAssemblyTimedOut(ResyncAssemblyTimedOut msg)
{
if (_assemblingResyncId != msg.ResyncId) return; // superseded already
_logger.LogWarning(
"S&F resync assembly {ResyncId} timed out with {Have}/{Want} chunks — discarding partial (a lost chunk; the next peer-track retries)",
msg.ResyncId, _assemblingChunks.Count, _assemblingTotalChunks);
ScadaBridgeTelemetry.RecordReplicationFailure();
_assemblingResyncId = null;
_assemblingChunks.Clear();
}
/// <summary>Internal: the resync snapshot finished loading; chunk and send to the requester.</summary>
internal sealed record SfResyncSnapshotLoaded(
IActorRef ReplyTo, List<StoreAndForwardMessage> Messages, bool Truncated);
/// <summary>Internal: a partial chunk assembly for <paramref name="ResyncId"/> exceeded its window.</summary>
internal sealed record ResyncAssemblyTimedOut(string ResyncId);
/// <summary>Internal: the active node's ack window for <paramref name="ResyncId"/> expired.</summary>
internal sealed record ResyncAckTimedOut(string ResyncId);
}
/// <summary>
/// Standby→active: request a full S&amp;F buffer snapshot for anti-entropy resync
/// (sent when a standby (re)tracks a peer). Crosses Akka remoting between the two
/// site nodes; the POCO rides the default serializer.
/// </summary>
public sealed record RequestSfBufferResync;
/// <summary>
/// Active→standby: full-buffer snapshot. <paramref name="Truncated"/> is true when
/// the active node's buffer exceeded <c>MaxResyncRows</c> (the standby logs a Warning —
/// divergence beyond the cap drains naturally as the active node delivers). Crosses
/// Akka remoting; the message list rides the default serializer.
/// </summary>
public sealed record SfBufferSnapshot(List<StoreAndForwardMessage> Messages, bool Truncated);
/// <summary>
/// Active→standby: one sequenced chunk of a full-buffer anti-entropy snapshot
/// (review 02 round 2, N2 — the monolithic <see cref="SfBufferSnapshot"/> exceeds Akka
/// remoting's default 128 000-byte frame for any realistic backlog). All chunks of one
/// resync share <paramref name="ResyncId"/>; <paramref name="Sequence"/> is 1-based up to
/// <paramref name="TotalChunks"/>. Additive message — the legacy monolithic snapshot
/// handler is retained for rolling upgrades. Crosses intra-site Akka remoting (NOT
/// ClusterClient — ClusterClientContractLockTests is intentionally not involved).
/// </summary>
public sealed record SfBufferSnapshotChunk(
string ResyncId, int Sequence, int TotalChunks,
List<StoreAndForwardMessage> Messages, bool Truncated);
/// <summary>
/// Standby→active: delivery confirmation — the standby assembled all chunks of
/// <paramref name="ResyncId"/> and applied them atomically (<paramref name="RowCount"/>
/// rows installed). Absence within the ack window is surfaced by the active node
/// (Warning + counter) — the silent-loss mode N2 flagged.
/// </summary>
public sealed record SfBufferResyncAck(string ResyncId, int RowCount);
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs
-44
View File
@@ -1,44 +0,0 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
// Outbound messages — sent by local DeploymentManagerActor/S&F service
// to the local SiteReplicationActor for forwarding to the peer node.
/// <summary>Outbound: tell the peer to fetch+apply a deployed instance config by id (notify-and-fetch; no inline config).</summary>
public record ReplicateConfigDeploy(
string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled,
string CentralFetchBaseUrl, string FetchToken);
/// <summary>Outbound: replicate removal of a deployed instance config to the peer node.</summary>
public record ReplicateConfigRemove(string InstanceName);
/// <summary>Outbound: replicate an instance enabled/disabled flag change to the peer node.</summary>
public record ReplicateConfigSetEnabled(string InstanceName, bool IsEnabled);
/// <summary>Outbound: replicate a system-wide artifact deployment (shared scripts, external systems, etc.) to the peer node.</summary>
public record ReplicateArtifacts(DeployArtifactsCommand Command);
/// <summary>Outbound: replicate a store-and-forward buffer mutation (enqueue/dequeue/park/etc.) to the peer node.</summary>
public record ReplicateStoreAndForward(ReplicationOperation Operation);
// Inbound messages — received from the peer's SiteReplicationActor
// and applied to local SQLite storage.
/// <summary>Inbound: peer-replicated config deploy — the standby fetches the config by id and writes it (guarded).</summary>
public record ApplyConfigDeploy(
string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled,
string CentralFetchBaseUrl, string FetchToken);
/// <summary>Inbound: apply peer-replicated removal of a deployed instance config to local SQLite.</summary>
public record ApplyConfigRemove(string InstanceName);
/// <summary>Inbound: apply a peer-replicated instance enabled/disabled flag change to local SQLite.</summary>
public record ApplyConfigSetEnabled(string InstanceName, bool IsEnabled);
/// <summary>Inbound: apply a peer-replicated system-wide artifact deployment to local SQLite.</summary>
public record ApplyArtifacts(DeployArtifactsCommand Command);
/// <summary>Inbound: apply a peer-replicated store-and-forward buffer mutation to the local buffer.</summary>
public record ApplyStoreAndForward(ReplicationOperation Operation);
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs
+169
View File
@@ -0,0 +1,169 @@
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
/// <summary>
/// DDL for the site's nine configuration tables, extracted from
/// <see cref="SiteStorageService"/> so it can be applied by whoever owns the database
/// file.
/// <para>
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb library.
/// The Host applies this DDL to a LocalDb-managed connection before
/// <c>RegisterReplicated</c> installs the capture triggers; nothing about the schema
/// itself is LocalDb-specific, and the service still calls it so a directly-constructed
/// service (tests, tooling) remains self-sufficient.
/// </para>
/// <para>
/// Journal-mode and other connection pragmas are deliberately NOT set here — LocalDb owns
/// the connection's pragmas, and <see cref="SiteStorageService"/> keeps its own WAL set-up
/// for the databases it opens itself.
/// </para>
/// </summary>
public static class SiteStorageSchema
{
/// <summary>
/// Creates the nine site configuration tables when absent, and additively upgrades
/// tables created by an older build. Idempotent — safe to run on every startup.
/// </summary>
/// <param name="connection">An open connection to the database that should hold the tables.</param>
public static void Apply(SqliteConnection connection)
{
ArgumentNullException.ThrowIfNull(connection);
using (var command = connection.CreateCommand())
{
command.CommandText = @"
CREATE TABLE IF NOT EXISTS deployed_configurations (
instance_unique_name TEXT PRIMARY KEY,
config_json TEXT NOT NULL,
deployment_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
is_enabled INTEGER NOT NULL DEFAULT 1,
deployed_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS static_attribute_overrides (
instance_unique_name TEXT NOT NULL,
attribute_name TEXT NOT NULL,
override_value TEXT NOT NULL,
updated_at TEXT NOT NULL,
PRIMARY KEY (instance_unique_name, attribute_name)
);
CREATE TABLE IF NOT EXISTS shared_scripts (
name TEXT PRIMARY KEY,
code TEXT NOT NULL,
parameter_definitions TEXT,
return_definition TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS external_systems (
name TEXT PRIMARY KEY,
endpoint_url TEXT NOT NULL,
auth_type TEXT NOT NULL,
auth_configuration TEXT,
method_definitions TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS database_connections (
name TEXT PRIMARY KEY,
connection_string TEXT NOT NULL,
max_retries INTEGER NOT NULL DEFAULT 3,
retry_delay_ms INTEGER NOT NULL DEFAULT 1000,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS notification_lists (
name TEXT PRIMARY KEY,
recipient_emails TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS data_connection_definitions (
name TEXT PRIMARY KEY,
protocol TEXT NOT NULL,
configuration TEXT,
backup_configuration TEXT,
failover_retry_count INTEGER NOT NULL DEFAULT 3,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS smtp_configurations (
name TEXT PRIMARY KEY,
server TEXT NOT NULL,
port INTEGER NOT NULL,
auth_mode TEXT NOT NULL,
from_address TEXT NOT NULL,
username TEXT,
password TEXT,
oauth_config TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS native_alarm_state (
instance_unique_name TEXT NOT NULL,
source_canonical_name TEXT NOT NULL,
source_reference TEXT NOT NULL,
condition_json TEXT NOT NULL,
last_transition_at TEXT NOT NULL,
metadata_json TEXT,
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
);
";
command.ExecuteNonQuery();
}
// Schema migrations — add columns that may not exist on older databases
MigrateSchema(connection);
}
private static void MigrateSchema(SqliteConnection connection)
{
// Add backup_configuration and failover_retry_count to data_connection_definitions
// (added in primary/backup data connections feature)
AddColumnIfMissing(connection, "data_connection_definitions", "backup_configuration", "TEXT");
AddColumnIfMissing(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3");
// Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition
// renders fully (type/category/message/values) before the first source snapshot arrives.
AddColumnIfMissing(connection, "native_alarm_state", "metadata_json", "TEXT");
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
// artifact pipeline so site-side calls honor it; 0 = use the site default.
AddColumnIfMissing(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
}
/// <summary>
/// Additively adds a column only when it is not already present. SQLite lacks
/// <c>ADD COLUMN IF NOT EXISTS</c>, so the schema is probed via
/// <c>PRAGMA table_info</c> first. Mirrors <c>OperationTrackingSchema</c>.
/// <para>
/// This replaces the previous try/catch on <c>SqliteException</c> message text
/// containing "duplicate column": probing is the same precedent the other schema
/// classes use, does not depend on an error string that is not part of SQLite's
/// contract, and does not swallow unrelated failures of the same exception type.
/// </para>
/// </summary>
private static void AddColumnIfMissing(
SqliteConnection connection, string table, string columnName, string columnDefinition)
{
using (var probe = connection.CreateCommand())
{
// Table + column names are caller-controlled constants, never user input —
// safe to interpolate (parameters are not permitted in DDL or in a
// pragma-function argument).
probe.CommandText = $"SELECT COUNT(*) FROM pragma_table_info('{table}') WHERE name = $name";
probe.Parameters.AddWithValue("$name", columnName);
if (Convert.ToInt32(probe.ExecuteScalar()) > 0)
{
return;
}
}
using var alter = connection.CreateCommand();
alter.CommandText = $"ALTER TABLE {table} ADD COLUMN {columnName} {columnDefinition}";
alter.ExecuteNonQuery();
}
}
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs
+78 -205
View File
@@ -1,5 +1,6 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
@@ -10,189 +11,68 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
/// </summary>
public class SiteStorageService
{
private readonly string _connectionString;
private readonly ILocalDb _localDb;
private readonly ILogger<SiteStorageService> _logger;
/// <summary>
/// Initializes a new instance of the SiteStorageService with the specified SQLite connection string and logger.
/// Initializes the service over the consolidated site database.
/// </summary>
/// <param name="connectionString">SQLite connection string for the site database.</param>
/// <param name="localDb">
/// The consolidated site database. Every connection it hands out is already open and carries
/// the per-connection pragmas (including the busy timeout that this class used to pin itself)
/// plus the <c>zb_hlc_next()</c> UDF that the config tables' capture triggers call — which is
/// exactly why the service no longer builds its own connection string. A raw connection would
/// lack the UDF and every write to a replicated table would fail closed.
/// </param>
/// <param name="logger">Logger instance for diagnostic messages.</param>
public SiteStorageService(string connectionString, ILogger<SiteStorageService> logger)
public SiteStorageService(ILocalDb localDb, ILogger<SiteStorageService> logger)
{
// Normalize the connection string and pin a busy-timeout floor (S8). WAL lets a reader
// and a writer proceed concurrently, but two writers still contend; Microsoft.Data.Sqlite
// drives its busy handler off the command timeout, so a floor of 5 s means a briefly
// busy database is waited-on rather than failing fast with SQLITE_BUSY. Only raise a
// caller-supplied value that is lower than the floor (the library default of 30 s stays).
var builder = new SqliteConnectionStringBuilder(connectionString);
if (builder.DefaultTimeout < BusyTimeoutFloorSeconds)
builder.DefaultTimeout = BusyTimeoutFloorSeconds;
_connectionString = builder.ToString();
ArgumentNullException.ThrowIfNull(localDb);
ArgumentNullException.ThrowIfNull(logger);
_localDb = localDb;
_logger = logger;
}
/// <summary>Busy-timeout floor in seconds applied to the site connection string (S8).</summary>
private const int BusyTimeoutFloorSeconds = 5;
/// <summary>
/// Returns an <b>already-open</b> connection against the site database.
/// Exposed so site-local repositories can get their own connections without reaching
/// into private state via reflection. The caller owns the connection and is
/// responsible for disposing it.
/// </summary>
/// <remarks>
/// <b>Contract change:</b> this used to return an <i>unopened</i> connection that the
/// caller opened. LocalDb hands out connections already open, pragma-configured, and
/// carrying the <c>zb_hlc_next()</c> UDF — calling <c>Open</c>/<c>OpenAsync</c> on one
/// throws. Callers must not open it.
/// </remarks>
/// <returns>An open <see cref="SqliteConnection"/> against the site database.</returns>
public SqliteConnection CreateConnection() => _localDb.CreateConnection();
/// <summary>
/// Creates a new (unopened) SQLite connection against the site database.
/// Exposed so site-local repositories can open their own connections without
/// reaching into private state via reflection. The caller owns
/// the connection and is responsible for opening and disposing it.
/// <para>
/// The database runs in WAL journal mode (set once in <see cref="InitializeAsync"/>) with a
/// busy-timeout floor (see the constructor), so connections handed out here inherit
/// concurrent-reader/writer behavior and wait out a briefly-busy database instead of
/// failing with SQLITE_BUSY.
/// </para>
/// Convenience alias used internally, mirroring the other LocalDb-backed stores.
/// </summary>
/// <returns>A new, unopened <see cref="SqliteConnection"/> against the site database.</returns>
public SqliteConnection CreateConnection() => new(_connectionString);
private SqliteConnection OpenConnection() => _localDb.CreateConnection();
/// <summary>
/// Creates the SQLite tables if they do not exist.
/// Called once on site startup.
/// </summary>
/// <returns>A task that completes when all tables have been created or verified.</returns>
public async Task InitializeAsync()
public Task InitializeAsync()
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
// No journal-mode pragma and no busy-timeout normalization here any more:
// LocalDb owns the file and sets WAL plus the per-connection pragmas on every
// connection it hands out.
using var connection = OpenConnection();
// Switch to WAL journal mode (S8). WAL is persistent (database-level, survives across
// connections) so this one-time set is enough. It lets readers and a writer proceed
// concurrently instead of every access serializing on the rollback journal. A
// :memory: database or a filesystem that cannot support WAL falls back to its prior
// mode — we log the result rather than throwing.
await using (var walCommand = connection.CreateCommand())
{
walCommand.CommandText = "PRAGMA journal_mode=WAL;";
var resultingMode = (await walCommand.ExecuteScalarAsync()) as string ?? "unknown";
if (!string.Equals(resultingMode, "wal", StringComparison.OrdinalIgnoreCase))
_logger.LogWarning(
"Site SQLite could not enable WAL journal mode (got '{Mode}') — concurrent access may serialize",
resultingMode);
}
// The DDL itself lives in SiteStorageSchema so the Host can apply it to a
// LocalDb-managed connection before RegisterReplicated installs the capture
// triggers. The service still calls it, so a directly-constructed service
// (tests, tooling) remains self-sufficient.
SiteStorageSchema.Apply(connection);
await using var command = connection.CreateCommand();
command.CommandText = @"
CREATE TABLE IF NOT EXISTS deployed_configurations (
instance_unique_name TEXT PRIMARY KEY,
config_json TEXT NOT NULL,
deployment_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
is_enabled INTEGER NOT NULL DEFAULT 1,
deployed_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS static_attribute_overrides (
instance_unique_name TEXT NOT NULL,
attribute_name TEXT NOT NULL,
override_value TEXT NOT NULL,
updated_at TEXT NOT NULL,
PRIMARY KEY (instance_unique_name, attribute_name)
);
CREATE TABLE IF NOT EXISTS shared_scripts (
name TEXT PRIMARY KEY,
code TEXT NOT NULL,
parameter_definitions TEXT,
return_definition TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS external_systems (
name TEXT PRIMARY KEY,
endpoint_url TEXT NOT NULL,
auth_type TEXT NOT NULL,
auth_configuration TEXT,
method_definitions TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS database_connections (
name TEXT PRIMARY KEY,
connection_string TEXT NOT NULL,
max_retries INTEGER NOT NULL DEFAULT 3,
retry_delay_ms INTEGER NOT NULL DEFAULT 1000,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS notification_lists (
name TEXT PRIMARY KEY,
recipient_emails TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS data_connection_definitions (
name TEXT PRIMARY KEY,
protocol TEXT NOT NULL,
configuration TEXT,
backup_configuration TEXT,
failover_retry_count INTEGER NOT NULL DEFAULT 3,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS smtp_configurations (
name TEXT PRIMARY KEY,
server TEXT NOT NULL,
port INTEGER NOT NULL,
auth_mode TEXT NOT NULL,
from_address TEXT NOT NULL,
username TEXT,
password TEXT,
oauth_config TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS native_alarm_state (
instance_unique_name TEXT NOT NULL,
source_canonical_name TEXT NOT NULL,
source_reference TEXT NOT NULL,
condition_json TEXT NOT NULL,
last_transition_at TEXT NOT NULL,
metadata_json TEXT,
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
);
";
await command.ExecuteNonQueryAsync();
// Schema migrations — add columns that may not exist on older databases
await MigrateSchemaAsync(connection);
_logger.LogInformation("Site SQLite storage initialized at {ConnectionString}", _connectionString);
}
private async Task MigrateSchemaAsync(SqliteConnection connection)
{
// Add backup_configuration and failover_retry_count to data_connection_definitions
// (added in primary/backup data connections feature)
await TryAddColumnAsync(connection, "data_connection_definitions", "backup_configuration", "TEXT");
await TryAddColumnAsync(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3");
// Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition
// renders fully (type/category/message/values) before the first source snapshot arrives.
await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT");
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
// artifact pipeline so site-side calls honor it; 0 = use the site default.
await TryAddColumnAsync(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
}
private async Task TryAddColumnAsync(SqliteConnection connection, string table, string column, string type)
{
try
{
await using var cmd = connection.CreateCommand();
cmd.CommandText = $"ALTER TABLE {table} ADD COLUMN {column} {type}";
await cmd.ExecuteNonQueryAsync();
_logger.LogInformation("Migrated: added column {Column} to {Table}", column, table);
}
catch (SqliteException ex) when (ex.Message.Contains("duplicate column"))
{
// Column already exists — no action needed
}
_logger.LogInformation("Site SQLite storage initialized");
return Task.CompletedTask;
}
// ── Deployed Configuration CRUD ──
@@ -203,8 +83,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of all deployed instance configurations.</returns>
public async Task<List<DeployedInstance>> GetAllDeployedConfigsAsync()
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -245,8 +124,7 @@ public class SiteStorageService
string revisionHash,
bool isEnabled)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -278,9 +156,22 @@ public class SiteStorageService
/// clause so the guard is atomic with no application-level read-modify-write.
/// </summary>
/// <remarks>
/// This is the standby-node write path for replicated configs. The active-node
/// apply path (<see cref="StoreDeployedConfigAsync"/>) remains unguarded and always
/// overwrites, because the active node's write is always authoritative.
/// <b>This is the reconciliation write path.</b> <c>SiteReconciliationActor</c> runs a
/// per-node startup self-heal against central: it asks central what this node should be
/// running and fetches anything missing. That fetch races real deploys, so the
/// <c>deployed_at</c> guard is what stops a slow reconcile response from overwriting a
/// newer config that landed while it was in flight. The active-node apply path
/// (<see cref="StoreDeployedConfigAsync"/>) remains unguarded and always overwrites,
/// because a deploy is always authoritative.
/// <para>
/// It was originally the <i>standby</i> write path as well, under notify-and-fetch: the
/// standby was told a deploy had happened and fetched the config itself. LocalDb Phase 2
/// replaced that with change-data-capture — the config row simply replicates — so the
/// standby no longer writes here at all, and last-writer-wins on the primary key (not
/// this guard) is what orders concurrent writes between the two nodes. Reconciliation is
/// the reason the method survives; do not port the <c>deployed_at</c> guard onto the
/// replication path, where it would fight the HLC rather than help it.
/// </para>
/// <para>
/// <paramref name="deployedAtOverride"/> is exposed for testing so that the exact
/// <c>deployed_at</c> value can be controlled without sleeping between calls.
@@ -308,8 +199,7 @@ public class SiteStorageService
{
var deployedAt = (deployedAtOverride ?? DateTimeOffset.UtcNow).ToString("O");
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -342,8 +232,7 @@ public class SiteStorageService
/// <returns>A task that completes when the configuration and its overrides have been removed.</returns>
public async Task RemoveDeployedConfigAsync(string instanceName)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var transaction = await connection.BeginTransactionAsync();
@@ -383,8 +272,7 @@ public class SiteStorageService
/// <returns>A task that completes when the enabled flag has been updated.</returns>
public async Task SetInstanceEnabledAsync(string instanceName, bool isEnabled)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -411,8 +299,7 @@ public class SiteStorageService
/// <returns>A task that resolves to a dictionary mapping attribute names to their override values.</returns>
public async Task<Dictionary<string, string>> GetStaticOverridesAsync(string instanceName)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -440,8 +327,7 @@ public class SiteStorageService
/// <returns>A task that completes when the override has been saved.</returns>
public async Task SetStaticOverrideAsync(string instanceName, string attributeName, string value)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -467,8 +353,7 @@ public class SiteStorageService
/// <returns>A task that completes when all overrides for the instance have been deleted.</returns>
public async Task ClearStaticOverridesAsync(string instanceName)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = "DELETE FROM static_attribute_overrides WHERE instance_unique_name = @name";
@@ -495,8 +380,7 @@ public class SiteStorageService
string instanceName, string sourceCanonicalName, string sourceReference,
string conditionJson, DateTimeOffset lastTransitionAt, string? metadataJson = null)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -535,8 +419,7 @@ public class SiteStorageService
if (rows.Count == 0)
return;
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
await using var command = connection.CreateCommand();
@@ -580,8 +463,7 @@ public class SiteStorageService
/// <returns>A task that completes when the alarm condition row has been deleted.</returns>
public async Task DeleteNativeAlarmAsync(string instanceName, string sourceCanonicalName, string sourceReference)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -605,8 +487,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of stored native alarm condition rows for the binding.</returns>
public async Task<IReadOnlyList<NativeAlarmRow>> GetNativeAlarmsAsync(string instanceName, string sourceCanonicalName)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -637,8 +518,7 @@ public class SiteStorageService
/// <returns>A task that completes when all native alarm rows for the instance have been deleted.</returns>
public async Task ClearNativeAlarmsForInstanceAsync(string instanceName)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = "DELETE FROM native_alarm_state WHERE instance_unique_name = @name";
@@ -660,8 +540,7 @@ public class SiteStorageService
/// <returns>A task that completes when the shared script has been stored or updated.</returns>
public async Task StoreSharedScriptAsync(string name, string code, string? parameterDefs, string? returnDef)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -689,8 +568,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of all stored shared scripts.</returns>
public async Task<List<StoredSharedScript>> GetAllSharedScriptsAsync()
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = "SELECT name, code, parameter_definitions, return_definition FROM shared_scripts";
@@ -727,8 +605,7 @@ public class SiteStorageService
string name, string endpointUrl, string authType, string? authConfig, string? methodDefs,
int timeoutSeconds = 0)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -766,8 +643,7 @@ public class SiteStorageService
public async Task StoreDatabaseConnectionAsync(
string name, string connectionString, int maxRetries, TimeSpan retryDelay)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -810,8 +686,7 @@ public class SiteStorageService
/// <returns>A task that completes when both tables have been emptied.</returns>
public async Task PurgeCentralOnlyNotificationConfigAsync()
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -834,8 +709,7 @@ public class SiteStorageService
public async Task StoreDataConnectionDefinitionAsync(
string name, string protocol, string? configJson, string? backupConfigJson = null, int failoverRetryCount = 3)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -865,8 +739,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of all stored data connection definitions.</returns>
public async Task<List<StoredDataConnectionDefinition>> GetAllDataConnectionDefinitionsAsync()
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = "SELECT name, protocol, configuration, backup_configuration, failover_retry_count FROM data_connection_definitions";
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs
+10 -5
View File
@@ -31,8 +31,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<IReadOnlyList<ExternalSystemDefinition>> GetAllExternalSystemsAsync(
CancellationToken cancellationToken = default)
{
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -63,8 +64,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<ExternalSystemDefinition?> GetExternalSystemByNameAsync(
string name, CancellationToken cancellationToken = default)
{
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -91,8 +93,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
if (system is null)
return Array.Empty<ExternalSystemMethod>();
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -140,8 +143,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<IReadOnlyList<DatabaseConnectionDefinition>> GetAllDatabaseConnectionsAsync(
CancellationToken cancellationToken = default)
{
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand();
command.CommandText = @"
@@ -178,8 +182,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<DatabaseConnectionDefinition?> GetDatabaseConnectionByNameAsync(
string name, CancellationToken cancellationToken = default)
{
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand();
command.CommandText = @"
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs
+9 -21
View File
@@ -1,6 +1,7 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
@@ -14,32 +15,19 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime;
public static class ServiceCollectionExtensions
{
/// <summary>
/// Registers Site Runtime services including SiteStorageService for SQLite persistence.
/// The caller must register an <see cref="ISiteStorageConnectionProvider"/> or call the
/// overload with an explicit connection string.
/// Registers Site Runtime services including SiteStorageService, which persists to the
/// consolidated site database resolved from <c>ILocalDb</c> (<c>LocalDb:Path</c>).
/// </summary>
/// <param name="services">The DI service collection to register services into.</param>
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
public static IServiceCollection AddSiteRuntime(this IServiceCollection services)
{
// SiteStorageService is registered by the Host using AddSiteRuntime(connectionString)
// This overload is for backward compatibility / skeleton placeholder
return services;
}
/// <summary>
/// Registers Site Runtime services with an explicit SQLite connection string.
/// </summary>
/// <param name="services">The DI service collection to register services into.</param>
/// <param name="siteDbConnectionString">The SQLite connection string for the site local storage database.</param>
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
public static IServiceCollection AddSiteRuntime(this IServiceCollection services, string siteDbConnectionString)
{
services.AddSingleton(sp =>
{
var logger = sp.GetRequiredService<ILogger<SiteStorageService>>();
return new SiteStorageService(siteDbConnectionString, logger);
});
// SiteStorageService takes ILocalDb (the consolidated site database at
// LocalDb:Path) rather than a connection string, so there is nothing left for a
// caller to supply — the string overload is gone and this is the only entry point.
services.AddSingleton(sp => new SiteStorageService(
sp.GetRequiredService<ILocalDb>(),
sp.GetRequiredService<ILogger<SiteStorageService>>()));
services.AddHostedService<SiteStorageInitializer>();
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs
-7
View File
@@ -60,13 +60,6 @@ public class SiteRuntimeOptions
/// <summary>HTTP timeout (seconds) for fetching a deployment config from central (notify-and-fetch).</summary>
public int ConfigFetchTimeoutSeconds { get; set; } = 30;
/// <summary>
/// Bounded attempt count (including the first) for the standby's replicated-config
/// fetch; a 2 s fixed delay separates attempts and superseded fetches never retry.
/// Consumed by <c>SiteReplicationActor.HandleApplyConfigDeploy</c> (UA2). Default: 3.
/// </summary>
public int ConfigFetchRetryCount { get; set; } = 3;
/// <summary>
/// Fixed interval (ms) at which an Instance Actor re-sends a tag-subscribe
/// request that either failed or whose response was lost (S4/UA6). The retry is
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs
-4
View File
@@ -53,10 +53,6 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase<SiteRunti
$"ScadaBridge:SiteRuntime:ConfigFetchTimeoutSeconds must be greater than 0 " +
$"(was {options.ConfigFetchTimeoutSeconds}).");
builder.RequireThat(options.ConfigFetchRetryCount >= 0,
$"ScadaBridge:SiteRuntime:ConfigFetchRetryCount must be >= 0 " +
$"(was {options.ConfigFetchRetryCount}).");
builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
$"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
$"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs
-194
View File
@@ -1,194 +0,0 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary>
/// Async replication of buffer operations to standby node.
///
/// - Forwards add/remove/park operations to standby via a replication handler.
/// - No ack wait (fire-and-forget per design).
/// - Standby applies operations to its own SQLite.
/// - On failover, standby resumes delivery from its replicated state.
/// </summary>
public class ReplicationService
{
private readonly StoreAndForwardOptions _options;
private readonly ILogger<ReplicationService> _logger;
private Func<ReplicationOperation, Task>? _replicationHandler;
/// <summary>Initializes a new instance of <see cref="ReplicationService"/>.</summary>
/// <param name="options">Store-and-forward configuration options.</param>
/// <param name="logger">Logger instance.</param>
public ReplicationService(
StoreAndForwardOptions options,
ILogger<ReplicationService> logger)
{
_options = options;
_logger = logger;
}
/// <summary>
/// Sets the handler for forwarding replication operations to the standby node.
/// Typically wraps Akka Tell to the standby's replication actor.
/// </summary>
/// <param name="handler">The async delegate that forwards each replication operation to the standby.</param>
public void SetReplicationHandler(Func<ReplicationOperation, Task> handler)
{
_replicationHandler = handler;
}
/// <summary>
/// Replicates an enqueue operation to standby (fire-and-forget).
/// </summary>
/// <param name="message">The message that was enqueued on the active node.</param>
public void ReplicateEnqueue(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Add,
message.Id,
message));
}
/// <summary>
/// Replicates a remove operation to standby (fire-and-forget).
/// </summary>
/// <param name="messageId">The identifier of the message to remove from the standby buffer.</param>
public void ReplicateRemove(string messageId)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Remove,
messageId,
null));
}
/// <summary>
/// Replicates a park operation to standby (fire-and-forget).
/// </summary>
/// <param name="message">The message that was parked on the active node.</param>
public void ReplicatePark(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Park,
message.Id,
message));
}
/// <summary>
/// Replicates an operator-initiated requeue (a parked
/// message moved back to the pending queue) to standby (fire-and-forget). The
/// carried message reflects the active node's post-requeue state (Pending,
/// retry_count = 0) so the standby's copy can be brought into sync.
/// </summary>
/// <param name="message">The message in its post-requeue (Pending, retry_count=0) state.</param>
public void ReplicateRequeue(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Requeue,
message.Id,
message));
}
/// <summary>
/// Applies a replicated operation received from the active node.
/// Used by the standby node to keep its SQLite in sync.
///
/// Add/Park/Requeue are applied as <b>upserts</b> (<see cref="StoreAndForwardStorage.UpsertMessageAsync"/>),
/// not blind INSERT/UPDATE: the full message rides in every one of those operations,
/// so a Park/Requeue whose original Add was lost (fire-and-forget replication is
/// best-effort) self-heals by materialising the row, and a duplicate Add (e.g.
/// re-issued by an anti-entropy resync) applies newest-wins instead of throwing a
/// primary-key violation. Remove is a plain delete.
/// </summary>
/// <param name="operation">The replication operation to apply.</param>
/// <param name="storage">The standby node's store-and-forward storage to update.</param>
/// <returns>A task representing the asynchronous apply operation.</returns>
public async Task ApplyReplicatedOperationAsync(
ReplicationOperation operation,
StoreAndForwardStorage storage)
{
switch (operation.OperationType)
{
case ReplicationOperationType.Add when operation.Message != null:
await storage.UpsertMessageAsync(operation.Message);
break;
case ReplicationOperationType.Remove:
await storage.RemoveMessageAsync(operation.MessageId);
break;
case ReplicationOperationType.Park when operation.Message != null:
operation.Message.Status = StoreAndForwardMessageStatus.Parked;
await storage.UpsertMessageAsync(operation.Message);
break;
case ReplicationOperationType.Requeue when operation.Message != null:
operation.Message.Status = StoreAndForwardMessageStatus.Pending;
operation.Message.RetryCount = 0;
await storage.UpsertMessageAsync(operation.Message);
break;
}
}
private void FireAndForget(ReplicationOperation operation)
{
// Invoked inline, NOT via Task.Run: the handler is a non-blocking Akka
// Tell, and thread-pool hand-off destroyed Add/Remove ordering for the
// same message id (arch review 02). Inline invocation preserves issue
// order; Akka's per-sender/receiver guarantee preserves it on the wire.
try
{
var task = _replicationHandler!.Invoke(operation);
if (!task.IsCompletedSuccessfully)
{
task.ContinueWith(t =>
{
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning(t.Exception,
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
operation.OperationType, operation.MessageId);
},
TaskContinuationOptions.OnlyOnFaulted);
}
}
catch (Exception ex)
{
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning(ex,
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
operation.OperationType, operation.MessageId);
}
}
}
/// <summary>
/// Represents a buffer operation to be replicated to standby.
/// </summary>
public record ReplicationOperation(
ReplicationOperationType OperationType,
string MessageId,
StoreAndForwardMessage? Message);
/// <summary>
/// Types of buffer operations that are replicated.
/// </summary>
public enum ReplicationOperationType
{
Add,
Remove,
Park,
/// <summary>
/// An operator moved a parked message back to the pending
/// queue. The standby resets its matching row to Pending with retry_count = 0.
/// </summary>
Requeue
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs
+7 -17
View File
@@ -1,6 +1,7 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
@@ -15,21 +16,18 @@ public static class ServiceCollectionExtensions
/// <returns>The same <paramref name="services"/> collection, for chaining.</returns>
public static IServiceCollection AddStoreAndForward(this IServiceCollection services)
{
services.AddSingleton<StoreAndForwardStorage>(sp =>
{
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
var logger = sp.GetRequiredService<ILogger<StoreAndForwardStorage>>();
return new StoreAndForwardStorage(
$"Data Source={options.SqliteDbPath}",
logger);
});
// The buffer now lives in the consolidated LocalDb database at LocalDb:Path, not
// at StoreAndForwardOptions.SqliteDbPath — that option survives only as the
// migrator's source location (Task 8).
services.AddSingleton<StoreAndForwardStorage>(sp => new StoreAndForwardStorage(
sp.GetRequiredService<ILocalDb>(),
sp.GetRequiredService<ILogger<StoreAndForwardStorage>>()));
services.AddSingleton<StoreAndForwardService>(sp =>
{
var storage = sp.GetRequiredService<StoreAndForwardStorage>();
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
var logger = sp.GetRequiredService<ILogger<StoreAndForwardService>>();
var replication = sp.GetRequiredService<ReplicationService>();
// Wire the cached-call lifecycle
// observer + site identity through DI so the S&F retry loop emits
// per-attempt + terminal telemetry under the same TrackedOperationId
@@ -54,19 +52,11 @@ public static class ServiceCollectionExtensions
storage,
options,
logger,
replication,
cachedCallObserver,
siteId,
siteEventLogger);
});
services.AddSingleton<ReplicationService>(sp =>
{
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
var logger = sp.GetRequiredService<ILogger<ReplicationService>>();
return new ReplicationService(options, logger);
});
return services;
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs
+7 -4
View File
@@ -5,12 +5,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// </summary>
public class StoreAndForwardOptions
{
/// <summary>Path to the SQLite database for S&amp;F message persistence.</summary>
/// <summary>
/// Path to the legacy standalone store-and-forward SQLite file. <b>Migration-only.</b>
/// The buffer itself lives in the consolidated LocalDb database (<c>LocalDb:Path</c>)
/// as of LocalDb Phase 2; this path is read once at boot by
/// <c>SiteLocalDbLegacyMigrator</c> to drain a pre-Phase-2 file, and is otherwise
/// unused. A node that has already migrated may leave it set or unset.
/// </summary>
public string SqliteDbPath { get; set; } = "./data/store-and-forward.db";
/// <summary>Whether to replicate buffer operations to standby node.</summary>
public bool ReplicationEnabled { get; set; } = true;
/// <summary>Default retry interval for messages without per-source settings.</summary>
public TimeSpan DefaultRetryInterval { get; set; } = TimeSpan.FromSeconds(30);
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs
+11 -9
View File
@@ -5,21 +5,23 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary>
/// Validates <see cref="StoreAndForwardOptions"/> at startup. The retry intervals
/// feed the background sweep timer (a zero/negative period trips
/// <see cref="ArgumentOutOfRangeException"/> in the timer constructor) and the
/// SQLite path is opened for the S&amp;F buffer; an empty path yields an opaque
/// connection failure at first enqueue. Registered with <c>ValidateOnStart()</c>
/// so a bad <c>ScadaBridge:StoreAndForward</c> section fails fast at boot with a
/// clear, key-naming message.
/// <see cref="ArgumentOutOfRangeException"/> in the timer constructor). Registered
/// with <c>ValidateOnStart()</c> so a bad <c>ScadaBridge:StoreAndForward</c> section
/// fails fast at boot with a clear, key-naming message.
/// <para>
/// <see cref="StoreAndForwardOptions.SqliteDbPath"/> is deliberately NOT validated.
/// Before LocalDb Phase 2 it was the live buffer file, so an empty value produced an
/// opaque connection failure at first enqueue; the buffer now lives in the
/// consolidated LocalDb database and the key survives only as the legacy migration
/// source. An empty value there is a legitimate "nothing to migrate", so requiring
/// it would make every already-migrated node carry a dead key forever.
/// </para>
/// </summary>
public sealed class StoreAndForwardOptionsValidator : OptionsValidatorBase<StoreAndForwardOptions>
{
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, StoreAndForwardOptions options)
{
builder.RequireThat(!string.IsNullOrWhiteSpace(options.SqliteDbPath),
"ScadaBridge:StoreAndForward:SqliteDbPath must be a non-empty path; " +
"it is the SQLite file backing the store-and-forward buffer.");
builder.RequireThat(options.DefaultRetryInterval > TimeSpan.Zero,
$"ScadaBridge:StoreAndForward:DefaultRetryInterval must be a positive duration " +
$"(was {options.DefaultRetryInterval}); it is the default per-message retry interval.");
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs
+124
View File
@@ -0,0 +1,124 @@
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary>
/// DDL for the <c>sf_messages</c> table, extracted from
/// <see cref="StoreAndForwardStorage"/> so it can be applied by whoever owns the
/// database file.
/// <para>
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb library.
/// The Host applies this DDL to a LocalDb-managed connection before
/// <c>RegisterReplicated</c> installs the capture triggers; nothing about the schema
/// itself is LocalDb-specific, and the store still calls it so a directly-constructed
/// store (tests, tooling) remains self-sufficient.
/// </para>
/// </summary>
public static class StoreAndForwardSchema
{
/// <summary>
/// Creates the <c>sf_messages</c> table and its indexes when absent, and additively
/// upgrades a table created by an older build. Idempotent — safe to run on every
/// startup.
/// </summary>
/// <param name="connection">An open connection to the database that should hold the table.</param>
public static void Apply(SqliteConnection connection)
{
ArgumentNullException.ThrowIfNull(connection);
using (var command = connection.CreateCommand())
{
command.CommandText = @"
CREATE TABLE IF NOT EXISTS sf_messages (
id TEXT PRIMARY KEY,
category INTEGER NOT NULL,
target TEXT NOT NULL,
payload_json TEXT NOT NULL,
retry_count INTEGER NOT NULL DEFAULT 0,
max_retries INTEGER NOT NULL DEFAULT 50,
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
created_at TEXT NOT NULL,
last_attempt_at TEXT,
status INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
origin_instance TEXT
);
CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status);
CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category);
";
command.ExecuteNonQuery();
}
// Additively add the execution_id /
// source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add
// columns to a table that already exists from before these fields, so a
// databases created by an older build needs the columns ALTER-ed in.
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
// probed first and the ALTER skipped when already there. Both columns
// are nullable with no default, so any row buffered before this
// migration reads back ExecutionId/SourceScript = null (back-compat).
AddColumnIfMissing(connection, "execution_id", "TEXT");
AddColumnIfMissing(connection, "source_script", "TEXT");
// Additively add the
// parent_execution_id column the same way — a sibling to execution_id.
// Nullable with no default, so any row buffered before this migration
// reads back ParentExecutionId = null (back-compat).
AddColumnIfMissing(connection, "parent_execution_id", "TEXT");
// Additively add the epoch-ms sibling of last_attempt_at. The
// ISO-8601 text column stays authoritative for reads / back-compat; this
// INTEGER column drives the due predicate so the sweep no longer parses
// julianday() per row.
AddColumnIfMissing(connection, "last_attempt_at_ms", "INTEGER");
// One-time backfill for rows persisted before the ms column existed: derive
// epoch-ms from the text timestamp. The "... IS NULL" guard makes this run
// once per legacy DB and never match again — this is the only remaining
// julianday() use in the store.
using (var backfill = connection.CreateCommand())
{
backfill.CommandText = @"
UPDATE sf_messages
SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER)
WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL";
backfill.ExecuteNonQuery();
}
// Covering index for the due query (status filter + ms ordering column).
// Created after the ALTER above so the column exists.
using (var dueIndex = connection.CreateCommand())
{
dueIndex.CommandText =
"CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)";
dueIndex.ExecuteNonQuery();
}
}
/// <summary>
/// Adds a column to <c>sf_messages</c>
/// only when it is not already present. SQLite lacks <c>ADD COLUMN IF NOT
/// EXISTS</c>, so the schema is probed via <c>PRAGMA table_info</c> first.
/// Idempotent — safe to run on every <see cref="Apply"/>.
/// </summary>
private static void AddColumnIfMissing(
SqliteConnection connection, string columnName, string columnType)
{
using (var probe = connection.CreateCommand())
{
probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name";
probe.Parameters.AddWithValue("@name", columnName);
if (Convert.ToInt32(probe.ExecuteScalar()) > 0)
{
return;
}
}
using var alter = connection.CreateCommand();
// Column name + type are caller-controlled constants, never user input —
// safe to interpolate (parameters are not permitted in DDL).
alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}";
alter.ExecuteNonQuery();
}
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs
+9 -23
View File
@@ -36,7 +36,6 @@ public class StoreAndForwardService
{
private readonly StoreAndForwardStorage _storage;
private readonly StoreAndForwardOptions _options;
private readonly ReplicationService? _replication;
private readonly ILogger<StoreAndForwardService> _logger;
/// <summary>
/// Site-side observer notified
@@ -107,7 +106,7 @@ public class StoreAndForwardService
/// <c>null</c> when no sweep is currently running. Captured when the timer
/// callback starts a sweep so <see cref="StopAsync"/> can wait for it to
/// finish before the host disposes downstream dependencies
/// (<see cref="_storage"/>, <see cref="_replication"/>) that the sweep is
/// (<see cref="_storage"/>) that the sweep is
/// still touching. Written from the timer thread and from
/// <see cref="StopAsync"/>, so reads are synchronised via the
/// <see cref="Volatile"/> APIs.
@@ -227,7 +226,6 @@ public class StoreAndForwardService
/// <param name="storage">The storage backend for buffered messages.</param>
/// <param name="options">Configuration options.</param>
/// <param name="logger">Logger instance.</param>
/// <param name="replication">Optional replication service for standby synchronization.</param>
/// <param name="cachedCallObserver">Optional observer for cached call lifecycle events.</param>
/// <param name="siteId">The site identifier this service belongs to.</param>
/// <param name="siteEventLogger">
@@ -240,7 +238,6 @@ public class StoreAndForwardService
StoreAndForwardStorage storage,
StoreAndForwardOptions options,
ILogger<StoreAndForwardService> logger,
ReplicationService? replication = null,
ICachedCallLifecycleObserver? cachedCallObserver = null,
string siteId = "",
ISiteEventLogger? siteEventLogger = null)
@@ -248,7 +245,6 @@ public class StoreAndForwardService
_storage = storage;
_options = options;
_logger = logger;
_replication = replication;
_cachedCallObserver = cachedCallObserver;
_siteId = string.IsNullOrWhiteSpace(siteId) ? UnknownSiteSentinel : siteId;
_siteEventLogger = siteEventLogger;
@@ -651,7 +647,6 @@ public class StoreAndForwardService
private async Task BufferAsync(StoreAndForwardMessage message)
{
await _storage.EnqueueAsync(message);
_replication?.ReplicateEnqueue(message);
Interlocked.Increment(ref _bufferedCount);
}
@@ -802,7 +797,6 @@ public class StoreAndForwardService
if (success)
{
await _storage.RemoveMessageAsync(message.Id);
_replication?.ReplicateRemove(message.Id);
Interlocked.Decrement(ref _bufferedCount);
RaiseActivity("Delivered", message.Category,
$"Delivered to {message.Target} after {message.RetryCount} retries");
@@ -833,7 +827,6 @@ public class StoreAndForwardService
return RetryOutcome.Skipped;
}
Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category,
$"Permanent failure for {message.Target}: handler returned false");
@@ -869,7 +862,6 @@ public class StoreAndForwardService
return RetryOutcome.Skipped;
}
Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category,
$"Max retries ({message.MaxRetries}) reached for {message.Target}");
_logger.LogWarning(
@@ -1077,17 +1069,12 @@ public class StoreAndForwardService
/// <summary>
/// Retries a parked message (moves back to pending queue).
///
/// An operator requeue is a buffer state change and is
/// replicated to the standby (as a <see cref="ReplicationOperationType.Requeue"/>)
/// so a failover preserves the operator's retry intent.
/// An operator requeue is a buffer state change, and the peer picks it up because
/// <c>sf_messages</c> is a replicated table: the row update is captured and shipped
/// like any other write, so a failover preserves the operator's retry intent. This
/// used to be an explicit Requeue operation sent to the standby.
/// The activity-log entry carries the message's true
/// category rather than a hard-coded one.
/// The parked row is captured <i>before</i> the local
/// requeue write rather than re-read after it, so a concurrent
/// <c>RemoveMessageAsync</c> or <c>DiscardParkedMessageAsync</c> running
/// between the two storage calls cannot leave the standby in <c>Parked</c>
/// while the active node has already requeued — we always have the row in
/// hand for the <c>Requeue</c> replication.
/// </summary>
/// <param name="messageId">The identifier of the message to retry.</param>
/// <returns>True if successfully retried, false otherwise.</returns>
@@ -1117,7 +1104,6 @@ public class StoreAndForwardService
captured.RetryCount = 0;
captured.LastError = null;
captured.LastAttemptAt = null;
_replication?.ReplicateRequeue(captured);
RaiseActivity("Retry", captured.Category,
$"Parked message {messageId} moved back to queue");
@@ -1127,9 +1113,10 @@ public class StoreAndForwardService
/// <summary>
/// Permanently discards a parked message.
///
/// An operator discard is a buffer removal and is replicated
/// to the standby (as a <see cref="ReplicationOperationType.Remove"/>) so the
/// discarded message does not reappear after a failover.
/// An operator discard is a buffer removal. The delete is captured as a tombstone on
/// the replicated <c>sf_messages</c> table and carries the later HLC, so the discarded
/// message cannot reappear after a failover regardless of arrival order. This used to
/// be an explicit Remove operation sent to the standby.
/// The activity-log entry carries the message's true
/// category rather than a hard-coded one.
/// </summary>
@@ -1143,7 +1130,6 @@ public class StoreAndForwardService
var success = await _storage.DiscardParkedMessageAsync(messageId);
if (success)
{
_replication?.ReplicateRemove(messageId);
RaiseActivity("Discard", message?.Category ?? StoreAndForwardCategory.ExternalSystem,
$"Parked message {messageId} discarded");
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs
+47 -201
View File
@@ -1,5 +1,6 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
@@ -11,17 +12,25 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// </summary>
public class StoreAndForwardStorage
{
private readonly string _connectionString;
private readonly ILocalDb _localDb;
private readonly ILogger<StoreAndForwardStorage> _logger;
/// <summary>
/// Initializes a new instance of <see cref="StoreAndForwardStorage"/> with the given SQLite connection string.
/// Initializes the store over the consolidated site database and applies the schema.
/// </summary>
/// <param name="connectionString">SQLite connection string for the store-and-forward database.</param>
/// <param name="localDb">
/// The consolidated site database. Every connection it hands out is already open and carries
/// the per-connection pragmas (including <c>busy_timeout</c>) plus the <c>zb_hlc_next()</c>
/// UDF that <c>sf_messages</c>' capture triggers call — which is exactly why the store no
/// longer opens its own <see cref="SqliteConnection"/> from a connection string. A raw
/// connection would lack the UDF and every write to the replicated table would fail closed.
/// </param>
/// <param name="logger">Logger for diagnostics.</param>
public StoreAndForwardStorage(string connectionString, ILogger<StoreAndForwardStorage> logger)
public StoreAndForwardStorage(ILocalDb localDb, ILogger<StoreAndForwardStorage> logger)
{
_connectionString = connectionString;
ArgumentNullException.ThrowIfNull(localDb);
ArgumentNullException.ThrowIfNull(logger);
_localDb = localDb;
_logger = logger;
}
@@ -29,164 +38,34 @@ public class StoreAndForwardStorage
/// Creates the sf_messages table if it does not exist.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task InitializeAsync()
public Task InitializeAsync()
{
EnsureDatabaseDirectoryExists();
// No directory creation and no journal-mode pragma here any more: LocalDb owns
// the file (it creates the directory) and sets WAL plus the per-connection
// pragmas on every connection it hands out.
using var connection = OpenConnection();
await using var connection = await OpenConnectionAsync();
// Enable WAL so the concurrent writers this store has by design (script
// enqueues, the retry sweep's per-target lanes, standby replication applies,
// central pull queries) read/write without "database is locked". WAL is
// persistent + file-scoped; in-memory DBs report "memory" instead — harmless,
// so it is not asserted here.
await using (var walCmd = connection.CreateCommand())
{
walCmd.CommandText = "PRAGMA journal_mode=WAL";
await walCmd.ExecuteNonQueryAsync();
}
await using var command = connection.CreateCommand();
command.CommandText = @"
CREATE TABLE IF NOT EXISTS sf_messages (
id TEXT PRIMARY KEY,
category INTEGER NOT NULL,
target TEXT NOT NULL,
payload_json TEXT NOT NULL,
retry_count INTEGER NOT NULL DEFAULT 0,
max_retries INTEGER NOT NULL DEFAULT 50,
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
created_at TEXT NOT NULL,
last_attempt_at TEXT,
status INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
origin_instance TEXT
);
CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status);
CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category);
";
await command.ExecuteNonQueryAsync();
// Additively add the execution_id /
// source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add
// columns to a table that already exists from before these fields, so a
// databases created by an older build needs the columns ALTER-ed in.
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
// probed first and the ALTER skipped when already there. Both columns
// are nullable with no default, so any row buffered before this
// migration reads back ExecutionId/SourceScript = null (back-compat).
await AddColumnIfMissingAsync(connection, "execution_id", "TEXT");
await AddColumnIfMissingAsync(connection, "source_script", "TEXT");
// Additively add the
// parent_execution_id column the same way — a sibling to execution_id.
// Nullable with no default, so any row buffered before this migration
// reads back ParentExecutionId = null (back-compat).
await AddColumnIfMissingAsync(connection, "parent_execution_id", "TEXT");
// Additively add the epoch-ms sibling of last_attempt_at. The
// ISO-8601 text column stays authoritative for reads / back-compat; this
// INTEGER column drives the due predicate so the sweep no longer parses
// julianday() per row.
await AddColumnIfMissingAsync(connection, "last_attempt_at_ms", "INTEGER");
// One-time backfill for rows persisted before the ms column existed: derive
// epoch-ms from the text timestamp. The "... IS NULL" guard makes this run
// once per legacy DB and never match again — this is the only remaining
// julianday() use in the store.
await using (var backfill = connection.CreateCommand())
{
backfill.CommandText = @"
UPDATE sf_messages
SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER)
WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL";
await backfill.ExecuteNonQueryAsync();
}
// Covering index for the due query (status filter + ms ordering column).
// Created after the ALTER above so the column exists.
await using (var dueIndex = connection.CreateCommand())
{
dueIndex.CommandText =
"CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)";
await dueIndex.ExecuteNonQueryAsync();
}
// The DDL itself lives in StoreAndForwardSchema so the Host can apply it to a
// LocalDb-managed connection before RegisterReplicated installs the capture
// triggers. The store still calls it, so a directly-constructed store (tests,
// tooling) remains self-sufficient.
StoreAndForwardSchema.Apply(connection);
_logger.LogInformation("Store-and-forward SQLite storage initialized");
return Task.CompletedTask;
}
/// <summary>
/// Adds a column to <c>sf_messages</c>
/// only when it is not already present. SQLite lacks <c>ADD COLUMN IF NOT
/// EXISTS</c>, so the schema is probed via <c>PRAGMA table_info</c> first.
/// Idempotent — safe to run on every <see cref="InitializeAsync"/>.
/// CreateConnection returns an ALREADY-OPEN, pragma-configured connection with the
/// <c>zb_hlc_next()</c> UDF registered. Calling <c>OpenAsync</c> on it throws, and a raw
/// <see cref="SqliteConnection"/> would lack the UDF, making every capture trigger fail
/// closed.
/// </summary>
private static async Task AddColumnIfMissingAsync(
SqliteConnection connection, string columnName, string columnType)
{
await using var probe = connection.CreateCommand();
probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name";
probe.Parameters.AddWithValue("@name", columnName);
var exists = Convert.ToInt32(await probe.ExecuteScalarAsync()) > 0;
if (exists)
{
return;
}
await using var alter = connection.CreateCommand();
// Column name + type are caller-controlled constants, never user input —
// safe to interpolate (parameters are not permitted in DDL).
alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}";
await alter.ExecuteNonQueryAsync();
}
/// <summary>
/// Ensures the directory for a file-backed SQLite database exists. SQLite creates
/// the database file on demand but not its parent directory, so a configured path
/// such as "./data/store-and-forward.db" fails to open ("unable to open database
/// file") when the "data" directory does not yet exist. In-memory databases and
/// bare filenames in the working directory have no directory to create and are
/// skipped.
/// </summary>
private void EnsureDatabaseDirectoryExists()
{
var builder = new SqliteConnectionStringBuilder(_connectionString);
if (builder.Mode == SqliteOpenMode.Memory)
return;
var dataSource = builder.DataSource;
if (string.IsNullOrEmpty(dataSource) || dataSource == ":memory:")
return;
var directory = System.IO.Path.GetDirectoryName(System.IO.Path.GetFullPath(dataSource));
if (!string.IsNullOrEmpty(directory) && !System.IO.Directory.Exists(directory))
{
System.IO.Directory.CreateDirectory(directory);
_logger.LogInformation("Created store-and-forward database directory: {Directory}", directory);
}
}
/// <summary>
/// Opens a connection with a 5s busy_timeout. Concurrent writers exist by
/// design (script enqueues, the sweep's lanes, standby replication applies,
/// central pull queries); with connection-per-operation the pragma must be
/// set per open (it is per-connection, and Microsoft.Data.Sqlite pooling
/// makes the extra statement cheap on a pooled physical connection).
/// </summary>
private async Task<SqliteConnection> OpenConnectionAsync()
{
var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var pragma = connection.CreateCommand();
pragma.CommandText = "PRAGMA busy_timeout = 5000";
await pragma.ExecuteNonQueryAsync();
return connection;
}
private SqliteConnection OpenConnection() => _localDb.CreateConnection();
/// <summary>
/// INSERT statement for a full message row. Shared by <see cref="EnqueueAsync"/>
/// and <see cref="ReplaceAllAsync"/>; bind with <see cref="BindMessageParameters"/>
/// and <see cref="UpsertMessageAsync"/>; bind with <see cref="BindMessageParameters"/>
/// so the column list and the parameters never drift apart.
/// </summary>
private const string InsertMessageSql = @"
@@ -235,7 +114,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task EnqueueAsync(StoreAndForwardMessage message)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = InsertMessageSql;
@@ -255,7 +134,7 @@ public class StoreAndForwardStorage
/// <returns>The oldest-first page (at most <paramref name="limit"/> rows) and whether more rows exist beyond it.</returns>
public async Task<(List<StoreAndForwardMessage> Messages, bool Truncated)> GetAllMessagesAsync(int limit)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -272,39 +151,6 @@ public class StoreAndForwardStorage
return (rows.Take(limit).ToList(), truncated);
}
/// <summary>
/// Atomically replaces the entire buffer with <paramref name="messages"/> in a
/// single transaction (delete-all then insert-all). Standby-side anti-entropy
/// apply: a peer-join resync overwrites the standby's divergent copy with the
/// active node's authoritative snapshot. <b>Never call on an active node</b> —
/// it discards every in-flight row.
/// </summary>
/// <param name="messages">The full buffer snapshot to install.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task ReplaceAllAsync(IReadOnlyList<StoreAndForwardMessage> messages)
{
await using var connection = await OpenConnectionAsync();
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
await using (var deleteCmd = connection.CreateCommand())
{
deleteCmd.Transaction = transaction;
deleteCmd.CommandText = "DELETE FROM sf_messages";
await deleteCmd.ExecuteNonQueryAsync();
}
foreach (var message in messages)
{
await using var insertCmd = connection.CreateCommand();
insertCmd.Transaction = transaction;
insertCmd.CommandText = InsertMessageSql;
BindMessageParameters(insertCmd, message);
await insertCmd.ExecuteNonQueryAsync();
}
await transaction.CommitAsync();
}
/// <summary>
/// Inserts a message, or updates every mutable column in place if a row with the
/// same id already exists (<c>ON CONFLICT(id) DO UPDATE</c>). Used by the standby
@@ -318,7 +164,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task UpsertMessageAsync(StoreAndForwardMessage message)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -375,7 +221,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the list of messages due for retry, ordered by creation time ascending.</returns>
public async Task<List<StoreAndForwardMessage>> GetMessagesForRetryAsync(int limit = 0)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -409,7 +255,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task UpdateMessageAsync(StoreAndForwardMessage message)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -446,7 +292,7 @@ public class StoreAndForwardStorage
StoreAndForwardMessage message,
StoreAndForwardMessageStatus expectedStatus)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -479,7 +325,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task RemoveMessageAsync(string messageId)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id";
@@ -500,7 +346,7 @@ public class StoreAndForwardStorage
int pageNumber = 1,
int pageSize = 50)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
@@ -545,7 +391,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to <c>true</c> if the message was found and reset to Pending; <c>false</c> if not found or not in Parked status.</returns>
public async Task<bool> RetryParkedMessageAsync(string messageId)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -569,7 +415,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to <c>true</c> if the message was found and deleted; <c>false</c> if not found or not in Parked status.</returns>
public async Task<bool> DiscardParkedMessageAsync(string messageId)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id AND status = @parked";
@@ -586,7 +432,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to a dictionary mapping each category to its pending message count.</returns>
public async Task<Dictionary<StoreAndForwardCategory, int>> GetBufferDepthByCategoryAsync()
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -616,7 +462,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the number of messages whose origin instance matches <paramref name="instanceName"/>.</returns>
public async Task<int> GetMessageCountByOriginInstanceAsync(string instanceName)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -635,7 +481,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the matching message, or <c>null</c> if not found.</returns>
public async Task<StoreAndForwardMessage?> GetMessageByIdAsync(string messageId)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = @"
@@ -656,7 +502,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the number of messages currently in Parked status.</returns>
public async Task<int> GetParkedMessageCountAsync()
{
await using var conn = await OpenConnectionAsync();
await using var conn = OpenConnection();
await using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @parked";
cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked);
@@ -674,7 +520,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the oldest parked row's creation time, or <c>null</c> if none are parked.</returns>
public async Task<DateTimeOffset?> GetOldestParkedCreatedAtAsync()
{
await using var conn = await OpenConnectionAsync();
await using var conn = OpenConnection();
await using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT MIN(created_at) FROM sf_messages WHERE status = @parked";
cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked);
@@ -691,7 +537,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the count of messages with the specified status.</returns>
public async Task<int> GetMessageCountByStatusAsync(StoreAndForwardMessageStatus status)
{
await using var connection = await OpenConnectionAsync();
await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @status";
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ParentExecutionIdCorrelationTests.cs
+173 -162
View File
@@ -35,6 +35,7 @@ using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Delivery;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Integration;
@@ -197,174 +198,184 @@ public class ParentExecutionIdCorrelationTests : TestKit, IClassFixture<MsSqlMig
stubClient);
// Site Store-and-Forward — Notify.Send buffers a NotificationSubmit here.
using var safKeepAlive = new Microsoft.Data.Sqlite.SqliteConnection(
$"Data Source=parentexec-saf-{Guid.NewGuid():N};Mode=Memory;Cache=Shared");
safKeepAlive.Open();
var safStorage = new StoreAndForwardStorage(
safKeepAlive.ConnectionString, NullLogger<StoreAndForwardStorage>.Instance);
await safStorage.InitializeAsync();
var storeAndForward = new StoreAndForwardService(
safStorage,
new StoreAndForwardOptions
// The storage takes an ILocalDb and LocalDb has no in-memory mode, so this is a
// real temp file, disposed (the master connection anchors the WAL) and deleted in
// the finally below.
var safLocalDb = TestLocalDb.CreateTemp("parentexec-saf");
var safDbPath = safLocalDb.Path;
try
{
var safStorage = new StoreAndForwardStorage(
safLocalDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await safStorage.InitializeAsync();
var storeAndForward = new StoreAndForwardService(
safStorage,
new StoreAndForwardOptions
{
DefaultRetryInterval = TimeSpan.Zero,
DefaultMaxRetries = 3,
RetryTimerInterval = TimeSpan.FromMinutes(10),
},
NullLogger<StoreAndForwardService>.Instance);
// ── Outbound external-system client (routed run): sync Call succeeds,
// CachedCall completes immediately (WasBuffered=false) so the script
// helper emits the Submit + Attempted + CachedResolve lifecycle. ──
var externalClient = Substitute.For<IExternalSystemClient>();
externalClient
.CallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(), Arg.Any<CancellationToken>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null));
externalClient
.CachedCallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(),
Arg.Any<string?>(), Arg.Any<CancellationToken>(),
Arg.Any<ZB.MOM.WW.ScadaBridge.Commons.Types.TrackedOperationId?>(),
Arg.Any<Guid?>(), Arg.Any<string?>(), Arg.Any<Guid?>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false));
// ── The routing transport stand-in: builds the routed ScriptRuntimeContext
// carrying RouteToCallRequest.ParentExecutionId — exactly what the
// production site handler (DeploymentManagerActor) does. ──
var router = new BridgingInstanceRouter(
siteId,
externalClient,
siteAuditWriter,
cachedForwarder,
storeAndForward);
// ── The inbound API method script: it calls Route.Call into the site
// instance. The real InboundScriptExecutor binds the inbound request's
// ExecutionId onto the RouteHelper, so the routed call carries it as
// ParentExecutionId. ──
var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod(
"submitOrder",
$"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});");
var locator = Substitute.For<IInstanceLocator>();
locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any<CancellationToken>())
.Returns(siteId);
var scriptExecutor = new InboundScriptExecutor(
NullLogger<InboundScriptExecutor>.Instance,
new ServiceCollection().BuildServiceProvider());
Assert.True(scriptExecutor.CompileAndRegister(inboundMethod));
// ── Act — issue the inbound HTTP request through a TestHost pipeline
// fronted by the real AuditWriteMiddleware. The endpoint handler reads
// the middleware-stashed inbound ExecutionId and runs the inbound
// method script with it as parentExecutionId. ──
using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx =>
{
DefaultRetryInterval = TimeSpan.Zero,
DefaultMaxRetries = 3,
RetryTimerInterval = TimeSpan.FromMinutes(10),
},
NullLogger<StoreAndForwardService>.Instance);
var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!;
var route = new RouteHelper(locator, router);
var result = await scriptExecutor.ExecuteAsync(
inboundMethod,
new Dictionary<string, object?>(),
route,
TimeSpan.FromSeconds(30),
ctx.RequestAborted,
parentExecutionId: inboundExecutionId);
// ── Outbound external-system client (routed run): sync Call succeeds,
// CachedCall completes immediately (WasBuffered=false) so the script
// helper emits the Submit + Attempted + CachedResolve lifecycle. ──
var externalClient = Substitute.For<IExternalSystemClient>();
externalClient
.CallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(), Arg.Any<CancellationToken>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null));
externalClient
.CachedCallAsync(ExternalSystemName, ExternalMethodName,
Arg.Any<IReadOnlyDictionary<string, object?>?>(),
Arg.Any<string?>(), Arg.Any<CancellationToken>(),
Arg.Any<ZB.MOM.WW.ScadaBridge.Commons.Types.TrackedOperationId?>(),
Arg.Any<Guid?>(), Arg.Any<string?>(), Arg.Any<Guid?>())
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false));
ctx.Response.StatusCode = result.Success ? 200 : 500;
await ctx.Response.WriteAsync(result.Success ? "ok" : "fail");
});
// ── The routing transport stand-in: builds the routed ScriptRuntimeContext
// carrying RouteToCallRequest.ParentExecutionId — exactly what the
// production site handler (DeploymentManagerActor) does. ──
var router = new BridgingInstanceRouter(
siteId,
externalClient,
siteAuditWriter,
cachedForwarder,
storeAndForward);
var client = host.GetTestClient();
var response = await client.PostAsync(
"/api/submitOrder",
new StringContent("{}", Encoding.UTF8, "application/json"));
Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode);
// ── The inbound API method script: it calls Route.Call into the site
// instance. The real InboundScriptExecutor binds the inbound request's
// ExecutionId onto the RouteHelper, so the routed call carries it as
// ParentExecutionId. ──
var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod(
"submitOrder",
$"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});");
var locator = Substitute.For<IInstanceLocator>();
locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any<CancellationToken>())
.Returns(siteId);
var scriptExecutor = new InboundScriptExecutor(
NullLogger<InboundScriptExecutor>.Instance,
new ServiceCollection().BuildServiceProvider());
Assert.True(scriptExecutor.CompileAndRegister(inboundMethod));
// The routed run emits its sync-ApiCall and NotifySend audit rows on a
// deliberately fire-and-forget path (alog.md §7 — an audit write must
// never block the user-facing script call). `Notify.Send` therefore
// returns — and the routed `RouteToCallAsync` completes — BEFORE the
// SqliteAuditWriter background loop has flushed the NotifySend row into
// the site hot-path. Wait for all five site rows to be durably present
// in SQLite before the central assertion: this is the production
// durability point (the row IS in SQLite before it is considered
// audited), and pinning it removes the emit-vs-drain race that
// otherwise let the SiteAuditTelemetryADrain forward only four rows on
// its first tick and leave NotifySend stranded for a full drain
// interval under heavy parallel load.
await WaitForSiteRowsPersistedAsync(sqliteWriter);
// ── Act — issue the inbound HTTP request through a TestHost pipeline
// fronted by the real AuditWriteMiddleware. The endpoint handler reads
// the middleware-stashed inbound ExecutionId and runs the inbound
// method script with it as parentExecutionId. ──
using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx =>
// The routed run produced a NotifySend that buffered a NotificationSubmit
// into S&F. Drain that genuine site-produced submission to the central
// NotificationOutboxActor so the NotifyDeliver dispatch rows materialise.
await ForwardBufferedNotificationToCentralAsync(
storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter);
// ── Assert ──────────────────────────────────────────────────────────
await AwaitAssertAsync(async () =>
{
await using var readContext = CreateContext();
var repo = new AuditLogRepository(readContext);
// Every audit row this site produced (sync ApiCall + cached lifecycle
// + NotifySend) plus the central NotifyDeliver rows.
var siteRows = await repo.QueryAsync(
new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }),
new AuditLogPaging(PageSize: 100));
// sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1)
// + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run.
Assert.True(siteRows.Count == 7,
"Expected 7 routed-run audit rows; saw: "
+ string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}")));
Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend);
Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver));
// CORE PROMISE: every routed-run row carries the SAME non-null
// ParentExecutionId — the inbound request's ExecutionId.
var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList();
Assert.Single(parentIds);
Assert.NotNull(parentIds[0]);
var inboundExecutionId = parentIds[0]!.Value;
// The routed run has its OWN distinct ExecutionId — not the parent's.
var routedExecutionIds = siteRows
.Select(r => r.AsRow().ExecutionId)
.Distinct()
.ToList();
Assert.Single(routedExecutionIds);
Assert.NotNull(routedExecutionIds[0]);
var routedExecutionId = routedExecutionIds[0]!.Value;
Assert.NotEqual(inboundExecutionId, routedExecutionId);
// The inbound request's own InboundRequest row is TOP-LEVEL —
// ExecutionId = the propagated id, ParentExecutionId = NULL.
var inboundRows = await repo.QueryAsync(
new AuditLogQueryFilter(ExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 10));
var inboundRow = Assert.Single(inboundRows,
r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest);
Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status);
Assert.Null(inboundRow.AsRow().ParentExecutionId);
// The parentExecutionId filter pulls the routed run's complete
// trust-boundary footprint (all 7 routed rows, none of the inbound).
var byParent = await repo.QueryAsync(
new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 100));
Assert.Equal(7, byParent.Count);
Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId));
// GetExecutionTreeAsync returns BOTH executions in one chain —
// inbound (root) and routed (child), regardless of entry point.
var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId);
AssertChain(treeFromChild, inboundExecutionId, routedExecutionId);
var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId);
AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId);
}, TimeSpan.FromSeconds(90));
}
finally
{
var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!;
var route = new RouteHelper(locator, router);
var result = await scriptExecutor.ExecuteAsync(
inboundMethod,
new Dictionary<string, object?>(),
route,
TimeSpan.FromSeconds(30),
ctx.RequestAborted,
parentExecutionId: inboundExecutionId);
ctx.Response.StatusCode = result.Success ? 200 : 500;
await ctx.Response.WriteAsync(result.Success ? "ok" : "fail");
});
var client = host.GetTestClient();
var response = await client.PostAsync(
"/api/submitOrder",
new StringContent("{}", Encoding.UTF8, "application/json"));
Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode);
// The routed run emits its sync-ApiCall and NotifySend audit rows on a
// deliberately fire-and-forget path (alog.md §7 — an audit write must
// never block the user-facing script call). `Notify.Send` therefore
// returns — and the routed `RouteToCallAsync` completes — BEFORE the
// SqliteAuditWriter background loop has flushed the NotifySend row into
// the site hot-path. Wait for all five site rows to be durably present
// in SQLite before the central assertion: this is the production
// durability point (the row IS in SQLite before it is considered
// audited), and pinning it removes the emit-vs-drain race that
// otherwise let the SiteAuditTelemetryADrain forward only four rows on
// its first tick and leave NotifySend stranded for a full drain
// interval under heavy parallel load.
await WaitForSiteRowsPersistedAsync(sqliteWriter);
// The routed run produced a NotifySend that buffered a NotificationSubmit
// into S&F. Drain that genuine site-produced submission to the central
// NotificationOutboxActor so the NotifyDeliver dispatch rows materialise.
await ForwardBufferedNotificationToCentralAsync(
storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter);
// ── Assert ──────────────────────────────────────────────────────────
await AwaitAssertAsync(async () =>
{
await using var readContext = CreateContext();
var repo = new AuditLogRepository(readContext);
// Every audit row this site produced (sync ApiCall + cached lifecycle
// + NotifySend) plus the central NotifyDeliver rows.
var siteRows = await repo.QueryAsync(
new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }),
new AuditLogPaging(PageSize: 100));
// sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1)
// + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run.
Assert.True(siteRows.Count == 7,
"Expected 7 routed-run audit rows; saw: "
+ string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}")));
Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve);
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend);
Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver));
// CORE PROMISE: every routed-run row carries the SAME non-null
// ParentExecutionId — the inbound request's ExecutionId.
var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList();
Assert.Single(parentIds);
Assert.NotNull(parentIds[0]);
var inboundExecutionId = parentIds[0]!.Value;
// The routed run has its OWN distinct ExecutionId — not the parent's.
var routedExecutionIds = siteRows
.Select(r => r.AsRow().ExecutionId)
.Distinct()
.ToList();
Assert.Single(routedExecutionIds);
Assert.NotNull(routedExecutionIds[0]);
var routedExecutionId = routedExecutionIds[0]!.Value;
Assert.NotEqual(inboundExecutionId, routedExecutionId);
// The inbound request's own InboundRequest row is TOP-LEVEL —
// ExecutionId = the propagated id, ParentExecutionId = NULL.
var inboundRows = await repo.QueryAsync(
new AuditLogQueryFilter(ExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 10));
var inboundRow = Assert.Single(inboundRows,
r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest);
Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status);
Assert.Null(inboundRow.AsRow().ParentExecutionId);
// The parentExecutionId filter pulls the routed run's complete
// trust-boundary footprint (all 7 routed rows, none of the inbound).
var byParent = await repo.QueryAsync(
new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId),
new AuditLogPaging(PageSize: 100));
Assert.Equal(7, byParent.Count);
Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId));
// GetExecutionTreeAsync returns BOTH executions in one chain —
// inbound (root) and routed (child), regardless of entry point.
var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId);
AssertChain(treeFromChild, inboundExecutionId, routedExecutionId);
var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId);
AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId);
}, TimeSpan.FromSeconds(90));
safLocalDb.Dispose();
TestLocalDb.DeleteFiles(safDbPath);
}
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/Telemetry/SiteAuditTelemetryActorTests.cs
+52
View File
@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using NSubstitute.ExceptionExtensions;
using NSubstitute.ReceivedExtensions;
using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
@@ -451,4 +452,55 @@ public class SiteAuditTelemetryActorTests : TestKit
await _client.DidNotReceiveWithAnyArgs().IngestCachedTelemetryAsync(default!, default);
await _queue.DidNotReceiveWithAnyArgs().ReadPendingCachedTelemetryAsync(default, default);
}
/// <summary>
/// Regression for the 2026-07-20 rig finding (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8):
/// both drain handlers await with <c>ConfigureAwait(false)</c>, so when a
/// dependency call completes off the actor thread (as real SQLite/gRPC
/// always do) the <c>finally</c>-block re-arm runs on a pool thread with no
/// active ActorContext. Depending on what that thread's thread-static cell
/// slot holds, <c>Context</c>/<c>Self</c> either throw
/// <c>NotSupportedException: There is no active ActorContext</c> (crashing
/// the actor once per drain — the rig's logged variant) or silently resolve
/// to a STALE cell of some other actor, misrouting the tick so the drain
/// loop stops. The two assertions below catch one variant each. Every other
/// test in this class masks the bug by returning already-completed tasks
/// from the mocks, which keeps the continuations on the actor thread.
/// </summary>
[Fact]
public async Task Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing()
{
// Task.Delay completes on a timer thread; with ConfigureAwait(false)
// everything after the await — including the finally-block re-arm —
// stays off the actor context.
_queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(async _ =>
{
await Task.Delay(25).ConfigureAwait(false);
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
});
_queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(async _ =>
{
await Task.Delay(25).ConfigureAwait(false);
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
});
// Variant 1 (throw → restart storm): any NotSupportedException logged
// during the run fails the filter. Variant 2 (silent misroute → the
// drain loop stalls after the first tick): the sustained-drain
// assertion inside fails because reads stop at 1.
await EventFilter.Exception<NotSupportedException>().ExpectAsync(0, async () =>
{
CreateActorWithCachedDrain(Opts(busySeconds: 1, idleSeconds: 1));
// Three full cycles prove the finally-block re-arm works from an
// off-context thread: tick → drain → re-arm → tick → …
await AwaitAssertAsync(async () =>
{
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
}, TimeSpan.FromSeconds(10));
});
}
}
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj
+2 -1
View File
@@ -76,7 +76,8 @@
central MSSQL AuditLog.
-->
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.InboundAPI/ZB.MOM.WW.ScadaBridge.InboundAPI.csproj" />
</ItemGroup>
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
<ItemGroup>
<!-- M4 Bundle E (Task E3): need ASP.NET Core for the TestHost-based middleware E2E. -->
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/HealthFailoverButtonTests.cs
+241
View File
@@ -0,0 +1,241 @@
using System.Security.Claims;
using Bunit;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Health;
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared;
using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Security;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Monitoring;
/// <summary>
/// bUnit tests for the admin "Trigger failover" control on the Health dashboard
/// (decision 2026-07-22). The control is destructive and privileged, so the three
/// things that must hold are: only an Administrator sees it, it is refused when
/// there is no standby to take over, and it never fires without an explicit
/// confirmation.
/// </summary>
public class HealthFailoverButtonTests : BunitContext
{
private readonly IManualFailoverService _failover = Substitute.For<IManualFailoverService>();
private readonly IDialogService _dialog = Substitute.For<IDialogService>();
private void Arrange(string role)
{
var claims = new[]
{
new Claim(JwtTokenService.UsernameClaimType, "tester"),
new Claim(JwtTokenService.RoleClaimType, role)
};
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
AuthorizationPolicies.AddScadaBridgeAuthorization(Services);
// BunitContext pre-registers a placeholder IAuthorizationService that throws when
// AuthorizeView evaluates a policy. Force the real service so the admin gate is
// genuinely exercised rather than trivially satisfied (same note as NavMenuTests).
Services.AddSingleton<IAuthorizationService, DefaultAuthorizationService>();
Services.AddSingleton(_failover);
Services.AddSingleton(_dialog);
}
/// <summary>
/// AuthorizeView requires a cascading Task&lt;AuthenticationState&gt;, which the app
/// supplies from its layout; a bare component render has to provide it explicitly.
/// </summary>
private IRenderedComponent<CentralFailoverControl> Render(int onlineNodes)
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<CentralFailoverControl>(0);
builder.AddAttribute(1, nameof(CentralFailoverControl.OnlineCentralNodeCount), onlineNodes);
builder.CloseComponent();
})));
return host.FindComponent<CentralFailoverControl>();
}
[Fact]
public void Button_is_hidden_for_a_non_admin()
{
Arrange("Viewer");
var cut = Render(onlineNodes: 2);
Assert.Empty(cut.FindAll("button"));
}
[Fact]
public void Button_is_rendered_and_enabled_for_an_admin_with_a_standby()
{
Arrange("Administrator");
var cut = Render(onlineNodes: 2);
var button = cut.Find("button");
Assert.False(button.HasAttribute("disabled"));
Assert.Contains("failover", button.TextContent, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void Button_is_disabled_with_an_explanatory_title_when_there_is_no_standby()
{
Arrange("Administrator");
var cut = Render(onlineNodes: 1);
var button = cut.Find("button");
Assert.True(button.HasAttribute("disabled"));
// The tooltip must say WHY, not just that it is unavailable.
Assert.Contains("outage", button.GetAttribute("title"), StringComparison.OrdinalIgnoreCase);
}
[Fact]
public async Task Confirming_triggers_the_failover_exactly_once()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverCentralAsync(Arg.Any<string>())
.Returns(Task.FromResult<string?>("akka.tcp://scadabridge@central-a:8081"));
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
await _failover.Received(1).FailOverCentralAsync("tester");
}
[Fact]
public async Task Cancelling_the_confirmation_does_not_trigger_a_failover()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(false);
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
await _failover.DidNotReceive().FailOverCentralAsync(Arg.Any<string>());
}
[Fact]
public async Task Confirmation_warns_that_this_page_will_disconnect()
{
// Traefik routes the UI to the ACTIVE node — the very node being failed over — so
// the admin's own circuit drops. If the dialog does not say so, a working failover
// looks like a crash they caused.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
await _dialog.Received(1).ConfirmAsync(
Arg.Any<string>(),
Arg.Is<string>(m => m.Contains("reconnect", StringComparison.OrdinalIgnoreCase)),
Arg.Any<bool>());
}
[Fact]
public async Task A_refused_failover_surfaces_the_refusal_instead_of_claiming_success()
{
// The service returns null when the peer guard refuses. The UI must not report a
// failover that never happened.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverCentralAsync(Arg.Any<string>()).Returns(Task.FromResult<string?>(null));
var cut = Render(onlineNodes: 2);
await cut.Find("button").ClickAsync(new());
Assert.Contains("no standby", cut.Markup, StringComparison.OrdinalIgnoreCase);
}
// ---- Site-pair failover (Task 10) ------------------------------------------
// Same control, SiteId set. The differences that matter: it calls the site path,
// and its confirmation must NOT claim the admin's own page will drop (it won't —
// the site is a different cluster).
private IRenderedComponent<CentralFailoverControl> RenderForSite(int onlineNodes, string siteId)
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<CentralFailoverControl>(0);
builder.AddAttribute(1, nameof(CentralFailoverControl.OnlineCentralNodeCount), onlineNodes);
builder.AddAttribute(2, nameof(CentralFailoverControl.SiteId), siteId);
builder.CloseComponent();
})));
return host.FindComponent<CentralFailoverControl>();
}
[Fact]
public async Task Site_card_confirming_triggers_the_site_failover_path()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(true, "akka.tcp://scadabridge@site-a-a:8082", null)));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
await _failover.Received(1).FailOverSiteAsync("SiteA", "tester");
await _failover.DidNotReceive().FailOverCentralAsync(Arg.Any<string>());
}
[Fact]
public async Task Site_card_confirmation_does_not_claim_this_page_disconnects()
{
// Central failover drops the admin's circuit; a SITE failover does not, and saying
// otherwise would train operators to distrust the warning that does matter.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(true, "addr", null)));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
await _dialog.Received(1).ConfirmAsync(
Arg.Any<string>(),
Arg.Is<string>(m => !m.Contains("reconnect", StringComparison.OrdinalIgnoreCase)),
Arg.Any<bool>());
}
[Fact]
public async Task Site_card_surfaces_a_refusal_from_the_site()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(
false, null, "No standby available — failing over a lone node would be an outage.")));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
Assert.Contains("No standby available", cut.Markup);
}
[Fact]
public async Task Site_card_surfaces_an_unreachable_site_distinctly_from_a_refusal()
{
// A timeout is not a "no" from the site — the operator must be able to tell the
// difference, because the failover may or may not have taken effect.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(
false, null, "Site did not respond: Ask timed out")));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
Assert.Contains("did not respond", cut.Markup);
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs
+45 -11
View File
@@ -2,11 +2,14 @@ using System.Security.Claims;
using ZB.MOM.WW.ScadaBridge.Security;
using Akka.Actor;
using Bunit;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared;
using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
@@ -105,12 +108,43 @@ public class HealthPageTests : BunitContext
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
// The failover control's AuthorizeView evaluates the named RequireAdmin policy, so the
// real policy set has to be registered or the page throws on any card that renders it.
AuthorizationPolicies.AddScadaBridgeAuthorization(Services);
// BunitContext pre-registers a placeholder IAuthorizationService that throws when a
// policy is evaluated; force the real one (same note as NavMenuTests).
Services.AddSingleton<IAuthorizationService, DefaultAuthorizationService>();
// The cluster cards embed CentralFailoverControl (Task 10), which injects these two.
// The page's DI graph genuinely requires them in production, so the test supplies
// them rather than the component making its dependencies optional. Behaviour of the
// control itself is covered by HealthFailoverButtonTests.
Services.AddSingleton(Substitute.For<IManualFailoverService>());
Services.AddSingleton(Substitute.For<IDialogService>());
}
/// <summary>
/// Renders the dashboard the way the app does — inside a
/// <see cref="CascadingAuthenticationState"/>, which the real layout supplies. The
/// cluster cards contain an AuthorizeView (the Task 10 failover control), and
/// AuthorizeView throws without that cascading value.
/// </summary>
private IRenderedComponent<HealthPage> RenderHealthPage()
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<HealthPage>(0);
builder.CloseComponent();
})));
return host.FindComponent<HealthPage>();
}
[Fact]
public void Renders_OutboxKpiTiles_WithValues()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
// KPI data arrives via an async actor Ask after first render.
cut.WaitForAssertion(() =>
@@ -129,7 +163,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void RendersLinkToTheNotificationKpisPage()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
var link = cut.Find("a[href='/notifications/kpis']");
Assert.Contains("View details", link.TextContent);
}
@@ -148,7 +182,7 @@ public class HealthPageTests : BunitContext
AsOfUtc: DateTime.UtcNow)));
Services.AddSingleton(auditService);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -166,7 +200,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void Renders_SiteCallKpiTiles_WithValues()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
// KPI data arrives via an async actor Ask after first render.
cut.WaitForAssertion(() =>
@@ -186,7 +220,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void RendersLinkToTheSiteCallsReportPage()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
var link = cut.Find("a[href='/site-calls/report']");
Assert.Contains("View details", link.TextContent);
}
@@ -197,7 +231,7 @@ public class HealthPageTests : BunitContext
_siteCallKpiReply = new SiteCallKpiResponse(
"k", false, "site call repository unavailable", 0, 0, 0, 0, null, 0);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -216,7 +250,7 @@ public class HealthPageTests : BunitContext
_kpiReply = new NotificationKpiResponse(
"k", false, "outbox repository unavailable", 0, 0, 0, 0, null);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -234,7 +268,7 @@ public class HealthPageTests : BunitContext
// default-site load produces charts.
SeedSites("site-a");
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -268,7 +302,7 @@ public class HealthPageTests : BunitContext
throw new InvalidOperationException("kpi history unavailable"));
Services.AddSingleton(faulting);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -294,7 +328,7 @@ public class HealthPageTests : BunitContext
LastReportReceivedAt = DateTimeOffset.UtcNow.AddMinutes(-5),
});
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -316,7 +350,7 @@ public class HealthPageTests : BunitContext
LastStatusChangeAt = changedAt,
});
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs
+4 -1
View File
@@ -10,7 +10,10 @@ public class ClusterOptionsTests
{
var options = new ClusterOptions();
Assert.Equal("keep-oldest", options.SplitBrainResolverStrategy);
// 'auto-down' is the default posture (decision 2026-07-21): a crash of either
// node fails over to the survivor; dual-active during a real partition is the
// accepted trade for pairs with no shared lease infrastructure.
Assert.Equal("auto-down", options.SplitBrainResolverStrategy);
Assert.Equal(TimeSpan.FromSeconds(15), options.StableAfter);
Assert.Equal(TimeSpan.FromSeconds(2), options.HeartbeatInterval);
Assert.Equal(TimeSpan.FromSeconds(10), options.FailureDetectionThreshold);
tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs
+30 -1
View File
@@ -153,9 +153,10 @@ public class ClusterOptionsValidatorTests
}
[Fact]
public void DownIfAloneFalse_FailsValidation()
public void DownIfAloneFalse_UnderKeepOldest_FailsValidation()
{
var options = ValidOptions();
options.SplitBrainResolverStrategy = "keep-oldest";
options.DownIfAlone = false;
var result = new ClusterOptionsValidator().Validate(null, options);
@@ -164,6 +165,34 @@ public class ClusterOptionsValidatorTests
Assert.Contains("DownIfAlone", result.FailureMessage);
}
[Fact]
public void AutoDownStrategy_Passes()
{
// Decision 2026-07-21: 'auto-down' is the supported availability-first
// posture — either-node crash fails over; dual-active on a real partition
// is the accepted trade.
var options = ValidOptions();
options.SplitBrainResolverStrategy = "auto-down";
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void DownIfAloneFalse_UnderAutoDown_Passes()
{
// DownIfAlone is a keep-oldest knob; under auto-down it is inert and must
// not block startup.
var options = ValidOptions();
options.SplitBrainResolverStrategy = "auto-down";
options.DownIfAlone = false;
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void Validate_AccumulatesAllFailures()
{
tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/Enums/AuditEnumTests.cs
+4 -3
View File
@@ -12,7 +12,7 @@ public class AuditEnumTests
[Fact]
public void AuditChannel_HasExactlyExpectedMembers()
{
var expected = new[] { "ApiOutbound", "DbOutbound", "Notification", "ApiInbound", "SecuredWrite" };
var expected = new[] { "ApiOutbound", "DbOutbound", "Notification", "ApiInbound", "SecuredWrite", "Cluster" };
var actual = Enum.GetValues(typeof(AuditChannel))
.Cast<AuditChannel>()
.Select(x => x.ToString())
@@ -23,7 +23,7 @@ public class AuditEnumTests
}
[Fact]
public void AuditKind_HasExactlySixteenExpectedMembers()
public void AuditKind_HasExactlySeventeenExpectedMembers()
{
var expected = new[]
{
@@ -33,13 +33,14 @@ public class AuditEnumTests
"SecuredWriteSubmit", "SecuredWriteApprove", "SecuredWriteReject", "SecuredWriteExecute",
"SecuredWriteExpire",
"ReconciliationAbandoned",
"ManualFailover",
};
var actual = Enum.GetValues(typeof(AuditKind))
.Cast<AuditKind>()
.Select(x => x.ToString())
.ToArray();
Assert.Equal(16, actual.Length);
Assert.Equal(17, actual.Length);
Assert.Equal(expected, actual);
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteCommunicationActorTests.cs
+85
View File
@@ -372,4 +372,89 @@ public class SiteCommunicationActorTests : TestKit
Assert.Equal(isActive, heartbeat.IsActive);
Assert.Equal("site1", heartbeat.SiteId);
}
// ---- Central→site failover relay (Task 10) ----------------------------------
// The failover action is injected for the same reason isActiveCheck is: performing a
// real Cluster.Leave would require Akka.Cluster in the TestKit ActorSystem. These
// tests pin the ROUTING and GUARD logic; the actual Leave against a real two-node
// site cluster is covered by SiteFailoverRelayTests in the integration suite.
[Fact]
public void TriggerSiteFailover_IssuesTheLeave_AndAcksWithTheTarget()
{
var dmProbe = CreateTestProbe();
string? roleAskedFor = null;
Func<string, string?> failOver = role =>
{
roleAskedFor = role;
return "akka.tcp://scadabridge@site1-a:8082";
};
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-1", "site1"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.True(ack.Accepted);
Assert.Equal("corr-1", ack.CorrelationId);
Assert.Equal("akka.tcp://scadabridge@site1-a:8082", ack.TargetAddress);
Assert.Null(ack.ErrorMessage);
// Site singletons are scoped to the site-specific role, not the base "Site" role;
// failing over the wrong scope would move the wrong node.
Assert.Equal("site-site1", roleAskedFor);
}
[Fact]
public void TriggerSiteFailover_RefusesWhenThereIsNoPeer()
{
var dmProbe = CreateTestProbe();
Func<string, string?> failOver = _ => null;
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-2", "site1"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.False(ack.Accepted);
Assert.Null(ack.TargetAddress);
Assert.NotNull(ack.ErrorMessage);
}
[Fact]
public void TriggerSiteFailover_RefusesACommandAddressedToAnotherSite()
{
// A misrouted command must be refused, not silently acted on — acting would fail
// over a site the operator never selected.
var dmProbe = CreateTestProbe();
var invoked = false;
Func<string, string?> failOver = _ =>
{
invoked = true;
return "addr";
};
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-3", "site2"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.False(ack.Accepted);
Assert.Contains("site2", ack.ErrorMessage);
Assert.False(invoked);
}
[Fact]
public void TriggerSiteFailover_FaultInTheLeave_IsReportedNotThrown()
{
var dmProbe = CreateTestProbe();
Func<string, string?> failOver = _ => throw new InvalidOperationException("cluster unavailable");
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-4", "site1"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.False(ack.Accepted);
Assert.Contains("cluster unavailable", ack.ErrorMessage);
}
}
tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs
+71 -50
View File
@@ -3,18 +3,56 @@ using System.Data.Common;
using System.Text.Json;
using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests;
/// <summary>
/// WP-9: Tests for Database access — connection resolution, cached writes.
/// </summary>
public class DatabaseGatewayTests
public class DatabaseGatewayTests : IDisposable
{
private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>();
/// <summary>
/// Temp local databases opened by the Store-and-Forward tests, torn down in
/// <see cref="Dispose"/>.
/// </summary>
private readonly List<TestLocalDb> _localDbs = new();
/// <summary>
/// Opens a real <c>ILocalDb</c> over a fresh temp file for a test that needs a live
/// <see cref="ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage"/>. The
/// store takes an <c>ILocalDb</c> and LocalDb has no in-memory mode, so each test gets
/// its own file rather than a shared-cache in-memory database held open by a
/// keep-alive connection.
/// </summary>
private TestLocalDb CreateLocalDb(string prefix)
{
var localDb = TestLocalDb.CreateTemp(prefix);
_localDbs.Add(localDb);
return localDb;
}
/// <summary>
/// Disposes every temp local database and then deletes its files — in that order,
/// because the master connection LocalDb holds anchors the WAL sidecars.
/// </summary>
public void Dispose()
{
foreach (var localDb in _localDbs)
{
var path = localDb.Path;
localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
GC.SuppressFinalize(this);
}
/// <summary>
/// Configures the repository substitute for the name-keyed connection-resolution
/// path used by <c>DatabaseGateway</c> (ExternalSystemGateway-011). A <c>null</c>
@@ -84,12 +122,9 @@ public class DatabaseGatewayTests
};
StubConnection(conn);
var dbName = $"EsgCachedWrite_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
keepAlive.Open();
var localDb = CreateLocalDb("EsgCachedWrite");
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
{
@@ -121,7 +156,7 @@ public class DatabaseGatewayTests
var depth = await storage.GetBufferDepthByCategoryAsync();
Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.CachedDbWrite]);
var buffered = ReadBufferedRetrySettings(connStr);
var buffered = ReadBufferedRetrySettings(localDb.Db);
Assert.Equal(5, buffered.MaxRetries);
Assert.Equal((long)TimeSpan.FromSeconds(12).TotalMilliseconds, buffered.RetryIntervalMs);
@@ -148,12 +183,9 @@ public class DatabaseGatewayTests
};
StubConnection(conn);
var dbName = $"EsgCachedWriteZero_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
keepAlive.Open();
var localDb = CreateLocalDb("EsgCachedWriteZero");
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
{
@@ -172,7 +204,7 @@ public class DatabaseGatewayTests
await gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)");
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr);
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db);
// Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever.
Assert.Equal(99, maxRetries);
Assert.NotEqual(0, maxRetries);
@@ -182,19 +214,15 @@ public class DatabaseGatewayTests
// cached-write attempt + the buffered retry path ──
/// <summary>
/// Builds a real, initialised in-memory store-and-forward service plus a
/// keep-alive connection (the SQLite shared-cache DB lives only while a
/// connection is open). The caller disposes <paramref name="keepAlive"/>.
/// Builds a real, initialised store-and-forward service over a fresh temp local
/// database. The database is torn down by <see cref="Dispose"/>.
/// </summary>
private static (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, string ConnStr, Microsoft.Data.Sqlite.SqliteConnection KeepAlive)
private (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, ILocalDb Db)
NewStoreAndForward()
{
var dbName = $"EsgCachedWriteClassify_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
keepAlive.Open();
var localDb = CreateLocalDb("EsgCachedWriteClassify");
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
storage.InitializeAsync().GetAwaiter().GetResult();
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
{
@@ -204,7 +232,7 @@ public class DatabaseGatewayTests
};
var sf = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService(
storage, sfOptions, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService>.Instance);
return (sf, connStr, keepAlive);
return (sf, localDb.Db);
}
[Fact]
@@ -217,8 +245,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward();
using var _ = keepAlive;
var (sf, sfDb) = NewStoreAndForward();
var gateway = new ExecuteStubGateway(
_repository,
@@ -233,7 +260,7 @@ public class DatabaseGatewayTests
Assert.NotNull(result.ErrorMessage);
// Nothing buffered — the permanent failure short-circuited S&F.
Assert.Equal(0, ReadBufferDepth(connStr));
Assert.Equal(0, ReadBufferDepth(sfDb));
}
[Fact]
@@ -249,8 +276,7 @@ public class DatabaseGatewayTests
};
StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward();
using var _ = keepAlive;
var (sf, sfDb) = NewStoreAndForward();
var gateway = new ExecuteStubGateway(
_repository,
@@ -265,7 +291,7 @@ public class DatabaseGatewayTests
Assert.True(result.WasBuffered); // handed to S&F, not synchronously failed
Assert.Null(result.ErrorMessage);
Assert.Equal(1, ReadBufferDepth(connStr));
Assert.Equal(1, ReadBufferDepth(sfDb));
}
[Fact]
@@ -277,8 +303,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward();
using var _ = keepAlive;
var (sf, sfDb) = NewStoreAndForward();
var gateway = new ExecuteStubGateway(_repository, sf, onExecute: () => { /* succeeds */ });
@@ -288,7 +313,7 @@ public class DatabaseGatewayTests
Assert.False(result.WasBuffered);
Assert.Null(result.ErrorMessage);
Assert.Equal(0, ReadBufferDepth(connStr));
Assert.Equal(0, ReadBufferDepth(sfDb));
}
[Fact]
@@ -381,8 +406,7 @@ public class DatabaseGatewayTests
};
StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward();
using var _ = keepAlive;
var (sf, sfDb) = NewStoreAndForward();
// RawExecuteStubGateway routes the raw throw through the PRODUCTION
// ExecuteWriteAsync classification (the seam under test), unlike
@@ -395,7 +419,7 @@ public class DatabaseGatewayTests
Assert.True(result.WasBuffered); // handed to S&F as transient
Assert.Null(result.ErrorMessage); // not a permanent Failed result
Assert.Equal(1, ReadBufferDepth(connStr));
Assert.Equal(1, ReadBufferDepth(sfDb));
}
[Fact]
@@ -408,8 +432,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward();
using var _ = keepAlive;
var (sf, sfDb) = NewStoreAndForward();
using var cts = new CancellationTokenSource();
cts.Cancel();
@@ -421,7 +444,7 @@ public class DatabaseGatewayTests
() => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)", cancellationToken: cts.Token));
// Cancellation is not a transient failure — nothing must have been buffered.
Assert.Equal(0, ReadBufferDepth(connStr));
Assert.Equal(0, ReadBufferDepth(sfDb));
}
[Fact]
@@ -434,8 +457,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward();
using var _ = keepAlive;
var (sf, sfDb) = NewStoreAndForward();
var gateway = new RawExecuteStubGateway(
_repository, sf, onRunSql: () => throw new ArgumentException("authoring bug"));
@@ -443,7 +465,7 @@ public class DatabaseGatewayTests
await Assert.ThrowsAsync<ArgumentException>(
() => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)"));
Assert.Equal(0, ReadBufferDepth(connStr));
Assert.Equal(0, ReadBufferDepth(sfDb));
}
[Fact]
@@ -486,8 +508,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward();
using var _ = keepAlive;
var (sf, sfDb) = NewStoreAndForward();
using var cts = new CancellationTokenSource();
cts.Cancel();
@@ -505,7 +526,7 @@ public class DatabaseGatewayTests
// The cancel won — it must NOT have been classified as transient (buffered)
// nor returned as a permanent Failed result.
Assert.Equal(0, ReadBufferDepth(connStr));
Assert.Equal(0, ReadBufferDepth(sfDb));
}
/// <summary>
@@ -542,10 +563,10 @@ public class DatabaseGatewayTests
/// Reads the current buffered-message count off the S&amp;F SQLite DB by
/// counting <c>sf_messages</c> rows (the engine's persistence table).
/// </summary>
private static int ReadBufferDepth(string connStr)
private static int ReadBufferDepth(ILocalDb localDb)
{
using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
conn.Open();
// CreateConnection hands back an already-open, pragma-configured connection.
using var conn = localDb.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages";
return Convert.ToInt32(cmd.ExecuteScalar());
@@ -615,10 +636,10 @@ public class DatabaseGatewayTests
}
private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript)
ReadBufferedRetrySettings(string connStr)
ReadBufferedRetrySettings(ILocalDb localDb)
{
using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
conn.Open();
// CreateConnection hands back an already-open, pragma-configured connection.
using var conn = localDb.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText =
"SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages";
tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs
+53 -32
View File
@@ -1,24 +1,60 @@
using System.Net;
using System.Net.Http.Headers;
using System.Text;
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests;
/// <summary>
/// WP-6/7: Tests for ExternalSystemClient — HTTP client, call modes, error handling.
/// </summary>
public class ExternalSystemClientTests
public class ExternalSystemClientTests : IDisposable
{
private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>();
private readonly IHttpClientFactory _httpClientFactory = Substitute.For<IHttpClientFactory>();
/// <summary>
/// Temp local databases opened by the Store-and-Forward tests, torn down in
/// <see cref="Dispose"/>.
/// </summary>
private readonly List<TestLocalDb> _localDbs = new();
/// <summary>
/// Opens a real <c>ILocalDb</c> over a fresh temp file for a test that needs a live
/// <see cref="StoreAndForwardStorage"/>. The store takes an <c>ILocalDb</c> and LocalDb
/// has no in-memory mode, so each test gets its own file rather than a shared-cache
/// in-memory database held open by a keep-alive connection.
/// </summary>
private TestLocalDb CreateLocalDb(string prefix)
{
var localDb = TestLocalDb.CreateTemp(prefix);
_localDbs.Add(localDb);
return localDb;
}
/// <summary>
/// Disposes every temp local database and then deletes its files — in that order,
/// because the master connection LocalDb holds anchors the WAL sidecars.
/// </summary>
public void Dispose()
{
foreach (var localDb in _localDbs)
{
var path = localDb.Path;
localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
GC.SuppressFinalize(this);
}
/// <summary>
/// Configures the repository substitute for the name-keyed resolution path used by
/// <c>ExternalSystemClient</c> (ExternalSystemGateway-011). A <c>null</c> system or
@@ -275,11 +311,8 @@ public class ExternalSystemClientTests
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
// A real S&F service with a registered delivery handler that counts invocations.
var dbName = $"EsgDoubleDispatch_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
var localDb = CreateLocalDb("EsgDoubleDispatch");
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions
{
@@ -422,11 +455,8 @@ public class ExternalSystemClientTests
var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom"));
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
var dbName = $"EsgRetry_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
var localDb = CreateLocalDb("EsgRetry");
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
// S&F defaults deliberately different from the system's settings.
var sfOptions = new StoreAndForwardOptions
@@ -454,7 +484,7 @@ public class ExternalSystemClientTests
var depth = await storage.GetBufferDepthByCategoryAsync();
Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.ExternalSystem]);
var buffered = ReadBufferedRetrySettings(connStr);
var buffered = ReadBufferedRetrySettings(localDb.Db);
Assert.Equal(7, buffered.MaxRetries);
Assert.Equal((long)TimeSpan.FromSeconds(42).TotalMilliseconds, buffered.RetryIntervalMs);
@@ -466,10 +496,10 @@ public class ExternalSystemClientTests
}
private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript)
ReadBufferedRetrySettings(string connStr)
ReadBufferedRetrySettings(ILocalDb localDb)
{
using var conn = new SqliteConnection(connStr);
conn.Open();
// CreateConnection hands back an already-open, pragma-configured connection.
using var conn = localDb.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText =
"SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages";
@@ -505,11 +535,8 @@ public class ExternalSystemClientTests
var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom"));
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
var dbName = $"EsgRetryZero_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
var localDb = CreateLocalDb("EsgRetryZero");
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions
{
@@ -525,7 +552,7 @@ public class ExternalSystemClientTests
await client.CachedCallAsync("TestAPI", "postData");
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr);
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db);
// Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever.
Assert.Equal(99, maxRetries);
Assert.NotEqual(0, maxRetries);
@@ -651,11 +678,8 @@ public class ExternalSystemClientTests
var httpClient = new HttpClient(new HangingHttpMessageHandler(TimeSpan.FromMinutes(10)));
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
var dbName = $"EsgCancel_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
var localDb = CreateLocalDb("EsgCancel");
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions
{
@@ -1060,11 +1084,8 @@ public class ExternalSystemClientTests
.Returns(_ => new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, hugeBody)));
// A real S&F service so we can assert the oversized response is NOT buffered.
var dbName = $"EsgOversize_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
var localDb = CreateLocalDb("EsgOversize");
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions
{
tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj
+2 -1
View File
@@ -24,6 +24,7 @@
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ZB.MOM.WW.ScadaBridge.StoreAndForward.csproj" />
</ItemGroup>
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
</Project>
tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs
+47 -42
View File
@@ -1,10 +1,10 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests;
@@ -191,50 +191,55 @@ public class HealthReportSenderTests
[Fact]
public async Task ReportsIncludeStoreAndForwardBufferDepthsFromStorage()
{
var dbName = $"HealthSfDepth_{Guid.NewGuid():N}";
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
// Keep one connection alive so the in-memory DB persists for the test.
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
// Two pending ExternalSystem messages and one pending Notification message.
await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem));
await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem));
await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification));
var transport = new FakeTransport();
var collector = new SiteHealthCollector();
collector.SetActiveNode(true);
var options = Options.Create(new HealthMonitoringOptions
{
ReportInterval = TimeSpan.FromMilliseconds(50)
});
var sender = new HealthReportSender(
collector,
transport,
options,
NullLogger<HealthReportSender>.Instance,
new FakeSiteIdentityProvider(),
sfStorage: storage);
using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300));
// LocalDb has no in-memory mode, so the store runs over a real temp file.
var localDb = TestLocalDb.CreateTemp("HealthSfDepth");
try
{
await sender.StartAsync(cts.Token);
await Task.Delay(250, CancellationToken.None);
await sender.StopAsync(CancellationToken.None);
}
catch (OperationCanceledException) { }
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync();
Assert.True(transport.SentReports.Count >= 1);
var depths = transport.SentReports[^1].StoreAndForwardBufferDepths;
Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]);
Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]);
Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite)));
// Two pending ExternalSystem messages and one pending Notification message.
await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem));
await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem));
await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification));
var transport = new FakeTransport();
var collector = new SiteHealthCollector();
collector.SetActiveNode(true);
var options = Options.Create(new HealthMonitoringOptions
{
ReportInterval = TimeSpan.FromMilliseconds(50)
});
var sender = new HealthReportSender(
collector,
transport,
options,
NullLogger<HealthReportSender>.Instance,
new FakeSiteIdentityProvider(),
sfStorage: storage);
using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300));
try
{
await sender.StartAsync(cts.Token);
await Task.Delay(250, CancellationToken.None);
await sender.StopAsync(CancellationToken.None);
}
catch (OperationCanceledException) { }
Assert.True(transport.SentReports.Count >= 1);
var depths = transport.SentReports[^1].StoreAndForwardBufferDepths;
Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]);
Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]);
Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite)));
}
finally
{
// The master connection anchors the WAL — dispose before deleting.
localDb.Dispose();
TestLocalDb.DeleteFiles(localDb.Path);
}
}
private static StoreAndForwardMessage MakePendingMessage(string id, StoreAndForwardCategory category) =>
tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj
+2 -1
View File
@@ -28,6 +28,7 @@
<ItemGroup>
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
</ItemGroup>
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
</Project>

Some files were not shown because too many files have changed in this diff Show More