127ec25425aca5e7bc283b3d788c2393b282455f
1339 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
7caa8bfd99 |
fix(localdb): disable SQLite pooling on legacy reads - a Windows site node could not boot
SiteLocalDbLegacyMigrator opens each pre-Phase-2 file read-only, copies its rows into
the consolidated site database, then renames the file so a later boot skips it.
Microsoft.Data.Sqlite POOLS connections, so disposing one returns it to the pool and
leaves the underlying sqlite3 handle - and the OS file handle - open. On Windows the
subsequent File.Move throws IOException ("being used by another process"), the
migration faults out of AddZbLocalDb's factory, and the site node does not start at
all. There is no partial-migration path: it is a hard boot failure, once, on the first
upgrade past LocalDb Phase 2.
Why it shipped: POSIX rename ignores open handles, so this cannot reproduce on Linux or
macOS. Every existing rename assertion in SiteLocalDbLegacyMigratorTests - including
LegacyTrackingRows_AreCopiedAndTheFileIsRenamed - passes with the bug fully present, and
the docker rig migrates cleanly. It was found by pre-flighting the wonder-app-vd03
upgrade against COPIES of that box's real site databases; an earlier pre-flight pass
with empty data directories had nothing to drain and passed clean.
Fix: both legacy read-only connection strings now go through one
LegacyReadOnlyConnectionString helper carrying Pooling=False.
The accompanying test is deliberately white-box. A behavioural assertion cannot
discriminate here on the platform this suite runs on, so it pins the connection string
instead; it is red without the fix.
Verified: Host.Tests 440/440, 0 warnings. The build deployed to wonder-app-vd03 carries
this change (as the then-uncommitted fix) and has been running since 2026-08-05.
|
||
|
Joseph Doherty
|
d0af884760 |
feat(scripts): add the Alarms.CurrentAsync() read accessor for site scripts
MES alarm-status API §5.2 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 tasks 2-4). Site `Call` scripts had NO way to read alarm condition
state: the `Alarm` global exists only inside an on-trigger handler and
describes the one alarm that fired, and native mirrored conditions were
reachable only from the Debug View. That gap blocked the CvdReactor
SimpleAlarmStatus/AlarmStatus scripts entirely -- they cannot be written
without it. `Alarms.CurrentAsync()` closes it.
The data was already local: the script runs inside its own Instance Actor's
context, so this is a LOCAL Ask -- the same mechanism attribute reads use, no
cross-cluster hop. A dedicated GetAlarmSnapshotRequest/Response is used rather
than reusing DebugSnapshotRequest, which would materialise every attribute
value on every alarm poll; both are served from the same
BuildAlarmStatesSnapshot(), so the script view and the operator's Debug View
can never disagree.
Deliberate shape decisions:
- NOT scope-prefixed, unlike Attributes. Alarm identity is not a
scope-relative attribute name (computed alarms are keyed by configured
name, native conditions by a source-supplied reference), so prefixing
would hand a composed script a silently truncated list.
- Read-only. Native alarms are a read-only mirror of the source (no
ack-back), so no acknowledge/shelve operation is exposed.
- Placeholder rows are NOT pre-filtered: a caller must be able to tell
"binding configured and quiet" from "binding unknown". The documented
filter is `Active && !IsConfiguredPlaceholder`.
- ScriptAlarm lives in Commons so the runtime accessor and the compile-only
surface project to the SAME type -- a script binding at the design-time
gate binds identically at the site. Condition is the authority for
active/acked/severity, so one filter expression works across computed and
native alarms.
Mirrored on BOTH design-time surfaces. ScriptCompileSurface is covered by the
reflection parity guard (AlarmsAccessor added to its mirror pairs). The Central
UI Test-Run SandboxScriptHost is the third, hand-maintained mirror that the
parity test cannot reach (Central UI does not reference Site Runtime); without
it the design page would false-flag CS1061 on scripts the deploy gate accepts.
It throws a labelled ScriptSandboxException at run time -- there is no central
route to per-instance alarm state, and returning an empty list would read as
"nothing is in alarm", which is worse than an error.
ScriptTrustPolicy needs NO change, and the reason is structural rather than
incidental: the trust boundary is a deny-list over API roots, not an allow-list
of context members. A test pins that no ForbiddenScopes entry prefixes the
Commons script-surface namespace, so a future deny-list entry cannot silently
make ScriptAlarm untouchable.
Tests: 6 accessor cases (Ask contract, full native projection incl. AckTime,
unacked, computed-alarm derivation, placeholder visibility, scope-independence),
2 InstanceActor snapshot cases incl. equality with the Debug View row set, the
full MES script shape compiling against ScriptCompileSurface, 2 trust cases,
and a sandbox diagnose-clean case reading every projected ScriptAlarm field.
|
||
|
Joseph Doherty
|
01bcca992c |
feat(alarms): thread an additive AckTime through the native-alarm mirror
MES alarm-status API §6.4 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 task 1). MES needs a real AckDT for a triggered alarm, and the mirror
carried acked-vs-unacked but never WHEN. AckTime now rides the whole path:
DCL transition -> AlarmStateChanged -> gRPC AlarmStateUpdate -> site SQLite.
Stamping rule, identical on both protocols: non-null ONLY while the condition
is active AND acknowledged. That single predicate yields all three required
behaviours -- null while unacked, cleared on re-raise (a re-raise arrives
unacknowledged), and no phantom ack on a return-to-normal. The last one is
load-bearing for MxGateway, which maps INACTIVE to Acknowledged = true; without
the active check every clear would claim an ack the system never observed.
Provenance is honest, never fabricated:
- OPC UA A&C supplies a TRUE ack instant, so we now select it:
AcknowledgeableConditionType/AckedState/TransitionTime at SelectClause
index 18, APPENDED so the positional reads at 0-17 keep their meaning.
Servers that omit the field fall back to the event's own Time.
- MxAccess Gateway supplies none, so the ack transition's own timestamp is
used -- accurate to when the system SAW the ack. An ACTIVE_ACKED
re-subscribe snapshot restores one from LastTransitionTimestamp rather
than dropping it.
The decision lives in pure mappers (Opc/Mx AlarmMapper.DeriveAckTime), so it is
unit-tested with no live server or gateway.
Additive-only throughout: init-only property on AlarmStateChanged, trailing
optional positional on NativeAlarmTransition (all 14-arg call sites untouched),
proto field 24 (never reusing a number) regenerated via docker/regen-proto.sh
sitestream with the csproj diff verified empty.
Persistence rides native_alarm_state's existing metadata_json blob rather than
a new column -- deliberately. That table is RegisterReplicated in
SiteLocalDbSetup and LocalDb builds its CDC triggers from the column list at
registration time, so an additive JSON property changes no schema, no triggers
and no replication contract; metadata_json is exactly the extension point UA4
introduced for this. Rows written before the field deserialize it as null.
Tests: 4 OPC UA + 6 MxGateway mapper cases, 3 NativeAlarmActor (emit,
failover rehydrate, pre-AckTime row), 1 proto round-trip incl. the null case,
4 Commons additive/back-compat. The OPC UA SelectClause count lock-in moves
18 -> 19 with an index-18 assertion -- intended, the clause is appended, which
is precisely what that guard exists to make visible.
|
||
|
Joseph Doherty
|
0c9dffed78 |
fix(transport): flush created-site ids before the connection pass — create-missing import failed FK 547 on real SQL Server
Caught live 2026-08-01 importing a bundle into the empty env2 cluster: a create-missing Site has Id == 0 until SaveChanges on a relational provider, and ApplyDataConnectionsAsync stamps DataConnection.SiteId as a raw scalar (no navigation, no EF fix-up), so the insert violated FK_DataConnections_Sites_SiteId. Every importer integration test runs on the EF in-memory provider, which assigns ids eagerly on AddAsync — the exact masking the in-code comment predicted. Fix: one SaveChangesAsync between the site pass and the connection pass, riding the same outer transaction (all-or-nothing preserved; the failed live import rolled back cleanly). Regression test runs the create-missing path on SQLite (CreateMissingSiteRelationalTests) — red without the fix, green with it. |
||
|
Joseph Doherty
|
316153dc98 |
feat(centralui): hide OffsetPager on a single page, opt-in — NotificationReport enabled (M10 residual)
DECISION: a pager whose only two controls are permanently disabled is noise, so
the conventional behaviour ships — the bar is hidden when the result set is a
single page. NotificationReport, which raised the finding, opts in.
RATIONALE for opt-in rather than default-on. OffsetPager is shared, and its
summary span ("Page N of Y · T total") doubles as the ONLY total-count readout
on ConfigurationAuditLog — auto-hiding there would silently delete the result
count for every query returning under a page, and the existing Playwright
fixture asserts "3 total" on exactly such a query. So the behaviour lives in the
shared component (reusable for the next consumer) behind HideWhenSinglePage,
default false, which keeps every current consumer byte-identical.
The guard fires only when single-page-ness is POSITIVELY established:
PageCount <= 1 AND Page <= 1 AND !HasNextPage. A null TotalCount means the host
cannot count, so the bar renders — the controls never vanish while a further
page still exists.
NotificationReport's own Playwright pagination test seeds 51 rows (2 pages) and
is unaffected. 5 new bUnit tests cover hide-on-single-page, hide-on-empty,
show-on-multi-page, show-when-TotalCount-unknown, and the default-off path.
|
||
|
Joseph Doherty
|
a506b19d17 |
refactor(centralui): migrate TemplateEdit's four page-embedded modals to the DialogService host (M10 residual)
TemplateEdit was the last page still hand-rolling its own modal chrome after
M10 — T34c only tokenized its backdrops. All four member-authoring forms
(Attribute, Alarm, Native Alarm Source, Script) now open through
IDialogService.ShowAsync, so the single DialogHost in MainLayout owns the
backdrop, focus trap, Escape and focus restoration.
Pattern copied from the already-migrated pages (MoveDataConnectionDialog +
Templates' Move/Rename dialogs): each body is its own component beside the page
taking a DialogContext<bool>, owning its form state, rendering validation and
server errors INLINE while staying open, and closing with Close(true) only once
the save succeeded — at which point the page reloads. An inline RenderFragment
would NOT have worked: DialogHost renders the captured fragment in its own tree,
so the page's StateHasChanged could never refresh it.
Persistence deliberately stayed on the page (it owns TemplateService, the
inherited-member rules, and the repository-direct native-source path) and is
reached through an OnSaveAsync delegate returning null on success or the message
to display. Behaviour preserved verbatim, including the List-attribute encode +
Decode round-trip check, the name/trigger-type read-only-on-edit rules, the
duplicate-native-source-name guard, and NormalizeExecutionTimeout.
TemplateScriptDialog keeps all four tab panels mounted (Monaco and the JSONJoy
island must not tear down on tab switch) and hosts the Test Run panel, which
needs the live unsaved editor buffer, so it injects ScriptAnalysisService
directly and cancels an in-flight run on dispose.
Extraction moved markup that three structural source-scanning tests pinned;
all three were repointed at the new files rather than weakened:
- TemplateNativeAlarmSourceEditorTests (+ a new test asserting the form is
host-mounted and the body renders no chrome of its own)
- AttributeListEditorTests (list-editor reveal now in the dialog body)
- TestRunWarningTests (Real I/O warning travelled with the script panel)
Build 0/0; CentralUI.Tests 973/973 green.
|
||
|
Joseph Doherty
|
fdfd5e1b27 |
feat(centralui): TreeView full WAI-ARIA keyboard navigation (M10 residual R7)
TreeView<TItem> handled only Enter/Space on the chevron; the tree itself was unreachable by keyboard. Implements the WAI-ARIA tree pattern: - Roving tabindex on li[role=treeitem] — exactly one node is tabbable, so the whole tree is a single Tab stop. Target resolved per render: last-focused node while still visible -> SelectedKey -> first visible node. Browser focus follows via a per-node ElementReference + FocusAsync in OnAfterRenderAsync, guarded by the same JSException/JSDisconnectedException/InvalidOperationException triple the context-menu focus already used (a no-op under bUnit). - ArrowDown/ArrowUp move between VISIBLE nodes; ArrowRight expands a collapsed branch else moves to first child; ArrowLeft collapses an expanded branch else moves to parent; Home/End jump to first/last; Enter/Space activate through the SAME path a click takes (OnContentClick / OnCheckboxToggle) so selection semantics never diverge between input modes. - ARIA: aria-level, aria-posinset, aria-setsize added; aria-selected now renders true/false on selectable+checkbox trees and stays absent on non-selectable ones. - Event scoping: @onkeydown:stopPropagation on the li, chevron, checkbox and content slot, so nested nodes do not double-handle and consumer controls inside NodeContent keep their own key handling. Browser scroll-on-Space/Arrow is suppressed by a minimal NATIVE inline onkeydown on the root ul, targeted at treeitems only — Blazor's preventDefault directive is all-or-nothing per element, so on the li it would trap Tab and cancel Enter/Space on consumer buttons. No CSP is configured, so the inline handler runs. BuildVisibleNodes() mirrors RenderNode's visibility rules and must stay in step. Tests: 31 new bUnit tests in TreeViewKeyboardNavigationTests; the 47 existing TreeView tests are unchanged and still green. Docs: new keyboard/a11y section in docs/components/TreeView.md. |
||
|
Joseph Doherty
|
8aa6bf2270 |
fix(audit): populate ParentExecutionId on alarm-triggered script runs
M5.4 T4 threaded a `parentExecutionId` parameter through
AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext,
but every call site passed null — so alarm on-trigger runs were silently always
execution-tree roots, contradicting the "tag-cascade coverage is complete"
claim in CLAUDE.md and Component-AuditLog.md.
Source the id where a spawner genuinely exists: a static attribute write issued
by a site script (`Instance.SetAttribute`) or by an inbound API request
(`Route.To(...).SetAttributes(...)`, whose ParentExecutionId was already carried
to the site and then dropped). The id rides site-locally through three additive,
nullable fields — no wire, proto or central schema change:
ScriptRuntimeContext.SetAttribute / RouteToSetAttributesRequest.ParentExecutionId
→ SetStaticAttributeCommand.SourceExecutionId
→ AttributeValueChanged.SourceExecutionId (InstanceActor static-write path)
→ AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext
All four computed trigger types participate. Expression triggers evaluate off
the dispatcher, so the writer of the newest value folded into the snapshot is
captured *with* the snapshot and echoed home on ExpressionEvalResult /
ExpressionEvalFailed — a change arriving mid-flight cannot mis-attribute the
raise.
Deliberately still roots (documented, not deferred): alarms fired by Data
Connection Layer values (external device data has no spawning execution — this
includes the device echo of a script write to a *data-sourced* attribute, so
only static writes cascade), and ScriptActor value-change/conditional/
expression/timer trigger runs (a timer tick has no spawner; a WhileTrue/interval
run has no single identifiable write).
Tests: new SiteRuntime.Tests/Actors/AlarmCascadeParentExecutionTests pins all
three hops — SetAttribute stamps the run's ExecutionId, InstanceActor publishes
it on the change (and publishes null when absent), and ValueMatch/HiLo/
Expression alarms parent the on-trigger run to the writer while a DCL-originated
change leaves it a root.
Docs: CLAUDE.md and Component-AuditLog.md corrected from "complete" to the true
behaviour; Component-SiteRuntime.md gains an "Audit correlation of an on-trigger
run" section with the hop table and the by-design root cases.
|
||
|
Joseph Doherty
|
88638d774a |
feat(cli,management): close area-move and template-folder CLI parity gaps
Two verified-absent parity gaps between the service layer and the CLI / ManagementActor command surface, both left as follow-ups by the 2026-05-11 design plans. (1) area move. AreaService.MoveAreaAsync had existed since the deployment topology page shipped but was reachable only from the Blazor UI. Adds MoveAreaCommand(AreaId, NewParentAreaId?) to Commons, a ManagementActor dispatch arm delegating straight to AreaService.MoveAreaAsync (not-found / self-parent / descendant-cycle / cross-site / name-collision all surface as the standard curated ManagementCommandException failure response; the service writes its own "Move" audit row), and the CLI verb `site area move --id [--parent-id]`. Omitting --parent-id moves the area to the site root, matching the command's nullable NewParentAreaId. The command carries the SAME any-of [Designer, Deployer] gate as CreateArea/UpdateArea/DeleteArea (arch-review C6): re-parenting is the same structural authoring act, exposed on the same two surfaces. Placed under the existing `site area` group rather than a new top-level `area` group, alongside its create/update/delete siblings. (2) template folder verbs. The five folder management commands have been handled by ManagementActor since the folder-hierarchy plan, but the promised CLI surface was never written. Adds `template folder list|create|rename|move|reorder|delete` mapping 1:1 onto ListTemplateFolders / CreateTemplateFolder / RenameTemplateFolder / MoveTemplateFolder / ReorderTemplateFolder / DeleteTemplateFolder. --parent-id is omitted to target the tree root; --direction takes the lowercase literals up/down, validated at parse time by AcceptOnlyFromAmong (same case-sensitive contract as the audit --channel/--kind/--status options). Follow-on updates: the frozen authorization matrix gains its MoveArea entry (reflection-driven, so a missing entry would have failed CI); CommandTreeTests pins both new verb sets plus the omit-parent-id-means-root parse behaviour and registry round-trips; ManagementActorTests covers the MoveArea role gate and the delegate-to-service success/root/cycle/not-found paths; the CLI README and Component-ManagementService.md document the new surface (the latter also gained the previously-undocumented ReorderTemplateFolder); both plan docs' follow-up lines are marked done. |
||
|
Joseph Doherty
|
a26d6ba317 |
feat(central-ui): native-alarm-source CSV import on InstanceConfigure
The Native Alarm Source Overrides card had no bulk affordance — the CSV path shipped CLI-only (instance native-alarm-source import --file), so the UI could only retarget one source at a time inline. Adds a second InputFile on that card, mirroring the attribute importer's UX (hidden input behind a button-styled label, 512 KB cap, success/error alert with the per-line error list, toast). Parsing reuses the SHARED NativeAlarmSourceOverrideCsvParser — the exact parser the CLI uses — and the new pure InstanceConfigure.BuildNativeAlarmSourceCsvImport applies the same batch rules the server enforces in ManagementActor.HandleSetInstanceNativeAlarmSourceOverrides: the source must resolve, must not be template-locked, and may appear at most once; any error rejects the whole file and applies nothing. Semantics match the CLI: merge, not full replace — sources absent from the file keep their existing override, a blank field keeps the inherited value, and an all-blank row clears that source's override (equivalent to the CLI's all-null override row, without leaving a dead row behind). Persistence reuses the inline editor's path: SaveNativeOverride's upsert body is extracted to UpsertNativeOverrideCore (no SaveChangesAsync inside), so the import commits the validated batch in one SaveChangesAsync — no new server method, no duplicated parsing. Tests: InstanceConfigureNativeAlarmCsvImportTests (9) — happy path, blank-field inheritance, all-blank clear, merge semantics, unknown/locked/duplicate source and parser-error rejection, plus structural pins on the InputFile wiring. Docs: Component-CentralUI.md native-alarm-source card gains the import bullet. |
||
|
Joseph Doherty
|
2d03f2d507 |
fix(site-runtime): reconcile artifact deletions on apply — central deletes no longer orphan site rows
The artifact apply (DeploymentManagerActor.HandleDeployArtifacts) was upsert-only, so deleting an external system (or shared script, DB connection, data connection) centrally never removed the site's SQLite row — a deleted external system stayed callable from site scripts forever. Central always ships the COMPLETE set of each artifact class (ArtifactDeploymentService GetAll* snapshots; the wire's presence-tracking wrapper lists preserve null-vs-empty), so the site now applies upsert-then-reconcile: after storing the incoming set, SiteStorageService.DeleteRowsExceptAsync removes any stored row absent from it, per artifact table. A null list still means 'field not shipped' and touches nothing. Runtime cleanup rides along: a reconciled-away shared script is unregistered from the compiled SharedScriptLibrary (a stale delegate would stay callable until restart), and a removed data connection is evicted from the DCL hash cache and its live connection actor stopped via the previously-caller-less RemoveConnectionCommand — both on the actor thread via the extended ApplyArtifactDataConnectionsToDcl message. All four tables are RegisterReplicated, so the deletes reach the standby as ordinary CDC row tombstones. Tests: storage-level reconcile per table (incl. empty-set-deletes-all and idempotency) in ArtifactStorageTests; actor-level pins in DeploymentManagerActorTests (orphan delete, null-set no-op, library unregistration, DCL stop for the removed connection only). Docs: Component-DeploymentManager + Component-SiteRuntime record the full-set/reconcile semantics. |
||
|
Joseph Doherty
|
663d03e89c | Merge branch 'fix/cached-telemetry-drain-hot-loop' | ||
|
Joseph Doherty
|
63c16d6912 |
refactor(comm): retire ClusterClient naming after the gRPC cutover
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports but left the naming behind: `ClusterClientSiteAuditClient` was transport- agnostic and worked unchanged, so it survived the deletion under a name that now describes a transport the repo no longer has. Same for a scatter of doc-comments still framing gRPC as "the new transport" beside an Akka one that is gone. Renames it to `SiteCommunicationAuditClient` (and its test file) and rewrites the stale comments to describe the single transport that exists. Also tightens CLAUDE.md: drops the self-describing directory listing and the 27-component enumeration in favour of the non-obvious parts only. Behaviour-neutral: names and prose only. Recorded as the Phase 4 follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md. |
||
|
Joseph Doherty
|
b3dc17a5fe | Merge feat/site-node-health: serve health on site nodes + shared active-node check (Health 0.3.0) | ||
|
Joseph Doherty
|
1c9159a7ce | fix(tests): qualify Health page type — Communication.Health namespace shadowed it | ||
|
Joseph Doherty
|
5d4853a1f0 |
refactor(health): adopt the shared active-node check (Health 0.3.0)
OldestNodeActiveHealthCheck existed because the shared ActiveNodeHealthCheck
selected by cluster leadership (review 01 [High]): leadership is address-ordered
and diverges from singleton placement after a restart, and during a partition
both sides compute themselves leader so Traefik served both. Health 0.3.0 makes
the shared check age-based — it is now this host's own rule, promoted — so the
private copy is deleted and central registers the shared type.
ActiveNodeEvaluator, the "THE single definition of active node" this repo already
maintained, now delegates to the shared ClusterActiveNode rather than
re-implementing it. That keeps the delivery gate, the heartbeat IsActive stamp,
the inbound-API gate and the /health/active tier on one rule, and it means the
rule is shared with OtOpcUa, which had independently written a third copy.
Communication takes a ZB.MOM.WW.Health.Akka reference for it; the layering
trade-off is recorded at the PackageReference.
SitePairActiveNodeHealthCheck is deliberately KEPT. It is not a duplicate of the
rule — it is a thin adapter over the site's own IClusterNodeProvider, which
scopes to site-{SiteId} and is itself now backed by ClusterActiveNode. Registering
the shared check directly would have to re-derive the site role and would lose the
property that the tier and singleton placement come from the same provider.
Central stays unscoped: every central member competes for one active slot.
No behaviour change on any node — same rule, one implementation instead of two.
Verified: Host.Tests 439/439, Communication ActiveNode 2/2.
|
||
|
Joseph Doherty
|
248676ed16 |
feat(cluster): simultaneous-cold-start split-brain guard (#33)
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true simultaneous cold start (shared site power event) races FirstSeedNodeProcess on both and forms two 1-node clusters that never merge. Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off, guard-off behavior byte-identical). When on: BuildHocon emits an empty seed list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService, registered in both the Central and Site composition roots) picks the join order from ClusterBootstrapGuard's pure decision core — the lower canonical host:port is the founder (self-first, forms immediately); the higher node TCP-probes the founder up to PartnerProbeSeconds and joins peer-first if reachable, else self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes, never re-forms mid-handshake. Review notes carried over: case-insensitive founder tie-break; fail-fast validation of probe timings when enabled; higher-node-cold-start-alone covered by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline both-cold-start-together-form-one-cluster). |
||
|
Joseph Doherty
|
47850c0f53 |
feat(health): serve MapZbHealth on site nodes
Site nodes served no health surface at all — gRPC on the HTTP/2-only listener and
/metrics on the HTTP/1.1 one — so nothing outside the cluster could ask a site
node whether it was ready or which half of the pair was active. The family
overview dashboard probes every instance the same way, and this is the one gap.
Three checks, registered in SiteServiceRegistration.Configure (not Program.cs, so
the composition-root tests that build this graph actually cover them) and mapped
by app.MapZbHealth() on the site's HTTP/1.1 listener (default :8084) alongside
/metrics:
akka-cluster [Ready] the shared AkkaClusterHealthCheck. Also carries the
cluster-view data (leader/memberCount/...) the dashboard
reads, free with ZB.MOM.WW.Health 0.2.0.
localdb [Ready] NEW SiteLocalDbHealthCheck — SELECT 1 through the
registered ILocalDb. Site has no EF context; central's
DatabaseHealthCheck<ScadaBridgeDbContext> is central-only.
Replication state rides along as data ENRICHMENT only:
replication is default-OFF, so failing on it would mark
every correctly-configured node unready.
active-node [Active] NEW SitePairActiveNodeHealthCheck, delegating to
IClusterNodeProvider.SelfIsPrimary. Deliberately NOT
central's OldestNodeActiveHealthCheck: that one calls
SelfIsOldest(cluster) with no role argument and would
compute "oldest" across the wrong member set on a mesh
carrying more than one site.
Anonymous, as central's are: the site pipeline runs no authentication middleware
and has no FallbackPolicy, so nothing extra was needed.
Also bumps ZB.MOM.WW.Health* 0.1.0 -> 0.2.0 (central gains data.leader for free).
Tests: SiteHealthCheckTests builds the REAL site container and activates every
registration through its factory (exact-set names, one tier tag each, resolved
types, central-only checks absent behind a positive control) plus behaviour for
both new checks. SiteHealthEndpointTests boots the real site Program over
WebApplicationFactory and proves the endpoints are MAPPED — without it, deleting
MapZbHealth would leave every registration test green while site nodes 404'd.
Prereq for scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Phase 1.
|
||
|
Joseph Doherty
|
86d129de6c |
fix(dcl): harden endpoint host-rewrite — IPv6 brackets + portless URLs
Code review of
|
||
|
Joseph Doherty
|
a1abbff760 |
fix(dcl): rewrite advertised OPC UA endpoint host to the reachable one
An OPC UA server advertises its base address from its own config, not the route the client took — commonly a 0.0.0.0 wildcard bind or an internal container/NAT hostname. The OPC Foundation session dials EndpointDescription.EndpointUrl verbatim, so a 0.0.0.0 advertisement resolved to the client's own loopback and the connect failed. RealOpcUaClient now swaps the advertised authority for the reachable host/port (ConnectAsync + VerifyEndpointAsync), preserving scheme+path; no-op when already reachable. Surfaced live-gating OtOpcUa v3 (#14). |
||
|
Joseph Doherty
|
0eb44314cb | feat(dcl): add OpcUaReferenceForm.IsDurable — flags bare ns= bindings (v3 cutover #14) | ||
|
Joseph Doherty
|
a5256e9b12 |
chore(comm): delete dead IntegrationCallRequest routing (Gitea #32)
'Pattern 4: Integration Routing' — RouteIntegrationCallAsync → SiteEnvelope(IntegrationCallRequest) → SiteCommunicationActor integration handler — was plumbed end to end but connected at neither end: no producer (zero callers) and no handler (AkkaHostedService never registered LocalHandlerType.Integration). It was an early scaffold the architecture routed around — the brokered External→Central→Site→Central round-trip is served by the Inbound API's routed-site-script path (the RouteTo* verbs, driven from CommunicationServiceInstanceRouter), which is live, tested, and shares IntegrationTimeout. Decision (#32): delete. Removed the IntegrationCall{Request,Response} messages, RouteIntegrationCallAsync, the SiteCommunicationActor receive block + _integrationHandler field + LocalHandlerType.Integration, and the four tests that covered them (2 actor, 2 message-contract, 1 dispatcher- reject, 1 mapper-reject). KEPT IntegrationTimeout — it is the live timeout for the RouteTo* verbs. Updated the exclusion-narrative comments (proto/mapper/dispatcher), design §4, the components doc timeout table, and marked the known-issue RESOLVED. Full solution build clean (0/0); Communication 634 + Host 421 green. Net -172/+20 across 14 files. Not in the gRPC proto (was deliberately excluded there), so no wire-format change. |
||
|
Joseph Doherty
|
8524a7f746 |
test(sms-e2e): valid Twilio SID fixture so the create path passes (Gitea #29)
TestAccountSid was 'ACtest123' (AC + 6 chars). The management create
path validates the Account SID against ^AC[0-9a-fA-F]{32}$
(ManagementActor.cs:2205, added 2026-07-10, commit
|
||
|
Joseph Doherty
|
c4dcd9bc02 |
fix(transport): run bundle import inside the execution strategy (Gitea #28)
BundleImporter.ApplyAsync opened a user-initiated transaction directly,
but the central ConfigurationDb context is configured with
EnableRetryOnFailure. SqlServerRetryingExecutionStrategy rejects
user-initiated transactions, so every bundle import threw on any real
SQL Server ('...does not support user-initiated transactions.'). The
whole Transport suite ran on the in-memory provider (no retrying
strategy, BeginTransaction is a no-op) so it never surfaced.
Extract the transactional apply into ApplyMergeAsync and drive it via
_dbContext.Database.CreateExecutionStrategy().ExecuteAsync, so the
strategy owns the BeginTransaction -> apply -> Commit unit as one
retriable block. The delegate resets per-attempt state (change tracker
+ ApplyMergeAsync rebuilds its own summary/resolution accumulators) so a
retried attempt cannot double-apply; a rollback failure is captured and
surfaced on the BundleImportFailed audit row exactly as before. Post-
commit side effects (ScriptArtifactsChanged publish, session zero/
remove) moved outside the retriable delegate so they run once.
Regression test: BundleImporterRetryingStrategyTests runs the import on
SQLite with a retrying execution strategy (RetriesOnFailure == true,
arming the same guard as production). It fails with the exact production
error on the pre-fix code and passes after. Existing rollback/apply
contracts unchanged (104 integration + 154 unit tests green).
|
||
|
Joseph Doherty
|
7fd5cb2b56 |
feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath. Deleted: - AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests) - ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it) - ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in AkkaHostedService; the RegisterCentralClient message + receive block - CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport coexistence flags; the CentralTransportMode/SiteTransportKind enums gRPC is now the only site↔central transport (site→central CentralControlService via GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default so TestKit command-dispatch suites still construct the site actor without a wired transport; production always injects GrpcCentralTransport. Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator rejects blank entries (role-agnostic), and StartupValidator requires a Site node to list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default, deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used). Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted the audit-push integration relay to an in-process bridge transport. Docs: Component-Communication/Host/StoreAndForward, components/Communication, topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired amendment), and CLAUDE.md transport decisions. Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles. |
||
|
Joseph Doherty
|
86ad4d5c8e |
feat(comm): T1B.3 — central-side gRPC site-command transport seam (default Akka)
Extract the central→site send path in CentralCommunicationActor behind a new ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport (Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's 27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope. - AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim (the _siteClients lookup + ClusterClient.Send with the reply-to sender preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path). - GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through ControlPlaneCredentials, per-command deadlines set EQUAL to today's CommunicationService Ask timeouts (per-command, not per-group: DeploymentState query and TriggerSiteFailover use QueryTimeout; the two parked relays map to QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds; WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout). - SitePairChannelProvider: per-site A/B channel pair with sticky failover (flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover may have run), background failback probe to the preferred node with 1s→60s doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll. Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site lifecycle across refreshes), ResolveDeadline pinned to each command's current Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process TestServers (PSK+header+deadline attached, Unavailable failover + stickiness, failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched (no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests 607 green with Akka default. |
||
|
Joseph Doherty
|
518c699b90 |
feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2)
Refactor SiteCommunicationActor's central→site routing table into one SiteCommandDispatcher — the single routing truth for the 28 migrated commands (IntegrationCallRequest, the dead 29th, stays on the actor and out of the dispatcher). The Akka actor and the new SiteCommandGrpcService both route through one dispatcher instance so the two transports can never drift on where a command goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3); ClusterClient remains the live path. Decisions worth recording: - Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded handlers with the exact same "handler not available" replies; the parked handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton proxy — pinned by a dispatcher test that asserts the target is the parked probe and NOT the dm proxy. - Sender preservation intact. The actor's command handlers became thin DispatchCommand delegations that still Forward (central Ask → reply routes straight back); the existing SiteCommunicationActorTests pass unchanged, which is the regression guard for that plumbing. UnsubscribeDebugView keeps its fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the synthetic UnsubscribeDebugViewAck so a unary RPC still answers. - Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred CommitLeave; the gRPC service returns the ack, then schedules the real Cluster.Leave — so a caller reaching the very node about to leave still gets its ack instead of a broken stream. The actor path keeps today's coupled resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is immaterial). Proven at both levels: a dispatcher test asserts the ack is built before CommitLeave runs, and a TestServer test asserts the recorded seam order is resolve-then-leave. - ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the one-public-ctor invariant and its test stay green. Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality and both failover paths) and SiteCommandGrpcService TestServer tests (auth, readiness→Unavailable, one command per oneof group, failover ordering). Full solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active <Protobuf> item. |
||
|
Joseph Doherty
|
59b13d317b |
feat(grpc): T1B.1 — site_command.proto + SiteCommandDtoMapper + round-trip goldens
Phase 1B's contract slice: the wire shape and the canonical translation for the 28 central→site commands that leave ClusterClient. No behaviour changes yet — SiteCommunicationActor and CentralCommunicationActor are untouched; the dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this. Protos/site_command.proto (package scadabridge.sitecommand.v1, service SiteCommandService): six domain RPCs, each with a `oneof` request/reply envelope. The grouping is what carries deadline policy — every command inside a group shares a CommunicationOptions timeout class today, so one RPC per group keeps the deadline choice in one place on the client and one dispatch switch on the server, while the oneof keeps each command individually typed: ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) · ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on SiteCommunicationActor's receive table — is deliberately excluded as dead code (2026-07-22-integration-call-routing-is-dead-code.md). Contract decisions worth knowing: - Nullable COLLECTIONS ride in per-collection wrapper messages (DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs, RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty, and that distinction is live at the site — the same silent-data-loss class the transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty and populated for each. - Nullable strings use the empty-string-means-null convention already set by AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's TargetValueEncoded, where "wait for the empty string" is a real target, so it carries a StringValue wrapper. Both behaviours are asserted, not assumed. - Nullable enums ride in one-field messages (proto3 enums have no presence and no stock wrapper). Enum translation is an explicit switch in both directions — never by ordinal — so reordering a C# enum cannot re-map the wire; every wire enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default rather than faulting a command from a version-skewed peer. - New LooseValueCodec carries the surviving `object?` members (script params and return values, attribute values, tag read/write values) as a type-tagged union so a boxed value keeps its runtime CLR type, as it does today under Akka's type-preserving JSON serializer. Dates ride as invariant round-trip strings, not Timestamp, which would silently normalise away DateTime.Kind and DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set falls back to JSON and is documented as CLR-type-lossy. - DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather than reusing sitestream's AlarmStateUpdate, which flattens values to display strings — right for a live stream, lossy for a snapshot the UI treats as authoritative. The encoder omits an AlarmStateChanged.Condition that already equals the record's derived default, so computed alarms round-trip exactly (record equality compares the nullable backing field, not the property). Tests are reflection-driven so the coverage cannot drift: the round-trip theory enumerates the mapper's own ToProto overloads, the envelope guards enumerate the generated oneof descriptors, and a missing golden fails the build. 216 new tests green (Communication 532 total, Commons 684 total, solution build 0/0). Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the <Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that ItemGroup instead of just sitestream, and re-comments idempotently. |
||
|
Joseph Doherty
|
0e162cb250 |
fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression)
Caught on the Phase 1A rig proof: central-a logged only "Now listening on: http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no startup error — the node booted, joined the cluster and served gRPC fine. Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod), so binding only the gRPC port silently deleted it. The site branch has the same shape but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly. Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2) alongside the gRPC port (Http2) in the one ConfigureKestrel call. New Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/ hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped). Why no test caught it originally: unit/E2E tests use TestServer, which never binds real Kestrel. Only a live node exposes the missing listener — which is exactly what the rig proof is for. |
||
|
Joseph Doherty
|
33b15f10a4 |
feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3)
Introduce ICentralTransport as the site->central choke point inside SiteCommunicationActor. The seven site->central sends (notification submit/ status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now delegate to an injected transport instead of owning ClusterClient.Send inline. - AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path, including the exact sender-forwarding that routes central's reply straight back to the waiting Ask. Default when no transport is injected -> behaviour unchanged, existing SiteCommunicationActorTests pass as-is. - GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with sticky failover + background failback (1s-doubling-cap-60s), PSK + x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring today's Ask timeouts. Cross-node retry ONLY on provably-unsent connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget. - StaticSitePskProvider: site's single own-key provider (fail-closed). - CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints (validator: required when transport=Grpc). Host selects the impl; the ClusterClient is created only on the Akka path. Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over in-process TestServer (failover flip, sticky, failback, PSK+header, deadline, no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391 green; the three above-seam suites pass unmodified. |
||
|
Joseph Doherty
|
780bb9c369 |
feat(grpc): host CentralControlService on the central node (T1A.2)
Central now ALSO listens for the seven site→central control messages over gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet — sites keep CentralTransport=Akka (T1A.3's job); central simply starts also accepting. - CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the SAME in-process message the ClusterClient path carries, Asks the existing CentralCommunicationActor (zero handler-logic changes), encodes the reply via the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady — Unavailable until AkkaHostedService hands the actor over. Heartbeat stays fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status mapping is retry-aware: Unavailable (never dispatched, safe to cross-node retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere). - CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site (verify the Bearer token against the key for the site in the required x-scadabridge-site header, via ISitePskProvider) where a site verifies its one own-key — a genuinely different model. Fail-closed on every branch: missing or blank header, unresolvable key, and mismatched token all → PermissionDenied, never pass-through. One public constructor only (the explicit-prefix ctor is internal), pinned by a reflection test — a second public ctor makes Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate. - Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort (default 8083, symmetric with sites), mirroring the Site branch. Additive to central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with MapGrpcService. Port range-validated by NodeOptionsValidator. - Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes so a later task can exercise it. Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor + real service over a stub actor) proves auth positives/negatives are distinguishable and covers unary + the ingest bridge shapes; the interceptor is registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests (Communication.Tests, TestKit) covers the readiness gate, fire-and-forget heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active <Protobuf> item. Communication.Tests (356) + Host.Tests (384) green. |
||
|
Joseph Doherty
|
d7455577a8 |
feat(grpc): central_control.proto + DTO mapper for the 7 site→central control RPCs (T1A.1)
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the seven messages SiteCommunicationActor forwards to /user/central-communication. This lands the contract and its mapper only — hosting (T1A.2) and the site-side transport seam (T1A.3) follow. `Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service CentralControlService) declares SubmitNotification, QueryNotificationStatus, IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and Heartbeat. Note the direction is the inverse of SiteStreamService: here the site dials and central serves. Decisions worth recording: - The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch / CachedTelemetryBatch / IngestAck rather than redeclaring them. The site telemetry actor already builds those messages, so a second copy would fork one wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc stay off-wire exactly as they are today. - Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never surface a fault onto the heartbeat timer path. - The three NULLABLE SiteHealthReport collections travel inside single-field wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto / NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but null and empty genuinely differ here — SiteHealthCollector emits `ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads null as "not reported", not as "reported empty". Same reasoning drives the BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected, LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null is not zero/false. - ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected onto it, and the decoder resolves anything unknown to ConnectionHealth.Error. An unrecognised connection state must not render as "healthy". - Guid? execution ids travel as "D" strings with empty meaning null; a malformed non-empty value throws rather than being laundered into "no correlation". - DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no offset). Lossless in practice — every producer stamps UTC — and documented + asserted rather than left implicit. SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously asserted such a method "would be dead code"; that held only while ClusterClient was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not SiteCallOperational) to central. Comment corrected alongside. Golden tests round-trip every message through a real protobuf encode/decode — DTO → proto → bytes → proto → DTO — with a fully-populated case and a null/empty/minimal case for every optional member. Verified to have teeth by mutation: dropping a scalar, collapsing an empty nullable collection to absent, and nulling a gauge each fail a test. Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the linux_arm64 Docker image); no active <Protobuf> item is committed. docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream| centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a comment, so it no longer depends on there being exactly one Protobuf line, and it restores the csproj verbatim on every exit path. |
||
|
Joseph Doherty
|
228ff8b428 |
fix(grpc): one public constructor on ControlPlaneAuthInterceptor — two made the gate inert
Caught by the Phase 0 live gate, not by the suite. Grpc.AspNetCore registers the interceptor BY TYPE, and InterceptorRegistration.GetFactory() throws "Multiple constructors accepting all given argument types have been found" when more than one public constructor is applicable. The interceptor had two: the DI one and a prefix-set overload added for later phases. The failure mode is nasty. The throw happens inside the interceptor pipeline on every call, so nothing fails at startup — the site node boots, joins, reports healthy. Every gated call then dies with Unknown / "Exception was thrown by handler", which reads as a handler bug rather than an auth bug. And it fails OPEN in the sense that matters least and closed in the sense that matters most: no call is ever authorized, but no call is ever correctly REFUSED either, so the rig showed identical errors for a correct key, a wrong key and no key at all. Live evidence, site-a: three PullAuditEvents calls, three identical InvalidOperationExceptions in the node log. Fix: the prefix-set constructor is internal (Host.Tests already has InternalsVisibleTo). Later phases extend DefaultGatedPrefixes rather than adding a second public registration shape. Why the tests missed it, and what changed: ControlPlaneAuthEndToEndTests registered the interceptor with AddSingleton alongside AddGrpc, so DI handed back the instance and Grpc.AspNetCore's activation path — the thing that throws — never ran. The harness now registers exactly as Program.cs does, by type and not in DI. Plus a direct reflection assertion that the type has exactly one public constructor, since that is the real invariant and it is cheap to pin. |
||
|
Joseph Doherty
|
2ee84af1c0 |
feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
|
||
|
Joseph Doherty
|
c8e2f4da02 |
feat(cluster): site-pair manual failover relayed from the central UI (Task 10)
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.
- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
SiteCommunicationActor cannot reference Host, and the two definitions must not
drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
site singletons are placed on the former, so the base role would move the wrong
node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
never fail over a site the operator did not select), refuses when there is no
peer, and reports a fault as an ack rather than throwing into supervision --
a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
confirmation deliberately does NOT claim the admin's page will disconnect --
it won't, and crying wolf there devalues the central warning that is real. A
site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
dead-letters and the Ask times out, reported as "site did not respond". That
is the honest outcome; documented on the contract.
Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
|
||
|
Joseph Doherty
|
caf14a3e03 |
feat(ui): admin manual-failover control on the health page
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles / SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without standing up the whole dashboard's DI graph. - Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is intentionally all-roles, so the gate belongs on the control, not the page. - Disabled with an explanatory title when the pair has no online standby; the authoritative guard remains server-side against live cluster membership. - Confirmation dialog (IDialogService, the page idiom) warns that singletons hand over, in-flight work on the active node is interrupted, and THIS PAGE will disconnect and reconnect against the new active node -- Traefik routes the UI to the node being restarted, so a working failover otherwise reads as a crash the admin caused. - A refused failover (service returns null) surfaces the refusal; the UI never reports a failover that did not happen. 7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a cascading AuthenticationState (the app supplies it from the layout), and BunitContext pre-registers a placeholder IAuthorizationService that throws on policy evaluation -- both handled the same way NavMenuTests documents. Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual Failover section) and docker/README.md. |
||
|
Joseph Doherty
|
f679d5c749 |
feat(cluster): manual central failover service — graceful Leave of the oldest Up member
Admin-triggered failover of the central pair. IManualFailoverService is declared in CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService implements it in the Host and is registered only in the Central branch. - Leave, never Down: singletons hand over instead of being killed. - Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so the node acted on is exactly the one hosting the singletons (never the leader, whose address-ordered definition diverges from singleton placement after a restart). - Peer guard: returns null when fewer than 2 Up Central members — failing over a lone node is an outage, not a failover. - Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be the one that goes away, and an audit written after could be lost to the shutdown it describes. Best-effort — audit failure never blocks the failover. New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated topology actions are not script trust-boundary crossings, but are exactly what an audit log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds. alog.md §4 updated per the lock-in tests' contract. Both tables were already stale -- the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had 16 -- so they are completed here, not merely appended to. ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer guard refuses with a positive still-running assert, dry-run probe does not perturb). |
||
|
Joseph Doherty
|
4a6341d871 |
feat(cluster): self-first seed ordering closes the boot-alone outage gap
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting central-b never came Up (the "registered outage gap"), and self-first ordering closes it using Akka's own protocol. - 6 node appsettings swapped (the *-node-b configs; the -a nodes were already self-first). All 14 shipped node configs now satisfy the invariant. - StartupValidator enforces it at boot, comparing host AND port -- the invariant fails silently when broken, so it is enforced loudly. NOTE: the gitignored deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or that node will refuse to boot. - SelfFirstSeedBootstrapTests: real in-process clusters at production failure-detection timings, incl. a falsifiability control proving the OLD peer-first ordering never forms. Rejected alternative (implemented, measured, discarded): an external self-form timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's join handshake and so cannot tell "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the node's own stale incarnation; a Join(self) during TryingToJoin abandons the in-flight join and forms a second cluster at the same address -- still split after 90s. Docs that claimed self-first ordering was unsafe for simultaneous cold start are corrected: while mutually reachable the InitJoin handshake converges them to one cluster (measured). |
||
|
Joseph Doherty
|
cf3bd52f93 |
feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
|
||
|
Joseph Doherty
|
62cddcfa56 |
fix(audit): let an unresolvable cached row leave the drain queue
A cached-telemetry audit row whose OperationTracking snapshot could not be
resolved was skipped and left Pending. Nothing ever removed it, so the next
drain re-read it, failed identically, and logged again — forever.
The severity was worse than the original write-up said. The queue is read
oldest-first with a fixed BatchSize (default 256), so once a batch's worth of
permanently-unresolvable rows collected at the head, every drain re-read
exactly those rows and NEVER REACHED the newer rows behind them. That is a
permanent stall of the cached-telemetry path, not the "log flood, no data loss"
the issue was first filed as. Measured on the rig at ~2,800 warnings/minute,
surviving both a process restart and a container restart.
Fix: after a grace period (CachedTrackingGraceSeconds, default 300 s) the
operational half is abandoned and the row marked Forwarded. A row with no
CorrelationId is abandoned immediately — it can never resolve.
Marking Forwarded does not drop audit data. That state means "no longer owed by
the drain, still eligible for reconciliation", which is exactly this situation:
ReadPendingSinceAsync covers Forwarded as well as Pending and central dedups on
EventId, so the reconciliation pull still delivers the audit half. What is lost
is the operational (SiteCalls) half, which is unrecoverable anyway once the
tracking row is gone.
Three deliberate boundaries:
- Inside the grace window the row is still retried — a missing snapshot is
normally a brief write race, and abandoning on the first failed lookup would
discard the operational half of every cached call that lost that race.
- A tracking-store THROW never abandons, however old the row: a throw is a
store fault (locked, corrupt, mid-restore), not a verdict about the row.
- Logging is per drain pass, not per row. Deferred rows dropped to Debug —
being inside the grace window is normal operation, not a warning.
CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash was RETARGETED
rather than deleted: it pinned the defect ("skipped and stays Pending"), so its
orphan assertion is inverted and the half that still holds is kept verbatim.
Three tests added, including the starvation regression. Both "must NOT abandon"
tests carry a positive control on the read count, so they cannot pass by virtue
of a drain that never ran.
Verified non-vacuous: with abandonment disabled the two abandonment tests go
red and the two guards stay green. Build 0 warnings; AuditLog 358, Host 330,
SiteRuntime 512, StoreAndForward 130, Communication 312, Commons 684,
Integration 94 — all pass.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
|
||
|
Joseph Doherty
|
166f07fa68 |
chore(localdb): adopt LocalDb 0.1.1 and drop the directory shim
LocalDb 0.1.1 creates the parent directory of LocalDb:Path itself, so the SiteLocalDbDirectory shim this repo carried through Phase 2 is deleted along with its call site. The gap was found here but was never ScadaBridge's alone — every LocalDb consumer had it — so the fix moved to the library. SiteLocalDbDirectoryTests is RETAINED and retargeted rather than deleted with the shim. It was already written against the site registration path, not the mechanism, so it needed only its Ensure() call removed: what a site node requires is that resolving ILocalDb not fail on a fresh machine, regardless of who provides that. Verified it still earns its place — pinned back to LocalDb 0.1.0 it fails inside SqliteLocalDb..ctor -> SqliteConnection.Open(), so it genuinely depends on the library behaviour and not on a coincidence. Also corrects the coverage-split note in StoreAndForwardStorageTests, which asserted that directory creation is "NOT LocalDb's" — true when written, wrong as of 0.1.1. Build 0 warnings; Host 330, StoreAndForward 130, LocalDb integration 20 pass. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
15013156bf |
test(localdb): two-node convergence for config tables + sf_messages
Four scenarios over the real site pair, driven through the REAL SiteStorageService rather than hand-written SQL. The sibling suites had to hand-write SQL because they were specifications written BEFORE the cutover; this suite runs after it, so it can drive the shipped writers — which is what makes the cascade scenario meaningful. - DeployedConfigRow_ConvergesToB_ColumnForColumn: the existing suite asserts config_json only, so a capture that dropped or defaulted any other column would still pass. deployed_at matters most — SiteReconciliationActor's guarded write compares it. - RemovingAnInstance_ConvergesAllThreeCascadeTables: the plan's flagged highest-risk case. RemoveDeployedConfigAsync deletes from three tables in one transaction, the schema has no foreign keys, and CDC ships three independent per-table streams. A dropped delete leaves a permanently stale override or alarm row on the standby, invisible until the instance name is redeployed. Carries a never-removed control instance, without which "the cascade converged" is indistinguishable from "node B lost these tables entirely". - ANativeAlarmBurst_Converges_AndTheOplogDrains: convergence alone would pass if entries replicated but were never acked, and an oplog that only grows trips the caps into a snapshot resync. - RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin: the union-survives property is per-table, and an unregistered table is silently local-only rather than an error, so it is re-proved on the tables the N1 scenario does not touch. Non-vacuity verified as the plan requires: with the eight Phase 2 RegisterReplicated calls commented out in SiteLocalDbSetup, all four go red (4 failed / 0 passed); restored, 20/20 pass across the three LocalDb suites. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
605e56829e |
chore(config): retire ReplicationEnabled, make legacy db paths migration-only
LocalDb Phase 2 deleted the bespoke replicators, so three config keys changed meaning or died outright: - ScadaBridge:StoreAndForward:ReplicationEnabled is fully dead. Deleted the property, its 10 config entries, and the 5 test references. - SqliteDbPath / SiteDbPath are now migration-only: they name the legacy files SiteLocalDbLegacyMigrator drains at boot, not live databases. Both mandatory rules are relaxed accordingly (StartupValidator's Site-only Require, and the S&F validator's non-empty rule) — an absent value now means "nothing to migrate", so an already-migrated node can drop the key. DatabaseOptions- Validator still rejects a present-but-blank value. - SiteRuntime:ConfigFetchRetryCount's only reader was SiteReplicationActor. Deleted with its validator rule. The two path keys stay present in every config, now with a comment explaining why: removing them would strand un-migrated data on a node that has not yet started once. Both relaxations are pinned by the inverse of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified to fail with the old rule restored. Note: deploy/wonder-app-vd03/appsettings.Site.json is under a gitignored deploy/ tree, so its edit is local-only and must be repeated on the box. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
037798b367 |
feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators
Tasks 14, 15 and 16, landed as ONE commit. PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both); DeploymentManagerActor Tells message types declared in ReplicationMessages.cs (Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any ordering leaves a broken intermediate. Combining them also strengthens the invariant Task 14 already stated for itself — the two mechanisms never both run, and never neither. Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site config tables. notification_lists and smtp_configurations are deliberately NOT registered — permanently empty by design, so registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Migrate stays the LAST call in OnReady, after all registrations, so migrated rows enter the oplog through live capture triggers. Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService, StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is not merely unused but unsafe to keep: a mass DELETE on a now-replicated table would be captured and shipped to the peer. Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor). The positional-argument hazard the plan flagged was real: removing DeploymentManagerActor's optional IActorRef? replicationActor shifted 6 trailing optionals, and 4 test call sites bound the wrong arguments with no compile error at some positions. Converted them to named arguments where possible — Props.Create builds an expression tree, which rejects out-of-position named args, so the rest are padded positionally with a comment saying why. The Task 7 'not yet registered' test was INVERTED rather than deleted, and is exact in both directions: too few means a table silently stops replicating, too many means the SMTP tables leak. Added a separate security-named test for those two, and a composite-PK test (LWW keys on the full PK, so a truncated key set would collapse distinct rows). The convergence suites now get their registrations from the real OnReady — their temporary harness registration is deleted, so they prove the cutover rather than agreeing with themselves. Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb integration 16 — all pass, 0 failures. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
79ce51612e |
test(site): pin the active-node notification-config purge; scope the guarded write
Task 12 + Task 13. No production behaviour change in either. Task 12: DeploymentManagerActor.HandleDeployArtifacts already purges notification_lists and smtp_configurations on every artifact apply, but nothing pinned the actor's CALL to it — ArtifactStorageTests covers the storage method only. Task 15 deletes SiteReplicationActor's copy, making this the sole remaining call site, and Task 16 edits this actor's wiring; dropping the call would leave plaintext SMTP passwords on disk with every suite still green. Verified red-first by commenting the call out: the pin fails with that message. Task 13: StoreDeployedConfigIfNewerAsync STAYS. Re-verified both callers — SiteReplicationActor:375 (dies at Task 15) and SiteReconciliationActor:166 (survives). Reconciliation is a per-node startup self-heal against central whose fetch races real deploys, so the deployed_at guard still does real work there. Doc comment rewritten to say so, and to warn against porting the guard onto the replication path where it would fight the HLC rather than help it. Also corrected a stale 'guarded standby write' section header in the tests. Re-ran the Task 13 step 2 scope check: ConfigFetchRetryCount's only production reader remains SiteReplicationActor:157, so its option + validator rule stay until Task 17, after Task 15 deletes the actor. Verified: build 0 warnings; SiteRuntime 533, Host 329, StoreAndForward 153, LocalDb integration 16 — all pass. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
c56bf4ae65 |
test(localdb): port resync + config replication intents as CDC convergence specs
Task 11. Companion to Task 10, covering the intents held by SiteReplicationActorTests and SfBufferResyncPredicateTests. N1 Critical is re-expressed, not ported literally (D2). SfBufferResyncPredicate existed because ReplaceAllAsync was a destructive DELETE-then-INSERT, so a wrong-direction resync wiped a live buffer and the code needed an oldest-Up predicate to decide authority. LocalDb's snapshot resync merges per row under LWW and never deletes, so this asserts the property the guard protected — no node loses rows to a peer's snapshot — with no directional-authority assertion, because there is no active/standby asymmetry left to enforce. The setup is the wipe scenario made concrete: each node writes rows the other never sees while partitioned, plus a contended row, then they resync. DEVIATION on the zero-fetch assertion. The plan asked for a fetcher test double recording zero invocations; this harness has no actor system, no central and no IDeploymentConfigFetcher in the graph, so the double would record zero calls whether or not the fetch path still existed. An assertion that cannot fail is worse than none, so the test proves the positive half — config reaches B over replication alone — and the file records that the negative half is proved by Task 15 deleting the code and the build still passing. It also records the D1 scope note: SiteReconciliationActor survives and legitimately fetches at startup, so 'the standby never fetches, ever' would be false. Non-vacuity verified by unregistering deployed_configurations: all 4 fail. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
2bbe66311d |
test(localdb): port store-and-forward replication intents as CDC convergence specs
Task 10. Specifications first, deletion second: the bespoke ReplicationService (explicit Add/Remove/Park/Requeue over Akka) dies at Task 14, so its behaviour is restated here as outcomes the CDC replacement must still deliver. Written in terms of ROWS, not operations — under CDC there is no Add or Park message to observe, only a row that must end up right on both nodes. Not ported: ReplicationOperations_AreDispatchedInIssueOrder. It asserts the mechanism (inline fire-and-forget dispatch), and CDC capture is asynchronous and batched by construction. Its portable content is the ordering OUTCOME — add-then-remove must never converge to present — which is a test here, with that reasoning recorded in the file so it does not read as an accidental drop. DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness rather than duplicating ~150 lines. Phase 1's tests now derive from it and still pass unchanged. The harness registers the Phase 2 tables itself, since production OnReady does not until Task 14; that method is marked for deletion at the cutover, and the 8-table list is written literally so a cutover registering the wrong set fails these tests instead of agreeing with itself. Non-vacuity verified by unregistering sf_messages: 6 of 7 failed. The 7th — the ordering test — PASSED, because an absent row is also what a pair that replicates nothing looks like. Fixed with a control row that must converge in the same window, so the absence is evidence rather than silence. Also corrected two comments from Task 9 that claimed Task 14 makes notification_lists/smtp_configurations replicated. It explicitly does not register them, for the same reason the migrator skips them. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
5ddc7eed6d |
feat(localdb): migrate legacy scadabridge.db config tables into the consolidated DB
Task 9. Seven of the nine site configuration tables are copied out of the legacy scadabridge.db in a single transaction, with one rename at the end: a partial config migration would leave a site node running against half its old configuration, which is worse than failing startup outright. notification_lists and smtp_configurations are deliberately NOT migrated. Both are purged on every deploy and permanently empty by design since the site write paths were removed (2026-07-10), but a pre-fix legacy file can still hold rows — and smtp_configurations.password is plaintext. Task 14 makes these tables replicated, so migrating them would push plaintext SMTP passwords across a channel whose only historical payload was exactly that. The tables are still created; only their historical contents stay behind. Generalizes Task 8's MigrateTable into MigrateFile(many tables, one transaction, one rename), keeping the legacy/current column intersection. Five tests, including a column-parity test asserting each declared column list equals the live SiteStorageSchema's. That mismatch is otherwise invisible: a typo'd column is silently dropped by the intersection, and a missing one silently leaves data behind. Non-vacuity verified twice — once by removing the Migrate call (3 fail), once by wrongly adding notification_lists and smtp_configurations to the table map (the skip test fails). Verified: solution build 0 warnings; Host 329, SiteRuntime 532, StoreAndForward 153 — all pass. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
bdc0dffea2 |
feat(localdb): migrate legacy store-and-forward.db into the consolidated DB
Task 8. Adds ResolveStoreAndForwardPath + MigrateStoreAndForward, wired as a third call in Migrate alongside the two Phase 1 files. Unlike the Phase 1 paths, the store-and-forward default sits INSIDE the data volume, so a real deployment has a real file here holding undelivered messages. This migration genuinely moves data; losing it would discard exactly the buffered calls store-and-forward exists to protect. No id synthesis: sf_messages.id is already a caller-assigned TEXT primary key, so INSERT OR IGNORE is idempotent across a crash-then-rerun. The copy intersects the legacy column set with the current one rather than naming all 16 columns outright. A file from an older build predates execution_id / parent_execution_id / last_attempt_at_ms, and naming a missing column throws 'no such column' — which the existing reader treats as an unrecognised shape and silently discards every row. A required-column (PK) guard keeps that tolerance from degrading into copying NULL-keyed rows. Four tests: copy+rename, crash-before-rename idempotence, the __localdb_oplog assertion that catches migrate-before-register, and the older-column-set case. All four verified to fail without the Migrate call. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
f8aa02e2a9 |
feat(localdb): create config + sf_messages tables in the consolidated DB
Task 7. SiteLocalDbSetup.OnReady now applies SiteStorageSchema and StoreAndForwardSchema alongside the Phase 1 schemas, so the nine site configuration tables and the store-and-forward buffer live in the consolidated LocalDb file. Deliberately NOT registered for replication. The bespoke SiteReplicationActor and the StoreAndForward ReplicationService still own these tables until the Task 14 cutover deletes both and registers them in one commit; registering early would run two replicators over the same rows and let either one's defects hide behind the other's writes. Pinned by two tests through the real composition root: one asserting all twelve tables exist, one asserting ReplicatedTables is EXACTLY the Phase 1 pair. Non-vacuity verified by removing the DDL and observing the failure. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |