817a72108efeb0df230716ac9e07cdbb38abccf9
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
8f2c2813ea |
fix(siteruntime): self-healed and newly-enabled instances now get their DCL connections (register row 85)
An Instance Actor started by a self-heal came up with NO DCL connection actors, so every tag it subscribed was answered "Unknown connection: <name>" by the DCL manager indefinitely. Proven live 2026-08-16 by the row-78 close-out probe: the sweep correctly started the replication-seeded instances, then every subscribe failed for 6+ min until a second pair restart. Root cause is narrower than the register assumed. A DCL connection actor is never created from data_connection_definitions -- that table has no production reader at all (GetAllDataConnectionDefinitionsAsync is called only by a test). It exists only as a side effect of starting an instance, from the Connections map inside that instance's own flattened config, via EnsureDclConnections. That ensure was the CALLER's job and only two of the four callers made it: CreateInstanceActorBatch (boot) ensured OK ApplyDeployment (deploy) ensured OK StartStoredInstance MISSING <- row-67 ensure-started + row-78 sweep HandleEnableResult MISSING <- enabling a disabled instance The two misses are the two paths added after the original pair -- the signature of a convention living at call sites instead of in the funnel. It was never replication-specific: the boot load reads only ENABLED configs, so a node booting with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. Fix: EnsureDclConnections moves into CreateInstanceActor, already documented as the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), and the two caller-side calls are removed. Placed before the already-exists guard so the two paths that already called it keep their exact previous semantics. No new message, option or DCL change -- it reuses EnsureDclConnection's existing per-name hash guard, so steady state is silent. Structural rather than another guard: a fifth start path cannot forget. Deliberately NOT done, with rationale in the memo: a sweep pass over data_connection_definitions. It would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. Two residual sibling gaps (connection-only artifact deploy replicating to a peer that never adopts it, and the mirror-image removal case) are recorded in the memo and the component docs as open, not silently skipped. Tests: SelfHealCreatesDclConnectionTests, 7 tests asserting on the CreateConnectionCommand messages sent to a recording DCL stand-in. Verified fail-before against a behavioral revert (call removed from the funnel, restored at the two original sites): 4 fail / 3 pass, the 3 being steady-state silence plus two refactor guards that must pass on both sides. Docs: sweep design memo gains section 7; Component-SiteRuntime, Component-DataConnectionLayer, docs/components/SiteRuntime and docs/components/DataConnectionLayer updated (the deploy-flow step ordering in docs/components/SiteRuntime was stale as written). |
||
|
Joseph Doherty
|
e606b24a55 |
fix(siteruntime): instance reconcile sweep closes config/runtime divergence (register rows 78, 72)
The site's deployed_configurations rows are the durable truth about what should be
running; the set of live Instance Actors is what IS running. Nothing kept the two in
agreement after boot, and nothing surfaced a disagreement.
Row 78 (availability, proven live 2026-08-15) -- deployed_configurations is one of
LocalDb's ten replicated tables, so on a replicated pair a config can appear in a
node's file with no local actor involved at all: the CDC sync applier writes the row
directly. Live sequence: site-a-b's reconcile fetched 4 configs at its solo boot; the
pair was then restarted with a fresh site-a-a file and snapshot replication seeded
site-a-a's table within ~5s -- AFTER its boot-time load had read an empty table, and
invisibly to its own reconcile (central rightly reported "0 fetched", because central's
view of the site was correct). Enabled instances sat dead with their configs present
until yet another restart. Row 67's EnsureInstanceStartedCommand only covers configs
the reconcile itself FETCHES; this is the same gap through a side door.
Row 72 (silent divergence) -- for a STALE ReconcileGapItem the pass writes the newer
config via StoreDeployedConfigIfNewerAsync, a pure file write, while the running
Instance Actor keeps executing the previously compiled config until the next restart
or redeploy. Nothing said "stored revision != running revision".
Both are closed by one periodic sweep owned by the Deployment Manager singleton
(ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0;
0 disables). Each tick re-reads the durable rows off-thread and, on the actor thread,
for every ENABLED stored config either starts a missing Instance Actor (row 78) or --
when the running one's revision no longer matches the stored one -- re-applies the
stored config through the ordinary redeploy path (row 72).
A sweep was chosen over a LocalDb change hook deliberately: a hook covers only the
replication arrival path, and only on the half of the fleet running with replication
enabled, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot
snapshot resync, a hand-restored file, any future path). It is also not available --
ILocalDb 0.3.0 exposes no change-notification or applied-rows callback, so a hook would
mean a new upstream capability in scadaproj plus an in-process bus into this singleton,
a cross-repo change to close a site-local gap.
Row 72's decision is to CONVERGE, not merely warn. The stored row is what central
successfully deployed; an actor serving the config central believes was replaced is the
actual fault, and a surface-only warning leaves the operator to issue by hand exactly
the redeploy the sweep would have performed. Reusing HandleDeploy means the compile
gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm
reset and all-or-nothing-per-instance are the production ones, not a second
implementation -- and a stop still does NOT cancel an in-flight script run, exactly as
for a central redeploy.
Guards, shared with row 67 via the extracted InstanceIsOwnedByADeployPipeline predicate
and the shared StartStoredInstance helper:
- a DISABLED instance is never started (config row, no actor is its correct state);
- the sweep defers to any deploy pipeline owning the instance (compile warm in flight,
predecessor still terminating, deployment awaiting its persistence/init join), so it
can never interleave with a real redeploy -- that window has the config stored and no
actor, which an unguarded sweep would answer by creating a child whose actor name is
still registered;
- inert until boot-time Instance Actor creation has finished, so a tick between
staggered batches does not "heal" instances the boot load is about to create;
- ONE repair attempt per (instance, revision) pair -- this is what stops a periodic
self-heal becoming a restart loop for an actor that dies during init, or for a stored
config the compile gate rejects (a rejection leaves the running instance untouched:
the gate returns before anything is stopped);
- reads never overlap, so a slow read cannot stack ticks into duplicate repairs.
A sweep that finds nothing is completely silent -- no log above Trace, no site event.
A repair emits the same Warning `deployment` site-event pattern as row 67 with the
trigger named ("deployment-manager sweep self-heal") so it is distinguishable from a
startup-reconcile heal; a stale-revision restart names BOTH revisions and a rejected
one adds an Error event. The synthetic redeploy replies to the singleton itself rather
than NoSender, because dead letters are a monitored site health metric.
CreateInstanceActor gains a revisionHash parameter and is the single creation funnel
that records it, so the stale check covers every path that can start an Instance Actor;
a missing entry reads as "unknown", never "stale", so a future path bypassing the funnel
degrades to no-op rather than to spurious restarts.
Fail-before by behavioral revert (sweep timer left unarmed): 5 of 10 new tests fail,
including both live symptoms -- the row-78 repro ("A replication-seeded config for an
enabled instance left no running Instance Actor") and the row-72 repro, whose snapshot
still shows the rev-1 value 111 while the store holds rev-2's 222. The 5 negative
guards (disabled / in-flight redeploy / matching revisions / silent-when-healthy /
interval-zero) are green in both states, as they must be.
SiteRuntime 626/626, Host 505/505, AuditLog 383/383.
Design note: docs/plans/2026-08-15-instance-sweep-self-heal-design.md
|
||
|
Joseph Doherty
|
744ff36a7b | merge: startup reconcile starts the healed InstanceActor (register row 67) — EnsureInstanceStartedCommand to the DM singleton, idempotent, disable-wins, Warning site event on self-heal | ||
|
Joseph Doherty
|
41a5ecf14a |
feat(external-systems): deploy per-system retry policy to sites + CLI/management surface (register row 27)
An external system's MaxRetries/RetryDelay existed on the central entity but never reached a site and had no management surface, so every ExternalSystem.CachedCall buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval 30s ~= 25 min to park) whatever central had configured. The sibling DatabaseConnectionArtifact had always carried its equivalents. The consumption end was already wired: ExternalSystemClient.CachedCallAsync already read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset convention. Only the transport and persistence were missing. Commons - ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params (0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going nullable, so it maps 1:1 onto the entity and the proto3 scalars. - Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following the existing TimeoutSeconds "omit to preserve" convention. Both additive. Central - ManagementActor create maps both; update preserves-if-null. - ArtifactDeploymentService projects them onto the artifact. Wire - site_command.proto ExternalSystemArtifactDto gains max_retries=7 / retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto. Additive: a pre-existing site decodes them as 0/absent, which IS the unset value, so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh sitecommand; csproj Protobuf item re-commented, no message types added or removed. Site - external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing, DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7 appended so existing ones are untouched), apply path, and the legacy migrator's explicit column list all carry them. Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables, so this was checked against the AckTime/native_alarm_state precedent. Real columns are correct here, not a JSON bag: the table has no general-purpose config column (auth_configuration and method_definitions are both semantically specific), timeout_seconds already set the additive-column precedent on this exact table, and sibling database_connections stores the same two values as plain columns. Verified in LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its install script is DROP-then-CREATE, so capture triggers rebuild over the new column set automatically, and SiteStorageSchema.Apply runs before registration on the same boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead of diverging silently. Both-nodes-together is already the standing site-pair rule, so no new operational constraint; recorded in topology-guide.md. Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay, so a bundle import that changed a system's retry budget previewed as "Identical" - harmless while the values were inert, a live behaviour change now. No EF migration: the central ExternalSystemDefinitions columns already exist (InitialSchema). Central UI already had both form fields. Tests: site storage round-trip + unset-default, site apply path, central artifact projection, proto round-trip (via the shared sample), management create/preserve, CLI option parse + alias, and diff reporting. Fail-before verified on the two core tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path). Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI, components/StoreAndForward, components/SiteRuntime, deployment/topology-guide, CLI README (which was also missing --timeout-seconds entirely). |
||
|
Joseph Doherty
|
abf864b4e5 |
docs(siteruntime): record reconcile's runtime heal semantics (register row 67)
Component-SiteRuntime.md gains a Startup Behavior note explaining why storing a config started nothing (the reconcile pass fires after the boot-time SQLite load, and Instance Actors are created only by that load or an explicit deploy/enable) and what the EnsureInstanceStartedCommand path now guarantees: singleton-proxy routing, idempotence, durable-row authority over enabled/disabled, deferral to in-flight deploys, quiet no-op on an unreplicated peer's heal, and the Warning log + `deployment` site event that make the heal visible. The command is also listed under Instance Lifecycle Commands, flagged site-internal and reply-less so it is not mistaken for a central-sent command. docs/components/SiteRuntime.md gets the matching implementation-level note beside the surviving-reconcile paragraph in Standby replication. |
||
|
Joseph Doherty
|
7804fe7958 |
docs: arch-review remediation — component docs sweep, execution log, residuals register
Final consistency sweep per plan §6: verified component docs against shipped WP1-WP3 + adversarial-review-fix state, corrected drift found in SiteRuntime (recursion-exempt run cap, stale ScriptExecutionActor/AlarmExecutionActor references), TemplateEngine (BundleImporter watermark path), DeploymentManager (phase-2 PendingDeployment staging), CentralUI (shared KPI cache, dedup'd alarm poll, render coalescing), StoreAndForward (rate-limited drop logging), and ConfigurationDatabase (documented DbContext-pooling non-adoption). Updated the docs/components/ developer-reference set (SiteRuntime, SiteEventLogging, InboundAPI) to drop the deleted per-run actor classes. Amended one known-issue for the superseding MaxBatchSize:64 read-page pin. Added CLAUDE.md bullets for stream graceful-completion reconnect, the required site audit DB path, honest CLI HTTP timeouts, bulk DeploySiteAsync, and LocalDb 0.2.1. New execution log records the phase→commit map, gate results, adversarial-review tally, the three test-flake root causes, and the nine-item residuals register. |
||
|
Joseph Doherty
|
7b5a5a6f34 |
docs(localdb): phase 2 truth pass across both repos
Normative first: Component-StoreAndForward.md:83 specified the whole chunked, ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It is rewritten rather than removed — the failure modes it reasoned about still exist, they are just bounded differently — and it now states the duplicate-delivery bound explicitly: a message can be delivered twice only when the OLD primary delivered it and the status change had not yet replicated when the gate flipped. One flush interval plus the in-flight ack, and unlike the old model it does NOT grow with backlog depth or with how long a node was absent. The N1 directional-authority and N5 orphan-row hazards are recorded as structurally gone, not merely unguarded. Frame-size known-issue amended: its 2026-06-26 resolution replaced the intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch itself, so the 128 KB Akka frame constraint no longer applies to that hop in any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which batches by ROW COUNT), including that the failure mode differs — an oversized gRPC message is rejected, not silently dropped. Deployment docs gain the two operational constraints that have no home in code: a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat handler that made a mixed-version pair converge went with the replicator, and a mixed pair now diverges silently), and a node offline beyond TombstoneRetention can resurrect deleted rows on rejoin. Both CLAUDE.md files corrected — each still said Phase 2 was NOT started. Definition of done closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures, 0 skips). Two DoD items needed amending rather than ticking: the stale-symbol grep still matches 4 lines, all deliberate comment prose recording what was deleted (a literal zero would delete the explanations that stop the old design coming back), and the live gate has 10 evidence items, not the 9 the checklist claimed. Deletes the phase2 resume-state scratch doc, which said to delete it on landing. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
d61881ffaf |
refactor(site-runtime): excise vestigial site-side notification-list surface — repo, DI registration, dead write paths (arch-review 08 §1.3/#23)
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj |
||
|
Joseph Doherty
|
25bae4e43b |
docs(components): accuracy fixes from deep review (batch 2)
TemplateEngine (alarm-script-ref ordering, native-alarm-sources not in revision hash, composition cycle checks, 9-step pipeline), SiteRuntime (alarm on-trigger scripts run with a restricted context; PreStart seeds children from defaults before overrides arrive), DataConnectionLayer (UnsubscribeAlarmsRequest stashed in Connecting), StoreAndForward (InFlight/ Delivered are dead enum values; notifications can park at 50 retries), ExternalSystemGateway (CachedWrite returns void + enqueues directly; log levels). |
||
|
Joseph Doherty
|
66f0f96328 |
docs(components): verification pass — fix cross-link targets, tag code fences, correct type names
- Fix 15 link-text/target mismatches (ConfigurationDatabase ×8 to Commons, NotificationOutbox ×4, ClusterInfrastructure case, HealthMonitoring, SiteCallAudit) caught by a link-text-vs-target consistency check. - Tag 14 untagged code-fence openers (ASCII diagrams/trees, JSON, HTTP). - Correct 4 type names to match source (ValidationService, HealthReportSender, CentralCommunicationActor, DebugSnapshotCommand set). - Soften Traefik version prose per the style guide. |
||
|
Joseph Doherty
|
8fb90ba400 | docs(components): reference docs batch 2/4 — TemplateEngine, DeploymentManager, SiteRuntime, DataConnectionLayer, StoreAndForward, ExternalSystemGateway |