merge(agent-a): rows 85+83 — EnsureDclConnections moved into the CreateInstanceActor funnel (self-healed/enabled instances get their DCL connections); ReconcileStartsInstanceTests gated on first instance-count report

This commit is contained in:
Joseph Doherty
2026-08-17 03:52:35 -04:00
10 changed files with 680 additions and 60 deletions
+1 -1
View File
@@ -161,7 +161,7 @@ A separate in-flight set `_subscribesInFlight` tracks the initial `SubscribeAsyn
### Creating a connection
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` during instance deployment:
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` whenever it creates an `InstanceActor` — the config comes from the `Connections` map inside that instance's flattened config, not from the site's `data_connection_definitions` table (which has no production reader). The manager never reads storage and has no independent view of which connections a site should have, so a start path that fails to push is indistinguishable here from a connection that genuinely does not exist: that was register row **85**, where two of the four Instance-Actor start paths omitted the push and every subscribe from the resulting instances was answered `Unknown connection: <name>` forever. The push now lives inside `CreateInstanceActor` on the Site Runtime side. Sends are hash-guarded there, so an unchanged connection produces no message at all:
```csharp
// Commons/Messages/DataConnection/CreateConnectionCommand.cs
+7 -4
View File
@@ -36,6 +36,8 @@ The staggered startup above reads storage exactly once. Anything that writes a `
`DeploymentManagerActor` therefore runs a periodic sweep (`InstanceSweepIntervalSeconds`, default 60; `0` disables) that re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config either starts a missing `InstanceActor` or — when the running one's revision no longer matches the stored one — re-applies the stored config **through the ordinary redeploy path**, so nothing about redeploy semantics is re-implemented. It defers to any in-flight deploy/warm/termination, never starts a disabled instance, stays inert until boot-time creation has finished, makes one repair attempt per (instance, revision) pair so it can never become a restart loop, and is completely silent when there is nothing to repair. Repairs emit Warning `deployment` site events naming the trigger (and, for a stale-revision restart, both revisions). Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
Register row **85** was the same gap one layer down: the sweep started the instance, but the instance came up with **no DCL connection actors**, so every tag subscribe was answered `Unknown connection: <name>`. The connection push (`EnsureDclConnections`) was the *caller's* job and two of the four Instance-Actor start paths omitted it; it now lives inside `CreateInstanceActor`. See *Deployment flow* below and §7 of the design note.
### Actor supervision and lifetimes
| Actor | Kind | Supervises children with | On exception |
@@ -83,10 +85,11 @@ Prior to WP3.1 (2026-08-15) each invocation spawned a short-lived `ScriptExecuti
Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration` to the site singleton. The singleton:
1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped).
2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`.
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
1. Calls `CreateInstanceActor`, which first calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped), then does `Context.ActorOf(props, instanceName)`.
2. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
3. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
> **`CreateInstanceActor` is the single funnel for the connection push too (register row 85)**: the `EnsureDclConnections` call moved *into* `CreateInstanceActor` on 2026-08-17. It used to sit at each caller, and only two of the four callers made it — the boot-time batch and this deploy path. `StartStoredInstance` (row-67 ensure-started and the row-78 sweep) and `HandleEnableResult` did not, so a self-healed or newly-enabled instance came up with no connection actors and every tag subscribe was answered `Unknown connection: <name>` indefinitely. See the *Instance reconcile sweep* section and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
For redeployment (instance already running), the existing actor is stopped and watched: