docs(siteruntime): record reconcile's runtime heal semantics (register row 67)
Component-SiteRuntime.md gains a Startup Behavior note explaining why storing a config started nothing (the reconcile pass fires after the boot-time SQLite load, and Instance Actors are created only by that load or an explicit deploy/enable) and what the EnsureInstanceStartedCommand path now guarantees: singleton-proxy routing, idempotence, durable-row authority over enabled/disabled, deferral to in-flight deploys, quiet no-op on an unreplicated peer's heal, and the Warning log + `deployment` site event that make the heal visible. The command is also listed under Instance Lifecycle Commands, flagged site-internal and reply-less so it is not mistaken for a central-sent command. docs/components/SiteRuntime.md gets the matching implementation-level note beside the surviving-reconcile paragraph in Standby replication.
This commit is contained in:
@@ -96,6 +96,8 @@ flowchart TD
|
||||
|
||||
> **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.
|
||||
|
||||
> **Startup reconcile heals the RUNTIME, not just the config file (register row 67)**: `SiteReconciliationActor` runs on **every** site node (not a singleton) and performs one best-effort pass shortly after boot — it reports the node's local deployed inventory to central and fetches + stores any config central reports as a gap. That pass fires **after** step 1's deployed-config read has already run, so storing a config started nothing: Instance Actors are created only by the boot-time load above and by an explicit deploy/enable. An instance whose config was missing at boot (lost volume, first boot after a restore, rehomed data dir) therefore stayed **dead until a second restart**, whose load finally picked it up — silently, with nothing surfacing "config stored, actor not running". The pass now raises a fire-and-forget `EnsureInstanceStartedCommand` for every **enabled** instance it healed, routed through the **deployment-manager singleton proxy** so Instance Actors are only ever created under the singleton regardless of which node's pass found the gap. The handler is deliberately advisory: it **no-ops when the actor is already running** (both nodes legitimately raise it for the same instance), **re-reads the durable row** — which is the authority on both the config body and enabled/disabled state, so a disable landing after the fetch still wins and the instance stays stopped — **defers to any in-flight deploy/redeploy** for that instance, preserving the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule (it never applies a config, only starts one already fully persisted), and treats a missing row as a quiet no-op (the normal case on an **unreplicated** site whose peer node healed its own file). A successful reconcile start is **surfaced**, not silent: a Warning log line plus an operational `deployment` site event, because reaching it means the node booted without a config for an instance that should have been running.
|
||||
|
||||
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
|
||||
|
||||
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
|
||||
@@ -124,6 +126,7 @@ flowchart TD
|
||||
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
|
||||
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
|
||||
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.
|
||||
- **Ensure Started** (`EnsureInstanceStartedCommand`, site-internal — never sent by central): advisory, idempotent request to make sure an already-persisted, enabled instance is actually running. Raised by the startup reconcile self-heal; see the note under **Startup Behavior**. Unlike the three commands above it has **no reply** and applies no configuration — it only starts an Instance Actor for a config that is already fully stored, and no-ops when the actor is running, the stored row says disabled, the row is absent, or a deploy for that instance is in flight.
|
||||
|
||||
### Debug Snapshot Routing
|
||||
- Receives `DebugSnapshotRequest` from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as `SubscribeDebugViewRequest`).
|
||||
|
||||
Reference in New Issue
Block a user