perf(deploy): flatten-session caching, bulk DeploySiteAsync, paged management queries
This commit is contained in:
Joseph Doherty
@@ -137,6 +137,57 @@ When deploying artifacts (shared scripts, external system definitions, etc.) to
|
||||
- Deployment is performed at the **individual instance level**.
|
||||
- The UI may provide convenience operations (e.g., "deploy all out-of-date instances at Site A"), but these decompose into individual instance deployments.
|
||||
|
||||
### Bulk site deployment (`DeploySiteAsync`, WP2.5)
|
||||
|
||||
`DeploySiteAsync(siteId, user)` deploys every instance at one site in a single
|
||||
operation, surfaced as the `MgmtDeploySite` management command and the CLI
|
||||
`deploy site --site-id`. It is the "deploy all at Site A" convenience above, made
|
||||
first-class — it still decomposes into individual instance deployments, and each
|
||||
instance keeps its own deployment id, revision hash, operation lock, and
|
||||
optimistically-concurrent status record.
|
||||
|
||||
It runs the ordinary deployment pipeline in three phases, of which only the middle
|
||||
one is parallel:
|
||||
|
||||
1. **Prepare (serial).** Validate transition, take the operation lock, flatten +
|
||||
validate, run query-before-redeploy reconciliation, stage the
|
||||
`PendingDeployment`, insert the `InProgress` record. Every step here touches the
|
||||
scoped, non-thread-safe `DbContext`, so the phase is strictly serial. All
|
||||
instances share ONE `FlattenSession`, so a template chain common to N instances
|
||||
is walked once and the session-global queries (shared scripts, schema library,
|
||||
the site's data connections) run once for the batch.
|
||||
2. **Send (bounded parallel).** The `RefreshDeploymentCommand` round-trips run
|
||||
concurrently up to `SiteDeploymentMaxParallelism` (default 4), each under a
|
||||
`SiteDeploymentTimeoutPerInstance` deadline (default 120 s). This phase touches
|
||||
no repository — that is exactly why it is the only phase allowed to run in
|
||||
parallel. Shape mirrors `ArtifactDeploymentService.DeployCoreAsync`.
|
||||
3. **Finalize (serial).** Commit terminal statuses, apply post-success side
|
||||
effects, write audit rows, release each operation lock.
|
||||
|
||||
**Not all-or-nothing.** An instance that fails for any reason (wrong state, failed
|
||||
validation, lock already held by another operation, site round-trip timed out) is
|
||||
reported as a failed row while the rest proceed, and is individually retryable via
|
||||
the ordinary single-instance deploy. This matches the artifact-deployment policy:
|
||||
successful targets are never rolled back because another target failed.
|
||||
|
||||
`DeployInstanceAsync` is composed from the same three phase helpers with a batch of
|
||||
one, so the two entry points cannot drift on deployment identity, idempotency, lock
|
||||
coverage, or optimistic concurrency.
|
||||
|
||||
### Terminal deployment-record retention
|
||||
|
||||
Deployment records are insert-only — one row per deploy attempt — so without a
|
||||
window the table grows for the life of the system and every unfiltered deployment
|
||||
query degrades with age rather than with page size.
|
||||
`TerminalDeploymentRecordRetention` (default **365 days**, deliberately generous
|
||||
because deployment history is operator forensics) bounds it, swept opportunistically
|
||||
and rate-limited on the deployment-list read path.
|
||||
|
||||
Only **terminal** rows (`Success` / `Failed`) are eligible. An `InProgress` row is
|
||||
never purged regardless of age: it is precisely the row the query-before-redeploy
|
||||
reconciliation reads to decide whether a prior deploy actually landed at the site,
|
||||
and expiring it by age would silently disable that idempotency guard.
|
||||
|
||||
## Diff View
|
||||
|
||||
Before deploying, the Deployment Manager can request a diff from the Template Engine showing:
|
||||
|
||||
@@ -87,7 +87,7 @@ Both endpoints honour any site-scope rules attached to the caller's audit role b
|
||||
|
||||
### Templates
|
||||
|
||||
- **ListTemplates** / **GetTemplate**: Query template definitions. `ListTemplates` supports additive offset paging via `Skip` / `Take` (arch-review P2): `Take = null` (the default) preserves the historical unpaged behaviour (unlimited), a set `Take` is clamped to `1..1000` and `Skip` floors at `0`.
|
||||
- **ListTemplates** / **GetTemplate**: Query template definitions. `ListTemplates` supports additive offset paging via `Skip` / `Take` (arch-review P2): `Take = null` (the default) preserves the historical unpaged behaviour (unlimited), a set `Take` is clamped to `1..1000` and `Skip` floors at `0`. **Paging and projection are DB-side (WP2.5):** the handler reads `TemplateSummary` rows (child collections reduced to counts, no script bodies) via `GetTemplateSummariesAsync`, instead of materialising every template's full five-Include child graph and then discarding all but one page in memory. A caller that needs a template's members already fetches it with `GetTemplate`.
|
||||
- **CreateTemplate** / **UpdateTemplate** / **DeleteTemplate**: Manage templates.
|
||||
- **ValidateTemplate**: Run on-demand pre-deployment validation (flattening, naming collisions, script compilation).
|
||||
- **GetTemplateDiff**: Compare deployed vs. template-derived configuration for an instance.
|
||||
@@ -115,9 +115,10 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
|
||||
|
||||
### Instances
|
||||
|
||||
- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2), applied **after** the in-memory site-scope filter so a site-scoped user never sees an out-of-scope instance surface into their page window; same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. (`QueryDeployments` and `ExportBundle` still load the full table internally — a logged scale follow-up, arch-review P2 deferred.)
|
||||
- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2), applied **after** the in-memory site-scope filter so a site-scoped user never sees an out-of-scope instance surface into their page window; same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. (`ExportBundle` still loads the full table internally — a logged scale follow-up, arch-review P2 deferred. `QueryDeployments` was closed by WP2.5, below.)
|
||||
- **CreateInstance**: Create a new instance from a template.
|
||||
- **UpdateInstanceOverrides**: Set attribute overrides on an instance.
|
||||
- **UpdateInstanceOverrides** (`SetInstanceOverrides`): Set attribute overrides on an instance. Every requested attribute is validated against the instance's template up front (unknown or locked ⇒ the whole batch is rejected before any write), then the batch is applied as **one bulk read of the existing override rows plus one commit** (WP2.5). It previously ran K independent read-modify-commit cycles — each re-reading the instance, the template's attributes and the existing overrides, then committing and writing its own audit row — so the all-or-nothing promise was only as good as the pre-validation. It is now a real single-transaction apply, with one audit row summarising the batch.
|
||||
- **MgmtDeploySite** (`SiteId`, WP2.5): Bulk-deploy every deployable instance at one site. Site scope is enforced on the SITE (the command's target); every instance the batch touches belongs to that site by construction. Same `Deployer` authority as `MgmtDeployInstance` — it is N of those, not a new class of privilege — and registered as a long-running command so the caller does not time out mid-batch. Semantics, phasing and failure policy: see `Component-DeploymentManager.md` → Bulk site deployment.
|
||||
- **SetInstanceAlarmOverride** / **DeleteInstanceAlarmOverride** / **ListInstanceAlarmOverrides**: Manage per-instance computed-alarm overrides.
|
||||
- **SetInstanceNativeAlarmSourceOverride** / **DeleteInstanceNativeAlarmSourceOverride**: Retarget or clear a per-instance native alarm source binding, keyed by `SourceCanonicalName` — `ConnectionNameOverride` / `SourceReferenceOverride` / `ConditionFilterOverride` each apply only when non-null. Gated to the **Deployment** role.
|
||||
- **ListInstanceNativeAlarmSourceOverrides** (`InstanceId`): List an instance's native alarm source overrides (read-only).
|
||||
@@ -144,7 +145,9 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
|
||||
|
||||
- **DeployInstance**: Deploy configuration to a specific instance (includes pre-deployment validation).
|
||||
- **DeployArtifacts**: Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections) to all sites or a specific site. The command's `SiteId` is honored (arch-review C2): a value routes to the single-site deploy path (`ArtifactDeploymentService.DeployToSiteAsync`), while `null` deploys fleet-wide (`DeployToAllSitesAsync`). Site scope is enforced — a site-scoped (non-Administrator) Deployer may only target a site within its `PermittedSiteIds`, and may **not** deploy fleet-wide (`SiteId is null` is rejected with `SiteScopeViolationException` → `ManagementUnauthorized`, since `EnforceSiteScope(null)` is a deliberate no-op); fleet-wide deployment requires a system-wide Deployer or Administrator.
|
||||
- **DeploySite** (`MgmtDeploySite`, WP2.5): Bulk-deploy every deployable instance at one site — see the Instances section above and `Component-DeploymentManager.md`.
|
||||
- **GetDeploymentStatus**: Query deployment status.
|
||||
- **QueryDeployments** (`InstanceId?`, `Status?`, `Page`, `PageSize`): List deployment records. **All four arguments are honoured DB-side (WP2.5)** and the result is a `DeploymentRecordSummary` projection. The handler previously ignored `Status`, `Page` and `PageSize` entirely (the CLI had been sending them all along) and loaded the whole insert-only `DeploymentRecords` table; for a site-scoped user it additionally loaded every instance and intersected the two sets in memory. Site scope is now resolved to the set of in-scope instance ids and pushed into the query as an id filter — an **empty** scope stays a real filter (a user permitted no in-scope instances sees nothing), never a no-op. This read path also drives the opportunistic, rate-limited terminal-record retention sweep described in `Component-DeploymentManager.md`.
|
||||
|
||||
### Secured Writes (MxGateway, two-person)
|
||||
|
||||
@@ -259,7 +262,7 @@ The ManagementActor receives the following services and repositories via DI (inj
|
||||
| Section | Options Class | Contents |
|
||||
|---------|--------------|----------|
|
||||
| `ScadaBridge:ManagementService` | `ManagementServiceOptions` | `CommandTimeout` (`TimeSpan`, default 30 s) — Ask timeout the HTTP endpoint applies when forwarding to the `ManagementActor`. A non-positive configured value falls back to the 30 s default. |
|
||||
| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. |
|
||||
| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`, `MgmtDeploySite`); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. |
|
||||
| | | `SecuredWritePendingTtl` (`TimeSpan`, default 24 h) — age after which a `Pending` secured write is transitioned to `Expired` and can no longer be approved/executed; enforced at approve/reject and swept opportunistically on list. A non-positive value disables expiry (arch-review S2). |
|
||||
|
||||
## Dependencies
|
||||
|
||||
@@ -158,6 +158,19 @@ Mirrors `TriggerExpressionGlobals` in the same way. Used by `ValidationService.C
|
||||
|
||||
`CheckExpressionSyntax` memoises its verdict in the Template Engine's process-wide `ScriptCompileVerdictCache`, whose key is the pair **(globals surface, SHA-256 of the code)** — not the code alone. The surface discriminator is load-bearing: a trigger expression vetted against `TriggerCompileSurface` is **not** interchangeable with a `ScriptCompileSurface` script-body verdict (different globals resolve different identifiers), so a code-only key could return a stale "clean" for code never compiled against the caller's surface. Keying by surface makes that cross-surface verdict reuse structurally impossible.
|
||||
|
||||
**Eviction is segmented, never wholesale (WP2.5).** The cache is bounded at two
|
||||
generations of 2048 entries. New verdicts land in the *hot* generation; a hit in
|
||||
*cold* is promoted back into hot; on overflow hot becomes the new cold and only the
|
||||
old cold — the half nothing has touched for a full generation — is dropped.
|
||||
|
||||
The bound must never be enforced by clearing the cache outright. That reads as
|
||||
harmless ("a verdict is cheap to recompute") but is not: a recompute is a fresh
|
||||
Roslyn compile, and every script compile loads an assembly through a
|
||||
**non-collectible `InteractiveAssemblyLoader`**. Bounding that leak is the entire
|
||||
reason this cache exists, so dropping every hot entry at the 4096th distinct script
|
||||
would re-open it for the whole working set at once. Segmenting retains anything in
|
||||
active use across an eviction while keeping the same ceiling.
|
||||
|
||||
#### Parity guard
|
||||
|
||||
A reflection-based parity test in `SiteRuntime.Tests` compares the public member names on `ScriptCompileSurface` against `ScriptGlobals` (and `TriggerCompileSurface` against `TriggerExpressionGlobals`). Any drift between the stub and the real globals causes this test to fail, ensuring the stubs cannot silently fall out of sync.
|
||||
|
||||
@@ -137,6 +137,64 @@ When an instance is deployed, the Template Engine resolves the full configuratio
|
||||
5. Resolve data connection bindings — replace connection name references with concrete connection details from the site.
|
||||
6. Output a flat structure: list of attributes with resolved values and data source addresses, list of alarms with resolved trigger definitions, list of scripts with resolved code and triggers.
|
||||
|
||||
### Flatten-session caching and the graph watermark (WP2.5)
|
||||
|
||||
The flatten above is driven per instance, so an unmemoised implementation re-walks
|
||||
the same template chain — one query per link, plus the compositions of every
|
||||
template it reaches and of every composed chain those reach — once per instance,
|
||||
and re-issues the three session-global queries (shared scripts, the shared-schema
|
||||
library, the target site's data connections) each time too.
|
||||
|
||||
A **`FlattenSession`** memoises all of that for the lifetime of ONE
|
||||
flatten/validate operation. Callers that flatten a batch (`DeploySiteAsync`) pass
|
||||
one shared session; a caller that passes none gets a private single-use session,
|
||||
which still collapses the repeated composed-chain loads inside a single instance's
|
||||
flatten.
|
||||
|
||||
Cache validity is decided by **`ITemplateGraphWatermark`**, a process-wide set of
|
||||
monotonic counters bumped by the configuration-database unit of work — the only
|
||||
place every template-graph writer funnels through (`TemplateService`, the
|
||||
`ManagementActor` native-alarm-source handlers, and the Transport bundle importer
|
||||
all commit via the same `SaveChangesAsync`, and the change tracker is inspected
|
||||
pre-commit to attribute each change to its owning template or instance):
|
||||
|
||||
- a memoised **template** is keyed on `(id, template version)`;
|
||||
- a memoised **chain** is valid only while BOTH the graph's `StructureVersion`
|
||||
(bumped on add/remove, re-parent, or any composition change — i.e. anything that
|
||||
can alter chain MEMBERSHIP) and every member's own version are unchanged. Both
|
||||
halves are load-bearing: structure alone misses an ordinary member edit, member
|
||||
versions alone miss a re-parent.
|
||||
|
||||
Sessions are short-lived by construction, so the design's "template state is
|
||||
captured at the time of flatten" guarantee is unchanged — the session narrows the
|
||||
capture window, it never widens it.
|
||||
|
||||
The watermark is **in-memory and process-local**, deliberately not persisted and
|
||||
not replicated between central nodes. A restart, a failover, or any mutation simply
|
||||
misses and falls back to the authoritative full flatten, so the watermark can only
|
||||
ever cause EXTRA work, never stale work.
|
||||
|
||||
The same watermark backs a **staleness fast path** in `StaleInstanceProbe`: a
|
||||
previously computed revision hash is reused when the instance's version, the
|
||||
structure version, and the version of every template that flatten walked are all
|
||||
unchanged. This matters most where the probe is called per-instance across a whole
|
||||
bundle import or a fleet-wide staleness sweep.
|
||||
|
||||
### Design-time analysis reads
|
||||
|
||||
The design-time checks — acyclicity (`CycleDetector`), naming collisions
|
||||
(`CollisionDetector`) and canonical-name resolution (`TemplateResolver`) — read the
|
||||
whole template graph but write through none of it and read no script bodies. They
|
||||
use `GetAllTemplatesForAnalysisAsync`: `AsNoTracking`, with `TemplateScript.Code`
|
||||
projected away.
|
||||
|
||||
The tracked, body-bearing `GetAllTemplatesAsync` remains the read for the two paths
|
||||
that genuinely need it — the inheritance reconciler (which compares and copies
|
||||
script bodies and writes through the loaded entities) and the flattener (which must
|
||||
observe rows an in-flight bundle import has staged on the shared change tracker).
|
||||
`ReconcileDescendantsAsync` additionally accepts an already-loaded tracked graph so
|
||||
a caller holding one does not force a second load.
|
||||
|
||||
### Native Alarm Source Resolution
|
||||
|
||||
The `FlatteningService` resolves native alarm sources alongside alarms, emitting a `ResolvedNativeAlarmSource` (CanonicalName, ConnectionName, SourceReference, ConditionFilter *(optional)*, and `Source` ∈ `Template` | `Inherited` | `Composed` | `Override`) for each. The resolved set is attached to `FlattenedConfiguration.NativeAlarmSources`.
|
||||
|
||||
Reference in New Issue
Block a user