# Component: Deployment Manager

## Purpose

The Deployment Manager orchestrates the process of deploying configurations from the central cluster to site clusters. It coordinates between the Template Engine (which produces flattened and validated configs), the Communication Layer (which delivers them), and tracks deployment status. It also manages system-wide artifact deployment and instance lifecycle commands (disable, enable, delete).

## Location

Central cluster only. The site-side deployment responsibilities (receiving configs, spawning Instance Actors) are handled by the Site Runtime component.

## Responsibilities

- Accept deployment requests from the Central UI for individual instances.
- Request flattened and validated configurations from the Template Engine.
- Request diffs between currently deployed and template-derived configurations from the Template Engine.
- Send flattened configurations to site clusters via the Communication Layer.
- Track deployment status (pending, in-progress, success, failed).
- Handle deployment failures gracefully — if a site is unreachable or the deployment fails, report the failure. No retry or buffering at central.
- If a central failover occurs during deployment, the deployment is treated as failed and must be re-initiated.
- Deploy system-wide artifacts (shared scripts, external system definitions, database connection definitions, data connection definitions) to all sites or to an individual site on explicit request.
- Send instance lifecycle commands (disable, enable, delete) to sites via the Communication Layer.

## Deployment Flow

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    engineer(["Engineer (UI)"])

    subgraph DMC["Deployment Manager (Central)"]
        step1["1. Request validated and flattened config from Template Engine<br/>(validation: flattening, script compilation, trigger references,<br/>connection binding completeness)"]
        step2{"2. Validation fails?"}
        step2fail(["Return errors to UI, stop"])
        step3["3. Send config to site via Communication Layer"]
        step8[("8. Update deployment status in config DB")]
    end

    subgraph SR["Site Runtime (Deployment Manager Singleton)"]
        step4[("4. Store new flattened config locally (SQLite)")]
        step5["5. Compile scripts at site"]
        step6["6. Create/update Instance Actor<br/>(with child Script + Alarm Actors)"]
        step7["7. Report success/failure back to central"]
    end

    engineer --> step1
    step1 --> step2
    step2 -->|yes| step2fail
    step2 -->|no| step3
    step3 -->|config| step4
    step4 --> step5
    step5 --> step6
    step6 --> step7
    step7 -. "report success/failure" .-> step8

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    class engineer start
    class step1,step5,step6,step7 dec
    class step2,step2fail bad
    class step3 dec
    class step8 proc
    class step4 start
```

### Validation Error Reporting

When step 2 fails, the returned error is a **grouped, capped summary** rather than a flat
semicolon-joined dump (followup #8). `ValidationResult.SummarizeErrors()` (Commons) leads
with the total error count, then lists one line per `ValidationCategory`; within a
category, entity-scoped findings (notably the unbound connection-binding case, which can
produce 50–194 entries for a richly data-sourced instance) are rolled up by **module** —
the attribute's canonical name up to its last dot — with per-module counts, and the breadth
is capped with a `… and N more module(s)` suffix. The complete per-entry list remains on
`ValidationResult.Errors` and is written to the deploy log (`LogWarning`) so operators can
still see every clause when needed. This keeps the UI/CLI failure toast scannable while
preserving full detail for diagnosis.

## Deployment Identity & Idempotency

- Every deployment is assigned a unique **deployment ID** and includes the flattened configuration's **revision hash** (from the Template Engine).
- Site-side apply is **idempotent on deployment ID** — if the same deployment is received twice (e.g., after a timeout where the site actually applied it), the site responds with "already applied" rather than re-applying.
- Sites **reject stale configurations** — if a deployment carries an older revision hash than what is already applied, the site rejects it and reports the current version.
- After a central failover or timeout, the Deployment Manager **queries the site for current deployment state** before allowing a re-deploy. This prevents duplicate application and out-of-order config changes.

## Operation Concurrency

All mutating operations on a single instance (deploy, disable, enable, delete) share a **per-instance operation lock**:

- Only one mutating operation per instance can be in-flight at a time. A second operation is rejected with an "operation in progress" error.
- **Different instances**: Operations on different instances can proceed **in parallel**, even at the same site. Each tracks status independently. This supports the bulk "deploy all out-of-date instances" operation efficiently.

> **`OperationLockManager` scope invariant (recorded decision, arch-review 05).** The
> per-instance locks are **per-node, in-memory**. Mutual exclusion therefore holds only
> because management traffic reaches a single node — the **active** central node, via
> Traefik's active-node routing. A direct mutation sent to the **standby** central port
> would take a *different* in-memory lock and could race the active node. Structurally
> rejecting mutating operations on the non-active node (so the standby can never accept
> them) is **not** owned here — it belongs to the cluster-routing / management-surface
> plans (01/07). The Deployment Manager records the invariant and assumes single-active-node
> delivery; it does not itself enforce node identity.

### Allowed State Transitions

| Current State | Deploy | Disable | Enable | Delete |
|---------------|--------|---------|--------|--------|
| Enabled | Yes | Yes | No (already enabled) | Yes |
| Disabled | Yes (enables on apply) | No (already disabled) | Yes | Yes |
| Not deployed | Yes (initial deploy) | No | No | Yes (removes the orphan record) |

> **Delete from Not deployed:** permitted so an instance that was previously
> undeployed (state `NotDeployed`) can have its record fully removed —
> deployment history, snapshot, attribute/alarm overrides, and connection
> bindings — rather than lingering as an unremovable orphan. There is no live
> site configuration to tear down in this state, so the delete is a
> central-side record cleanup (no site round-trip required).

## System-Wide Artifact Deployment Failure Handling

When deploying artifacts (shared scripts, external system definitions, etc.) to all sites, each site reports success or failure **independently**:

- The deployment status shows a per-site result matrix.
- Successful sites are **not rolled back** if other sites fail.
- The engineer can retry failed sites individually (e.g., when an offline site comes back online).
- This is consistent with the hub-and-spoke independence model — one site's unavailability does not affect others.

## Deployment Status Persistence

- Only the **current deployment status** per instance is stored in the configuration database (pending, in-progress, success, failed).
- No deployment history table — the audit log (via IAuditService) already captures every deployment action with who, what, when, and result.
- **Post-success audit isolation (arch-review 05, Task 17):** the audit write that records a successful deployment runs **outside** the deployment's own success path — an audit-write failure (DB blip, transient error) is logged and swallowed, and NEVER flips a genuinely-successful deployment to failed or triggers a spurious rollback. Audit is best-effort; the deployment's own apply result is authoritative (mirrors the system-wide "audit-write failure never aborts the user-facing action" rule).
- The Deployment Manager uses current status to determine staleness (is this instance up-to-date?) and display deployment results in the UI.

## Deployment Scope

- Deployment is performed at the **individual instance level**.
- The UI may provide convenience operations (e.g., "deploy all out-of-date instances at Site A"), but these decompose into individual instance deployments.

### Bulk site deployment (`DeploySiteAsync`, WP2.5)

`DeploySiteAsync(siteId, user)` deploys every instance at one site in a single
operation, surfaced as the `MgmtDeploySite` management command and the CLI
`deploy site --site-id`. It is the "deploy all at Site A" convenience above, made
first-class — it still decomposes into individual instance deployments, and each
instance keeps its own deployment id, revision hash, operation lock, and
optimistically-concurrent status record.

It runs the ordinary deployment pipeline in three phases, of which only the middle
one is parallel:

1. **Prepare (serial).** Validate transition, take the operation lock, flatten +
   validate, run query-before-redeploy reconciliation, insert the `InProgress`
   record. Every step here touches the scoped, non-thread-safe `DbContext`, so the
   phase is strictly serial. All instances share ONE `FlattenSession`, so a
   template chain common to N instances is walked once and the session-global
   queries (shared scripts, schema library, the site's data connections) run once
   for the batch.
2. **Send (bounded parallel).** Concurrently up to `SiteDeploymentMaxParallelism`
   (default 4), each under a `SiteDeploymentTimeoutPerInstance` deadline
   (default 120 s): stage the `PendingDeployment` row **immediately before** that
   instance's `RefreshDeploymentCommand` round-trip, not upfront in phase 1 —
   `PendingDeployment` carries a 5-minute TTL, and staging every instance in the
   batch serially in phase 1 would burn a chunk of that TTL for the tail
   instances before their round-trip even starts (review fix, commit
   `e0e4b246`). Staging touches the repository but is scoped per-instance, so it
   does not reintroduce serialization. Shape mirrors
   `ArtifactDeploymentService.DeployCoreAsync`.
3. **Finalize (serial).** Commit terminal statuses, apply post-success side
   effects, write audit rows, release each operation lock.

**Not all-or-nothing.** An instance that fails for any reason (wrong state, failed
validation, lock already held by another operation, site round-trip timed out) is
reported as a failed row while the rest proceed, and is individually retryable via
the ordinary single-instance deploy. This matches the artifact-deployment policy:
successful targets are never rolled back because another target failed.

**Cancellation is lock-safe (review fix, commit `e0e4b246`).** A cancelled bulk
deploy no longer leaks the per-instance operation lock of any instance that had
already taken one — a leaked lock is a wedged semaphore that never releases for
the life of the process. Phase 2 does not throw on cancellation; each in-flight
instance is instead recorded with a `Failed` outcome so phase 3 (finalize) still
runs for it. An escape from phase 1 or phase 3 unwinds every not-yet-finalised
entry the same way: `Failed` status plus lock release, so no instance touched by
a cancelled bulk deploy can be left holding its lock.

`DeployInstanceAsync` is composed from the same three phase helpers with a batch of
one, so the two entry points cannot drift on deployment identity, idempotency, lock
coverage, or optimistic concurrency.

### Terminal deployment-record retention

Deployment records are insert-only — one row per deploy attempt — so without a
window the table grows for the life of the system and every unfiltered deployment
query degrades with age rather than with page size.
`TerminalDeploymentRecordRetention` (default **365 days**, deliberately generous
because deployment history is operator forensics) bounds it, swept opportunistically
and rate-limited on the deployment-list read path.

Only **terminal** rows (`Success` / `Failed`) are eligible. An `InProgress` row is
never purged regardless of age: it is precisely the row the query-before-redeploy
reconciliation reads to decide whether a prior deploy actually landed at the site,
and expiring it by age would silently disable that idempotency guard.

## Diff View

Before deploying, the Deployment Manager can request a diff from the Template Engine showing:
- **Added** attributes, alarms, or scripts (new in the template since last deploy).
- **Removed** members (removed from template since last deploy).
- **Changed** values (attribute values, alarm thresholds, script code that differ).
- **Connection binding changes** (data connection references that changed).

## Deployed vs. Template-Derived State

The system maintains two views per instance:
- **Deployed Configuration**: What is currently running at the site, as of the last successful deployment.
- **Template-Derived Configuration**: What the instance would look like if deployed now, based on the current state of its template hierarchy and instance overrides.

These are compared to determine staleness and generate diffs.

The comparison path is **hash-only by design**. When the Template-Derived
Configuration is re-flattened for a comparison (the Deployments comparison page)
or for the Transport stale-instance probe
(`IStaleInstanceProbe.GetCurrentRevisionHashAsync`, used per-instance across a
bundle import), the pipeline is invoked with `validateScripts: false`, which
skips **only** the expensive Roslyn script-compilation stage
(`ValidationService.ValidateScriptCompilation` — a non-collectible assembly load
per script). Structural and semantic validation still run. These read-only paths
need only the flattened config and its revision hash, not a compile, so a script
that does not currently compile still yields a comparison. The **deploy gate
remains the authoritative compile**: instance deployment re-flattens with
`validateScripts: true` (the default), and a script that fails to compile blocks
the deployment (see Site-Side Apply Atomicity).

## Deployable Artifacts

A deployment to a site includes the flattened instance configuration plus any system-wide artifacts that have changed:
- Shared scripts
- External system definitions
- Database connection definitions
- Data connection definitions

System-wide artifact deployment is a **separate action** from instance deployment, triggered explicitly by a user with the Deployment role. Artifacts can be deployed to all sites at once or to an individual site (per-site deployment via the Sites admin page).

Each artifact class in the per-site command carries the **complete system-wide set** (the site-scoped full set, for data connections), and the site applies it as **upsert-then-reconcile**: every incoming artifact is stored, then any stored row absent from the set is deleted — the artifact was deleted centrally. This is how central deletes reach sites; there is no per-artifact delete command. Reconciled removals also clean up runtime state on the site: a removed shared script is unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped (any deployed instance still referencing it sees bad quality — the standard disconnected signal — until redeployed against current central config). A `null` artifact list on the command means "field not shipped" and touches nothing.

Notification lists and SMTP configuration are **not** deployable artifacts — they
are central-only definitions managed by the Notification Service (see
Component-NotificationService.md). Notification delivery happens on the central
cluster, so no notification artifact or SMTP credential is ever distributed to
sites.

## Site-Side Apply Atomicity

Applying a deployment at the site is **all-or-nothing per instance**:

- The site stores the new config, compiles all scripts, and creates/updates the Instance Actor as a single operation.
- If any step fails (e.g., script compilation), the entire deployment for that instance is rejected. The previous configuration remains active and unchanged.
- The site reports the specific failure reason (e.g., compilation error details) back to central.

## System-Wide Artifact Version Compatibility

- Cross-site version skew for artifacts (shared scripts, external system definitions, data connection definitions, etc.) is **supported** — sites can temporarily run different artifact versions after a partial deployment.
- Artifacts are self-contained and site-independent. A site running an older version of shared scripts continues to operate correctly with its current instance configurations.
- The Central UI clearly indicates which sites have pending artifact updates so engineers can remediate.

## Instance Lifecycle Commands

The Deployment Manager sends the following commands to sites via the Communication Layer:

- **Disable**: Instructs the site to stop the Instance Actor's data subscriptions, script triggers, and alarm evaluation. The deployed configuration is retained for re-enablement.
- **Enable**: Instructs the site to re-activate a disabled instance.
- **Delete**: Instructs the site to remove the running configuration and destroy the Instance Actor and its children. Store-and-forward messages are not cleared. If the site is unreachable, the delete command **fails** — the central side does not mark the instance as deleted until the site confirms.

## Dependencies

- **Template Engine**: Produces flattened configurations, diffs, and validation results.
- **Communication Layer**: Delivers configurations and lifecycle commands to sites.
- **Configuration Database (MS SQL)**: Stores deployment status and deployed configuration snapshots.
- **Security & Auth**: Enforces Deployment role (with optional site scoping).
- **Configuration Database (via IAuditService)**: Logs all deployment actions, system-wide artifact deployments, and instance lifecycle changes.

## Interactions

- **Central UI**: Engineers trigger deployments, view diffs/status, manage instance lifecycle, and deploy system-wide artifacts.
- **Template Engine**: Provides resolved and validated configurations.
- **Site Runtime**: Receives and applies configurations and lifecycle commands.
- **Health Monitoring**: Deployment failures contribute to site health status.
- **Transport (#24)**: Does not invoke Deployment Manager directly. Transport-driven template changes propagate to deployed instances through the existing revision-hash drift detection in `DeploymentService.CompareAsync`; the Deployments page surfaces affected instances as stale automatically.
