ScadaBridge/docs/requirements/Component-DeploymentManager.md

Component: Deployment Manager

Purpose

The Deployment Manager orchestrates the process of deploying configurations from the central cluster to site clusters. It coordinates between the Template Engine (which produces flattened and validated configs), the Communication Layer (which delivers them), and tracks deployment status. It also manages system-wide artifact deployment and instance lifecycle commands (disable, enable, delete).

Location

Central cluster only. The site-side deployment responsibilities (receiving configs, spawning Instance Actors) are handled by the Site Runtime component.

Responsibilities

• Accept deployment requests from the Central UI for individual instances.
• Request flattened and validated configurations from the Template Engine.
• Request diffs between currently deployed and template-derived configurations from the Template Engine.
• Send flattened configurations to site clusters via the Communication Layer.
• Track deployment status (pending, in-progress, success, failed).
• Handle deployment failures gracefully — if a site is unreachable or the deployment fails, report the failure. No retry or buffering at central.
• If a central failover occurs during deployment, the deployment is treated as failed and must be re-initiated.
• Deploy system-wide artifacts (shared scripts, external system definitions, database connection definitions, data connection definitions) to all sites or to an individual site on explicit request.
• Send instance lifecycle commands (disable, enable, delete) to sites via the Communication Layer.

Deployment Flow

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    engineer(["Engineer (UI)"])

    subgraph DMC["Deployment Manager (Central)"]
        step1["1. Request validated and flattened config from Template Engine<br/>(validation: flattening, script compilation, trigger references,<br/>connection binding completeness)"]
        step2{"2. Validation fails?"}
        step2fail(["Return errors to UI, stop"])
        step3["3. Send config to site via Communication Layer"]
        step8[("8. Update deployment status in config DB")]
    end

    subgraph SR["Site Runtime (Deployment Manager Singleton)"]
        step4[("4. Store new flattened config locally (SQLite)")]
        step5["5. Compile scripts at site"]
        step6["6. Create/update Instance Actor<br/>(with child Script + Alarm Actors)"]
        step7["7. Report success/failure back to central"]
    end

    engineer --> step1
    step1 --> step2
    step2 -->|yes| step2fail
    step2 -->|no| step3
    step3 -->|config| step4
    step4 --> step5
    step5 --> step6
    step6 --> step7
    step7 -. "report success/failure" .-> step8

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    class engineer start
    class step1,step5,step6,step7 dec
    class step2,step2fail bad
    class step3 dec
    class step8 proc
    class step4 start

Validation Error Reporting

When step 2 fails, the returned error is a grouped, capped summary rather than a flat semicolon-joined dump (followup #8). ValidationResult.SummarizeErrors() (Commons) leads with the total error count, then lists one line per ValidationCategory; within a category, entity-scoped findings (notably the unbound connection-binding case, which can produce 50–194 entries for a richly data-sourced instance) are rolled up by module — the attribute's canonical name up to its last dot — with per-module counts, and the breadth is capped with a … and N more module(s) suffix. The complete per-entry list remains on ValidationResult.Errors and is written to the deploy log (LogWarning) so operators can still see every clause when needed. This keeps the UI/CLI failure toast scannable while preserving full detail for diagnosis.

Deployment Identity & Idempotency

• Every deployment is assigned a unique deployment ID and includes the flattened configuration's revision hash (from the Template Engine).
• Site-side apply is idempotent on deployment ID — if the same deployment is received twice (e.g., after a timeout where the site actually applied it), the site responds with "already applied" rather than re-applying.
• Sites reject stale configurations — if a deployment carries an older revision hash than what is already applied, the site rejects it and reports the current version.
• After a central failover or timeout, the Deployment Manager queries the site for current deployment state before allowing a re-deploy. This prevents duplicate application and out-of-order config changes.

Operation Concurrency

All mutating operations on a single instance (deploy, disable, enable, delete) share a per-instance operation lock:

• Only one mutating operation per instance can be in-flight at a time. A second operation is rejected with an "operation in progress" error.
• Different instances: Operations on different instances can proceed in parallel, even at the same site. Each tracks status independently. This supports the bulk "deploy all out-of-date instances" operation efficiently.

OperationLockManager scope invariant (recorded decision, arch-review 05). The per-instance locks are per-node, in-memory. Mutual exclusion therefore holds only because management traffic reaches a single node — the active central node, via Traefik's active-node routing. A direct mutation sent to the standby central port would take a different in-memory lock and could race the active node. Structurally rejecting mutating operations on the non-active node (so the standby can never accept them) is not owned here — it belongs to the cluster-routing / management-surface plans (01/07). The Deployment Manager records the invariant and assumes single-active-node delivery; it does not itself enforce node identity.

Allowed State Transitions

Current State	Deploy	Disable	Enable	Delete
Enabled	Yes	Yes	No (already enabled)	Yes
Disabled	Yes (enables on apply)	No (already disabled)	Yes	Yes
Not deployed	Yes (initial deploy)	No	No	Yes (removes the orphan record)

Delete from Not deployed: permitted so an instance that was previously undeployed (state NotDeployed) can have its record fully removed — deployment history, snapshot, attribute/alarm overrides, and connection bindings — rather than lingering as an unremovable orphan. There is no live site configuration to tear down in this state, so the delete is a central-side record cleanup (no site round-trip required).

System-Wide Artifact Deployment Failure Handling

When deploying artifacts (shared scripts, external system definitions, etc.) to all sites, each site reports success or failure independently:

• The deployment status shows a per-site result matrix.
• Successful sites are not rolled back if other sites fail.
• The engineer can retry failed sites individually (e.g., when an offline site comes back online).
• This is consistent with the hub-and-spoke independence model — one site's unavailability does not affect others.

Deployment Status Persistence

• Only the current deployment status per instance is stored in the configuration database (pending, in-progress, success, failed).
• No deployment history table — the audit log (via IAuditService) already captures every deployment action with who, what, when, and result.
• Post-success audit isolation (arch-review 05, Task 17): the audit write that records a successful deployment runs outside the deployment's own success path — an audit-write failure (DB blip, transient error) is logged and swallowed, and NEVER flips a genuinely-successful deployment to failed or triggers a spurious rollback. Audit is best-effort; the deployment's own apply result is authoritative (mirrors the system-wide "audit-write failure never aborts the user-facing action" rule).
• The Deployment Manager uses current status to determine staleness (is this instance up-to-date?) and display deployment results in the UI.

Deployment Scope

• Deployment is performed at the individual instance level.
• The UI may provide convenience operations (e.g., "deploy all out-of-date instances at Site A"), but these decompose into individual instance deployments.

Diff View

Before deploying, the Deployment Manager can request a diff from the Template Engine showing:

• Added attributes, alarms, or scripts (new in the template since last deploy).
• Removed members (removed from template since last deploy).
• Changed values (attribute values, alarm thresholds, script code that differ).
• Connection binding changes (data connection references that changed).

Deployed vs. Template-Derived State

The system maintains two views per instance:

• Deployed Configuration: What is currently running at the site, as of the last successful deployment.
• Template-Derived Configuration: What the instance would look like if deployed now, based on the current state of its template hierarchy and instance overrides.

These are compared to determine staleness and generate diffs.

The comparison path is hash-only by design. When the Template-Derived Configuration is re-flattened for a comparison (the Deployments comparison page) or for the Transport stale-instance probe (IStaleInstanceProbe.GetCurrentRevisionHashAsync, used per-instance across a bundle import), the pipeline is invoked with validateScripts: false, which skips only the expensive Roslyn script-compilation stage (ValidationService.ValidateScriptCompilation — a non-collectible assembly load per script). Structural and semantic validation still run. These read-only paths need only the flattened config and its revision hash, not a compile, so a script that does not currently compile still yields a comparison. The deploy gate remains the authoritative compile: instance deployment re-flattens with validateScripts: true (the default), and a script that fails to compile blocks the deployment (see Site-Side Apply Atomicity).

Deployable Artifacts

A deployment to a site includes the flattened instance configuration plus any system-wide artifacts that have changed:

• Shared scripts
• External system definitions
• Database connection definitions
• Data connection definitions

System-wide artifact deployment is a separate action from instance deployment, triggered explicitly by a user with the Deployment role. Artifacts can be deployed to all sites at once or to an individual site (per-site deployment via the Sites admin page).

Each artifact class in the per-site command carries the complete system-wide set (the site-scoped full set, for data connections), and the site applies it as upsert-then-reconcile: every incoming artifact is stored, then any stored row absent from the set is deleted — the artifact was deleted centrally. This is how central deletes reach sites; there is no per-artifact delete command. Reconciled removals also clean up runtime state on the site: a removed shared script is unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped (any deployed instance still referencing it sees bad quality — the standard disconnected signal — until redeployed against current central config). A null artifact list on the command means "field not shipped" and touches nothing.

Notification lists and SMTP configuration are not deployable artifacts — they are central-only definitions managed by the Notification Service (see Component-NotificationService.md). Notification delivery happens on the central cluster, so no notification artifact or SMTP credential is ever distributed to sites.

Site-Side Apply Atomicity

Applying a deployment at the site is all-or-nothing per instance:

• The site stores the new config, compiles all scripts, and creates/updates the Instance Actor as a single operation.
• If any step fails (e.g., script compilation), the entire deployment for that instance is rejected. The previous configuration remains active and unchanged.
• The site reports the specific failure reason (e.g., compilation error details) back to central.

System-Wide Artifact Version Compatibility

• Cross-site version skew for artifacts (shared scripts, external system definitions, data connection definitions, etc.) is supported — sites can temporarily run different artifact versions after a partial deployment.
• Artifacts are self-contained and site-independent. A site running an older version of shared scripts continues to operate correctly with its current instance configurations.
• The Central UI clearly indicates which sites have pending artifact updates so engineers can remediate.

Instance Lifecycle Commands

The Deployment Manager sends the following commands to sites via the Communication Layer:

• Disable: Instructs the site to stop the Instance Actor's data subscriptions, script triggers, and alarm evaluation. The deployed configuration is retained for re-enablement.
• Enable: Instructs the site to re-activate a disabled instance.
• Delete: Instructs the site to remove the running configuration and destroy the Instance Actor and its children. Store-and-forward messages are not cleared. If the site is unreachable, the delete command fails — the central side does not mark the instance as deleted until the site confirms.

Dependencies

• Template Engine: Produces flattened configurations, diffs, and validation results.
• Communication Layer: Delivers configurations and lifecycle commands to sites.
• Configuration Database (MS SQL): Stores deployment status and deployed configuration snapshots.
• Security & Auth: Enforces Deployment role (with optional site scoping).
• Configuration Database (via IAuditService): Logs all deployment actions, system-wide artifact deployments, and instance lifecycle changes.

Interactions

• Central UI: Engineers trigger deployments, view diffs/status, manage instance lifecycle, and deploy system-wide artifacts.
• Template Engine: Provides resolved and validated configurations.
• Site Runtime: Receives and applies configurations and lifecycle commands.
• Health Monitoring: Deployment failures contribute to site health status.
• Transport (#24): Does not invoke Deployment Manager directly. Transport-driven template changes propagate to deployed instances through the existing revision-hash drift detection in DeploymentService.CompareAsync; the Deployments page surfaces affected instances as stale automatically.