ScadaBridge/docs/requirements/Component-DeploymentManager.md

Component: Deployment Manager

Purpose

The Deployment Manager orchestrates the process of deploying configurations from the central cluster to site clusters. It coordinates between the Template Engine (which produces flattened and validated configs), the Communication Layer (which delivers them), and tracks deployment status. It also manages system-wide artifact deployment and instance lifecycle commands (disable, enable, delete).

Location

Central cluster only. The site-side deployment responsibilities (receiving configs, spawning Instance Actors) are handled by the Site Runtime component.

Responsibilities

• Accept deployment requests from the Central UI for individual instances.
• Request flattened and validated configurations from the Template Engine.
• Request diffs between currently deployed and template-derived configurations from the Template Engine.
• Send flattened configurations to site clusters via the Communication Layer.
• Track deployment status (pending, in-progress, success, failed).
• Handle deployment failures gracefully — if a site is unreachable or the deployment fails, report the failure. No retry or buffering at central.
• If a central failover occurs during deployment, the deployment is treated as failed and must be re-initiated.
• Deploy system-wide artifacts (shared scripts, external system definitions, database connection definitions, data connection definitions) to all sites or to an individual site on explicit request.
• Send instance lifecycle commands (disable, enable, delete) to sites via the Communication Layer.

Deployment Flow

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    engineer(["Engineer (UI)"])

    subgraph DMC["Deployment Manager (Central)"]
        step1["1. Request validated and flattened config from Template Engine<br/>(validation: flattening, script compilation, trigger references,<br/>connection binding completeness)"]
        step2{"2. Validation fails?"}
        step2fail(["Return errors to UI, stop"])
        step3["3. Send config to site via Communication Layer"]
        step8[("8. Update deployment status in config DB")]
    end

    subgraph SR["Site Runtime (Deployment Manager Singleton)"]
        step4[("4. Store new flattened config locally (SQLite)")]
        step5["5. Compile scripts at site"]
        step6["6. Create/update Instance Actor<br/>(with child Script + Alarm Actors)"]
        step7["7. Report success/failure back to central"]
    end

    engineer --> step1
    step1 --> step2
    step2 -->|yes| step2fail
    step2 -->|no| step3
    step3 -->|config| step4
    step4 --> step5
    step5 --> step6
    step6 --> step7
    step7 -. "report success/failure" .-> step8

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    class engineer start
    class step1,step5,step6,step7 dec
    class step2,step2fail bad
    class step3 dec
    class step8 proc
    class step4 start

Validation Error Reporting

When step 2 fails, the returned error is a grouped, capped summary rather than a flat semicolon-joined dump (followup #8). ValidationResult.SummarizeErrors() (Commons) leads with the total error count, then lists one line per ValidationCategory; within a category, entity-scoped findings (notably the unbound connection-binding case, which can produce 50–194 entries for a richly data-sourced instance) are rolled up by module — the attribute's canonical name up to its last dot — with per-module counts, and the breadth is capped with a … and N more module(s) suffix. The complete per-entry list remains on ValidationResult.Errors and is written to the deploy log (LogWarning) so operators can still see every clause when needed. This keeps the UI/CLI failure toast scannable while preserving full detail for diagnosis.

Deployment Identity & Idempotency

• Every deployment is assigned a unique deployment ID and includes the flattened configuration's revision hash (from the Template Engine).
• Site-side apply is idempotent on deployment ID — if the same deployment is received twice (e.g., after a timeout where the site actually applied it), the site responds with "already applied" rather than re-applying.
• Sites reject stale configurations — if a deployment carries an older revision hash than what is already applied, the site rejects it and reports the current version.
• After a central failover or timeout, the Deployment Manager queries the site for current deployment state before allowing a re-deploy. This prevents duplicate application and out-of-order config changes.

Operation Concurrency

All mutating operations on a single instance (deploy, disable, enable, delete) share a per-instance operation lock:

• Only one mutating operation per instance can be in-flight at a time. A second operation is rejected with an "operation in progress" error.
• Different instances: Operations on different instances can proceed in parallel, even at the same site. Each tracks status independently. This supports the bulk "deploy all out-of-date instances" operation efficiently.

OperationLockManager scope invariant (recorded decision, arch-review 05). The per-instance locks are per-node, in-memory. Mutual exclusion therefore holds only because management traffic reaches a single node — the active central node, via Traefik's active-node routing. A direct mutation sent to the standby central port would take a different in-memory lock and could race the active node. Structurally rejecting mutating operations on the non-active node (so the standby can never accept them) is not owned here — it belongs to the cluster-routing / management-surface plans (01/07). The Deployment Manager records the invariant and assumes single-active-node delivery; it does not itself enforce node identity.

Allowed State Transitions

Current State	Deploy	Disable	Enable	Delete
Enabled	Yes	Yes	No (already enabled)	Yes
Disabled	Yes (enables on apply)	No (already disabled)	Yes	Yes
Not deployed	Yes (initial deploy)	No	No	Yes (removes the orphan record)

Delete from Not deployed: permitted so an instance that was previously undeployed (state NotDeployed) can have its record fully removed — deployment history, snapshot, attribute/alarm overrides, and connection bindings — rather than lingering as an unremovable orphan. There is no live site configuration to tear down in this state, so the delete is a central-side record cleanup (no site round-trip required).

System-Wide Artifact Deployment Failure Handling

When deploying artifacts (shared scripts, external system definitions, etc.) to all sites, each site reports success or failure independently:

• The deployment status shows a per-site result matrix.
• Successful sites are not rolled back if other sites fail.
• The engineer can retry failed sites individually (e.g., when an offline site comes back online).
• This is consistent with the hub-and-spoke independence model — one site's unavailability does not affect others.

Deployment Status Persistence

• Only the current deployment status per instance is stored in the configuration database (pending, in-progress, success, failed).
• No deployment history table — the audit log (via IAuditService) already captures every deployment action with who, what, when, and result.
• Post-success audit isolation (arch-review 05, Task 17): the audit write that records a successful deployment runs outside the deployment's own success path — an audit-write failure (DB blip, transient error) is logged and swallowed, and NEVER flips a genuinely-successful deployment to failed or triggers a spurious rollback. Audit is best-effort; the deployment's own apply result is authoritative (mirrors the system-wide "audit-write failure never aborts the user-facing action" rule).
• The Deployment Manager uses current status to determine staleness (is this instance up-to-date?) and display deployment results in the UI.

Deployment Scope

• Deployment is performed at the individual instance level.
• The UI may provide convenience operations (e.g., "deploy all out-of-date instances at Site A"), but these decompose into individual instance deployments.

Bulk site deployment (DeploySiteAsync, WP2.5)

DeploySiteAsync(siteId, user) deploys every instance at one site in a single operation, surfaced as the MgmtDeploySite management command and the CLI deploy site --site-id. It is the "deploy all at Site A" convenience above, made first-class — it still decomposes into individual instance deployments, and each instance keeps its own deployment id, revision hash, operation lock, and optimistically-concurrent status record.

It runs the ordinary deployment pipeline in three phases, of which only the middle one is parallel:

1. Prepare (serial). Validate transition, take the operation lock, flatten + validate, run query-before-redeploy reconciliation, insert the InProgress record. Every step here touches the scoped, non-thread-safe DbContext, so the phase is strictly serial. All instances share ONE FlattenSession, so a template chain common to N instances is walked once and the session-global queries (shared scripts, schema library, the site's data connections) run once for the batch.
2. Send (bounded parallel). Concurrently up to SiteDeploymentMaxParallelism (default 4), each under a SiteDeploymentTimeoutPerInstance deadline (default 120 s): stage the PendingDeployment row immediately before that instance's RefreshDeploymentCommand round-trip, not upfront in phase 1 — PendingDeployment carries a 5-minute TTL, and staging every instance in the batch serially in phase 1 would burn a chunk of that TTL for the tail instances before their round-trip even starts (review fix, commit e0e4b246). Staging touches the repository but is scoped per-instance, so it does not reintroduce serialization. Shape mirrors ArtifactDeploymentService.DeployCoreAsync.
3. Finalize (serial). Commit terminal statuses, apply post-success side effects, write audit rows, release each operation lock.

Disabled instances are skipped. A bulk site deploy never touches an instance in Disabled — no flatten, no send, no state change — because deploy is a legal transition out of Disabled and forces Enabled on success, so an unfiltered site-wide push would silently restart every piece of equipment an operator had deliberately stopped; they are reported in the summary's SkippedInstances with the reason "Disabled — bulk deploy preserves disabled state; deploy individually to re-enable", and the single-instance deploy remains the explicit re-enable path.

Reconciliation is bounded per batch. Phase 1 is serial, so an unreachable site would otherwise cost the batch one full communication-layer QueryTimeout (30 s) per instance before a single deploy was sent — hours for a large site, past the server's bulk scheduling deadline. Repeated client timeouts must not re-arm the identical cost on another attempt. Each reconcile in a bulk deploy therefore runs under the short SiteDeploymentReconcileTimeout (default 5 s), and the FIRST one to hit that deadline opens a once-per-batch circuit-breaker that skips the remaining reconciles (one summary log line, not N). Skipping costs nothing: a timed-out reconcile already falls through to a normal deploy on central's recorded state, with the site's own stale-revision rejection as the safety net. The single-instance path is unbounded by this option and keeps the full QueryTimeout.

Not all-or-nothing. An instance that fails for any reason (wrong state, failed validation, lock already held by another operation, site round-trip timed out) is reported as a failed row while the rest proceed, and is individually retryable via the ordinary single-instance deploy. This matches the artifact-deployment policy: successful targets are never rolled back because another target failed.

Cancellation is lock-safe (review fix, commit e0e4b246). A cancelled bulk deploy no longer leaks the per-instance operation lock of any instance that had already taken one — a leaked lock is a wedged semaphore that never releases for the life of the process. Phase 2 does not throw on cancellation; each in-flight instance is instead recorded with a Failed outcome so phase 3 (finalize) still runs for it. An escape from phase 1 or phase 3 unwinds every not-yet-finalised entry the same way: Failed status plus lock release, so no instance touched by a cancelled bulk deploy can be left holding its lock.

DeployInstanceAsync is composed from the same three phase helpers with a batch of one, so the two entry points cannot drift on deployment identity, idempotency, lock coverage, or optimistic concurrency.

Terminal deployment-record retention

Deployment records are insert-only — one row per deploy attempt — so without a window the table grows for the life of the system and every unfiltered deployment query degrades with age rather than with page size. TerminalDeploymentRecordRetention (default 365 days, deliberately generous because deployment history is operator forensics) bounds it, swept opportunistically and rate-limited on the deployment-list read path.

Only terminal rows (Success / Failed) are eligible. An InProgress row is never purged regardless of age: it is precisely the row the query-before-redeploy reconciliation reads to decide whether a prior deploy actually landed at the site, and expiring it by age would silently disable that idempotency guard.

Diff View

Before deploying, the Deployment Manager can request a diff from the Template Engine showing:

• Added attributes, alarms, or scripts (new in the template since last deploy).
• Removed members (removed from template since last deploy).
• Changed values (attribute values, alarm thresholds, script code that differ).
• Connection binding changes (data connection references that changed).

Deployed vs. Template-Derived State

The system maintains two views per instance:

• Deployed Configuration: What is currently running at the site, as of the last successful deployment.
• Template-Derived Configuration: What the instance would look like if deployed now, based on the current state of its template hierarchy and instance overrides.

These are compared to determine staleness and generate diffs.

The comparison path is hash-only by design. When the Template-Derived Configuration is re-flattened for a comparison (the Deployments comparison page) or for the Transport stale-instance probe (IStaleInstanceProbe.GetCurrentRevisionHashAsync, used per-instance across a bundle import), the pipeline is invoked with validateScripts: false, which skips only the expensive Roslyn script-compilation stage (ValidationService.ValidateScriptCompilation — a non-collectible assembly load per script). Structural and semantic validation still run. These read-only paths need only the flattened config and its revision hash, not a compile, so a script that does not currently compile still yields a comparison. The deploy gate remains the authoritative compile: instance deployment re-flattens with validateScripts: true (the default), and a script that fails to compile blocks the deployment (see Site-Side Apply Atomicity).

Preview flatten (no instance row)

IFlatteningPipeline.PreviewAsync(templateId, siteId) flattens a template against a site as if an instance with no overrides and no bindings existed there. It returns the FlattenedConfiguration alone — no validation, no revision hash, nothing persisted, and the instance repository is never touched. It exists for the Central UI's Create Instance page, which needs the instance's module groups (and therefore a real flatten, composed members included) before the instance row is created; see Component-CentralUI.md → Instance Management. It shares the template chain + nested-composition walk with FlattenAndValidateAsync (LoadTemplateGraphAsync), so the two cannot drift, and each call gets a private FlattenSession.

Deployable Artifacts

A deployment to a site includes the flattened instance configuration plus any system-wide artifacts that have changed:

• Shared scripts
• External system definitions
• Database connection definitions
• Data connection definitions

System-wide artifact deployment is a separate action from instance deployment, triggered explicitly by a user with the Deployment role. Artifacts can be deployed to all sites at once or to an individual site (per-site deployment via the Sites admin page).

Each artifact class in the per-site command carries the complete system-wide set (the site-scoped full set, for data connections), and the site applies it as upsert-then-reconcile: every incoming artifact is stored, then any stored row absent from the set is deleted — the artifact was deleted centrally. This is how central deletes reach sites; there is no per-artifact delete command. Reconciled removals also clean up runtime state on the site: a removed shared script is unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped (any deployed instance still referencing it sees bad quality — the standard disconnected signal — until redeployed against current central config). A null artifact list on the command means "field not shipped" and touches nothing.

Notification lists and SMTP configuration are not deployable artifacts — they are central-only definitions managed by the Notification Service (see Component-NotificationService.md). Notification delivery happens on the central cluster, so no notification artifact or SMTP credential is ever distributed to sites.

A site's secrets are a fifth, derived artifact class riding the same command — see Site secrets below.

Per-site artifact revision (2026-09-14)

Site carries two nullable columns — ArtifactRevisionHash (string?, max 100) and ArtifactsDeployedAt (DateTimeOffset?) — recording the artifact set last successfully deployed to that site (migration AddSiteArtifactRevision; see Component-ConfigurationDatabase.md). Both are null on a site that has never had artifacts deployed, including every site immediately after an upgrade that ships this migration — a redeploy landed the columns, it did not retroactively stamp them, so the Sites page reads every site Never deployed until one artifact deploy runs. The two columns are deliberately not transported: like an imported instance landing NotDeployed, they are environment-specific runtime state and the Transport round-trip guard excludes them by name. An import that changes a site's data connections therefore correctly makes the site read out of date until artifacts are redeployed there.

ArtifactRevisionHashService computes a deterministic SHA-256 over a canonical projection of a per-site DeployArtifactsCommand — every artifact class sorted by Name (ordinal), DeploymentId/timestamp excluded (so two deploys of identical definitions hash identically), an external system's MethodDefinitionsJson re-canonicalised with its methods sorted by name, and TimeSpan retry intervals carried as ticks rather than a formatted string. Internal Hashable* records MUST declare their properties in alphabetical (ordinal) order — System.Text.Json serializes in declaration order and does not sort — pinned by a HashableRecords_PropertiesDeclaredAlphabetically test, mirroring RevisionHashService's own determinism contract. Because a data connection's configuration is plaintext in memory (the ciphertext is an EF value converter applied only at rest), a no-op re-save that merely re-encrypts does not change the hash.

ArtifactDeploymentService stamps ArtifactRevisionHash + ArtifactsDeployedAt on the Site row only for a site whose deploy succeeded, on both the all-sites/single-site sweep (DeployCoreAsync) and the per-site retry (RetryForSiteAsync) paths. A failed site keeps its previous stamp (or none), so it keeps reading out of date. Ordering is deliberate: the SystemArtifactDeploymentRecord audit row is persisted first, and the stamp is applied — and saved — afterward, inside its own try/catch. Stamping first would let a stamp-save failure (concurrency conflict, constraint violation) escape uncaught through the record's own SaveChangesAsync and turn a wholly successful deploy into a reported failure; running the record's save first keeps a stamp failure best-effort — logged at Warning, deploy still reports success, the site simply keeps reading out of date until the next deploy.

ArtifactDeploymentService.ComputeCurrentArtifactHashesAsync(sites) returns siteId → hash for what a deploy would send right now — the global artifact queries are hoisted once for the whole batch, mirroring DeployCoreAsync's own hoist. A failing per-site data-connection query aborts the whole batch by design: a partial map would silently report the unqueried sites as up to date, so the pending-updates cache instead discards the faulted flight and the UI shows the aggregate as unavailable.

Site secrets (2026-09-28)

Site secrets ride the artifact deploy instead of the (now retired) gRPC secrets hub — design docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md; see that doc's Implementation notes (as built) section for the deviations called out below.

• Derived, not configured. SiteSecretsCollector scans the site's data connection artifacts (SecretReferenceScanner over PrimaryConfigurationJson / BackupConfigurationJson) for secret:<name> string values and resolves each name against the central ISecretStore, producing the ciphertext-only SecretArtifact rows the command carries (DeployArtifactsCommand.Secrets, proto field 9) plus the set of unresolved references. A site with no secret: references ships no secrets and is unaffected by any of this.
• Fail closed, per site. A site with one or more unresolved (absent or tombstoned) references fails its deploy before _communicationService is called — nothing is sent, the site keeps its previous stamp, and the error names the secret and the referencing data connection(s): Secret 'X' referenced by data connection(s) A, B is not in the central secret store (semicolon-joined for more than one). Other sites in the same sweep are unaffected.
• Stamp gating. A site is stamped ArtifactRevisionHash/ArtifactsDeployedAt only when every shipped secret actually applied. Two failure shapes are distinguished from an ordinary site failure and never stamped: an old site (wire Success = true, secrets were shipped, but the response has no SecretsApplied at all — a pre-feature site build) fails with Site build predates secret deployment — upgrade the site before deploying secrets; a partial apply (SecretsApplied less than the shipped count) fails with Site applied {n} of {m} secrets. Both apply identically on the all-sites sweep (DeployCoreAsync) and the per-site retry (RetryForSiteAsync).
• Staleness hash. ArtifactRevisionHashService folds secrets into the same hash as every other artifact class (HashableSecret: a SHA-256 digest of the ciphertext/nonce/tag/wrapped-DEK material, never plaintext, plus KekId and Revision) so a KEK rewrap correctly flips a site to out of date. A referenced-but-not-yet-created secret hashes as an explicit Missing marker (DeployArtifactsCommand.MissingSecrets, central-only — never on the wire), so that site reads Out of date, never Current, until the secret exists. A null secrets list and an empty one hash identically (the existing null-when-empty convention for every artifact class), so a site with no secret references keeps its pre-existing hash across the upgrade that added this feature.
• Audit. One best-effort DeploySiteSecrets / SiteSecrets audit row is written per site deploy that carried secrets or a missing-secret failure (never for a site with no secret references at all) — after-state is { deploymentId, secrets: [{name, revision}], missing: [name], success }, which structurally excludes ciphertext. Written on both DeployCoreAsync and RetryForSiteAsync, including the fail-closed path (audited even though nothing was sent).
• Central store. Central resolves secrets from ISecretStore, composed by the Host per Secrets:Store:Provider — see Component-Host.md → Secrets.
• Site apply and the per-node mirror — see Component-SiteRuntime.md → Deployed Secrets.

Site-Side Apply Atomicity

Applying a deployment at the site is all-or-nothing per instance:

• The site stores the new config, compiles all scripts, and creates/updates the Instance Actor as a single operation.
• If any step fails (e.g., script compilation), the entire deployment for that instance is rejected. The previous configuration remains active and unchanged.
• The site reports the specific failure reason (e.g., compilation error details) back to central.

System-Wide Artifact Version Compatibility

• Cross-site version skew for artifacts (shared scripts, external system definitions, data connection definitions, etc.) is supported — sites can temporarily run different artifact versions after a partial deployment.
• Artifacts are self-contained and site-independent. A site running an older version of shared scripts continues to operate correctly with its current instance configurations.
• The Central UI clearly indicates which sites have pending artifact updates so engineers can remediate — see Per-site artifact revision above and Pending-updates aggregate below for the concrete mechanism (the Sites-page Artifacts badge and the rail indicator).

Pending-updates aggregate (2026-09-14)

DeploymentService.GetInstanceStalenessAsync(instanceIds) answers instance staleness for a whole batch cheaply enough to run fleet-wide on a cache-TTL cadence, agreeing exactly with the per-instance Topology badge (GetDeploymentComparisonAsync):

• All the flattens share one FlattenSession for the batch, so a template chain used by N instances is walked once — the same trick DeploySiteAsync uses.
• The deployed side is read as the (InstanceId, RevisionHash) projection (IDeploymentManagerRepository.GetDeployedRevisionHashesAsync), never the fat ConfigurationJson for the whole fleet.
• Mismatch-only normalisation: only when the projected hash disagrees with the re-flatten is the full DeployedConfigSnapshot read and the I-1 List-value normalisation applied (see Deployed vs. Template-Derived State above) — the same normalisation GetDeploymentComparisonAsync applies, so the aggregate never disagrees with the Topology badge on an old-form List snapshot, without paying the fat read fleet-wide.
• null means "could not evaluate" (the instance's flatten failed) — counted as unknown, never as stale; a fleet-wide read must not manufacture a pending update out of a read failure. An instance with no deployed snapshot at all is omitted, not counted as unknown.
• Per-instance failures are not logged individually — the whole batch runs at most once per cache TTL, so one broken shared template fanning out to N instances would otherwise emit N Warning lines every minute forever. A single Warning summarises the batch's failure count instead; per-instance detail stays at Debug.

PendingUpdatesEvaluator (scoped) combines that with ArtifactDeploymentService.ComputeCurrentArtifactHashesAsync into one PendingUpdatesSnapshot: ComputedAt, plus per-site SitePendingUpdates (ArtifactState — NeverDeployed / Current / OutOfDate —, ArtifactsDeployedAt, DeployedInstanceCount, StaleInstanceCount, UnknownInstanceCount) and fleet totals StaleInstanceCount / StaleArtifactSiteCount. Only instances with a deployed snapshot (State != NotDeployed, the same gate Topology uses) are evaluated. NeverDeployed is deliberately not a pending update — a freshly-created site the operator has not gotten to yet has nothing drifted, so it is surfaced on the Sites page but never counted toward the fleet totals or the rail indicator's badge count. Nothing in the evaluator swallows an exception: a faulted evaluation propagates so the cache's single-flight memo discards the flight and the Central UI renders the aggregate as unavailable, rather than manufacturing a false "up to date".

The Central UI holds the snapshot behind a process-wide singleton IPendingUpdatesCache (PendingUpdatesCache, mirroring KpiSnapshotCache): one SingleFlightMemo<PendingUpdatesSnapshot> keyed by ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl (default 60 s, validated positive by DeploymentManagerOptionsValidator), each flight running the evaluator in its own DI scope. However many Blazor circuits poll GetAsync(forceRefresh), the fleet is flattened at most once per TTL, and not at all when no Deployer/Admin circuit is open. See docs/plans/2026-09-14-pending-updates-design.md for the full design, including the Sites-page badge/re-check UI and the rail-footer indicator (Component-CentralUI.md → Site & Data Connection Management and Technology).

Not in scope for this slice (deferred-work register rows 124–125): a template-graph-watermark short-circuit so an unchanged fleet costs no re-flatten per TTL, and a deploy CLI / Health-dashboard KPI surface for the counts.

Instance Lifecycle Commands

The Deployment Manager sends the following commands to sites via the Communication Layer:

• Disable: Instructs the site to stop the Instance Actor's data subscriptions, script triggers, and alarm evaluation. The deployed configuration is retained for re-enablement.
• Enable: Instructs the site to re-activate a disabled instance.
• Delete: Instructs the site to remove the running configuration and destroy the Instance Actor and its children. Store-and-forward messages are not cleared. If the site is unreachable, the delete command fails — the central side does not mark the instance as deleted until the site confirms.

Dependencies

• Template Engine: Produces flattened configurations, diffs, and validation results.
• Communication Layer: Delivers configurations and lifecycle commands to sites.
• Configuration Database (MS SQL): Stores deployment status and deployed configuration snapshots.
• Security & Auth: Enforces Deployment role (with optional site scoping).
• Configuration Database (via IAuditService): Logs all deployment actions, system-wide artifact deployments, and instance lifecycle changes.
• ZB.MOM.WW.Secrets.Abstractions (ISecretStore, SecretName): SiteSecretsCollector resolves a site's referenced secrets against the Host-composed central store — see Component-Host.md → Secrets.

Interactions

• Central UI: Engineers trigger deployments, view diffs/status, manage instance lifecycle, and deploy system-wide artifacts.
• Template Engine: Provides resolved and validated configurations.
• Site Runtime: Receives and applies configurations and lifecycle commands.
• Health Monitoring: Deployment failures contribute to site health status.
• Transport (#24): Does not invoke Deployment Manager directly. Transport-driven template changes propagate to deployed instances through the existing revision-hash drift detection in DeploymentService.CompareAsync; the Deployments page surfaces affected instances as stale automatically.

Authoritative staleness and bulk deadlines

StaleInstanceProbe no longer memoizes revision hashes against process-local watermarks. Every probe starts a fresh flatten session, so a configuration edit through another central is visible to the next deployment skip decision. This trades extra flattening work for correctness across shared-database writers.

Management bulk deployment uses LongRunningCommandTimeout as an overall scheduling/send deadline. Cancellation stops scheduling more instances, fails unfinished eligible rows, and releases prepared operation locks. Completed and deliberately skipped outcomes remain in the response. HTTP Ask reserves an additional60 seconds for cleanup and response; the bulk CLI defers its timeout to this server-owned deadline.