docs(components): correct the three component docs against the code they describe

CLAUDE.md was corrected in 34227991 but the component docs were not, so the
same understatements — plus several outright wrong statements — survived where
a reader is most likely to meet them.

ClusterInfrastructure.md carried the worst of it. It described active/standby as
cluster leadership and showed an IsActiveNode snippet doing
`cluster.State.Leader == self.Address`. No such code exists: ActiveNodeGate
returns ClusterActivityEvaluator.SelfIsOldest, and ActiveNodeEvaluator's own doc
comment says "never cluster.State.Leader". A second snippet (siteCallAuditShutdown
.AddTask) described a hand-rolled drain that has since been folded into
SingletonRegistrar.Start. Both snippets are replaced with what the code does. The
central singleton table listed 3 of the 7 registered singletons. The downing
section still described keep-oldest as the strategy and omitted
downing-provider-class entirely; it now shows the branching block cf3bd52f
introduced, with the accepted dual-active trade stated rather than the old
"impossible to boot" framing. Note the requirements-side spec,
docs/requirements/Component-ClusterInfrastructure.md, was already rewritten by
cf3bd52f — this is the components-side doc, which was untouched.

Communication.md described two transports; there are three — the deployment
config fetch over HTTP was missing. Each row now names which side dials, because
the gRPC entry gave a data direction (site to central) without saying who hosts
the server, which reads as the opposite of the truth: the only MapGrpcService in
the tree is in Program.cs's Site branch, and central dials in. The SiteEnvelope
snippet called DeployInstanceAsync, which is not a member of CommunicationService;
the heartbeat bullet claimed a cluster-leadership check; the proto summary listed
4 of 6 RPCs.

DeploymentManager.md still said the pipeline sends DeployInstanceCommand carrying
FlattenedConfigurationJson. It stages a PendingDeployment and sends a
RefreshDeploymentCommand; the config travels over the HTTP fetch. That is the
change the 128 KB frame-size issue forced, and the doc predated it.

Two of the five briefed drifts turned out not to be errors: nothing claimed the
transports were authenticated, and nothing asserted per-site ActorSystem names —
both were simply unstated. They are now stated, since silence about an
unauthenticated boundary is its own problem.

Docs only; no code changed.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 18:45:56 -04:00
parent cf3bd52f93
commit 69b3ccfc37
3 changed files with 170 additions and 91 deletions
+20 -6
View File
@@ -26,10 +26,20 @@ Every instance deployment carries two correlated identifiers:
- **`DeploymentId`** — a new `Guid` (formatted `"N"`) minted by `DeploymentService` at the start of each `DeployInstanceAsync` call.
- **`RevisionHash`** — computed by the Template Engine's `RevisionHashService` over the fully resolved `FlattenedConfiguration`. The hash captures the template state at the moment of flattening, so concurrent last-write-wins template edits do not affect an in-flight deployment.
The pair travels inside `DeployInstanceCommand` to the site. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
The pair travels to the site inside the `RefreshDeploymentCommand` notify and is echoed back on the fetched config. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
Central stores the `RevisionHash` on `DeploymentRecord` and, after a confirmed success, on `DeployedConfigSnapshot`. Comparing the snapshot hash against the current-template hash determines whether an instance is stale without a site round-trip.
### Notify-and-fetch: the config does not travel in the Akka message
A deployment crosses the central↔site boundary over **two** transports, not one. Central stages the flattened configuration in a `PendingDeployment` row (config JSON, a generated `DeploymentFetchToken`, and an expiry of `CommunicationOptions.PendingDeploymentTtl`, default 5 minutes) and then sends only a small `RefreshDeploymentCommand` over ClusterClient carrying the deployment id, instance name, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton fetches the config back over plain HTTP — `GET {CentralFetchBaseUrl}/api/internal/deployments/{deploymentId}/config` with an `X-Deployment-Token` header — and only then runs its normal apply path.
This exists because a flattened configuration can exceed the default 128 KB Akka frame size, and an over-limit message is dropped silently without tearing down the association — the deploy then simply hangs to its Ask timeout. `CentralFetchBaseUrl` is therefore mandatory: `DeployInstanceAsync` fails fast with "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)" rather than attempting a deploy that cannot complete. Note that `DeployArtifactsCommand` was **not** moved to this path — artifact deployment still carries its payload inline and remains exposed to the frame limit.
Staged rows are cleaned up by **TTL only** — they are deliberately not deleted on success or in the failure path. Three things keep that safe: `AddPendingDeploymentAsync` supersedes (deletes) any prior pending row for the same instance before inserting, so at most one row exists per instance; the fetch endpoint enforces the TTL itself, so an un-purged row is not a usable one; and the central `pending-deployment-purge` singleton sweeps expired rows on `PendingDeploymentPurgeInterval` (default 1 hour).
The site's **startup reconciliation** path uses the same endpoint but stages its own rows: a site node reports its local instance→revision-hash map on boot, and central's `ReconcileService` diffs it against the expected deployed set, stages a fresh `PendingDeployment` (with a new token) for each missing or stale instance, and returns the gap plus `CentralFetchBaseUrl` for the node to fetch. Intra-site replication to the standby node does **not** use this path — `deployed_configurations` is a replicated LocalDb table, so the active node's write reaches the peer as an ordinary row change.
### Per-instance operation lock
`OperationLockManager` holds a `Dictionary<string, LockEntry>` keyed by instance `UniqueName`. Each `LockEntry` wraps a `SemaphoreSlim(1,1)` with a reference count so the semaphore is created on first contention and disposed when the last waiter clears. The lock covers all four mutating operations — deploy, disable, enable, delete — so they can never interleave on a single instance. Operations on different instances proceed in parallel.
@@ -67,7 +77,7 @@ The operation lock is in-memory. If the active central node fails mid-deployment
3. **Flatten and validate**`IFlatteningPipeline.FlattenAndValidateAsync` runs the Template Engine pipeline and returns a `FlatteningPipelineResult` containing the `FlattenedConfiguration`, `RevisionHash`, and a `ValidationResult`. Semantic validation failures (call targets, argument types, trigger operand types, connection binding completeness) are returned to the caller before any record is written.
4. **Pre-deploy site reconciliation** — when the prior `DeploymentRecord` for the instance is `InProgress` or `Failed` with a timeout marker (`"Communication failure:"`), the service queries the site via `CommunicationService.QueryDeploymentStateAsync`. If the site already holds the target revision hash, the prior record is updated to `Success` and no new deployment is sent.
5. **Write `InProgress` record** — a single `DeploymentRecord` insert directly at `InProgress` status (no transient `Pending` hop). `IDeploymentStatusNotifier.NotifyStatusChanged` fires to push the status to the UI.
6. **Send `DeployInstanceCommand`** — the command carries `DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `FlattenedConfigurationJson`, `DeployedBy`, and `Timestamp`.
6. **Stage and notify** — insert a `PendingDeployment` row holding the flattened config JSON and a fresh fetch token, then send `RefreshDeploymentCommand` (`DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `DeployedBy`, staging timestamp, `CentralFetchBaseUrl`, `FetchToken`) via `CommunicationService.RefreshDeploymentAsync`. The site fetches the config over HTTP and replies with the same `DeploymentStatusResponse` as before.
7. **Commit terminal status** — the `DeploymentRecord` is updated to `Success` or `Failed` and saved before any post-success side effects run. This ordering ensures the recorded outcome can never be lost if a post-success write fails.
8. **Post-success side effects**`ApplyPostSuccessSideEffectsAsync` sets `Instance.State = Enabled` (or preserves `Disabled` on the reconciliation path) and upserts the `DeployedConfigSnapshot`. These writes are best-effort: a failure here is logged at `Error` but does not flip the already-committed `Success` record back to `Failed`.
9. **Audit log**`IAuditService.LogAsync` records `Deploy` / `DeployFailed` / `DeployReconciled` with the `DeploymentId`, status, and user.
@@ -76,7 +86,7 @@ Any exception in the site round-trip (steps 67) writes `DeploymentStatus.Fail
```csharp
// DeploymentService.DeployInstanceAsync — exception handler
var isTimeout = ex is TimeoutException or OperationCanceledException;
var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException;
record.Status = DeploymentStatus.Failed;
record.ErrorMessage = isTimeout
@@ -171,11 +181,11 @@ Options are registered via `AddDeploymentManager` and bound from `ScadaBridge:De
- [Template Engine (#1)](./TemplateEngine.md) — `FlatteningPipeline` delegates to `FlatteningService`, `ValidationService`, and `RevisionHashService`. Template state is captured at flatten time; last-write-wins edits made after flatten do not affect the in-flight deployment. `DiffService.ComputeDiff` powers the deployment diff view.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — owns the EF Core implementation of `IDeploymentManagerRepository`, which stores `DeploymentRecord`, `DeployedConfigSnapshot`, and `SystemArtifactDeploymentRecord`. `IAuditService` (also registered by the Configuration Database component) writes all deployment audit rows.
- [CentralSite Communication (#5)](./Communication.md) — `CommunicationService` provides `DeployInstanceAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `DeploymentStatus`, `InstanceState`, `DeployInstanceCommand`, `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [CentralSite Communication (#5)](./Communication.md) — `CommunicationService` provides `RefreshDeploymentAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`, all over the ClusterClient command/control transport. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. `CommunicationOptions.CentralFetchBaseUrl` / `PendingDeploymentTtl` (also owned by that component) parameterise the notify-and-fetch HTTP leg.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `PendingDeployment`, `DeploymentFetchToken`, `DeploymentStatus`, `InstanceState`, `RefreshDeploymentCommand`, `DeployInstanceCommand` (retained as the site-side in-process apply DTO), `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Site Runtime (#3)](./SiteRuntime.md) — receives `DeployInstanceCommand` and `DeployArtifactsCommand` via the Communication Layer. Site-side apply is all-or-nothing per instance: the Deployment Manager singleton at the site stores the config, compiles all scripts, and creates or replaces the Instance Actor as a unit. A failure at any step is reported back with the specific error message and the previous configuration remains active.
- [Central UI (#9)](./CentralUI.md) — engineers trigger deployments, view diffs, manage instance lifecycle, and deploy system-wide artifacts through the UI. The deployment status page subscribes to `IDeploymentStatusNotifier.StatusChanged` for real-time push updates via Blazor Server SignalR.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. It also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route a site calls to fetch a staged config. That endpoint is `AllowAnonymous`; the per-deployment token, compared in constant time, is the entire security boundary, and existence/TTL are checked before the token so unknown, superseded and expired ids are indistinguishable `404`s.
- [Security & Auth (#10)](./Security.md) — the Deployment role is required for all deploy and artifact operations; site-scoped permissions are enforced by the Central UI and Management Service before commands reach `DeploymentService`.
## Troubleshooting
@@ -188,6 +198,10 @@ The operation lock is in-memory. On failover the new active node has no lock ent
The site round-trip timed out or was cancelled before a response arrived. The site may or may not have applied the config. On the next deploy attempt the reconciliation query determines the ground truth. If the query also fails (site unreachable), a new `DeployInstanceCommand` is sent; the site rejects it with "already applied" if it ran the previous one.
### A deployment fails with a config-fetch error
The notify reached the site but the HTTP leg did not complete. `CentralFetchBaseUrl` must be resolvable and reachable **from the site** — a value that only works inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl` (existence is hidden, so those are indistinguishable); a `401` means the row is live but the presented token did not match. A fetch failure applies nothing, and the site replies `Failed` rather than letting central's Ask hang to timeout.
### DeleteOrphaned audit entry
The site destroyed the Instance Actor but the central DB removal failed. The instance record exists in the central DB but has no corresponding site actor. It cannot be deleted through the normal UI path (the site will reject the delete command because the instance does not exist). Reconcile by removing the central record directly via the Management API or database, referencing the `CommandId` in the audit entry.