ScadaBridge/docs/components/TemplateEngine.md

# Template Engine

The Template Engine models the machine blueprints — templates — from which all deployed instances are created. It enforces inheritance, composition, locking, naming rules, and acyclicity at authoring time, then flattens the resulting graph plus instance overrides into a concrete, revision-hashed `FlattenedConfiguration` that the Deployment Manager sends to sites.

## Overview

Template Engine (#1) runs on the central cluster only. Sites receive flattened output and have no awareness of template structure. The component code lives in `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/`, organized as follows:

- Root — `TemplateService`, `SharedScriptService`, `TemplateResolver`, `CycleDetector`, `CollisionDetector`, `LockEnforcer`, `TemplateNaming` — core authoring operations and graph invariant enforcement.
- `Flattening/` — `FlatteningService`, `RevisionHashService`, `DiffService` — produce and compare the deployment-ready representation.
- `Validation/` — `ValidationService`, `SemanticValidator`, `ScriptCompiler`, `CSharpDelimiterScanner` — pre-deployment and on-demand correctness checks.
- `Services/` — `InstanceService`, `SiteService`, `AreaService`, `TemplateFolderService`, `TemplateDeletionService` — domain operations that depend on the scoped `ITemplateEngineRepository`.

The single DI entry point is `ServiceCollectionExtensions.AddTemplateEngine`. `TemplateService` and `SharedScriptService` are scoped; the flattening and validation utilities are transient; static helpers (`CycleDetector`, `CollisionDetector`, `LockEnforcer`, `TemplateResolver`) are not registered.

## Key Concepts

### Template graph

The full set of templates forms a directed graph with two independent edge types:

- **Inheritance** — `Template.ParentTemplateId` (nullable `int?`). A null value means no parent; a non-null value sets the defining ancestor. The parent is set at creation time and is immutable thereafter; `UpdateTemplateAsync` rejects any attempt to change it.
- **Composition** — `TemplateComposition` rows, each pointing from an owner `TemplateId` to a `ComposedTemplateId` with a slot name (`InstanceName`). Only base (non-derived) templates may be composed.

Both edge types are enforced acyclic on every mutating call. `CycleDetector` provides three checks:

- `DetectInheritanceCycle` — walks the proposed parent chain upward looking for the template being modified.
- `DetectCompositionCycle` — BFS from the proposed composed template through its own compositions.
- `DetectCrossGraphCycle` — BFS across both inheritance and composition edges simultaneously, catching cycles that neither pure check alone would find.

All three run before any composition is written. Because the graph can contain not-yet-saved templates (Id = 0), `CycleDetector.BuildLookup` uses `TryAdd` rather than `ToDictionary` so duplicate Ids do not throw.

### Derived templates and the composition model

When `AddCompositionAsync` composes template B into template A under slot name `Pump`, the engine calls `CreateCascadedCompositionAsync`, which:

1. Creates a derived `Template` (`IsDerived = true`, `ParentTemplateId = B.Id`, `Name = "Pump"`) as the slot-owned backing record.
2. Copies B's attributes, alarms, and scripts onto the derived template with `IsInherited = true`.
3. Creates a `TemplateComposition` row linking A to the derived template.
4. Sets `derived.OwnerCompositionId` so the slot can be deleted as a unit.
5. Recurses into B's own compositions to replicate them under the new derived template.

Derived templates are hidden from the main template tree and cannot be directly composed or deleted by name — removal goes through `DeleteCompositionAsync`, which calls `CascadeDeleteDerivedAsync` to tear the whole subtree down.

### Canonical naming and path qualification

Members of composed modules are addressed by **path-qualified canonical names**: `[ModuleInstanceName].[MemberName]`. For deeper nesting the dot chain extends: `Pump.Motor.Speed`. Direct members of the owning template carry no prefix.

`TemplateResolver.ResolveAllMembers` builds a `Dictionary<string, ResolvedTemplateMember>` in inheritance-chain order (root first) then adds composed-module members with their prefix. The last writer on a given canonical name wins — child overrides shadow parent definitions. `FindMemberByCanonicalName` is the entry point for lock checks during `UpdateAttribute/Alarm/ScriptAsync`.

`TemplateNaming.QualifiedName` computes the full dotted path of a derived template at read time by walking the `OwnerCompositionId` chain — the derived template stores only its contained name (`InstanceName`), so the full path is never stored and cannot drift.

### Locking and override granularity

`LockEnforcer` enforces three classes of rules for attributes, alarms, and scripts:

| Rule | Mechanism |
|------|-----------|
| `IsLocked` = true blocks all downstream overrides | `ValidateLockChange`: once set, cannot be cleared (one-way ratchet). |
| `LockedInDerived` = true blocks derived-template overrides of that specific member | `ValidateLockedInDerivedChange`: also a one-way ratchet — cannot be cleared on a base template. |
| Fixed fields cannot change at any level | `ValidateAttributeOverride` / `ValidateAlarmOverride` / `ValidateScriptOverride`. |

**Attribute fixed fields**: `DataType`, `DataSourceReference`. Overridable: `Value`, `Description`.

**Alarm fixed fields**: `Name`, `TriggerType`. Overridable: `PriorityLevel`, `TriggerConfiguration`, `Description`, `OnTriggerScriptId`.

**Script fixed fields**: `Name`. Overridable: `Code`, `TriggerType`, `TriggerConfiguration`, `MinTimeBetweenRuns`, `ParameterDefinitions`, `ReturnDefinition`.

Intermediate locking is permitted: a child template can lock an unlocked member inherited from its parent. Unlocking is never permitted at any level.

### Naming collisions

Adding a member (attribute, alarm, script, or composition) triggers `CollisionDetector.DetectCollisions` on a speculative clone of the template. The detector collects all canonical names — direct members plus path-qualified composed-module members plus inherited members — groups them by canonical name, and reports any group where two entries come from different origin descriptions. A collision is a design-time error and blocks the operation.

Because each composition slot has a unique `InstanceName` prefix, members from different slots can never collide by canonical name. Collisions arise only when a directly defined member shares an unqualified name with an inherited or composed member under the same owner.

### Flattening

`FlatteningService.Flatten` takes the instance, its template inheritance chain (most-derived first), a composition map, per-composed-template chains, and available data connections, and produces a `FlattenedConfiguration`. The resolution order is:

1. Instance overrides (highest priority, respects locks).
2. Most-derived template in the inheritance chain.
3. Parent templates, walking to the root.
4. Composed module members, path-qualified.

The eight steps in order:

1. Validate `LockedInDerived` is not violated across each chain.
2. Resolve attributes from the inheritance chain (base-to-derived; `IsInherited` placeholders never shadow live base values).
3. Resolve composed-module attributes with path-qualified canonical names, recursively.
4. Apply `InstanceAttributeOverride` records (locked attributes are silently skipped).
5. Apply `InstanceConnectionBinding` records (data-sourced attributes only).
6. Resolve alarms; for `HiLo` trigger type, merge setpoints key-by-key so a derived template can override just `hi` while inheriting `loLo`.
7. Resolve scripts; wire `ScriptScope` (self- and parent-path) into each composed script so `Attributes["X"]` resolves to the right path-prefix at runtime.
8. Resolve native alarm source bindings (`TemplateNativeAlarmSource`), apply `InstanceNativeAlarmSourceOverride`.

Between steps 6 and 7, `ResolveAlarmScriptReferences` resolves each alarm's `OnTriggerScriptId` FK to the canonical name of the corresponding resolved script. A dangling reference (script Id has no resolved script) produces a null `OnTriggerScriptCanonicalName` and is caught by `SemanticValidator`.

### Revision hash

`RevisionHashService.ComputeHash` produces a deterministic `sha256:<hex>` string over the canonical JSON serialization of the `FlattenedConfiguration`. Volatile fields (`GeneratedAtUtc`) are excluded. Collections are sorted by `CanonicalName` before hashing. Internal `Hashable*` records declare their properties in alphabetical order because `System.Text.Json` emits them in declaration order — out-of-order additions would silently break determinism. The hash is included in the deployment identity and lets the Deployment Manager detect whether a re-flatten has changed anything before pushing to sites.

### Diff

`DiffService.ComputeDiff` compares two `FlattenedConfiguration` snapshots by canonical name, producing `Added`, `Removed`, and `Changed` entries for attributes, alarms, and scripts. `ComputeConnectionsDiff` produces the same shape for data-connection configurations. The diff is used by the Deployment Manager to decide whether a full redeploy is needed.

### Concurrent editing

Template edits use **last-write-wins** — there is no optimistic concurrency token on `Template` or its member rows. Two simultaneous edits to the same template produce one winner. This is by design and is documented in the `InstanceService` comment: "Concurrent editing uses last-write-wins — no pessimistic locking or conflict detection." Optimistic concurrency (`RowVersion`) applies to deployment status records in the Deployment Manager, not to template authoring.

## Architecture

### Service map

```text
AddTemplateEngine()
├── TemplateService        (scoped)  — template + member CRUD, collision/acyclicity pre-checks
├── SharedScriptService    (scoped)  — system-wide shared script CRUD + syntax validation
├── InstanceService        (scoped)  — instance CRUD, overrides, connection bindings
├── SiteService            (scoped)  — site CRUD
├── AreaService            (scoped)  — area CRUD
├── TemplateFolderService  (scoped)  — folder hierarchy, sibling-name uniqueness, acyclicity on move
├── TemplateDeletionService(scoped)  — deletion constraints; called from TemplateService.DeleteTemplateAsync
├── FlatteningService      (transient)
├── RevisionHashService    (transient)
├── DiffService            (transient)
├── ValidationService      (transient) — full pipeline: 8 stages merged into one ValidationResult
├── SemanticValidator      (transient) — call-target, argument-count, operand-type, cross-call rules
└── ScriptCompiler         (transient) — advisory forbidden-API scan + delimiter balance check
```

Static helpers — `CycleDetector`, `CollisionDetector`, `LockEnforcer`, `TemplateResolver`, `TemplateNaming` — are not registered with DI.

### Validation pipeline

`ValidationService.Validate` runs eight stages in sequence and merges results via `ValidationResult.Merge`:

| Stage | Category | Outcome |
|-------|----------|---------|
| `ValidateFlatteningSuccess` | `FlatteningFailure` | Error on missing name; warning on empty configuration |
| `ValidateNamingCollisions` | `NamingCollision` | Error per duplicate canonical name within entity type |
| `ValidateScriptCompilation` | `ScriptCompilation` | Error per script that fails `ScriptCompiler.TryCompile` |
| `ValidateAlarmTriggerReferences` | `AlarmTriggerReference` | Error when `attributeName` / `attribute` key not in flattened attributes |
| `ValidateScriptTriggerReferences` | `ScriptTriggerReference` | Same check for script triggers |
| `ValidateExpressionTriggers` | `ScriptTriggerReference` / `AlarmTriggerReference` | Blank warning, syntax error, or missing `Attributes["X"]` reference |
| `ValidateConnectionBindingCompleteness` | `ConnectionBinding` | Warning per data-sourced attribute with no binding |
| `SemanticValidator.Validate` | Multiple | Call targets, argument counts, `RangeViolation`/`HiLo` operand types, on-trigger script existence, cross-call violations, native alarm source completeness |

`ValidationResult` is defined in Commons: `IsValid` is true when `Errors` is empty; `Warnings` do not block deployment. Each `ValidationEntry` carries a `ValidationCategory`, a human-readable `Message`, and an optional `EntityName` (canonical name of the offending entity).

### Key entity types (defined in Commons)

| Type | Namespace | Role |
|------|-----------|------|
| `Template` | `Commons.Entities.Templates` | Base and derived template rows; `IsDerived` distinguishes slot-owned derived templates |
| `TemplateAttribute` | same | Attribute definition with `IsInherited`, `LockedInDerived`, `DataType`, `DataSourceReference` |
| `TemplateAlarm` | same | Alarm definition; `TriggerType` and `Name` are fixed fields |
| `TemplateScript` | same | Script definition; `Name` is a fixed field |
| `TemplateComposition` | same | Slot row linking owner to composed (or derived) template by `InstanceName` |
| `TemplateNativeAlarmSource` | same | Read-only native alarm binding; `SourceReference` is a raw connection address |
| `FlattenedConfiguration` | `Commons.Types.Flattening` | Deployment-ready snapshot; fields `Attributes`, `Alarms`, `Scripts`, `NativeAlarmSources`, `Connections` |
| `ResolvedAttribute` / `ResolvedAlarm` / `ResolvedScript` / `ResolvedNativeAlarmSource` | same | Flattened member records carrying `CanonicalName` and `Source` provenance |
| `ConfigurationDiff` / `DiffEntry<T>` | same | Diff output keyed by canonical name |
| `ValidationResult` / `ValidationEntry` / `ValidationCategory` | same | Validation output |

## Usage

### Authoring a template with inheritance and composition

The normal flow goes through `TemplateService` methods; each call validates graph invariants before persisting and audits after:

```csharp
// Create a base template
var base = await templateService.CreateTemplateAsync(
    "MotorBase", description: null, parentTemplateId: null, user: "alice");

// Add an attribute to the base
await templateService.AddAttributeAsync(base.Value.Id, new TemplateAttribute("Speed")
{
    DataType = DataType.Float,
    DataSourceReference = "/Motor/Speed"
}, user: "alice");

// Create a child that inherits from the base
var child = await templateService.CreateTemplateAsync(
    "PumpMotor", description: null, parentTemplateId: base.Value.Id, user: "alice");

// Compose a feature module (AlarmsModule) into the base — acyclicity and collision
// checks run; a derived template is auto-created to back the slot
await templateService.AddCompositionAsync(
    templateId: base.Value.Id,
    composedTemplateId: alarmsModule.Id,
    instanceName: "Alarms",
    user: "alice");
```

After composition, `Alarms.HighTemp` is the canonical name of `HighTemp` from `AlarmsModule` as it appears in the flattened output.

### Resolving members and checking overrides

```csharp
// Returns all effective members with canonical names for templateId
var members = await templateService.ResolveTemplateMembersAsync(templateId);

// Returns TemplateResolver.ResolvedTemplateMember with CanonicalName, MemberType,
// IsLocked, and ModulePath (null for direct members, slot prefix for composed members).
foreach (var m in members)
{
    Console.WriteLine($"{m.CanonicalName} ({m.MemberType}) locked={m.IsLocked}");
}
```

### Flattening and hashing

`FlatteningService` and `RevisionHashService` are transient services called by the Deployment Manager, not directly from the Central UI. The caller builds the template chain (most-derived first) from the repository and passes it:

```csharp
var flatResult = flatteningService.Flatten(
    instance,
    templateChain,          // IReadOnlyList<Template>, index 0 = instance's template
    compositionMap,         // Dictionary<int, IReadOnlyList<TemplateComposition>>
    composedTemplateChains, // Dictionary<int, IReadOnlyList<Template>>
    dataConnections);       // Dictionary<int, DataConnection>

if (flatResult.IsSuccess)
{
    var hash = revisionHashService.ComputeHash(flatResult.Value);
    // hash: "sha256:3a7f..."
}
```

### Validating before deployment

```csharp
var validationResult = validationService.Validate(flatResult.Value, sharedScripts);
if (!validationResult.IsValid)
{
    foreach (var err in validationResult.Errors)
        logger.LogError("{Category} {Entity}: {Msg}", err.Category, err.EntityName, err.Message);
}
```

## Dependencies & Interactions

- [Commons (#16)](./Commons.md) — owns all entity types (`Template`, `TemplateAttribute`, `TemplateAlarm`, `TemplateScript`, `TemplateComposition`, `TemplateNativeAlarmSource`), the flattening types (`FlattenedConfiguration`, `ResolvedAttribute`, `ResolvedAlarm`, `ResolvedScript`, `ResolvedNativeAlarmSource`, `ConfigurationDiff`), the `ValidationResult`/`ValidationEntry`/`ValidationCategory` hierarchy, the `ITemplateEngineRepository` interface, and the `IAuditService` interface. Template Engine imports from Commons; it never holds a direct EF Core dependency.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides the `ITemplateEngineRepository` implementation backed by the central MS SQL database. `TemplateService`, `InstanceService`, and all `Services/` classes resolve this via constructor injection. EF Core migrations for template tables live in this project.
- [Deployment Manager (#2)](./DeploymentManager.md) — consumes `FlatteningService`, `RevisionHashService`, `DiffService`, and `ValidationService` to prepare deployment packages. It also calls `TemplateDeletionService.CanDeleteTemplateAsync` to check constraints before removing a template that has active deployments.
- [Site Runtime (#3)](./SiteRuntime.md) — receives the `FlattenedConfiguration` (via the Deployment Manager) and uses `ResolvedScript.Scope` to set the path-prefix context for script attribute accessors. `ResolvedNativeAlarmSource` records drive the site's `NativeAlarmActor` bindings.
- [Central UI (#9)](./CentralUI.md) — the primary authoring surface. All template CRUD, instance management, shared script editing, and folder organization go through the Management Service, which delegates to `TemplateService`, `InstanceService`, and `SharedScriptService`. The Central UI calls `ValidateAsync` on-demand so designers see errors before deployment.
- [Management Service (#18)](./ManagementService.md) — the Akka.NET actor that exposes template operations over the cluster boundary. `TemplateService` and related services are injected into its DI scope per request.
- [Transport (#24)](./Transport.md) — exports and imports templates as encrypted bundles. On import, the Transport component calls `TemplateService.CreateTemplateAsync` (and member-add methods) for each template in the bundle; acyclicity and collision checks run identically to manual authoring.
- Design spec: [Component-TemplateEngine.md](../requirements/Component-TemplateEngine.md).

## Troubleshooting

### Composition fails with a cycle error

`CycleDetector.DetectCrossGraphCycle` rejects edges that would create a cycle across either inheritance or composition edges. The most common trigger is composing a template that already (transitively) includes the owner — for example, A composes B, and then trying to compose A into B. The error message identifies the template by name. Remove or restructure the graph to break the circular dependency.

### `LockedInDerived` cannot be cleared

`LockEnforcer.ValidateLockedInDerivedChange` enforces a one-way ratchet: once a base template sets `LockedInDerived = true` on a member, the flag cannot be cleared. This is intentional — clearing it retroactively would make previously blocked derived overrides legal without any visible signal to derived-template authors. The only remediation is to create a new base template without the flag.

### Revision hash changes unexpectedly between deploys

The SHA-256 hash covers all resolved attributes, alarms, scripts, and connection configurations. Changes to any member anywhere in the inheritance or composition chain — including in parent templates or feature modules the instance does not directly own — will change the hash. Use `DiffService.ComputeDiff` to identify exactly which canonical names changed and why.

### Naming collision on composed member add

`CollisionDetector.DetectCollisions` fires when a member's unqualified name collides with a direct or inherited member on the same owner template. Because composed members carry a slot-name prefix, a collision can arise between a directly defined member (`Speed`) and a composed member that resolves to the same unqualified name in an ancestor. The fix is to rename one of the conflicting members before adding the composition.

### Semantic validation: `CallScript` target not found

`SemanticValidator.ExtractCallTargets` uses a substring scan for `CallScript("name", ...)` and `CallShared("name", ...)`. If the target name does not match any resolved script canonical name, the error `CallTargetNotFound` is reported. Check that the call uses the full canonical name, including any composition prefix (e.g., `Alarms.HandleFault`, not just `HandleFault`).

## Related Documentation

- [Template Engine design specification](../requirements/Component-TemplateEngine.md)
- [Commons](./Commons.md)
- [Configuration Database](./ConfigurationDatabase.md)
- [Deployment Manager](./DeploymentManager.md)
- [Site Runtime](./SiteRuntime.md)
- [Central UI](./CentralUI.md)
- [Management Service](./ManagementService.md)
- [Transport](./Transport.md)