# Phase 3A: Runtime Foundation & Persistence Model

**Date**: 2026-03-16
**Status**: Draft
**Prerequisites**: Phase 0 (Solution Skeleton), Phase 1 (Central Platform Foundations — Akka.NET bootstrap via REQ-HOST-6), Phase 2 (Modeling & Validation — deployment package contract defines the serialized format stored in SQLite)

---

## Scope

**Goal**: Prove the Akka.NET cluster, singleton, and local persistence model work correctly — including failover.

**Components**:
- **Cluster Infrastructure** (full) — Akka.NET cluster setup, split-brain resolution, failure detection, graceful shutdown, dual-node recovery.
- **Host** (site-role Akka bootstrap) — Site nodes use generic `IHost` (no Kestrel), Akka.NET actor system with Remoting, Clustering, Persistence (SQLite), and SBR.
- **Site Runtime** (partial) — Deployment Manager singleton skeleton, basic Instance Actor (holds attribute state, static attribute persistence). Script Actors, Alarm Actors, stream, and script execution are Phase 3B.
- **Local SQLite persistence model** — Schema for deployed configurations, static attribute overrides.

**Testable Outcome**: Two-node site cluster forms. Singleton starts on oldest node. Failover migrates singleton to surviving node. Singleton reads deployed configs from SQLite and recreates Instance Actors. Static attribute overrides persist across restart. `min-nr-of-members=1` verified. CoordinatedShutdown enables fast handover.

---

## Prerequisites

| Prerequisite | Phase | What's Needed |
|-------------|-------|---------------|
| Solution structure with all component projects | 0 | ClusterInfrastructure, Host, SiteRuntime, Commons projects exist and compile |
| Commons shared types | 0 | `NodeRole` enum, `Result<T>`, UTC timestamp types |
| Commons entity POCOs | 0 | Deployed configuration entity, attribute entity |
| Commons message contracts | 0 | Base message types with correlation IDs |
| Host skeleton with role detection | 0 | `Program.cs` reads `NodeOptions` and selects service registration path |
| Host Akka.NET bootstrap | 1 | REQ-HOST-6 baseline: Akka.Hosting, Remoting, Clustering wired |
| Host configuration binding | 1 | `ClusterOptions`, `NodeOptions`, `DatabaseOptions` bound via Options pattern |
| Host CoordinatedShutdown wiring | 1 | REQ-HOST-9 baseline wired into service lifecycle |
| Host structured logging | 1 | Serilog with SiteId/NodeHostname/NodeRole enrichment |
| Deployment package contract | 2 | Stable serialization format for flattened configs that Site Runtime will store in SQLite |

---

## Requirements Checklist

### Section 1.1 — Central vs. Site Responsibilities (partial — this phase covers site-side bullets only)

- [ ] `[1.1-1]` Central cluster is the single source of truth for all template authoring, configuration, and deployment decisions.
  - *Phase 3A scope*: Not directly implemented here (central-side concern), but the site-side design must **not** include any local authoring capability. **Negative requirement**: Site has no mechanism to create or edit configurations locally.
- [ ] `[1.1-2]` Site clusters receive **flattened configurations** — fully resolved attribute sets with no template structure. Sites do not need to understand templates, inheritance, or composition.
  - *Phase 3A scope*: SQLite schema stores flattened configs. Deployment Manager reads them. Instance Actor loads attributes from them. No template resolution logic on site.
- [ ] `[1.1-3]` Sites **do not** support local/emergency configuration overrides. All configuration changes originate from central.
  - *Phase 3A scope*: **Negative requirement** — no API or mechanism for local configuration changes. Static attribute writes (SetAttribute) are runtime value overrides, not configuration overrides.

**Split-section note**: Section 1.1 is primarily Phase 3A. All three bullets are addressed here. No bullets deferred to other phases (central-side truth enforcement is implicit in system design — central is the only entity that sends configs).

### Section 1.2 — Failover (partial — Phase 3A covers mechanism; Phase 8 covers full-system validation)

- [ ] `[1.2-1]` Failover is managed at the **application level** using **Akka.NET** (not Windows Server Failover Clustering).
- [ ] `[1.2-2]` Each cluster (central and site) runs an **active/standby** pair where Akka.NET manages node roles and failover detection.
- [ ] `[1.2-3]` **Site failover**: The standby node takes over data collection and script execution seamlessly, including responsibility for the store-and-forward buffers.
  - *Phase 3A scope*: Singleton migration to standby and Instance Actor recreation proven. Data collection (DCL) seamless takeover is Phase 3B. Script execution resumption is Phase 3B. S&F buffer takeover is Phase 3C. This phase proves the **foundation**: singleton migrates, deployed configs are read from SQLite, and Instance Actors are recreated — which is the prerequisite for all higher-level takeover behaviors.
- [ ] `[1.2-4]` The Site Runtime Deployment Manager singleton is restarted on the new active node, which reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy.
  - *Phase 3A scope*: "Full hierarchy" in this phase means Deployment Manager → Instance Actors. Script Actors and Alarm Actors (the lower levels of the hierarchy) are added in Phase 3B. The recreation pattern established here is extended in Phase 3B to include script compilation and child actor creation.
- [ ] `[1.2-5]` **Central failover**: The standby node takes over central responsibilities. Deployments that are in-progress during a failover are treated as **failed** and must be re-initiated by the engineer.
  - *Phase 3A scope*: Central failover is not tested here (Phase 8). But the cluster infrastructure must support both central and site cluster topologies.

**Split-section note**: Section 1.2 contains 4 prose bullets. Phase 3A decomposes them into 5 atomic requirements for finer traceability:
- Phase 3A owns: `[1.2-1]` (app-level failover), `[1.2-2]` (active/standby pair), `[1.2-3]` (site failover — singleton migration and Instance Actor recreation foundation), `[1.2-4]` (singleton restart and hierarchy recreation from SQLite).
- Phase 3B owns: `[1.2-3]` completion (DCL reconnection, script execution resumption, alarm re-evaluation).
- Phase 3C owns: `[1.2-3]` completion (S&F buffer takeover).
- Phase 8 owns: `[1.2-3]` full-system validation, `[1.2-5]` (central failover end-to-end).

### Section 2.3 — Site-Level Storage & Interface (partial)

- [ ] `[2.3-1]` Sites have **no user interface** — they are headless collectors, forwarders, and script executors.
  - *Phase 3A scope*: Site-role Host uses generic `IHost` (no Kestrel, no HTTP). **Negative requirement**.
- [ ] `[2.3-2]` Sites require local storage for: the current deployed (flattened) configurations, deployed scripts, shared scripts, external system definitions, database connection definitions, and notification lists.
  - *Phase 3A scope*: SQLite schema for deployed flattened configurations. Scripts, shared scripts, external system defs, DB connection defs, and notification lists are stored in later phases (3B, 3C, 7) but the schema should be extensible.
- [ ] `[2.3-3]` Store-and-forward buffers are persisted to a **local SQLite database on each node** and replicated between nodes via application-level replication.
  - *Phase 3A scope*: Not implemented here (Phase 3C). But SQLite infrastructure established here is reused.

**Split-section note**: `[2.3-2]` is a compound bullet listing 6 storage categories. Phase ownership by sub-item:
- Phase 3A: deployed (flattened) configurations.
- Phase 3B: deployed scripts, shared scripts (stored when deployments are received and scripts compiled).
- Phase 3C/7: external system definitions, database connection definitions, notification lists (stored when system-wide artifacts are deployed).
- Phase 3C: `[2.3-3]` store-and-forward buffers.

---

## Design Constraints Checklist

### From CLAUDE.md Key Design Decisions

- [ ] `[KDD-runtime-1]` Instance modeled as Akka actor (Instance Actor) — single source of truth for runtime state.
- [ ] `[KDD-runtime-2]` Site Runtime actor hierarchy: Deployment Manager singleton → Instance Actors → Script Actors + Alarm Actors.
  - *Phase 3A scope*: Deployment Manager → Instance Actors. Script/Alarm Actors are Phase 3B.
- [ ] `[KDD-runtime-8]` Staggered Instance Actor startup on failover to prevent reconnection storms (e.g., 20 at a time with short delay between batches).
- [ ] `[KDD-runtime-9]` Supervision: Resume for coordinator actors, Stop for short-lived execution actors.
  - *Phase 3A scope*: Deployment Manager supervises Instance Actors with OneForOneStrategy. Instance Actor supervision of Script/Alarm Actors is Phase 3B.
- [ ] `[KDD-data-5]` Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- [ ] `[KDD-data-7]` Tell for hot-path internal communication; Ask reserved for system boundaries.
  - *Phase 3A scope*: Establish the convention. Instance Actor attribute updates use Tell.
- [ ] `[KDD-cluster-1]` Keep-oldest SBR with `down-if-alone=on`, 15s stable-after.
- [ ] `[KDD-cluster-2]` Both nodes are seed nodes. `min-nr-of-members=1`.
- [ ] `[KDD-cluster-3]` Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s.
- [ ] `[KDD-cluster-4]` CoordinatedShutdown for graceful singleton handover.
- [ ] `[KDD-cluster-5]` Automatic dual-node recovery from persistent storage.

### From docs/requirements/Component-ClusterInfrastructure.md

- [ ] `[CD-CI-1]` Two-node cluster (active/standby) using Akka.NET Cluster.
- [ ] `[CD-CI-2]` Leader election and role assignment (active vs. standby).
- [ ] `[CD-CI-3]` Cluster singleton hosting for Site Runtime Deployment Manager.
- [ ] `[CD-CI-4]` Cluster seed nodes: both nodes listed; either can start first.
- [ ] `[CD-CI-5]` Cluster role configuration: Central or Site (plus site identifier for site clusters).
- [ ] `[CD-CI-6]` Akka.NET remoting: hostname/port for inter-node communication.
- [ ] `[CD-CI-7]` Local storage paths: SQLite database locations (site nodes only).
- [ ] `[CD-CI-8]` `akka.coordinated-shutdown.run-by-clr-shutdown-hook = on`.
- [ ] `[CD-CI-9]` `akka.cluster.run-coordinated-shutdown-when-down = on`.
- [ ] `[CD-CI-10]` Dual-node recovery: no manual intervention required. First node forms cluster, second joins.
- [ ] `[CD-CI-11]` Deployment Manager singleton reads deployed configurations from local SQLite on recovery.
- [ ] `[CD-CI-12]` Alarm states re-evaluated from incoming values on recovery (alarm state is in-memory only).
  - *Phase 3A scope*: Establish the pattern — no alarm persistence. Alarm Actors are Phase 3B, but the design must not persist alarm state.
- [ ] `[CD-CI-13]` Keep-oldest SBR rationale: with two nodes, quorum-based strategies cause total shutdown. Keep-oldest with `down-if-alone` ensures at most one node runs the singleton.

### From docs/requirements/Component-SiteRuntime.md

- [ ] `[CD-SR-1]` Deployment Manager is an Akka.NET cluster singleton — guaranteed to run on exactly one node.
- [ ] `[CD-SR-2]` Startup behavior step 1: Read all deployed configurations from local SQLite.
- [ ] `[CD-SR-3]` Startup behavior step 4: Create Instance Actors for all deployed, **enabled** instances as child actors in **staggered batches** (e.g., 20 at a time with short delay).
- [ ] `[CD-SR-4]` Instance Actor: single source of truth for all runtime state of a deployed instance.
- [ ] `[CD-SR-5]` Instance Actor initialization: Load all attribute values from flattened configuration (static defaults).
- [ ] `[CD-SR-6]` Instance Actor SetAttribute (static): Updates in-memory value and **persists override to local SQLite**. On restart/failover, loads persisted overrides on top of deployed config. Redeployment resets all persisted overrides.
- [ ] `[CD-SR-7]` Deployment Manager supervises Instance Actors with **OneForOneStrategy** — one Instance Actor's failure does not affect others.
- [ ] `[CD-SR-8]` Instance lifecycle: Disable stops actor, retains config in SQLite. Enable re-creates actor. Delete stops actor, removes config from SQLite. Delete does **not** clear S&F messages.
  - *Phase 3A scope*: Skeleton lifecycle — disable/enable/delete message handling in Deployment Manager. Full lifecycle with DCL/scripts is Phase 3B/3C.
- [ ] `[CD-SR-9]` When Instance Actor is stopped (disable, delete, redeployment), Akka.NET automatically stops all child actors.

### From docs/requirements/Component-Host.md

- [ ] `[CD-HOST-1]` REQ-HOST-6: Site-role Akka bootstrap with Remoting, Clustering, Persistence (SQLite), Split-Brain Resolver.
- [ ] `[CD-HOST-2]` REQ-HOST-7: Site nodes use `Host.CreateDefaultBuilder` — generic `IHost`, **not** `WebApplication`. No Kestrel, no HTTP port, no web endpoints.
- [ ] `[CD-HOST-3]` REQ-HOST-2: Site-role service registration includes SiteRuntime, DataConnectionLayer, StoreAndForward, SiteEventLogging (only SiteRuntime is wired in this phase; others are stubs).
- [ ] `[CD-HOST-4]` `ClusterOptions`: SeedNodes, SplitBrainResolverStrategy, StableAfter, HeartbeatInterval, FailureDetectionThreshold, MinNrOfMembers.
- [ ] `[CD-HOST-5]` `DatabaseOptions`: Site SQLite paths.

---

## Work Packages

### WP-1: Akka.NET Cluster Configuration (HOCON/Akka.Hosting)

**Description**: Implement the full Akka.NET cluster configuration for site nodes using Akka.Hosting, driven by `ClusterOptions`. This includes Remoting, Clustering, Split-Brain Resolver, failure detection, and CoordinatedShutdown settings.

**Acceptance Criteria**:
- [ ] Cluster configured with keep-oldest SBR, `down-if-alone = on`, 15s stable-after. (`[KDD-cluster-1]`, `[CD-CI-13]`)
- [ ] Both nodes configured as seed nodes. Either node can start first. (`[KDD-cluster-2]`, `[CD-CI-4]`)
- [ ] `min-nr-of-members = 1` — surviving node can operate alone. (`[KDD-cluster-2]`)
- [ ] Failure detection: 2s heartbeat interval, 10s failure threshold. (`[KDD-cluster-3]`)
- [ ] Total failover time ~25s (detection + stable-after + singleton restart). (`[KDD-cluster-3]`)
- [ ] `akka.coordinated-shutdown.run-by-clr-shutdown-hook = on`. (`[CD-CI-8]`)
- [ ] `akka.cluster.run-coordinated-shutdown-when-down = on`. (`[CD-CI-9]`)
- [ ] Remoting configured with hostname/port from `NodeOptions`. (`[CD-CI-6]`)
- [ ] Cluster role set to "site" with SiteId from `NodeOptions`. Site identifier is included in the cluster role tag for site clusters. (`[CD-CI-5]`)
- [ ] All cluster settings driven by `ClusterOptions` (Options pattern). (`[CD-HOST-4]`)
- [ ] Failover is application-level (Akka.NET), not WSFC. (`[1.2-1]`)

**Estimated Complexity**: M

**Requirements Traced**: `[1.2-1]`, `[1.2-2]`, `[KDD-cluster-1]`, `[KDD-cluster-2]`, `[KDD-cluster-3]`, `[KDD-cluster-4]`, `[CD-CI-1]`, `[CD-CI-2]`, `[CD-CI-4]`, `[CD-CI-5]`, `[CD-CI-6]`, `[CD-CI-8]`, `[CD-CI-9]`, `[CD-CI-13]`, `[CD-HOST-1]`, `[CD-HOST-4]`

---

### WP-2: Site-Role Host Bootstrap

**Description**: Implement the site-role startup path in `Program.cs`. Site nodes use generic `IHost` (no Kestrel), configure Akka.NET with Remoting, Clustering, SQLite Persistence, and SBR. Register the SiteRuntime component services and actors.

**Acceptance Criteria**:
- [ ] Site nodes use `Host.CreateDefaultBuilder` — no `WebApplication`, no Kestrel, no HTTP port. (`[CD-HOST-2]`, `[2.3-1]`)
- [ ] Site node **cannot** accept inbound HTTP connections. (`[2.3-1]` — negative)
- [ ] Akka.NET actor system boots with Remoting, Clustering, SQLite Persistence, and SBR. (`[CD-HOST-1]`)
- [ ] SiteRuntime `AddSiteRuntime()` / `AddSiteRuntimeActors()` extension methods called for site role. (`[CD-HOST-3]`)
- [ ] SQLite paths read from `DatabaseOptions`. (`[CD-HOST-5]`)
- [ ] Akka.NET Persistence configured with SQLite journal and snapshot store. (`[CD-HOST-1]`)
- [ ] Active/standby pair forms when two site nodes start. (`[1.2-2]`, `[CD-CI-1]`)

**Estimated Complexity**: M

**Requirements Traced**: `[1.2-2]`, `[2.3-1]`, `[CD-HOST-1]`, `[CD-HOST-2]`, `[CD-HOST-3]`, `[CD-HOST-4]`, `[CD-HOST-5]`, `[CD-CI-1]`

---

### WP-3: Local SQLite Persistence Schema

**Description**: Design and implement the local SQLite schema for site nodes. This phase covers deployed configurations and static attribute overrides. The schema must be extensible for future additions (scripts, shared scripts, S&F buffers, event logs).

**Acceptance Criteria**:
- [ ] `deployed_configurations` table stores flattened configuration blobs keyed by instance unique name. Stores the deployment package format defined in Phase 2. (`[1.1-2]`, `[2.3-2]`, `[CD-SR-2]`)
- [ ] `static_attribute_overrides` table stores per-instance, per-attribute runtime value overrides. (`[KDD-data-5]`, `[CD-SR-6]`)
- [ ] Schema includes an `enabled` flag per deployed instance to support disable/enable lifecycle. (`[CD-SR-8]`)
- [ ] Schema supports efficient lookup: all configs for startup, single config for deployment/lifecycle. (`[CD-SR-2]`)
- [ ] SQLite database file created at path from `DatabaseOptions`. (`[CD-HOST-5]`)
- [ ] Schema migration strategy for SQLite (code-first or explicit migration scripts).
- [ ] No template structure in site storage — only flattened configs. Schema does not include tables for templates, inheritance relationships, or composition relationships. Stored configs are the deployment package format (flat attribute sets). (`[1.1-2]`)
- [ ] No local configuration authoring or editing capability. (`[1.1-3]` — negative)

**Estimated Complexity**: M

**Requirements Traced**: `[1.1-2]`, `[1.1-3]`, `[2.3-2]`, `[KDD-data-5]`, `[CD-SR-2]`, `[CD-SR-6]`, `[CD-SR-8]`, `[CD-HOST-5]`

---

### WP-4: Deployment Manager Singleton

**Description**: Implement the Deployment Manager as an Akka.NET cluster singleton on site nodes. On startup (or failover recovery), it reads all deployed configurations from SQLite and creates Instance Actors for enabled instances in staggered batches.

**Acceptance Criteria**:
- [ ] Deployment Manager registered as an Akka.NET cluster singleton via `ClusterSingletonManager`. (`[CD-CI-3]`, `[CD-SR-1]`)
- [ ] Cluster singleton proxy registered for communication with the singleton. (`[CD-CI-3]`)
- [ ] On startup: reads all deployed configurations from local SQLite. (`[CD-SR-2]`, `[1.2-4]`, `[CD-CI-11]`)
- [ ] Creates Instance Actors only for **enabled** instances. (`[CD-SR-3]`)
- [ ] Instance Actors created in **staggered batches** (configurable batch size, e.g., 20, with configurable delay between batches). (`[KDD-runtime-8]`, `[CD-SR-3]`)
- [ ] Supervises Instance Actors with **OneForOneStrategy** — one failure does not affect others. (`[CD-SR-7]`)
- [ ] Supervision directive for Instance Actors is **Resume** (coordinator-level actors retain state across child failures). Verify: an Instance Actor that throws an unhandled exception resumes with its pre-exception state intact. (`[KDD-runtime-9]`)
- [ ] Actor hierarchy: Deployment Manager → Instance Actors (children). (`[KDD-runtime-2]`)
- [ ] Handles skeleton lifecycle messages: Deploy (store config, create actor), Disable (stop actor, mark disabled), Enable (re-create actor), Delete (stop actor, remove config). (`[CD-SR-8]`)
- [ ] Deploy does **not** include any local authoring — configs come from central only. (`[1.1-1]`, `[1.1-3]`)
- [ ] Delete does **not** clear store-and-forward messages. Implementation: delete logic only removes the `deployed_configurations` and `static_attribute_overrides` rows for the instance. It does not touch any other tables. When S&F tables are added in Phase 3C, this constraint is verified end-to-end. (`[CD-SR-8]` — negative)
- [ ] When an Instance Actor is stopped, Akka.NET automatically stops all child actors. (`[CD-SR-9]`)

**Estimated Complexity**: L

**Requirements Traced**: `[1.1-1]`, `[1.1-3]`, `[1.2-4]`, `[KDD-runtime-2]`, `[KDD-runtime-8]`, `[KDD-runtime-9]`, `[CD-CI-3]`, `[CD-CI-11]`, `[CD-SR-1]`, `[CD-SR-2]`, `[CD-SR-3]`, `[CD-SR-7]`, `[CD-SR-8]`, `[CD-SR-9]`

---

### WP-5: Instance Actor Skeleton

**Description**: Implement the basic Instance Actor that holds attribute state from the flattened configuration. In this phase, it loads attributes, supports GetAttribute/SetAttribute for static attributes (with SQLite persistence), and establishes the single-source-of-truth pattern. DCL integration, Script/Alarm Actors, and stream publishing are Phase 3B.

**Acceptance Criteria**:
- [ ] Instance Actor is the single source of truth for all runtime state of a deployed instance. (`[KDD-runtime-1]`, `[CD-SR-4]`)
- [ ] On initialization: loads all attribute values from flattened configuration (static defaults). (`[CD-SR-5]`)
- [ ] On initialization: loads persisted static attribute overrides from SQLite and applies them on top of deployed config defaults. (`[CD-SR-6]`, `[KDD-data-5]`)
- [ ] GetAttribute returns current in-memory value for requested attribute. (`[CD-SR-5]`)
- [ ] SetAttribute for static attributes: updates in-memory value and persists override to local SQLite. (`[CD-SR-6]`, `[KDD-data-5]`)
- [ ] Static attribute overrides survive restart and failover. (`[KDD-data-5]`)
- [ ] Static attribute overrides are reset when the instance is redeployed (new deployment clears previous overrides). (`[CD-SR-6]`, `[KDD-data-5]`)
- [ ] Internal communication uses Tell pattern for attribute updates. (`[KDD-data-7]`)
- [ ] Alarm state is **not** persisted — design explicitly excludes alarm persistence. (`[CD-CI-12]` — negative)

**Estimated Complexity**: M

**Requirements Traced**: `[KDD-runtime-1]`, `[KDD-data-5]`, `[KDD-data-7]`, `[CD-CI-12]`, `[CD-SR-4]`, `[CD-SR-5]`, `[CD-SR-6]`

---

### WP-6: CoordinatedShutdown & Graceful Singleton Handover

**Description**: Implement and verify CoordinatedShutdown for graceful singleton handover. When a site node is stopped (service stop, Ctrl+C), the Deployment Manager singleton hands over to the other node in seconds rather than waiting for full failure detection timeout.

**Acceptance Criteria**:
- [ ] CoordinatedShutdown triggers graceful leave from cluster on service stop. (`[KDD-cluster-4]`, `[CD-CI-8]`, `[CD-CI-9]`)
- [ ] Graceful shutdown enables singleton handover in seconds (hand-over retry interval), not ~25s. (`[KDD-cluster-4]`)
- [ ] Host does not call `Environment.Exit()` or forcibly terminate the actor system without coordinated shutdown. (REQ-HOST-9)
- [ ] CoordinatedShutdown wired into Windows Service lifecycle. (`[CD-CI-8]`)

**Estimated Complexity**: S

**Requirements Traced**: `[KDD-cluster-4]`, `[CD-CI-8]`, `[CD-CI-9]`

---

### WP-7: Dual-Node Recovery

**Description**: Implement and verify automatic recovery when both nodes in a site cluster fail simultaneously (e.g., power outage). Whichever node starts first forms a new cluster; the second joins. No manual intervention required.

**Acceptance Criteria**:
- [ ] Both nodes are seed nodes — either can start first and form a cluster. (`[KDD-cluster-2]`, `[CD-CI-4]`, `[CD-CI-10]`)
- [ ] `min-nr-of-members=1` allows single surviving node to operate. (`[KDD-cluster-2]`)
- [ ] First node starts, forms cluster, Deployment Manager singleton starts and rebuilds from SQLite. (`[CD-CI-10]`, `[CD-CI-11]`, `[KDD-cluster-5]`)
- [ ] Second node joins the cluster as standby. (`[CD-CI-10]`)
- [ ] No manual intervention required for recovery. (`[CD-CI-10]`)

**Estimated Complexity**: S

**Requirements Traced**: `[KDD-cluster-2]`, `[KDD-cluster-5]`, `[CD-CI-4]`, `[CD-CI-10]`, `[CD-CI-11]`

---

### WP-8: Failover Acceptance Tests

**Description**: Comprehensive integration/acceptance tests proving failover, recovery, and persistence semantics. These are the primary verification gate for Phase 3A.

**Acceptance Criteria**:

**Test: Active node crash → singleton migration**
- [ ] Kill the active node process. Standby detects failure within ~25s. (`[KDD-cluster-3]`)
- [ ] Deployment Manager singleton restarts on surviving node. (`[1.2-4]`, `[CD-CI-11]`)
- [ ] Singleton reads deployed configs from SQLite and recreates Instance Actors. (`[1.2-4]`, `[CD-SR-2]`)
- [ ] Instance Actors have correct attribute state (deployed defaults + persisted overrides). (`[CD-SR-5]`, `[CD-SR-6]`)

**Test: Graceful shutdown → fast singleton handover**
- [ ] Stop the active node gracefully (service stop). (`[KDD-cluster-4]`)
- [ ] Singleton hands over to standby in seconds (faster than crash scenario). (`[KDD-cluster-4]`)
- [ ] Instance Actors recreated on new active node. (`[1.2-4]`)

**Test: Both nodes down → first up forms cluster, rebuilds from SQLite**
- [ ] Both nodes stopped. (`[CD-CI-10]`)
- [ ] First node starts and forms cluster alone (seed node + `min-nr-of-members=1`). (`[KDD-cluster-2]`, `[CD-CI-10]`)
- [ ] Deployment Manager singleton starts and rebuilds Instance Actor hierarchy from SQLite. (`[KDD-cluster-5]`, `[CD-CI-11]`)
- [ ] Second node starts and joins as standby. (`[CD-CI-10]`)

**Test: Static attribute overrides survive failover**
- [ ] Set a static attribute via SetAttribute on Instance Actor. (`[KDD-data-5]`)
- [ ] Kill active node. Wait for failover. (`[KDD-cluster-3]`)
- [ ] On new active node, Instance Actor loads with persisted override value. (`[KDD-data-5]`, `[CD-SR-6]`)

**Test: Static attribute overrides reset on redeployment**
- [ ] Set a static attribute override. (`[KDD-data-5]`)
- [ ] Redeploy the instance (send new flattened config). (`[CD-SR-6]`)
- [ ] Instance Actor loads with new deployed defaults; persisted override is cleared. (`[CD-SR-6]`)

**Test: Staggered Instance Actor startup**
- [ ] Deploy many instances (e.g., 50+). (`[KDD-runtime-8]`)
- [ ] On startup/failover, verify Instance Actors are created in batches (default batch size configurable, e.g., 20) with observable delays between batches. (`[KDD-runtime-8]`, `[CD-SR-3]`)
- [ ] Verify batch size and delay are configurable via options. (`[CD-SR-3]`)

**Test: Instance lifecycle (disable/enable/delete)**
- [ ] Disable an instance: actor stopped, config retained in SQLite, not recreated on restart. (`[CD-SR-8]`)
- [ ] Enable a disabled instance: actor re-created from stored config. (`[CD-SR-8]`)
- [ ] Delete an instance: actor stopped, config removed from SQLite. (`[CD-SR-8]`)

**Test: Singleton on single node**
- [ ] Start only one node. Cluster forms. Singleton starts. Instances created. (`[KDD-cluster-2]`)
- [ ] Confirms `min-nr-of-members=1` works correctly. (`[KDD-cluster-2]`)

**Test: Negative — no local configuration authoring or overrides**
- [ ] Verify the Deployment Manager accepts configuration only via deployment messages (the central-to-site message path). No public method, message type, or API endpoint exists for local config creation or modification. (`[1.1-1]`, `[1.1-3]`)
- [ ] Verify that the only way to modify deployed config structure is via a new deployment from central. Static attribute SetAttribute modifies runtime values only, not config structure (attributes, scripts, alarms). (`[1.1-3]`)

**Test: Negative — site nodes are headless (no UI, no inbound HTTP)**
- [ ] Verify site node process does not bind any TCP listener on HTTP/HTTPS ports. Scan for open ports after startup. (`[2.3-1]`)
- [ ] Verify site-role Host uses `Host.CreateDefaultBuilder` (not `WebApplication.CreateBuilder`). (`[2.3-1]`)

**Test: Negative — no alarm state persistence**
- [ ] Verify no alarm state table or column exists in the SQLite schema. (`[CD-CI-12]`)
- [ ] Verify Instance Actor initialization does not attempt to load alarm state from any persistent store. (`[CD-CI-12]`)

**Test: Supervision — OneForOneStrategy isolation and Resume directive**
- [ ] One Instance Actor throws an unhandled exception. Other Instance Actors continue processing unaffected. (`[CD-SR-7]`)
- [ ] The failing Instance Actor resumes with its pre-exception in-memory state intact (Resume directive). (`[KDD-runtime-9]`)

**Estimated Complexity**: L

**Requirements Traced**: `[1.1-1]`, `[1.1-3]`, `[1.2-4]`, `[2.3-1]`, `[KDD-runtime-8]`, `[KDD-data-5]`, `[KDD-cluster-2]`, `[KDD-cluster-3]`, `[KDD-cluster-4]`, `[KDD-cluster-5]`, `[CD-CI-10]`, `[CD-CI-11]`, `[CD-CI-12]`, `[CD-CI-13]`, `[CD-SR-2]`, `[CD-SR-3]`, `[CD-SR-5]`, `[CD-SR-6]`, `[CD-SR-7]`, `[CD-SR-8]`

---

## Test Strategy

### Unit Tests

| Area | Tests |
|------|-------|
| SQLite schema | Table creation, CRUD operations for deployed configs and attribute overrides |
| Deployment Manager | Startup reads configs, creates correct number of Instance Actors, staggering logic |
| Instance Actor | Attribute loading from flattened config, static override load/save, override reset on redeploy |
| Cluster config | HOCON/Akka.Hosting configuration generates correct settings (SBR, seed nodes, timings) |
| Lifecycle commands | Deploy/Disable/Enable/Delete state transitions, SQLite side effects |

### Integration Tests

| Area | Tests |
|------|-------|
| Two-node cluster formation | Two processes join cluster, leader elected, singleton starts on oldest |
| Singleton migration on crash | Kill active process, singleton restarts on standby |
| Graceful handover | CoordinatedShutdown on active, measure handover time |
| Dual-node recovery | Both down, first up forms cluster, second joins |
| Static attribute persistence | Set override, restart, verify value survives |
| Staggered startup | Deploy 50+ instances, verify batch creation with timing |

### Negative Tests

| Requirement | Test |
|-------------|------|
| `[1.1-1]` No local authoring | Verify Deployment Manager only accepts configs via deployment messages from central. No local creation path exists. |
| `[1.1-3]` No local overrides | Verify no mechanism to modify deployed config structure locally. SetAttribute modifies runtime values only, not config structure. |
| `[2.3-1]` No HTTP on site | Verify no TCP listener on any HTTP port. Verify `Host.CreateDefaultBuilder` used (not `WebApplication`). |
| `[CD-CI-12]` No alarm persistence | Verify no alarm state table/column in SQLite. Verify Instance Actor does not load alarm state from storage. |
| `[CD-SR-8]` Delete does not clear S&F | Verify delete only removes deployed_configurations and static_attribute_overrides rows. Does not touch other tables. |
| `[KDD-runtime-9]` Resume directive | Verify Instance Actor resumes with intact state after unhandled exception (not restarted from scratch). |

### Failover Tests

See WP-8 for complete failover test scenarios.

---

## Verification Gate

Phase 3A is complete when **all** of the following pass:

1. Two-node site cluster forms reliably with keep-oldest SBR.
2. Deployment Manager singleton starts on oldest node and creates Instance Actors from SQLite.
3. Instance Actors hold correct attribute state (deployed defaults + persisted overrides).
4. Active node crash triggers failover: singleton migrates, Instance Actors recreated within ~25s.
5. Graceful shutdown triggers fast handover (seconds, not ~25s).
6. Both-nodes-down recovery works with no manual intervention.
7. `min-nr-of-members=1` allows single-node operation.
8. Static attribute overrides persist across restart and failover.
9. Static attribute overrides reset on redeployment.
10. Instance lifecycle (disable/enable/delete) works correctly.
11. Staggered Instance Actor startup is observable.
12. All negative tests pass (no HTTP on site, no local authoring, no alarm persistence).
13. All unit and integration tests pass.

---

## Open Questions

| # | Question | Context | Impact | Status |
|---|----------|---------|--------|--------|
| Q-P3A-1 | What is the optimal batch size and delay for staggered Instance Actor startup? | docs/requirements/Component-SiteRuntime.md suggests 20 with a "short delay." Actual values depend on OPC UA server capacity. | Performance tuning. Default to 20/100ms, make configurable. | Deferred — tune during Phase 3B when DCL is integrated. |
| Q-P3A-2 | Should the SQLite schema use a single database file or separate files per concern (configs, overrides, S&F, events)? | Single file is simpler. Separate files isolate concerns and allow independent backup/maintenance. | Schema design. | Recommend single file with separate tables. Simpler transaction management. Final decision during implementation. |
| Q-P3A-3 | Should Akka.Persistence (event sourcing / snapshotting) be used for the Deployment Manager singleton, or is direct SQLite access sufficient? | Akka.Persistence adds complexity (journal, snapshots) but provides built-in recovery. Direct SQLite is simpler for this use case (singleton reads all configs on startup). | Architecture. | Recommend direct SQLite — Deployment Manager recovery is a full read-all-configs-and-rebuild pattern, not event replay. Akka.Persistence is overkill here. |

---

## Orphan Check Result

### Forward Check (Requirements → Work Packages)

Every item in the Requirements Checklist and Design Constraints Checklist is verified against work packages:

| Requirement/Constraint | Work Package(s) | Verified |
|----------------------|-----------------|----------|
| `[1.1-1]` Central is single source of truth | WP-4 (negative: no local authoring), WP-8 (negative test) | Yes |
| `[1.1-2]` Sites receive flattened configs | WP-3 (schema), WP-4 (reads configs), WP-5 (loads attributes) | Yes |
| `[1.1-3]` Sites do not support local overrides | WP-3 (negative), WP-4 (negative), WP-8 (negative test) | Yes |
| `[1.2-1]` Failover at application level (Akka.NET) | WP-1 (config) | Yes |
| `[1.2-2]` Active/standby pair | WP-1, WP-2 | Yes |
| `[1.2-3]` Site failover: standby takes over | WP-8 (failover tests) | Yes |
| `[1.2-4]` Singleton restarts, reads SQLite, recreates hierarchy | WP-4, WP-8 | Yes |
| `[1.2-5]` Central failover (Phase 8) | Out of scope — noted in split-section | Yes |
| `[2.3-1]` Sites are headless | WP-2 (no Kestrel), WP-8 (negative test) | Yes |
| `[2.3-2]` Local storage for deployed configs | WP-3 (schema) | Yes |
| `[2.3-3]` S&F buffers (Phase 3C) | Out of scope — noted in split-section | Yes |
| `[KDD-runtime-1]` Instance as Akka actor | WP-5 | Yes |
| `[KDD-runtime-2]` Actor hierarchy | WP-4, WP-5 | Yes |
| `[KDD-runtime-8]` Staggered startup | WP-4, WP-8 | Yes |
| `[KDD-runtime-9]` Supervision strategies | WP-4 | Yes |
| `[KDD-data-5]` Static attribute SQLite persistence | WP-3, WP-5, WP-8 | Yes |
| `[KDD-data-7]` Tell for hot-path | WP-5 | Yes |
| `[KDD-cluster-1]` Keep-oldest SBR | WP-1 | Yes |
| `[KDD-cluster-2]` Both seed nodes, min-nr=1 | WP-1, WP-7, WP-8 | Yes |
| `[KDD-cluster-3]` Failure detection timing | WP-1, WP-8 | Yes |
| `[KDD-cluster-4]` CoordinatedShutdown | WP-1, WP-6, WP-8 | Yes |
| `[KDD-cluster-5]` Dual-node recovery | WP-7, WP-8 | Yes |
| `[CD-CI-1]` through `[CD-CI-13]` | WP-1, WP-2, WP-4, WP-6, WP-7, WP-8 | Yes |
| `[CD-SR-1]` through `[CD-SR-9]` | WP-3, WP-4, WP-5, WP-8 | Yes |
| `[CD-HOST-1]` through `[CD-HOST-5]` | WP-1, WP-2, WP-3 | Yes |

**Result**: All checklist items map to at least one work package. **No orphans.**

### Reverse Check (Work Packages → Requirements)

| Work Package | Requirements Traced | Verified |
|-------------|-------------------|----------|
| WP-1 | `[1.2-1]`, `[1.2-2]`, `[KDD-cluster-1–4]`, `[CD-CI-1,2,4,5,6,8,9,13]`, `[CD-HOST-1,4]` | Yes |
| WP-2 | `[1.2-2]`, `[2.3-1]`, `[CD-HOST-1,2,3,4,5]`, `[CD-CI-1]` | Yes |
| WP-3 | `[1.1-2]`, `[1.1-3]`, `[2.3-2]`, `[KDD-data-5]`, `[CD-SR-2,6,8]`, `[CD-HOST-5]` | Yes |
| WP-4 | `[1.1-1]`, `[1.1-3]`, `[1.2-4]`, `[KDD-runtime-2,8,9]`, `[CD-CI-3,11]`, `[CD-SR-1,2,3,7,8,9]` | Yes |
| WP-5 | `[KDD-runtime-1]`, `[KDD-data-5,7]`, `[CD-CI-12]`, `[CD-SR-4,5,6]` | Yes |
| WP-6 | `[KDD-cluster-4]`, `[CD-CI-8,9]` | Yes |
| WP-7 | `[KDD-cluster-2,5]`, `[CD-CI-4,10,11]` | Yes |
| WP-8 | All requirements verified via tests | Yes |

**Result**: Every work package traces to at least one requirement or constraint. **No untraceable work.**

### Split-Section Check

| Section | This Phase Covers | Other Phase Covers | Gap |
|---------|------------------|-------------------|-----|
| 1.1 | `[1.1-1]`, `[1.1-2]`, `[1.1-3]` (all bullets) | — | None |
| 1.2 | `[1.2-1]`, `[1.2-2]`, `[1.2-3]` (singleton portion), `[1.2-4]` | Phase 8: `[1.2-3]` (full DCL/S&F), `[1.2-5]` (central) | None |
| 2.3 | `[2.3-1]`, `[2.3-2]` (deployed configs) | Phase 3B/3C: `[2.3-2]` (scripts, artifacts), `[2.3-3]` (S&F) | None |

**Result**: No gaps in split-section coverage.

### Negative Requirement Check

| Negative Requirement | Acceptance Criterion | Sufficient |
|---------------------|---------------------|------------|
| `[1.1-1]` No local authoring | WP-4: Configs accepted only via deployment messages. WP-8: Verify no local creation path exists. | Yes |
| `[1.1-3]` No local overrides | WP-3: No config structure modification API. WP-8: Verify SetAttribute modifies runtime values only, not config structure. | Yes |
| `[2.3-1]` No HTTP on site | WP-2: `Host.CreateDefaultBuilder` (no Kestrel). WP-8: Port scan + builder type verification. | Yes |
| `[CD-CI-12]` No alarm persistence | WP-5: No alarm state in SQLite. WP-8: Verify no alarm table/column and no alarm state load on init. | Yes |
| `[CD-SR-8]` Delete does not clear S&F | WP-4: Delete only removes deployed_configurations and static_attribute_overrides rows. End-to-end S&F verification in Phase 3C. | Yes |
| `[KDD-runtime-9]` Resume directive | WP-4: Resume directive on Instance Actors. WP-8: Verify Instance Actor retains state after exception. | Yes |

**Result**: All negative requirements have explicit behavioral acceptance criteria. Criteria strengthened after Codex review.

---

## Codex MCP Verification

**Model**: gpt-5.4
**Date**: 2026-03-16
**Result**: Pass with corrections applied.

### Step 1 — Requirements Coverage Review

Codex identified 15 findings. Disposition:

| # | Finding | Disposition |
|---|---------|-------------|
| 1 | `[1.2-3]` not fully covered (DCL, scripts, S&F) | **Acknowledged — by design.** Phase 3A covers singleton migration foundation only. Split-section notes updated to explicitly list Phase 3B (DCL, scripts) and Phase 3C (S&F) ownership. |
| 2 | "Full hierarchy" means Script/Alarm Actors too | **Acknowledged — clarified.** Added scope note to `[1.2-4]` explaining "full hierarchy" in this phase means DM → Instance Actors; Script/Alarm Actors added in Phase 3B. |
| 3 | `[2.3-2]` missing deployed scripts, ext sys defs, etc. | **Acknowledged — by design.** Split-section note updated to list all 6 storage categories with per-phase ownership. |
| 4 | `[2.3-3]` S&F not covered | **Acknowledged — by design.** Explicitly deferred to Phase 3C in split-section. |
| 5 | `[1.2-5]` central failover not covered | **Acknowledged — by design.** Deferred to Phase 8 per phase definition. |
| 6 | REQ-HOST-2 only partially covered (missing DCL, S&F, SiteEventLogging registration) | **Acknowledged.** `[CD-HOST-3]` already notes "only SiteRuntime is wired in this phase; others are stubs." Stub registrations are sufficient for Phase 3A. |
| 7 | Script compilation not covered in startup | **Acknowledged — by design.** Script compilation is Phase 3B. Startup step 3 (compile scripts) is deferred. |
| 8 | Site identifier not explicit in cluster role | **Corrected.** WP-1 acceptance criterion updated to include site identifier. |
| 9 | Database path configuration not explicit | **Dismissed.** Already covered by `[CD-HOST-5]` in WP-2 and WP-3 acceptance criteria. |
| 10 | Supervision strategy (Resume) not verified | **Corrected.** WP-4 and WP-8 updated with explicit Resume directive verification. |
| 11 | Ask boundary rule not tested | **Dismissed.** `[KDD-data-7]` in Phase 3A scope note says "Establish the convention." Ask pattern usage is Phase 3B (CallScript). WP-5 verifies Tell usage. |
| 12 | Delete S&F negative not verifiable yet | **Corrected.** WP-4 criterion strengthened to specify delete only removes config/override rows. End-to-end S&F verification deferred to Phase 3C. |
| 13 | Alarm re-evaluation not tested | **Dismissed.** Alarm Actors are Phase 3B. `[CD-CI-12]` in Phase 3A scope is limited to "no alarm persistence." Re-evaluation is Phase 3B concern. |
| 14 | Flattened config not explicitly verified | **Corrected.** WP-3 criterion strengthened to verify schema has no template/inheritance/composition tables. |
| 15 | `[1.1-3]` negative test too narrow | **Corrected.** WP-8 negative test expanded to verify SetAttribute modifies runtime values only, not config structure. |

### Step 2 — Negative Requirement Review

Codex flagged all 5 negative requirements as weak. Disposition:

| # | Finding | Disposition |
|---|---------|-------------|
| 1 | `[1.1-1]` only checks one mechanism | **Corrected.** Test expanded to verify Deployment Manager only accepts configs via deployment messages. |
| 2 | `[1.1-3]` misses override paths | **Corrected.** Test expanded to verify SetAttribute modifies runtime values only. |
| 3 | `[2.3-1]` "headless" vs "no HTTP" misaligned | **Partially corrected.** Added Host builder type verification. The "no HTTP" test is the practical enforcement of "headless" — site nodes have no web framework loaded. |
| 4 | `[CD-CI-12]` schema check insufficient | **Corrected.** Added verification that Instance Actor does not attempt to load alarm state from storage. |
| 5 | `[CD-SR-8]` delete S&F just restated | **Corrected.** Specified delete only removes config/override rows. |

### Step 3 — Split-Section Gap Review

Codex found:
- `[1.2-3]` double-assigned (Phase 3A and Phase 8): **Intentional** — Phase 3A proves the foundation, Phase 8 validates the full behavior. Different aspects.
- `[2.3-2]` triple-assigned: **Intentional** — compound bullet decomposed into sub-items across phases. Split-section note now lists all 6 storage categories with explicit per-phase ownership.
- `[1.2-5]` numbering concern: **Clarified.** Section 1.2 has 4 prose bullets but bullet 3 contains two distinct requirements (site failover mechanics + singleton restart behavior), hence 5 atomic IDs. Split-section note updated.
