scadalink-design/docs/plans/phase-8-production-readiness.md

Phase 8: Production Readiness & Hardening

Date: 2026-03-16 Status: Plan complete Goal: System is validated at target scale and ready for production deployment.

---

Scope

Components: Cross-cutting (all 17 components)

Features:

• Full-system failover testing (central + site)
• Dual-node failure recovery
• Load/performance testing at target scale
• Security hardening
• Script sandboxing verification
• Recovery drills
• Observability validation
• Message contract compatibility
• Installer/deployment packaging
• Operational documentation

Note: This phase is for comprehensive resilience and scale testing. Basic failover testing is embedded in Phases 3A and 3C. This phase validates the full system under realistic conditions.

---

Prerequisites

Phase	What must be complete
Phases 0–7	All components fully implemented and individually tested

---

Requirements Checklist

Section 1.2 — Failover (full-system validation)

• [1.2-1-val] Failover managed at application level using Akka.NET (not WSFC) — verified under load
• [1.2-2-val] Active/standby pair per cluster — verified for both central and site
• [1.2-3-val] Site failover: standby takes over data collection, scripts, S&F buffers seamlessly — verified end-to-end
• [1.2-4-val] Site failover: Deployment Manager singleton restarts, reads SQLite, re-creates full Instance Actor hierarchy — verified at scale
• [1.2-5-val] Central failover: standby takes over. In-progress deployments treated as failed — verified
• [1.2-6-val] Central failover: JWT sessions survive (shared Data Protection keys) — verified

Section 2.5 — Scale

• [2.5-1] Approximately 10 sites — system tested with 10 site clusters
• [2.5-2] 50–500 machines per site — tested at upper bound (500 instances per site)
• [2.5-3] 25–75 live data point tags per machine — tested at upper bound (75 tags per instance)

Cross-Cutting Validation (all sections, final pass)

• [xc-1] All 8 Communication Layer message patterns work correctly under load
• [xc-2] Health monitoring accurate at scale (30s reports, 60s offline detection)
• [xc-3] Site event logging handles high event volume within retention/storage limits
• [xc-4] Audit logging captures all system-modifying actions without performance degradation
• [xc-5] Template flattening/validation performs within acceptable time at scale
• [xc-6] Debug view streams data at scale without impacting site performance
• [xc-7] Store-and-forward handles concurrent buffering from multiple instances
• [xc-8] All UI workflows responsive under load

---

Design Constraints Checklist

ID	Constraint	Source	Mapped WP
KDD-runtime-8	Staggered Instance Actor startup on failover to prevent reconnection storms	CLAUDE.md	WP-1
KDD-runtime-9	Supervision: Resume for coordinator actors, Stop for short-lived execution actors	CLAUDE.md	WP-1
KDD-cluster-1	Keep-oldest SBR with down-if-alone=on, 15s stable-after	CLAUDE.md	WP-1
KDD-cluster-2	Both nodes are seed nodes, min-nr-of-members=1	CLAUDE.md	WP-3
KDD-cluster-3	Failure detection: 2s heartbeat, 10s threshold, ~25s total failover	CLAUDE.md	WP-1
KDD-cluster-4	CoordinatedShutdown for graceful singleton handover	CLAUDE.md	WP-1
KDD-cluster-5	Automatic dual-node recovery from persistent storage	CLAUDE.md	WP-3
KDD-sec-1	LDAPS/StartTLS required, no Kerberos/NTLM	CLAUDE.md	WP-5
KDD-sec-2	JWT HMAC-SHA256, 15-min expiry, 30-min idle timeout	CLAUDE.md	WP-5
KDD-sec-3	LDAP failure: new logins fail; active sessions continue	CLAUDE.md	WP-5
KDD-sec-4	Load balancer + JWT + shared Data Protection keys for failover	CLAUDE.md	WP-1
KDD-code-9	Script trust model: forbidden APIs (System.IO, Process, Threading, Reflection, raw network)	CLAUDE.md	WP-6
KDD-code-4	Message contracts follow additive-only evolution rules	CLAUDE.md	WP-9
KDD-data-8	Application-level correlation IDs on all request/response messages	CLAUDE.md	WP-8
KDD-sf-2	Async best-effort replication to standby (no ack wait)	CLAUDE.md	WP-2
KDD-deploy-6	Deployment ID + revision hash for idempotency	CLAUDE.md	WP-7
KDD-deploy-8	Site-side apply is all-or-nothing per instance	CLAUDE.md	WP-7
KDD-deploy-9	System-wide artifact version skew across sites supported	CLAUDE.md	WP-9

---

Work Packages

WP-1: Full-System Failover Testing — Central (L)

Description: Comprehensive failover testing of the central cluster under realistic conditions.

Acceptance Criteria:

• Central active node crash → standby takes over central responsibilities ([1.2-5-val])
• In-progress deployments treated as failed on central failover ([1.2-5-val])
• JWT sessions survive failover — users not required to re-login ([1.2-6-val], KDD-sec-4)
• Blazor Server SignalR circuit reconnects automatically
• Load balancer routes to new active node via /health/ready endpoint
• Inbound API available on new active node
• Debug view streams interrupted — user must re-open (verified)
• Graceful shutdown via CoordinatedShutdown enables fast handover (KDD-cluster-4)
• Keep-oldest SBR with down-if-alone works correctly (KDD-cluster-1)
• Failure detection timing verified: ~25s total failover (KDD-cluster-3)
• All tests executed with 10 sites connected and active traffic ([2.5-1])

Complexity: L Traces: [1.2-2-val], [1.2-5-val], [1.2-6-val], KDD-cluster-1, KDD-cluster-3, KDD-cluster-4, KDD-sec-4

---

WP-2: Full-System Failover Testing — Site (L)

Description: Comprehensive failover testing of site clusters under realistic conditions.

Acceptance Criteria:

• Site active node crash → standby takes over data collection, script execution, alarm evaluation ([1.2-3-val])
• Deployment Manager singleton restarts, reads SQLite, re-creates Instance Actor hierarchy ([1.2-4-val])
• Staggered Instance Actor startup prevents reconnection storms (KDD-runtime-8)
• Supervision strategies verified: coordinator actors resume, execution actors stop (KDD-runtime-9)
• S&F buffer takeover seamless — standby has replicated copy ([1.2-3-val], KDD-sf-2)
• Data Connection Layer re-establishes subscriptions after failover
• Alarm states re-evaluated from incoming values (not persisted)
• Static attribute overrides survive failover (persisted to SQLite)
• Health reporting resumes from new active node
• Failover managed at application level (Akka.NET), not WSFC ([1.2-1-val])
• All tests executed with 500 instances and 75 tags per instance ([2.5-2], [2.5-3])

Complexity: L Traces: [1.2-1-val]–[1.2-4-val], KDD-runtime-8, KDD-runtime-9, KDD-sf-2

---

WP-3: Dual-Node Failure Recovery (M)

Description: Verify both nodes in a cluster can fail simultaneously and recover automatically.

Acceptance Criteria:

• Both central nodes down → first node up forms cluster, resumes from MS SQL (KDD-cluster-5)
• Both site nodes down → first node up forms cluster, Deployment Manager reads SQLite, rebuilds hierarchy (KDD-cluster-5)
• Both nodes configured as seed nodes — either can start first (KDD-cluster-2)
• min-nr-of-members=1 allows single-node operation (KDD-cluster-2)
• Second node joins when it starts — no manual intervention required
• S&F buffer recovery from local SQLite on each node

Complexity: M Traces: KDD-cluster-2, KDD-cluster-5

---

WP-4: Load/Performance Testing at Target Scale (L)

Description: Validate system performance at the documented scale targets.

Acceptance Criteria:

• 10 sites simultaneously connected and operational ([2.5-1])
• 500 instances per site with active data subscriptions ([2.5-2])
• 75 live tags per instance (37,500 subscriptions per site, 375,000 total) ([2.5-3])
• All Communication Layer message patterns function correctly under load ([xc-1])
• Health reports arrive at central within expected intervals ([xc-2])
• Site event logging handles volume within 30-day/1GB limits ([xc-3])
• Audit logging does not degrade central performance ([xc-4])
• Template flattening/validation completes within acceptable time for large templates ([xc-5])
• Debug view streams without impacting site performance ([xc-6])
• S&F handles concurrent buffering from multiple instances ([xc-7])
• UI workflows remain responsive ([xc-8])
• Deployment of 500 instances to a site completes within acceptable time
• Memory usage, CPU, and disk I/O within acceptable bounds on target hardware

Complexity: L Traces: [2.5-1]–[2.5-3], [xc-1]–[xc-8]

---

WP-5: Security Hardening (M)

Description: Validate and harden all security surfaces.

Acceptance Criteria:

• LDAPS/StartTLS enforced — unencrypted LDAP connections rejected (KDD-sec-1)
• No Kerberos/NTLM — only direct LDAP bind (KDD-sec-1)
• JWT HMAC-SHA256 signing verified, 15-min sliding refresh works correctly (KDD-sec-2)
• 30-minute idle timeout enforced (KDD-sec-2)
• LDAP failure: new logins fail, active sessions continue with current roles (KDD-sec-3)
• LDAP recovery: next token refresh re-queries group memberships
• JWT signing key rotation procedure documented and tested
• Secrets management: connection strings, API keys, LDAP credentials, JWT signing key stored securely
• API key auth for Inbound API validated (invalid/disabled/unapproved keys rejected)
• Site-scoped Deployment role permissions enforced correctly
• All credential fields in external system definitions and DB connection definitions are not exposed in API responses or logs

Complexity: M Traces: KDD-sec-1, KDD-sec-2, KDD-sec-3

---

WP-6: Script Sandboxing Verification (M)

Description: Verify script trust model under adversarial conditions.

Acceptance Criteria:

• Forbidden APIs blocked at compilation: System.IO, System.Diagnostics.Process, System.Threading (except async/await), System.Reflection, System.Net.Sockets, System.Net.Http, assembly loading, unsafe code (KDD-code-9)
• Scripts that attempt to use forbidden APIs fail compilation with clear error
• Execution timeout enforced — runaway scripts canceled and error logged
• Recursion limit enforced (default 10 levels) — exceeded calls fail with error
• Scripts cannot access other instances' attributes or scripts
• Alarm on-trigger scripts can call instance scripts; reverse not possible
• Adversarial test scripts (attempting forbidden operations) verified blocked

Complexity: M Traces: KDD-code-9

---

WP-7: Recovery Drills (M)

Description: Simulate realistic failure scenarios that span multiple components.

Acceptance Criteria:

• Mid-deploy failover: central crashes during deployment → deployment treated as failed → re-initiate succeeds
• Mid-deploy failover: site crashes during deployment → deployment fails → redeploy after recovery succeeds
• Deployment ID + revision hash idempotency verified — duplicate deployment ID is not re-applied (KDD-deploy-6)
• Site-side all-or-nothing apply: script compilation failure rejects entire deployment (KDD-deploy-8)
• Communication drop during system-wide artifact deployment → partial success → retry failed sites
• Site restart during S&F delivery → buffer recovered from SQLite → delivery resumes
• Central-to-site communication loss → site continues operating with last config → reconnect → operations resume

Complexity: M Traces: KDD-deploy-6, KDD-deploy-8

---

WP-8: Observability Validation (S)

Description: Validate that structured logging, correlation IDs, and health dashboard provide sufficient operational visibility.

Acceptance Criteria:

• All log entries include SiteId, NodeHostname, NodeRole enrichment
• Application-level correlation IDs present on all request/response messages (KDD-data-8)
• Correlation ID traceable across central → Communication Layer → site → response
• Health dashboard accurately reflects site status at scale
• Dead letter monitoring reports dead letter count as health metric
• Site event log captures all expected event categories
• Log output is structured and machine-parseable (Serilog)

Complexity: S Traces: KDD-data-8

---

WP-9: Message Contract Compatibility (M)

Description: Verify message contract versioning and cross-version compatibility.

Acceptance Criteria:

• Message contracts follow additive-only evolution rules (KDD-code-4)
• Central running version N can communicate with site running version N-1 (and vice versa)
• System-wide artifact version skew works — sites with different artifact versions operate correctly (KDD-deploy-9)
• New fields added to messages do not break older receivers (additive-only)
• Missing optional fields handled gracefully by newer receivers

Complexity: M Traces: KDD-code-4, KDD-deploy-9

---

WP-10: Installer/Deployment Packaging (M)

Description: Create production deployment packaging for the single binary.

Acceptance Criteria:

• Windows Service installer/setup for both central and site nodes
• appsettings.json templates for central and site configurations
• Role-based configuration documented (which settings differ between central and site)
• EF Core migration: manual SQL scripts for production (KDD-code-8 from Phase 1)
• SQLite database paths configurable and validated at startup
• Connection string management for production environments

Complexity: M Traces: (cross-cutting — all components)

---

WP-11: Operational Documentation (M)

Description: Create runbooks and operational documentation for production operation.

Acceptance Criteria:

• Failover procedures documented (expected behavior, recovery steps)
• Monitoring guide: health dashboard interpretation, alert thresholds
• Troubleshooting guide: common failure scenarios and resolution
• Security administration: LDAP group mapping, API key management, JWT key rotation
• Deployment procedures: instance deployment, system-wide artifact deployment
• Backup/recovery: database backup procedures, SQLite recovery
• Scaling guide: adding new sites, capacity planning
• Configuration reference: all appsettings.json sections documented

Complexity: M Traces: (cross-cutting — all components)

---

Test Strategy

Failover Tests (WP-1, WP-2, WP-3)

• Central active node kill during deployment → verify failed status → redeploy
• Central active node kill during UI session → verify JWT survival and SignalR reconnect
• Site active node kill during script execution → verify standby takeover
• Site active node kill during S&F delivery → verify buffer takeover
• Both nodes simultaneous kill → recovery from persistent storage
• Graceful shutdown → verify fast singleton handover

Performance Tests (WP-4)

• Sustained operation at 10 sites × 500 instances × 75 tags for 1 hour minimum
• Measure: tag update latency, deployment time, memory growth, CPU utilization
• Health report delivery timing at scale
• Debug view latency at scale
• Concurrent deployment to multiple instances at same site

Security Tests (WP-5)

• Attempt unencrypted LDAP connection → verify rejected
• Token expiry and refresh cycle verification
• Idle timeout enforcement
• LDAP server unavailable → verify login fails, active sessions continue
• Cross-site permission enforcement (site-scoped Deployment role)

Sandboxing Tests (WP-6)

• Script attempting File.ReadAllText → compilation fails
• Script attempting Process.Start → compilation fails
• Script attempting Thread creation → compilation fails
• Script attempting HttpClient direct use → compilation fails
• Script with infinite loop → timeout cancellation
• Script exceeding recursion limit → error logged
• Script attempting cross-instance access → fails

Recovery Drills (WP-7)

• Power outage simulation (both nodes down, cold start)
• Network partition between central and site → reconnection and resume
• Mid-deployment central failover → failed → redeploy
• S&F buffer recovery after site restart

---

Verification Gate

Phase 8 is complete when:

1. All 11 work packages pass acceptance criteria
2. System operates correctly at target scale (10 sites, 500 machines, 75 tags) for sustained period
3. All failover scenarios pass (central, site, dual-node, mid-operation)
4. Security hardening verified (LDAPS, JWT lifecycle, script sandboxing)
5. Observability provides sufficient operational visibility (correlation IDs traceable, health dashboard accurate)
6. Message contract compatibility verified across version skew
7. Deployment packaging ready for production
8. Operational documentation complete
9. No workflow requires direct database access — all operations available through UI or API
10. Documented pass on all test scenarios

---

Open Questions

No new questions discovered during Phase 8 plan generation.

---

Split-Section Verification

Section	Phase 8 Bullets	Other Phase(s)	Other Phase Bullets
1.2	[1.2-1-val]–[1.2-6-val] (full-system validation)	Phase 3A	Site failover mechanics (basic)
2.5	[2.5-1]–[2.5-3] (all — Phase 8 owns entirely)	—	No split

Phase 8 is primarily a validation/hardening phase. It does not introduce new functional requirements but validates that all existing requirements (from all sections across all phases) work correctly under realistic conditions. The Requirements Checklist focuses on the sections explicitly assigned to Phase 8 in the traceability matrix (1.2 validation, 2.5 scale). Cross-cutting validation items ([xc-*]) cover the full-system verification pass.

---

Orphan Check Result

Forward check: Every Requirements Checklist item and Design Constraints Checklist item maps to at least one work package with acceptance criteria that would fail if the requirement were not implemented. PASS.

Reverse check: Every work package traces back to at least one requirement or design constraint. WP-10 and WP-11 are cross-cutting packaging/documentation work that serves all components and all requirements — they do not map to individual requirement bullets but are necessary for the "production ready" outcome. PASS.

Split-section check: Phase 8 shares section 1.2 with Phase 3A. Phase 3A covers the basic failover mechanics (singleton migration, SQLite recovery). Phase 8 validates these at full scale under realistic conditions with all components running. No unassigned bullets. PASS.

Negative requirement check: Negative requirements validated in this phase:

• Script forbidden APIs → adversarial sandboxing tests (WP-6)
• No Kerberos/NTLM → verified in security hardening (WP-5)
• No unencrypted LDAP → verified in security hardening (WP-5)
• Scripts cannot access other instances → verified in sandboxing (WP-6)
• No workflow requires direct DB access → verified in verification gate item 9

PASS.

Codex MCP verification: Skipped — external tool verification deferred.