021817930b
All phases (0-8) now have detailed implementation plans with: - Bullet-level requirement extraction from HighLevelReqs sections - Design constraint traceability (KDD + Component Design) - Work packages with acceptance criteria mapped to every requirement - Split-section ownership verified across phases - Orphan checks (forward, reverse, negative) all passing - Codex MCP (gpt-5.4) external verification completed per phase Total: 7,549 lines across 11 plan documents, ~160 work packages, ~400 requirements traced, ~25 open questions logged for follow-up.
505 lines
26 KiB
Markdown
505 lines
26 KiB
Markdown
# Phase 7: Integration Surfaces
|
||
|
||
**Date**: 2026-03-16
|
||
**Status**: Plan complete
|
||
**Goal**: External systems can call in and site scripts can call out.
|
||
|
||
---
|
||
|
||
## Scope
|
||
|
||
**Components**:
|
||
- Inbound API (full runtime)
|
||
- External System Gateway (full site-side execution)
|
||
- Notification Service (full site-side delivery)
|
||
|
||
**Features**:
|
||
- Inbound API: ASP.NET endpoint, X-API-Key auth, method routing, parameter validation, script execution on central, Route.To() cross-site calls, batch attribute operations, error handling, failures-only logging
|
||
- External System Gateway: HTTP/REST client, API key + Basic Auth, per-system timeout, dual call modes (Call/CachedCall), error classification, database access (synchronous + cached write), dedicated blocking I/O dispatcher
|
||
- Notification Service: SMTP with OAuth2 Client Credentials (M365) + Basic Auth, token lifecycle, BCC delivery plain text, timeout + max connections, error classification, S&F integration
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
| Phase | What must be complete |
|
||
|-------|-----------------------|
|
||
| Phase 1 | Security & Auth (API key storage in config DB), Configuration Database |
|
||
| Phase 2 | Template Engine (instance/template model for Route.To resolution) |
|
||
| Phase 3A | Cluster Infrastructure, Site Runtime (Instance Actor, Script Actor, Script Execution Actor) |
|
||
| Phase 3B | Communication Layer (integration routing pattern), Site Runtime (script runtime API framework, Script Execution Actor lifecycle, dedicated dispatcher) |
|
||
| Phase 3C | Store-and-Forward Engine (buffering, retry, parking, replication) |
|
||
| Phase 4 | API key management UI (Admin role) |
|
||
| Phase 5 | External system definition UI, DB connection definition UI, notification list UI, inbound API method definition UI |
|
||
|
||
---
|
||
|
||
## Requirements Checklist
|
||
|
||
### Section 5.1 — External System Definitions (runtime portion)
|
||
- [ ] `[5.1-1-rt]` Definitions are deployed uniformly to all sites — site-side can load deployed definitions
|
||
- [ ] `[5.1-2-rt]` Connection details (URL, auth, protocol) used at runtime for HTTP calls
|
||
|
||
### Section 5.2 — Site-to-External-System Communication
|
||
- [ ] `[5.2-1]` Sites communicate with external systems directly (not through central)
|
||
- [ ] `[5.2-2]` Scripts invoke external system methods by referencing predefined definitions
|
||
|
||
### Section 5.3 — Store-and-Forward for External Calls (integration portion)
|
||
- [ ] `[5.3-1-int]` If external system unavailable, message buffered locally at site
|
||
- [ ] `[5.3-2-int]` Retry per message — individual failed messages retry independently
|
||
- [ ] `[5.3-3-int]` Configurable retry settings: max retry count, fixed time between retries
|
||
- [ ] `[5.3-4-int]` After max retries exhausted, message is parked
|
||
- [ ] `[5.3-5-int]` No maximum buffer size
|
||
|
||
### Section 5.5 — Database Connections (runtime portion)
|
||
- [ ] `[5.5-1-rt]` Definitions deployed to sites and loadable at runtime
|
||
- [ ] `[5.5-2-rt]` Retry settings applied to cached writes
|
||
|
||
### Section 5.6 — Database Access Modes
|
||
- [ ] `[5.6-1]` Real-time (synchronous): `Database.Connection("name")` returns raw MS SQL client connection (ADO.NET)
|
||
- [ ] `[5.6-2]` Full ADO.NET control: queries, updates, transactions, stored procedures
|
||
- [ ] `[5.6-3]` Cached write (store-and-forward): `Database.CachedWrite("name", "sql", parameters)`
|
||
- [ ] `[5.6-4]` Cached entry stores: database connection name, SQL statement, parameter values
|
||
- [ ] `[5.6-5]` If database unavailable, write buffered locally and retried per retry settings
|
||
- [ ] `[5.6-6]` After max retries exhausted, cached write is parked
|
||
|
||
### Section 6.1 — Notification Lists (runtime portion)
|
||
- [ ] `[6.1-1-rt]` Notification lists deployed to sites and loadable at runtime
|
||
- [ ] `[6.1-2-rt]` List name resolves to recipients at site
|
||
|
||
### Section 6.2 — Email Support
|
||
- [ ] `[6.2-1]` Predefined support for sending email as notification delivery mechanism
|
||
- [ ] `[6.2-2]` SMTP settings defined centrally and deployed to all sites
|
||
|
||
### Section 6.3 — Script API
|
||
- [ ] `[6.3-1]` `Notify.To("list name").Send("subject", "message")` — simplified script API
|
||
- [ ] `[6.3-2]` Available to instance scripts, alarm on-trigger scripts, and shared scripts
|
||
|
||
### Section 6.4 — Store-and-Forward for Notifications
|
||
- [ ] `[6.4-1]` If email server unavailable, notifications buffered locally
|
||
- [ ] `[6.4-2]` Same retry pattern: configurable max retry count, fixed time between retries
|
||
- [ ] `[6.4-3]` After max retries exhausted, notification parked
|
||
- [ ] `[6.4-4]` No maximum buffer size
|
||
|
||
### Section 7.1 — Inbound API Purpose
|
||
- [ ] `[7.1-1]` Web API on central cluster for external systems to call in
|
||
- [ ] `[7.1-2]` Counterpart to outbound External System Gateway
|
||
|
||
### Section 7.2 — API Key Management (already in Phase 4; Phase 7 uses the keys)
|
||
- [ ] `[7.2-rt]` API keys stored in config DB and retrievable at runtime for validation
|
||
|
||
### Section 7.3 — Authentication
|
||
- [ ] `[7.3-1]` Inbound requests authenticated via API key (not LDAP/AD)
|
||
- [ ] `[7.3-2]` API key included with each request
|
||
- [ ] `[7.3-3]` Invalid or disabled keys rejected
|
||
|
||
### Section 7.4 — API Method Definitions (runtime portion)
|
||
- [ ] `[7.4-1-rt]` Method definitions loadable at runtime for routing and validation
|
||
- [ ] `[7.4-2-rt]` Approved API keys checked at runtime per method
|
||
- [ ] `[7.4-3-rt]` Parameter validation enforced at runtime (type checking against definitions)
|
||
- [ ] `[7.4-4-rt]` Return value serialized per return definition
|
||
- [ ] `[7.4-5-rt]` Timeout enforced per method (max execution time including routed calls)
|
||
- [ ] `[7.4-6-rt]` Implementation script executed on central cluster
|
||
- [ ] `[7.4-7-rt]` Route.To() routes calls to any instance at any site
|
||
- [ ] `[7.4-8-rt]` API scripts cannot call shared scripts directly (shared scripts are site-only)
|
||
|
||
### Section 7.5 — Availability
|
||
- [ ] `[7.5-1]` Inbound API hosted only on central cluster (active node)
|
||
- [ ] `[7.5-2]` On central failover, API becomes available on new active node
|
||
|
||
### Section 4.4 — Script Capabilities (Phase 7 portion)
|
||
- [ ] `[4.4-6]` `ExternalSystem.Call()` — synchronous request/response
|
||
- [ ] `[4.4-7]` `ExternalSystem.CachedCall()` — fire-and-forget with S&F on transient failure
|
||
- [ ] `[4.4-8]` Send notifications via `Notify.To().Send()`
|
||
- [ ] `[4.4-9]` `Database.Connection()` for raw ADO.NET access; `Database.CachedWrite()` for S&F delivery
|
||
|
||
---
|
||
|
||
## Design Constraints Checklist
|
||
|
||
| ID | Constraint | Source | Mapped WP |
|
||
|----|-----------|--------|-----------|
|
||
| KDD-ext-1 | External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth | CLAUDE.md | WP-6 |
|
||
| KDD-ext-2 | Dual call modes: Call() synchronous and CachedCall() store-and-forward | CLAUDE.md | WP-7 |
|
||
| KDD-ext-3 | Error classification: HTTP 5xx/408/429/connection = transient; other 4xx = permanent | CLAUDE.md | WP-8 |
|
||
| KDD-ext-4 | Notification Service: SMTP with OAuth2 Client Credentials (M365) or Basic Auth. BCC delivery, plain text | CLAUDE.md | WP-10, WP-11 |
|
||
| KDD-ext-5 | Inbound API: POST /api/{methodName}, X-API-Key header, flat JSON, extended type system (Object/List) | CLAUDE.md | WP-1, WP-2, WP-3 |
|
||
| KDD-sf-4 | CachedCall idempotency is the caller's responsibility | CLAUDE.md | WP-7 |
|
||
| CD-ESG-1 | ESG acts as HTTP client; base URL + method relative path | Component-ESG | WP-6 |
|
||
| CD-ESG-2 | Request params serialized as JSON body (POST/PUT) or query params (GET/DELETE) | Component-ESG | WP-6 |
|
||
| CD-ESG-3 | Credentials (API key header or Basic Auth) attached to every request | Component-ESG | WP-6 |
|
||
| CD-ESG-4 | Per-system timeout applies to all method calls | Component-ESG | WP-8 |
|
||
| CD-ESG-5 | CachedCall: on transient failure, routed to S&F. Script does not block | Component-ESG | WP-7 |
|
||
| CD-ESG-6 | CachedCall: on permanent failure (4xx), error returned synchronously. No retry | Component-ESG | WP-7 |
|
||
| CD-ESG-7 | Dedicated dispatcher for Script Execution Actors to isolate blocking I/O | Component-ESG | WP-9 |
|
||
| CD-ESG-8 | Database connections use standard ADO.NET connection pooling per named connection | Component-ESG | WP-14 |
|
||
| CD-ESG-9 | Synchronous DB failures return error to calling script | Component-ESG | WP-14 |
|
||
| CD-NS-1 | Single email per Send() call, all recipients in BCC, from address in To | Component-NotificationService | WP-11 |
|
||
| CD-NS-2 | No per-recipient deduplication | Component-NotificationService | WP-11 |
|
||
| CD-NS-3 | Transient SMTP failures (connection refused, timeout, SMTP 4xx) → S&F. Script does not block | Component-NotificationService | WP-12 |
|
||
| CD-NS-4 | Permanent SMTP failures (5xx) → error returned synchronously. No retry | Component-NotificationService | WP-12 |
|
||
| CD-NS-5 | No application-level rate limiting; SMTP server throttling handled as transient failure | Component-NotificationService | WP-11 |
|
||
| CD-NS-6 | OAuth2 token lifecycle: fetch, cache, refresh on expiry | Component-NotificationService | WP-10 |
|
||
| CD-NS-7 | Connection timeout (default 30s) and max concurrent connections (default 5) | Component-NotificationService | WP-10 |
|
||
| CD-IA-1 | All calls are POST — RPC-style, not RESTful | Component-InboundAPI | WP-1 |
|
||
| CD-IA-2 | Parameters as top-level JSON fields in request body | Component-InboundAPI | WP-3 |
|
||
| CD-IA-3 | Success: 200 with return value JSON. Failure: 4xx/5xx with error object | Component-InboundAPI | WP-5 |
|
||
| CD-IA-4 | Only failures (500) logged centrally. Successful calls not logged | Component-InboundAPI | WP-5 |
|
||
| CD-IA-5 | No rate limiting | Component-InboundAPI | WP-1 |
|
||
| CD-IA-6 | Route.To() resolves instance site from config DB, routes via Communication Layer | Component-InboundAPI | WP-4 |
|
||
| CD-IA-7 | Route.To() calls are synchronous from API caller perspective — blocks until site responds or timeout | Component-InboundAPI | WP-4 |
|
||
| CD-IA-8 | No S&F buffering for inbound API calls — site unreachable = error to caller | Component-InboundAPI | WP-4 |
|
||
| CD-IA-9 | Route.To().GetAttributes/SetAttributes batch operations | Component-InboundAPI | WP-4 |
|
||
| CD-IA-10 | Database.Connection() available to Inbound API scripts for central DB access | Component-InboundAPI | WP-4 |
|
||
|
||
---
|
||
|
||
## Work Packages
|
||
|
||
### WP-1: Inbound API — ASP.NET Endpoint Registration (M)
|
||
|
||
**Description**: Register the `POST /api/{methodName}` ASP.NET Core endpoint on central. Wire into Host via `MapInboundAPI()`.
|
||
|
||
**Acceptance Criteria**:
|
||
- `POST /api/{methodName}` endpoint registered and reachable (`[7.1-1]`, `KDD-ext-5`, `CD-IA-1`)
|
||
- API hosted only on central cluster active node (`[7.5-1]`)
|
||
- On central failover, API becomes available on new active node (`[7.5-2]`)
|
||
- No rate limiting (`CD-IA-5`)
|
||
- Content-Type: application/json
|
||
|
||
**Complexity**: M
|
||
**Traces**: `[7.1-1]`, `[7.1-2]`, `[7.5-1]`, `[7.5-2]`, KDD-ext-5, CD-IA-1, CD-IA-5
|
||
|
||
---
|
||
|
||
### WP-2: Inbound API — X-API-Key Authentication (S)
|
||
|
||
**Description**: Extract and validate API key from the `X-API-Key` header. Check key exists, is enabled, and is approved for the method.
|
||
|
||
**Acceptance Criteria**:
|
||
- API key extracted from `X-API-Key` HTTP header (`[7.3-2]`, `KDD-ext-5`)
|
||
- Key validated against config DB: exists and is enabled (`[7.3-1]`, `[7.3-3]`, `[7.2-rt]`)
|
||
- Key checked against method's approved list (`[7.4-2-rt]`)
|
||
- Invalid key → 401 Unauthorized
|
||
- Valid key but not approved for method → 403 Forbidden
|
||
- Disabled key → 401 Unauthorized
|
||
|
||
**Complexity**: S
|
||
**Traces**: `[7.2-rt]`, `[7.3-1]`–`[7.3-3]`, `[7.4-2-rt]`, KDD-ext-5
|
||
|
||
---
|
||
|
||
### WP-3: Inbound API — Method Routing & Parameter Validation (M)
|
||
|
||
**Description**: Resolve method by name, validate and deserialize parameters against definitions (extended type system).
|
||
|
||
**Acceptance Criteria**:
|
||
- Method resolved by name from URL path segment (`[7.4-1-rt]`)
|
||
- Parameters validated against method definition: count, names, data types (`[7.4-3-rt]`, `CD-IA-2`)
|
||
- Extended type system (Object, List) supported for parameters (`KDD-ext-5`)
|
||
- Invalid parameters → 400 Bad Request
|
||
- Unknown method → 404 Not Found
|
||
|
||
**Complexity**: M
|
||
**Traces**: `[7.4-1-rt]`, `[7.4-3-rt]`, KDD-ext-5, CD-IA-2
|
||
|
||
---
|
||
|
||
### WP-4: Inbound API — Script Execution & Route.To() (L)
|
||
|
||
**Description**: Execute the method's C# implementation script on central. Implement Route.To() for cross-site calls and batch attribute operations.
|
||
|
||
**Acceptance Criteria**:
|
||
- Implementation script executed on central cluster (`[7.4-6-rt]`)
|
||
- Per-method timeout enforced (including routed calls to sites) (`[7.4-5-rt]`)
|
||
- `Route.To("instanceCode").Call("scriptName", params)` routes to site via Communication Layer (`[7.4-7-rt]`, `CD-IA-6`)
|
||
- Route.To() resolves instance site from config DB (`CD-IA-6`)
|
||
- Calls are synchronous from caller perspective — blocks until response or timeout (`CD-IA-7`)
|
||
- No S&F buffering — site unreachable returns error to caller (`CD-IA-8`)
|
||
- `Route.To().GetAttribute()` / `Route.To().GetAttributes()` batch read (`CD-IA-9`)
|
||
- `Route.To().SetAttribute()` / `Route.To().SetAttributes()` batch write (`CD-IA-9`)
|
||
- `Database.Connection("name")` available for central DB access (`CD-IA-10`)
|
||
- API scripts cannot call shared scripts directly (`[7.4-8-rt]`)
|
||
- Return value serialized per return definition (`[7.4-4-rt]`)
|
||
|
||
**Complexity**: L
|
||
**Traces**: `[7.4-4-rt]`–`[7.4-8-rt]`, CD-IA-6–CD-IA-10
|
||
|
||
---
|
||
|
||
### WP-5: Inbound API — Error Handling & Logging (S)
|
||
|
||
**Description**: Implement error response format and failures-only logging.
|
||
|
||
**Acceptance Criteria**:
|
||
- Success: 200 with return value JSON (`CD-IA-3`)
|
||
- Failure responses: error object with "error" and "code" fields (`CD-IA-3`)
|
||
- 401 for invalid/disabled key, 403 for unapproved key, 400 for invalid params, 500 for script failure
|
||
- Only 500 errors logged centrally (`CD-IA-4`)
|
||
- Successful calls not logged (audit log reserved for config changes) (`CD-IA-4`)
|
||
- Script execution errors: safe error message, no internal details exposed
|
||
|
||
**Complexity**: S
|
||
**Traces**: CD-IA-3, CD-IA-4
|
||
|
||
---
|
||
|
||
### WP-6: External System Gateway — HTTP/REST Client (M)
|
||
|
||
**Description**: Implement the HTTP client that executes external system API calls with JSON serialization.
|
||
|
||
**Acceptance Criteria**:
|
||
- HTTP/REST only, JSON serialization (`KDD-ext-1`, `CD-ESG-1`)
|
||
- Base URL from definition + method relative path (`CD-ESG-1`)
|
||
- Request params serialized as JSON body for POST/PUT, query params for GET/DELETE (`CD-ESG-2`)
|
||
- Response body deserialized from JSON into method return type
|
||
- Authentication: API key header or Basic Auth per system definition (`KDD-ext-1`, `CD-ESG-3`)
|
||
- Sites communicate directly with external systems (not through central) (`[5.2-1]`)
|
||
- Scripts invoke methods by referencing predefined definitions (`[5.2-2]`)
|
||
- Deployed definitions loaded at site runtime (`[5.1-1-rt]`, `[5.1-2-rt]`)
|
||
|
||
**Complexity**: M
|
||
**Traces**: `[5.1-1-rt]`, `[5.1-2-rt]`, `[5.2-1]`, `[5.2-2]`, KDD-ext-1, CD-ESG-1, CD-ESG-2, CD-ESG-3
|
||
|
||
---
|
||
|
||
### WP-7: External System Gateway — Dual Call Modes (M)
|
||
|
||
**Description**: Implement `ExternalSystem.Call()` (synchronous) and `ExternalSystem.CachedCall()` (S&F) in the script runtime API.
|
||
|
||
**Acceptance Criteria**:
|
||
- `ExternalSystem.Call("system", "method", params)` — synchronous. Script blocks until response or timeout. All failures return to script (`[4.4-6]`, `KDD-ext-2`)
|
||
- `ExternalSystem.CachedCall("system", "method", params)` — fire-and-forget. On transient failure, routed to S&F Engine. Script does not block (`[4.4-7]`, `KDD-ext-2`, `CD-ESG-5`)
|
||
- CachedCall on permanent failure (4xx except 408/429): error returned synchronously to script. No retry (`CD-ESG-6`)
|
||
- CachedCall on success: response discarded (fire-and-forget)
|
||
- CachedCall idempotency is caller's responsibility — documented (`KDD-sf-4`)
|
||
- S&F integration: buffered, retried per system's retry settings, parked after max retries (`[5.3-1-int]`–`[5.3-5-int]`)
|
||
|
||
**Complexity**: M
|
||
**Traces**: `[4.4-6]`, `[4.4-7]`, `[5.3-1-int]`–`[5.3-5-int]`, KDD-ext-2, KDD-sf-4, CD-ESG-5, CD-ESG-6
|
||
|
||
---
|
||
|
||
### WP-8: External System Gateway — Error Classification & Timeout (S)
|
||
|
||
**Description**: Implement transient/permanent error classification and per-system timeout enforcement.
|
||
|
||
**Acceptance Criteria**:
|
||
- Per-system timeout applied to all HTTP calls (`CD-ESG-4`)
|
||
- Transient failures: connection refused, timeout, HTTP 408, 429, 5xx (`KDD-ext-3`)
|
||
- Permanent failures: HTTP 4xx except 408/429 (`KDD-ext-3`)
|
||
- Transient → CachedCall buffers for retry; Call returns error to script
|
||
- Permanent → always returned to calling script regardless of call mode
|
||
- Permanent failures logged to Site Event Logging
|
||
|
||
**Complexity**: S
|
||
**Traces**: KDD-ext-3, CD-ESG-4
|
||
|
||
---
|
||
|
||
### WP-9: Blocking I/O Dispatcher Isolation (S)
|
||
|
||
**Description**: Configure a dedicated Akka.NET dispatcher for Script Execution Actors to isolate blocking I/O (HTTP calls, DB operations) from the default dispatcher.
|
||
|
||
**Acceptance Criteria**:
|
||
- Dedicated dispatcher configured for Script Execution Actors (`CD-ESG-7`)
|
||
- HTTP calls and database operations execute on dedicated dispatcher threads
|
||
- Default dispatcher (used by coordination actors) is not starved by blocking I/O
|
||
|
||
**Complexity**: S
|
||
**Traces**: CD-ESG-7
|
||
|
||
---
|
||
|
||
### WP-10: Notification Service — SMTP Client (M)
|
||
|
||
**Description**: Implement the SMTP client with support for OAuth2 Client Credentials (M365) and Basic Auth.
|
||
|
||
**Acceptance Criteria**:
|
||
- SMTP client supports OAuth2 Client Credentials flow: Tenant ID, Client ID, Client Secret (`KDD-ext-4`)
|
||
- OAuth2 token lifecycle: fetch on first use, cache, refresh on expiry (`CD-NS-6`)
|
||
- SMTP client supports Basic Auth (username/password) (`KDD-ext-4`)
|
||
- TLS modes: None, StartTLS, SSL
|
||
- Connection timeout configurable (default 30s) (`CD-NS-7`)
|
||
- Max concurrent connections configurable (default 5) (`CD-NS-7`)
|
||
- SMTP settings loaded from deployed configuration (`[6.2-1]`, `[6.2-2]`)
|
||
- Deployed notification lists loadable at runtime (`[6.1-1-rt]`, `[6.1-2-rt]`)
|
||
|
||
**Complexity**: M
|
||
**Traces**: `[6.1-1-rt]`, `[6.1-2-rt]`, `[6.2-1]`, `[6.2-2]`, KDD-ext-4, CD-NS-6, CD-NS-7
|
||
|
||
---
|
||
|
||
### WP-11: Notification Service — Email Delivery Behavior (S)
|
||
|
||
**Description**: Implement the email composition and delivery behavior: BCC, plain text, from address.
|
||
|
||
**Acceptance Criteria**:
|
||
- `Notify.To("listName").Send("subject", "message")` script API implemented (`[6.3-1]`)
|
||
- Available to instance scripts, alarm on-trigger scripts, and shared scripts (`[6.3-2]`, `[4.4-8]`)
|
||
- Single email per Send() call with all recipients in BCC (`CD-NS-1`, `KDD-ext-4`)
|
||
- From address placed in To field (`CD-NS-1`)
|
||
- Plain text only, no HTML (`KDD-ext-4`)
|
||
- No per-recipient deduplication (`CD-NS-2`)
|
||
- No application-level rate limiting (`CD-NS-5`)
|
||
|
||
**Complexity**: S
|
||
**Traces**: `[4.4-8]`, `[6.3-1]`, `[6.3-2]`, KDD-ext-4, CD-NS-1, CD-NS-2, CD-NS-5
|
||
|
||
---
|
||
|
||
### WP-12: Notification Service — Error Classification & S&F Integration (S)
|
||
|
||
**Description**: Implement transient/permanent SMTP error classification and integration with the Store-and-Forward Engine.
|
||
|
||
**Acceptance Criteria**:
|
||
- Transient failures (connection refused, timeout, SMTP 4xx): notification handed to S&F Engine. Script does not block (`CD-NS-3`, `[6.4-1]`)
|
||
- Permanent failures (SMTP 5xx): error returned synchronously to script. No retry (`CD-NS-4`)
|
||
- S&F retry per SMTP config retry settings: max retry count, fixed interval (`[6.4-2]`)
|
||
- After max retries exhausted, notification parked (`[6.4-3]`)
|
||
- No maximum buffer size (`[6.4-4]`)
|
||
|
||
**Complexity**: S
|
||
**Traces**: `[6.4-1]`–`[6.4-4]`, CD-NS-3, CD-NS-4
|
||
|
||
---
|
||
|
||
### WP-13: End-to-End Integration Tests (L)
|
||
|
||
**Description**: End-to-end tests covering the full integration paths.
|
||
|
||
**Acceptance Criteria**:
|
||
- **Test 1**: External system calls Inbound API → Route.To() → site script executes → response flows back (`[7.1-1]`, `[7.4-7-rt]`)
|
||
- **Test 2**: Script calls `ExternalSystem.Call()` → HTTP request → response returned to script (`[4.4-6]`)
|
||
- **Test 3**: Script calls `ExternalSystem.CachedCall()` → transient failure → S&F buffer → retry → success (`[4.4-7]`, `[5.3-1-int]`)
|
||
- **Test 4**: Script calls `Notify.To().Send()` → SMTP delivery succeeds (`[4.4-8]`, `[6.3-1]`)
|
||
- **Test 5**: Script calls `Notify.To().Send()` → SMTP unavailable → S&F buffer → retry → success (`[6.4-1]`)
|
||
- **Test 6**: Script calls `Database.Connection()` → ADO.NET query succeeds (`[4.4-9]`, `[5.6-1]`, `[5.6-2]`)
|
||
- **Test 7**: Script calls `Database.CachedWrite()` → DB unavailable → S&F buffer → retry → success (`[5.6-3]`–`[5.6-6]`)
|
||
- **Test 8**: CachedCall permanent failure (4xx) → error returned to script, not buffered
|
||
- **Test 9**: Inbound API → site unreachable → error returned to external caller (`CD-IA-8`)
|
||
- **Test 10**: Inbound API → disabled API key → 401 (`[7.3-3]`)
|
||
- **Test 11**: Batch attribute operations via Route.To().GetAttributes/SetAttributes (`CD-IA-9`)
|
||
|
||
**Complexity**: L
|
||
**Traces**: All requirements from this phase
|
||
|
||
---
|
||
|
||
### WP-14: Database Access — Connection() and CachedWrite() (M)
|
||
|
||
**Description**: Implement `Database.Connection()` for synchronous ADO.NET access and `Database.CachedWrite()` for S&F database writes in the script runtime API.
|
||
|
||
**Acceptance Criteria**:
|
||
- `Database.Connection("name")` returns raw ADO.NET SqlConnection (`[4.4-9]`, `[5.6-1]`)
|
||
- Full ADO.NET control: queries, updates, transactions, stored procedures (`[5.6-2]`)
|
||
- Standard ADO.NET connection pooling per named connection (`CD-ESG-8`)
|
||
- Synchronous DB failures return error to calling script (`CD-ESG-9`)
|
||
- `Database.CachedWrite("name", "sql", params)` submits to S&F Engine (`[5.6-3]`)
|
||
- Cached entry stores: connection name, SQL statement, parameter values (`[5.6-4]`)
|
||
- If DB unavailable, write buffered and retried per connection retry settings (`[5.6-5]`, `[5.5-2-rt]`)
|
||
- After max retries, cached write parked (`[5.6-6]`)
|
||
- Deployed database connection definitions loaded at site runtime (`[5.5-1-rt]`)
|
||
|
||
**Complexity**: M
|
||
**Traces**: `[4.4-9]`, `[5.5-1-rt]`, `[5.5-2-rt]`, `[5.6-1]`–`[5.6-6]`, CD-ESG-8, CD-ESG-9
|
||
|
||
---
|
||
|
||
## Test Strategy
|
||
|
||
### Unit Tests
|
||
- API key validation logic (valid, invalid, disabled, not approved for method)
|
||
- Method routing and parameter validation (type checking, missing params, extra params)
|
||
- Extended type system serialization/deserialization (Object, List)
|
||
- Error classification: transient vs. permanent for HTTP status codes
|
||
- Error classification: transient vs. permanent for SMTP status codes
|
||
- OAuth2 token fetch, cache, refresh-on-expiry logic
|
||
- BCC email composition
|
||
- Database.Connection() pooling behavior
|
||
- Database.CachedWrite() S&F submission
|
||
|
||
### Integration Tests
|
||
- Inbound API full flow: auth → route → method resolve → script execute → return
|
||
- External System Gateway: HTTP client against test REST server (infra/restapi)
|
||
- Notification Service: SMTP client against Mailpit test server (infra)
|
||
- Store-and-Forward integration: CachedCall → buffer → retry → deliver
|
||
- Database access: Connection() query against test MS SQL, CachedWrite() with simulated failure
|
||
- Route.To() cross-site call via Communication Layer
|
||
|
||
### Negative Tests
|
||
- `ExternalSystem.Call()` with unreachable system → error returned to script (no buffer)
|
||
- `ExternalSystem.CachedCall()` with permanent 4xx → error returned synchronously, not buffered
|
||
- Inbound API with invalid key → 401
|
||
- Inbound API with unapproved key → 403
|
||
- Inbound API with bad params → 400
|
||
- Inbound API script failure → 500 with safe error message, no internal details
|
||
- Inbound API script calls shared script → fails (shared scripts site-only)
|
||
- Inbound API route to unreachable site → error, no S&F
|
||
- `Notify.To().Send()` with SMTP 5xx → error returned to script, not buffered
|
||
- `Database.Connection()` with unreachable server → error returned to script
|
||
|
||
### Failover Tests
|
||
- Central failover → Inbound API becomes available on new active node
|
||
- Site failover → S&F buffer takeover for pending external calls, notifications, DB writes
|
||
|
||
---
|
||
|
||
## Verification Gate
|
||
|
||
Phase 7 is complete when:
|
||
1. All 14 work packages pass acceptance criteria
|
||
2. All unit, integration, negative, and failover tests pass
|
||
3. External systems can call `POST /api/{method}` with API key auth and receive scripted responses
|
||
4. Site scripts can call external systems via both `Call()` and `CachedCall()` modes
|
||
5. Notifications send via SMTP with both OAuth2 and Basic Auth
|
||
6. Database access works synchronously and via CachedWrite with S&F
|
||
7. Error classification correctly routes transient failures to S&F and permanent failures to the script
|
||
8. End-to-end integration paths verified (all 11 test scenarios in WP-13)
|
||
|
||
---
|
||
|
||
## Open Questions
|
||
|
||
| # | Question | Context | Impact | Status |
|
||
|---|----------|---------|--------|--------|
|
||
| Q12 | What Microsoft 365 tenant/app registration is available for SMTP OAuth2 testing? | Affects Notification Service OAuth2 implementation. | Phase 7. | Deferred — implement against Basic Auth first; OAuth2 tested when tenant available. |
|
||
|
||
(Existing question from questions.md — no new questions discovered.)
|
||
|
||
---
|
||
|
||
## Split-Section Verification
|
||
|
||
| Section | Phase 7 Bullets | Other Phase(s) | Other Phase Bullets |
|
||
|---------|----------------|-----------------|---------------------|
|
||
| 4.4 | `[4.4-6]`–`[4.4-9]` (external calls, notifications, DB access) | Phase 3B | `[4.4-1]`–`[4.4-5]`, `[4.4-10]` (read/write/call scripts) |
|
||
| 5.1 | `[5.1-1-rt]`, `[5.1-2-rt]` (runtime) | Phase 5 | `[5.1-1]`–`[5.1-5]` (definition UI) |
|
||
| 5.3 | `[5.3-1-int]`–`[5.3-5-int]` (integration) | Phase 3C | S&F Engine infrastructure |
|
||
| 5.5 | `[5.5-1-rt]`, `[5.5-2-rt]` (runtime) | Phase 5 | `[5.5-1]`–`[5.5-5]` (definition UI) |
|
||
| 6.1 | `[6.1-1-rt]`, `[6.1-2-rt]` (runtime) | Phase 5 | `[6.1-1]`–`[6.1-5]` (definition UI) |
|
||
| 7.4 | `[7.4-1-rt]`–`[7.4-8-rt]` (runtime) | Phase 5 | `[7.4-1]`–`[7.4-8]` (definition UI) |
|
||
|
||
---
|
||
|
||
## Orphan Check Result
|
||
|
||
**Forward check**: Every Requirements Checklist item and Design Constraints Checklist item maps to at least one work package with acceptance criteria that would fail if the requirement were not implemented. PASS.
|
||
|
||
**Reverse check**: Every work package traces back to at least one requirement or design constraint. No untraceable work. PASS.
|
||
|
||
**Split-section check**: All split sections verified above. Phase 7 covers runtime execution for all integration components. Definition management UI is in Phase 5. S&F infrastructure is in Phase 3C. Core script capabilities (read/write/call) are in Phase 3B. No unassigned bullets found. PASS.
|
||
|
||
**Negative requirement check**: The following negative requirements have explicit acceptance criteria:
|
||
- `[7.4-8-rt]` API scripts cannot call shared scripts directly → tested in WP-4 and negative tests
|
||
- `[4.4-10]` (Phase 3B) Scripts cannot access other instances → not this phase's concern
|
||
- `CD-IA-8` No S&F for inbound API calls → tested in WP-4 and WP-13 test 9
|
||
- `CD-NS-2` No per-recipient deduplication → verified in WP-11
|
||
- `CD-NS-5` No application-level rate limiting → verified in WP-11
|
||
- `CD-IA-5` No rate limiting on inbound API → verified in WP-1
|
||
- `CD-IA-4` Successful calls not logged → verified in WP-5
|
||
|
||
PASS.
|
||
|
||
**Codex MCP verification**: Skipped — external tool verification deferred.
|