dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: reconcile design docs with shipped code (enums, Sms, per-channel retention, SiteCalls schema, conventions)

Browse Source 
PLAN-04 Task 24. Component-Commons: AuditChannel gains SecuredWrite (5), AuditKind full
15-value list (+SecuredWrite*/ReconciliationAbandoned), NotificationType Email/Sms (Teams
evaluated+dropped), AuditLogRow persistence-only-projection deviation note. ConfigDB:
Type discriminator Email/Sms, data-layer conventions (UTC DateTime + Utc converter, MSSQL-
only repos/fixtures, AuditLog clustered-key + P3/P4/P6 tracked follow-ups). SiteCallAudit:
schema table rewritten to shipped columns (Channel/Target/AuditStatus-Status/SourceNode/
HttpStatus/IngestedAtUtc; no Kind/TargetSummary/provenance) + stored-vs-tracking-view note.
AuditLog: per-channel retention config lists all 5 channels incl SecuredWrite. Also fixed
two stale 'Teams planned' refs surfaced by the verification grep (HighLevelReqs, CentralUI).
This commit is contained in:
Joseph Doherty
2026-07-09 10:03:13 -04:00
parent 3a21e77c03
commit 7f1b59ff84
6 changed files with 56 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-ConfigurationDatabase.md
+10 -3
View File
@@ -55,13 +55,20 @@ The configuration database stores all central system data, organized by domain a
- **Database Connection Definitions**: Named database connections (name, connection details, retry settings).
### Notifications
- **Notification Lists**: List definitions (name, `Type` discriminator — `Email` / `Teams` / … — selecting the list's notification type and type-specific targets).
- **Notification Lists**: List definitions (name, `Type` discriminator — `Email` / `Sms` — selecting the list's notification type and type-specific targets).
- **Notification Recipients**: Recipients per list (name, email address).
- **SMTP Configuration**: Email server settings.
- **Notifications**: The durable central notification queue owned by the Notification Outbox — one row per notification, the single source of audit truth. The schema is **type-agnostic** so it records any notification type the system supports (email today, Microsoft Teams and others later): a `Type` discriminator selects the type, and a `TypeData` JSON column (`nvarchar(max)`) carries any future per-type fields without a schema change. Columns: `NotificationId` (GUID, primary key — generated at the site, used as the idempotency key), `Type`, `ListName`, `Subject`, `Body`, `TypeData`, `Status`, `RetryCount`, `LastError`, `ResolvedTargets`, `SourceSiteId`, `SourceInstanceId`, `SourceScript`, `SiteEnqueuedAt`, `CreatedAt`, `LastAttemptAt`, `NextAttemptAt`, `DeliveredAt`. `Status` is a `NotificationStatus` enum stored with values `Pending`, `Retrying`, `Delivered`, `Parked`, `Discarded` (the site-local `Forwarding` state is never persisted centrally). Indexed on `Status` and `NextAttemptAt` for efficient dispatcher polling of due rows, and on `SourceSiteId` and `CreatedAt` for KPI computation and the Central UI query page. Terminal rows are removed by a daily purge job — see Scheduled Maintenance below. See Component-NotificationOutbox.md for the full lifecycle.
- **Notifications**: The durable central notification queue owned by the Notification Outbox — one row per notification, the single source of audit truth. The schema is **type-agnostic** so it records any notification type the system supports (email and SMS today; the type-agnostic shape absorbs future types without a migration): a `Type` discriminator selects the type, and a `TypeData` JSON column (`nvarchar(max)`) carries any future per-type fields without a schema change. Columns: `NotificationId` (GUID, primary key — generated at the site, used as the idempotency key), `Type`, `ListName`, `Subject`, `Body`, `TypeData`, `Status`, `RetryCount`, `LastError`, `ResolvedTargets`, `SourceSiteId`, `SourceInstanceId`, `SourceScript`, `SiteEnqueuedAt`, `CreatedAt`, `LastAttemptAt`, `NextAttemptAt`, `DeliveredAt`. `Status` is a `NotificationStatus` enum stored with values `Pending`, `Retrying`, `Delivered`, `Parked`, `Discarded` (the site-local `Forwarding` state is never persisted centrally). Indexed on `Status` and `NextAttemptAt` for efficient dispatcher polling of due rows, and on `SourceSiteId` and `CreatedAt` for KPI computation and the Central UI query page. Terminal rows are removed by a daily purge job — see Scheduled Maintenance below. See Component-NotificationOutbox.md for the full lifecycle.
### Site Calls
- **SiteCalls**: The central audit table for cached site calls — `ExternalSystem.CachedCall()` and `Database.CachedWrite()` — owned by the Site Call Audit component and a sibling of the `Notifications` table. One row per cached operation. Columns: `TrackedOperationId` (`varchar(36)`, primary key — the GUID idempotency key stored in "D"-format string, matching the site SQLite and gRPC wire format), `Channel` (`varchar(32)` — `ApiOutbound` or `DbOutbound`; the trust-boundary channel that produced the call), `Target` (`varchar(256)` — human-readable target, e.g. `ERP.GetOrder` for an external call), `SourceSite` (`varchar(64)` — the site that submitted the cached call), `SourceNode` (`varchar(64)` NULL — cluster node that emitted the call; null for reconciled rows from retired nodes), `Status` (`varchar(32)` — `AuditStatus` enum name: `Submitted`, `Forwarded`, `Attempted`, `Delivered`, `Failed`, `Parked`, `Discarded`; lifecycle is monotonic), `RetryCount` (`int`), `LastError` (`nvarchar(1024)` NULL), `HttpStatus` (`int` NULL — last HTTP status code for API calls), `CreatedAtUtc` (`datetime2`), `UpdatedAtUtc` (`datetime2`), `TerminalAtUtc` (`datetime2` NULL), `IngestedAtUtc` (`datetime2` — central ingest timestamp). The table is populated **only** by Site Call Audit telemetry and reconciliation pulls — sites are the source of truth and the row is an eventually-consistent mirror, never written by a central dispatcher. Ingestion is **insert-if-not-exists** keyed on `TrackedOperationId`, then **upsert-on-newer-status**; the lifecycle is monotonic, so at-least-once and out-of-order telemetry are harmless. Indexes: `IX_SiteCalls_Source_Created (SourceSite, CreatedAtUtc DESC)` for per-site queries and `IX_SiteCalls_Status_Updated (Status, UpdatedAtUtc DESC)` for status-filtered views. Terminal rows are removed by a daily purge job — see Scheduled Maintenance below. See Component-SiteCallAudit.md for the full lifecycle.
- **SiteCalls**: The central audit table for cached site calls — `ExternalSystem.CachedCall()` and `Database.CachedWrite()` — owned by the Site Call Audit component and a sibling of the `Notifications` table. One row per cached operation. Columns: `TrackedOperationId` (`varchar(36)`, primary key — the GUID idempotency key stored in "D"-format string, matching the site SQLite and gRPC wire format), `Channel` (`varchar(32)` — `ApiOutbound` or `DbOutbound`; the trust-boundary channel that produced the call), `Target` (`varchar(256)` — human-readable target, e.g. `ERP.GetOrder` for an external call), `SourceSite` (`varchar(64)` — the site that submitted the cached call), `SourceNode` (`varchar(64)` NULL — cluster node that emitted the call; null for reconciled rows from retired nodes), `Status` (`varchar(32)` — `AuditStatus` enum name: `Submitted`, `Forwarded`, `Attempted`, `Delivered`, `Failed`, `Parked`, `Discarded`; lifecycle is monotonic), `RetryCount` (`int`), `LastError` (`nvarchar(1024)` NULL), `HttpStatus` (`int` NULL — last HTTP status code for API calls), `CreatedAtUtc` (`datetime2`), `UpdatedAtUtc` (`datetime2`), `TerminalAtUtc` (`datetime2` NULL), `IngestedAtUtc` (`datetime2` — central ingest timestamp). The table is populated **only** by Site Call Audit telemetry and reconciliation pulls — sites are the source of truth and the row is an eventually-consistent mirror, never written by a central dispatcher. Ingestion is **insert-if-not-exists** keyed on `TrackedOperationId`, then **upsert-on-newer-status**; the lifecycle is monotonic, so at-least-once and out-of-order telemetry are harmless. Indexes: `IX_SiteCalls_Source_Created (SourceSite, CreatedAtUtc DESC)` for per-site queries, `IX_SiteCalls_Status_Updated (Status, UpdatedAtUtc DESC)` for status-filtered views, and the filtered `IX_SiteCalls_NonTerminal` (on `CreatedAtUtc`, `WHERE TerminalAtUtc IS NULL`, INCLUDE `SourceSite, SourceNode, Status`) that backs the live-queue KPI predicates. Terminal rows are removed by a daily purge job — see Scheduled Maintenance below. See Component-SiteCallAudit.md for the full lifecycle.
### Data-layer conventions & tracked follow-ups (arch-review 04)
- **New tables use UTC `DateTime` + the UTC value converter**, not `DateTimeOffset`. The `Notification` entity's `DateTimeOffset` timestamps are the **documented legacy exception** — on the SQLite test harness the `DateTimeOffset`→ISO-8601-string converter forced in-memory reductions in a few KPI paths (now pushed server-side where translatable). Prefer `datetime2` UTC columns for any new table.
- **New repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (`SkippableFact` gated on the live SQL Server container). The `NotificationOutboxRepository` dual SQLite/T-SQL dialect is the **documented legacy exception** (finding S8) — it predates the MSSQL-fixture convention; do not copy that pattern.
- **AuditLog clustered key.** The `dbo.AuditLog` primary/clustered key is `(EventId, OccurredAtUtc)` — `OccurredAtUtc` is required by the monthly partition scheme (`ps_AuditLog_Month`), and `EventId` carries a separate unique index for `InsertIfNotExists` idempotency. **Tracked follow-up (P3):** benchmark whether flipping the clustered key to `(OccurredAtUtc, EventId)` improves range-scan locality once real volume exists.
- **Tracked follow-ups (P4/P6):** the per-node KPI sampling cadence and the AuditLog reconciliation-pull keyset paging (the SiteCalls reconciliation pull already uses a composite keyset cursor; `PullAuditEvents` remains inclusive-`>=`, idempotent on `EventId`) are deferred, lower-urgency hardening items.
### Audit Log
- **AuditLog**: The central, append-only audit table owned by the Audit Log component — one row per script-trust-boundary lifecycle event across all channels (outbound API calls, outbound DB writes/reads, notifications, and inbound API requests). Sibling of the `Notifications` and `SiteCalls` tables but distinct: `AuditLog` is the immutable history that observes the other subsystems, not an operational state store.