fix(central): review findings — no client-side audit truncation, insert-first upsert, QI-safe scripts, honest operator replies

Six adversarial-review findings in the central SQL/ingest layer.

F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.

F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).

F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.

F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).

F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.

F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.

Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.

docs/plans/sql/AddAuditLogAndNotificationCoveringIndexes.sql

@@ -1,4 +1,14 @@
-BEGIN TRANSACTION;
+-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
+-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
+-- refuses BOTH the filtered-index DDL below and any later DML on those tables
+-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
+-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
+-- in their own batch so they are in force when the statements after the GO parse.
+SET QUOTED_IDENTIFIER ON;
+SET ANSI_NULLS ON;
+GO
+
+BEGIN TRANSACTION;
 IF NOT EXISTS (
     SELECT * FROM [__EFMigrationsHistory]
     WHERE [MigrationId] = N'20260814235335_AddAuditLogAndNotificationCoveringIndexes'

docs/plans/sql/AddSiteCallsNonTerminalIndex.sql

@@ -1,4 +1,14 @@
-BEGIN TRANSACTION;
+-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
+-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
+-- refuses BOTH the filtered-index DDL below and any later DML on those tables
+-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
+-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
+-- in their own batch so they are in force when the statements after the GO parse.
+SET QUOTED_IDENTIFIER ON;
+SET ANSI_NULLS ON;
+GO
+
+BEGIN TRANSACTION;
 IF NOT EXISTS (
     SELECT * FROM [__EFMigrationsHistory]
     WHERE [MigrationId] = N'20260709110614_AddSiteCallsNonTerminalIndex'

docs/plans/sql/AddSiteCallsTerminalIndex.sql

@@ -1,4 +1,14 @@
-BEGIN TRANSACTION;
+-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
+-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
+-- refuses BOTH the filtered-index DDL below and any later DML on those tables
+-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
+-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
+-- in their own batch so they are in force when the statements after the GO parse.
+SET QUOTED_IDENTIFIER ON;
+SET ANSI_NULLS ON;
+GO
+
+BEGIN TRANSACTION;
 IF NOT EXISTS (
     SELECT * FROM [__EFMigrationsHistory]
     WHERE [MigrationId] = N'20260713142234_AddSiteCallsTerminalIndex'

docs/plans/sql/AlignAuditLogEventIdUniqueness.sql

@@ -1,4 +1,14 @@
-BEGIN TRANSACTION;
+-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
+-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
+-- refuses BOTH the filtered-index DDL below and any later DML on those tables
+-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
+-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
+-- in their own batch so they are in force when the statements after the GO parse.
+SET QUOTED_IDENTIFIER ON;
+SET ANSI_NULLS ON;
+GO
+
+BEGIN TRANSACTION;
 IF NOT EXISTS (
     SELECT * FROM [__EFMigrationsHistory]
     WHERE [MigrationId] = N'20260815004957_AlignAuditLogEventIdUniqueness'

docs/requirements/Component-AuditLog.md

@@ -303,6 +303,24 @@ per-row / per-entry path — so the documented invariant that one bad row cannot
 sink the rest of the batch still holds, it is simply no longer paid for on the
 healthy path.
 
+The set-based path binds its string parameters at the VALUE's own length, never
+at the column width. Declaring the width makes the client TRUNCATE an over-long
+value at bind time and commit the shortened row — silent mutilation in an
+append-only store, with no `PayloadTruncated` flag to admit it, and inconsistent
+with the per-row and reconciliation paths, which send the value in full and let
+the server reject it. Length enforcement belongs to the server on every path:
+reject everywhere, truncate nowhere. (Deliberate, policy-driven truncation of
+`RequestSummary`/`ResponseSummary` under the payload caps is a different thing
+entirely — it happens before the write and always sets `PayloadTruncated`.)
+
+The per-row fallback runs on its OWN short cancellation budget rather than the
+batch's. Sharing it meant that a batch failing *because* the ingest budget
+expired handed every fallback insert an already-cancelled token: N instant
+failures, N counter bumps, nothing accepted — the fallback defeated at exactly
+the moment it was needed. A blown budget is also counted ONCE for the batch
+rather than once per row, so the health counter reads as one timeout instead of
+a burst of write failures.
+
 **Timeout ladder.** The ingest budget is deliberately the smallest on the path:
 the site's Ask and the central gRPC handler's Ask are both 30 s, the actor's
 own database budget is 20 s and the per-statement SQL timeout is 15 s. Before

docs/requirements/Component-ConfigurationDatabase.md

@@ -379,6 +379,22 @@ dotnet ef migrations script FromMigration ToMigration --output migration.sql
 
 Generated scripts are idempotent — they can be safely re-run without causing errors or duplicate changes.
 
+### Running a Script: QUOTED_IDENTIFIER
+
+`SiteCalls`, `Notifications` and `AuditLog` all carry **filtered indexes**, and `AuditLog` additionally carries persisted computed columns. SQL Server refuses both the filtered-index DDL and any subsequent `INSERT`/`UPDATE`/`DELETE` on such a table with **error 1934** unless the session has `QUOTED_IDENTIFIER ON`.
+
+`sqlcmd` defaults that option **OFF** — including the `docker exec … /opt/mssql-tools18/bin/sqlcmd` invocation used throughout the test-infra docs — so a script that applies cleanly in SSMS (which defaults ON) fails there. Two mitigations, both in place:
+
+- Every checked-in script under `docs/plans/sql/` begins with a `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO` header, in its own batch so the setting is in force when the following batch parses.
+- Run `sqlcmd` with **`-I`** (enable quoted identifiers) regardless — it is the correct default for every script this repo generates, and it protects hand-written one-off statements that carry no header:
+
+```bash
+docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
+  -S localhost -U sa -P 'ScadaBridge_Dev1#' -C -I -d ScadaBridgeConfig -i /path/to/script.sql
+```
+
+The application itself is unaffected: SqlClient sets `QUOTED_IDENTIFIER ON` on every connection, so EF Core migrations and all runtime DML already run with it on.
+
 ---
 
 ## Seed Data

docs/requirements/Component-NotificationOutbox.md

@@ -167,6 +167,8 @@ A notification is **stuck** if it is `Pending` or `Retrying` and older than a co
 - **Health Monitoring dashboard** — headline KPI tiles: queue depth, stuck count, parked count. These are central-computed and are not part of the site health report. The site S&F notification backlog remains a separate site health metric covering the site→central leg.
 - **Central UI "Notification Outbox" page** — KPI tiles plus a queryable notification list: filter by status, type, source site, list, and time range; a stuck-only toggle; keyword search on subject. Parked notifications offer **Retry** (→ `Pending`, reset `RetryCount` / `NextAttemptAt`) and **Discard** (→ `Discarded`) actions. Stuck rows are badged.
 
+Both operator actions are **read-then-write against a row the daily retention purge may delete in between**, so the write reports whether it matched a row and the actor answers `Success: false` / `"notification not found"` when it did not — never a success against a notification that no longer exists (and no audit row is emitted for the action that did not happen). The dispatcher's own delivery-state write takes the opposite stance on the same signal: the delivery has already happened, there is nothing left to retry, so a vanished row is logged as a warning and the audit rows stand as the durable record.
+
 ## Configuration
 
 The component is configured via `NotificationOutboxOptions`, bound from an `appsettings.json` section on the central host (Options pattern):

docs/requirements/Component-SiteCallAudit.md

@@ -148,15 +148,27 @@ and the mailbox supplies the memory barrier between consecutive passes. The
 daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s
 dispatch sweep is the in-repo reference.)
 
-**The central upsert is one statement.** `SiteCallAuditRepository.UpsertAsync`
-issues a single batch that runs the monotonic UPDATE first and INSERTs only when
-nothing matched *and* the row genuinely does not exist — instead of an
-unconditional insert-if-absent followed by the update, which cost two round trips
-on every packet and wasted the insert half for every packet after the first. The
-existence re-check is load-bearing: a zero row count also means "the monotonic
+**The central upsert is one round trip, INSERT-first.**
+`SiteCallAuditRepository.UpsertAsync` ships both statements in a single command
+text — `IF NOT EXISTS … INSERT;` then the monotonic `UPDATE` — so a packet costs
+one round trip, not two. The insert leg is gated on the row's existence *alone*,
+never on "the update matched nothing": a zero row count also means "the monotonic
 guard rejected this packet", and inserting there would fork the mirror with a
 second row for an id that already exists.
 
+The ORDER is load-bearing, and running the UPDATE first is a data-loss bug. The
+two writers — the cached dual-write and the reconciliation pull — routinely carry
+*different* lifecycle states for the same `TrackedOperationId`, and both can race
+its very first packet. Under UPDATE-first both find no row, so both updates match
+nothing; the loser then either fails its existence re-check and skips its insert,
+or attempts it and takes a duplicate-key fault — either way its
+`Status`/`RetryCount`/`HttpStatus`/`TerminalAtUtc` are dropped, because it never
+ran an update against the winner's row. Under INSERT-first the loser's insert is
+skipped or faults and its monotonic update still applies to whichever row won, so
+the newer state survives every interleaving while a stale one is still rejected by
+the rank guard. The duplicate-key catch re-runs the monotonic update for the same
+reason (it is idempotent and rank-guarded, so a redundant re-run is inert).
+
 ## Retry / Discard Relay
 
 Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard