fix: outbox KPI per-status legs, composite event-log indexes, AuditLog ingest FORCESEEK (#122, #123, #124)

#122: the Notification Outbox KPI snapshots ORed every status in one
aggregation, which scanned the clustered index once the table held ~1 M
Parked rows (0.7-2.6 s; Health dashboard up to 5 s). Each snapshot now runs
three status-equality legs that seek their own index (~0.12 s); a cold Health
load on the rig is 0.12-0.52 s.

#123: site_events event_type/severity indexes are now composites with
timestamp, so a filtered page is read in order and stops at the limit
(severity=Error over 1 GB: 2,287 -> 15 ms cold, 115 -> 1.6 ms warm). Costs
454 -> 530 B/row, ~14 % fewer rows under the same cap. The schema apply drops
the old indexes in place; the cap size model samples timestamp three times
and re-scores inside its band with unchanged constants.

#124: the set-based AuditLog insert's NOT EXISTS probe scanned the EventId
index and concurrent writers deadlocked on S/IX page locks. WITH (FORCESEEK):
4 writers 775 rows/s + 76 deadlocks in 30 s -> 5,214 rows/s + 0. New
concurrent-writer regression test fails 4/4 without the hint.

src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs

@@ -250,7 +250,18 @@ VALUES
         }
 
         sql.Append("\n) AS v (").Append(CanonicalColumnList).Append(")\n");
-        sql.Append("WHERE NOT EXISTS (SELECT 1 FROM dbo.AuditLog x WHERE x.EventId = v.EventId);");
+        // FORCESEEK (#124): left to itself the optimizer answered the anti-semi-join for
+        // a 256-row packet by SCANNING the EventId index. Under READ COMMITTED a scan
+        // holds its current page's S lock while it requests the next page, so two
+        // concurrent inserts — each holding IX on the pages it had just written —
+        // deadlocked on each other's pages (error 1205; captured graph: two INSERTs,
+        // S/IX page locks on AuditLog). Measured with 4 concurrent writers on a fresh
+        // database (ResidualBench audit-contention): 76 victims in 30 s, 775 rows/s.
+        // One index seek per VALUES row locks only the key it reads, and only while
+        // reading it: 0 victims, 5,214 rows/s. A seek can still wait on another
+        // writer's uncommitted row with the SAME EventId — a genuine duplicate, which
+        // the duplicate-key fallback below already resolves.
+        sql.Append("WHERE NOT EXISTS (SELECT 1 FROM dbo.AuditLog x WITH (FORCESEEK) WHERE x.EventId = v.EventId);");
 
         var conn = _context.Database.GetDbConnection();
         var openedHere = false;

src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs

@@ -428,63 +428,50 @@ VALUES
         return total;
     }
 
+    // ── Outbox KPIs (#122) ────────────────────────────────────────────────────────────
+    //
+    // Every snapshot is computed as THREE status-equality legs instead of one
+    // aggregation over an OR'd predicate:
+    //   * live    — Status IN (Pending, Retrying): queue depth, stuck count, oldest age;
+    //   * parked  — Status = Parked: the parked count;
+    //   * delivered — Status = Delivered AND DeliveredAt >= since: the interval count.
+    // The OR'd pre-filter (the shape before #122) made SQL Server scan the clustered
+    // index: on the rig's 4.35 M-row table (945 k Parked, 3.4 M Delivered) the global
+    // snapshot took 0.7-2.6 s and 76 k logical reads. Split, each leg seeks its own index
+    // — (Status, NextAttemptAt) for live and parked, the filtered IX_Notifications_Delivered
+    // for the interval leg — and the global snapshot takes ~0.12 s (parked count 18 k
+    // reads, live 195, delivered 5). The grouped per-site/per-node snapshots still read
+    // every parked row to group it, but the live and delivered legs no longer ride along.
+
     /// <inheritdoc />
     public async Task<NotificationKpiSnapshot> ComputeKpisAsync(
         DateTimeOffset stuckCutoff, DateTimeOffset deliveredSince, CancellationToken cancellationToken = default)
     {
         var now = DateTimeOffset.UtcNow;
 
-        // One conditional-aggregation pass replaces four sequential COUNT round trips:
-        // each metric is a COUNT(CASE WHEN <predicate> THEN 1 END) over the same scan
-        // (arch-review 04). GroupBy(_ => 1) yields a single group (no rows → no group).
-        //
-        // The aggregation is PREDICATE-RESTRICTED instead of scanning the
-        // whole table. Every KPI here is about the live queue (Pending/Retrying), the
-        // parked backlog, or the last delivery interval; the overwhelming bulk of the
-        // Notifications table is historical Delivered and Discarded rows that
-        // contribute to NONE of them. The pre-filter below is the union of the
-        // metric-contributing predicates, which lets the optimizer seek the status
-        // index for the live/parked legs and the filtered
-        // (DeliveredAt) WHERE Status='Delivered' index for the interval leg,
-        // instead of paying a full scan whose cost grows with retained history.
-        // Mirrors the pre-filter ComputePerSiteKpisAsync/ComputePerNodeKpisAsync
-        // already use — the global snapshot was the odd one out.
-        var counts = await _context.Notifications
-            .Where(n => n.Status == NotificationStatus.Pending
-                || n.Status == NotificationStatus.Retrying
-                || n.Status == NotificationStatus.Parked
-                || (n.Status == NotificationStatus.Delivered
-                    && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince))
+        // Live leg: counts and the oldest CreatedAt in one aggregation over the (small)
+        // non-terminal set. GroupBy(_ => 1) yields a single group, or none when the queue is empty.
+        var live = await LiveQueue()
             .GroupBy(_ => 1)
             .Select(g => new
             {
-                QueueDepth = g.Count(n => n.Status == NotificationStatus.Pending
-                    || n.Status == NotificationStatus.Retrying),
-                StuckCount = g.Count(n => (n.Status == NotificationStatus.Pending
-                        || n.Status == NotificationStatus.Retrying)
-                    && n.CreatedAt < stuckCutoff),
-                ParkedCount = g.Count(n => n.Status == NotificationStatus.Parked),
-                DeliveredLastInterval = g.Count(n => n.Status == NotificationStatus.Delivered
-                    && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince),
+                QueueDepth = g.Count(),
+                StuckCount = g.Count(n => n.CreatedAt < stuckCutoff),
+                OldestCreatedAt = g.Min(n => (DateTimeOffset?)n.CreatedAt),
             })
             .FirstOrDefaultAsync(cancellationToken);
 
-        // Oldest non-terminal CreatedAt — TOP(1) via ordered FirstOrDefault, so the
-        // live queue is never materialized. The nullable cast lets FirstOrDefault
-        // return null (no non-terminal rows) instead of throwing.
-        var oldestCreatedAt = await _context.Notifications
-            .Where(n => n.Status == NotificationStatus.Pending
-                || n.Status == NotificationStatus.Retrying)
-            .OrderBy(n => n.CreatedAt)
-            .Select(n => (DateTimeOffset?)n.CreatedAt)
-            .FirstOrDefaultAsync(cancellationToken);
+        var parkedCount = await _context.Notifications
+            .CountAsync(n => n.Status == NotificationStatus.Parked, cancellationToken);
+
+        var deliveredCount = await DeliveredSince(deliveredSince).CountAsync(cancellationToken);
 
         return new NotificationKpiSnapshot(
-            QueueDepth: counts?.QueueDepth ?? 0,
-            StuckCount: counts?.StuckCount ?? 0,
-            ParkedCount: counts?.ParkedCount ?? 0,
-            DeliveredLastInterval: counts?.DeliveredLastInterval ?? 0,
-            OldestPendingAge: oldestCreatedAt is null ? null : now - oldestCreatedAt.Value);
+            QueueDepth: live?.QueueDepth ?? 0,
+            StuckCount: live?.StuckCount ?? 0,
+            ParkedCount: parkedCount,
+            DeliveredLastInterval: deliveredCount,
+            OldestPendingAge: live?.OldestCreatedAt is null ? null : now - live.OldestCreatedAt.Value);
     }
 
     /// <inheritdoc />
@@ -493,47 +480,31 @@ VALUES
     {
         var now = DateTimeOffset.UtcNow;
 
-        // One grouped conditional-aggregation query per scope: counts AND the oldest
-        // non-terminal CreatedAt (server-side MIN over a CASE, so the live queue is
-        // never materialized) computed in a single scan. Pre-filter to rows that
-        // contribute to at least one metric so a site with only fully-uncounted rows
-        // (e.g. Discarded, or out-of-window Delivered) does not surface as an
-        // all-zero row — preserving the prior union-of-metric-keys semantics.
-        var rows = await _context.Notifications
-            .Where(n => n.Status == NotificationStatus.Pending
-                || n.Status == NotificationStatus.Retrying
-                || n.Status == NotificationStatus.Parked
-                || (n.Status == NotificationStatus.Delivered
-                    && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince))
+        var live = await LiveQueue()
             .GroupBy(n => n.SourceSiteId)
-            .Select(g => new
-            {
-                Site = g.Key,
-                QueueDepth = g.Count(n => n.Status == NotificationStatus.Pending
-                    || n.Status == NotificationStatus.Retrying),
-                StuckCount = g.Count(n => (n.Status == NotificationStatus.Pending
-                        || n.Status == NotificationStatus.Retrying)
-                    && n.CreatedAt < stuckCutoff),
-                ParkedCount = g.Count(n => n.Status == NotificationStatus.Parked),
-                DeliveredLastInterval = g.Count(n => n.Status == NotificationStatus.Delivered
-                    && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince),
-                OldestCreatedAt = g.Min(n => (n.Status == NotificationStatus.Pending
-                        || n.Status == NotificationStatus.Retrying)
-                    ? (DateTimeOffset?)n.CreatedAt
-                    : null),
-            })
+            .Select(g => new LiveRow(
+                g.Key,
+                g.Count(),
+                g.Count(n => n.CreatedAt < stuckCutoff),
+                g.Min(n => (DateTimeOffset?)n.CreatedAt)))
             .ToListAsync(cancellationToken);
 
-        return rows
-            .OrderBy(r => r.Site, StringComparer.Ordinal)
-            .Select(r => new SiteNotificationKpiSnapshot(
-                SourceSiteId: r.Site,
-                QueueDepth: r.QueueDepth,
-                StuckCount: r.StuckCount,
-                ParkedCount: r.ParkedCount,
-                DeliveredLastInterval: r.DeliveredLastInterval,
-                OldestPendingAge: r.OldestCreatedAt is null ? null : now - r.OldestCreatedAt.Value))
-            .ToList();
+        var parked = await _context.Notifications
+            .Where(n => n.Status == NotificationStatus.Parked)
+            .GroupBy(n => n.SourceSiteId)
+            .Select(g => new CountRow(g.Key, g.Count()))
+            .ToListAsync(cancellationToken);
+
+        var delivered = await DeliveredSince(deliveredSince)
+            .GroupBy(n => n.SourceSiteId)
+            .Select(g => new CountRow(g.Key, g.Count()))
+            .ToListAsync(cancellationToken);
+
+        // A key appears only when it contributes to at least one leg, so a site with only
+        // uncounted rows (Discarded, out-of-window Delivered) never surfaces as an all-zero row.
+        return MergeLegs(live, parked, delivered, now,
+            (key, depth, stuck, parkedCount, deliveredCount, age) =>
+                new SiteNotificationKpiSnapshot(key, depth, stuck, parkedCount, deliveredCount, age));
     }
 
     /// <inheritdoc />
@@ -542,44 +513,83 @@ VALUES
     {
         var now = DateTimeOffset.UtcNow;
 
-        // Same single-scan grouped aggregation as ComputePerSiteKpisAsync, keyed by
-        // SourceNode. Rows with a NULL SourceNode (legacy / unstamped) are excluded —
-        // per-node KPIs are only meaningful when node identity is known.
-        var rows = await _context.Notifications
-            .Where(n => n.SourceNode != null
-                && (n.Status == NotificationStatus.Pending
-                    || n.Status == NotificationStatus.Retrying
-                    || n.Status == NotificationStatus.Parked
-                    || (n.Status == NotificationStatus.Delivered
-                        && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince)))
+        // Same three legs as ComputePerSiteKpisAsync, keyed by SourceNode. Rows with a NULL
+        // SourceNode (legacy / unstamped) are excluded — per-node KPIs are only meaningful
+        // when node identity is known.
+        var live = await LiveQueue()
+            .Where(n => n.SourceNode != null)
             .GroupBy(n => n.SourceNode!)
-            .Select(g => new
-            {
-                Node = g.Key,
-                QueueDepth = g.Count(n => n.Status == NotificationStatus.Pending
-                    || n.Status == NotificationStatus.Retrying),
-                StuckCount = g.Count(n => (n.Status == NotificationStatus.Pending
-                        || n.Status == NotificationStatus.Retrying)
-                    && n.CreatedAt < stuckCutoff),
-                ParkedCount = g.Count(n => n.Status == NotificationStatus.Parked),
-                DeliveredLastInterval = g.Count(n => n.Status == NotificationStatus.Delivered
-                    && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince),
-                OldestCreatedAt = g.Min(n => (n.Status == NotificationStatus.Pending
-                        || n.Status == NotificationStatus.Retrying)
-                    ? (DateTimeOffset?)n.CreatedAt
-                    : null),
-            })
+            .Select(g => new LiveRow(
+                g.Key,
+                g.Count(),
+                g.Count(n => n.CreatedAt < stuckCutoff),
+                g.Min(n => (DateTimeOffset?)n.CreatedAt)))
             .ToListAsync(cancellationToken);
 
-        return rows
-            .OrderBy(r => r.Node, StringComparer.Ordinal)
-            .Select(r => new NodeNotificationKpiSnapshot(
-                SourceNode: r.Node,
-                QueueDepth: r.QueueDepth,
-                StuckCount: r.StuckCount,
-                ParkedCount: r.ParkedCount,
-                DeliveredLastInterval: r.DeliveredLastInterval,
-                OldestPendingAge: r.OldestCreatedAt is null ? null : now - r.OldestCreatedAt.Value))
+        var parked = await _context.Notifications
+            .Where(n => n.Status == NotificationStatus.Parked && n.SourceNode != null)
+            .GroupBy(n => n.SourceNode!)
+            .Select(g => new CountRow(g.Key, g.Count()))
+            .ToListAsync(cancellationToken);
+
+        var delivered = await DeliveredSince(deliveredSince)
+            .Where(n => n.SourceNode != null)
+            .GroupBy(n => n.SourceNode!)
+            .Select(g => new CountRow(g.Key, g.Count()))
+            .ToListAsync(cancellationToken);
+
+        return MergeLegs(live, parked, delivered, now,
+            (key, depth, stuck, parkedCount, deliveredCount, age) =>
+                new NodeNotificationKpiSnapshot(key, depth, stuck, parkedCount, deliveredCount, age));
+    }
+
+    /// <summary>The non-terminal queue: <c>Status IN (Pending, Retrying)</c>, two index seek ranges.</summary>
+    private IQueryable<Notification> LiveQueue() =>
+        _context.Notifications.Where(n => n.Status == NotificationStatus.Pending
+            || n.Status == NotificationStatus.Retrying);
+
+    /// <summary>
+    /// Delivered rows inside the KPI interval. The literal <c>Status = 'Delivered'</c> plus the
+    /// <c>DeliveredAt</c> range matches the filtered <c>IX_Notifications_Delivered</c> index.
+    /// </summary>
+    private IQueryable<Notification> DeliveredSince(DateTimeOffset deliveredSince) =>
+        _context.Notifications.Where(n => n.Status == NotificationStatus.Delivered
+            && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince);
+
+    private sealed record LiveRow(string Key, int QueueDepth, int StuckCount, DateTimeOffset? OldestCreatedAt);
+
+    private sealed record CountRow(string Key, int Count);
+
+    /// <summary>
+    /// Joins the three per-key legs into one snapshot per key, ordered ordinally by key.
+    /// </summary>
+    private static List<TSnapshot> MergeLegs<TSnapshot>(
+        IReadOnlyList<LiveRow> live,
+        IReadOnlyList<CountRow> parked,
+        IReadOnlyList<CountRow> delivered,
+        DateTimeOffset now,
+        Func<string, int, int, int, int, TimeSpan?, TSnapshot> create)
+    {
+        var liveByKey = live.ToDictionary(r => r.Key, StringComparer.Ordinal);
+        var parkedByKey = parked.ToDictionary(r => r.Key, r => r.Count, StringComparer.Ordinal);
+        var deliveredByKey = delivered.ToDictionary(r => r.Key, r => r.Count, StringComparer.Ordinal);
+
+        return liveByKey.Keys
+            .Concat(parkedByKey.Keys)
+            .Concat(deliveredByKey.Keys)
+            .Distinct(StringComparer.Ordinal)
+            .OrderBy(k => k, StringComparer.Ordinal)
+            .Select(key =>
+            {
+                liveByKey.TryGetValue(key, out var l);
+                return create(
+                    key,
+                    l?.QueueDepth ?? 0,
+                    l?.StuckCount ?? 0,
+                    parkedByKey.GetValueOrDefault(key),
+                    deliveredByKey.GetValueOrDefault(key),
+                    l?.OldestCreatedAt is null ? null : now - l.OldestCreatedAt.Value);
+            })
             .ToList();
     }
 

src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs

@@ -89,7 +89,11 @@ public class EventLogPurgeService : BackgroundService
     /// </summary>
     private const double TablePageFillFactor = 0.78;
 
-    /// <summary>Number of indexes on <c>site_events</c> (timestamp, event_type, instance_id, severity).</summary>
+    /// <summary>
+    /// Number of secondary indexes on <c>site_events</c>: timestamp, instance_id,
+    /// (event_type, timestamp) and (severity, timestamp). The sampled <c>indexed_bytes</c>
+    /// charges every covered column once per index, hence <c>3 * octet_length(timestamp)</c>.
+    /// </summary>
     private const int IndexedColumnCount = 4;
 
     /// <summary>Per-entry bookkeeping in an index B-tree leaf: key-length varint, rowid and cell pointer.</summary>
@@ -680,8 +684,8 @@ public class EventLogPurgeService : BackgroundService
     /// <para>
     /// <b>The model.</b> Rows are sampled from both ends of the table (<see cref="SizeSampleRows"/>
     /// from each — the head is what the oldest-first trim will actually delete, the tail catches a
-    /// workload whose event shape has drifted) to get an average payload width and an average
-    /// width across the four indexed columns. Table cost is then computed as whole pages —
+    /// workload whose event shape has drifted) to get an average payload width and the average
+    /// summed width of the columns each of the four indexes covers. Table cost is then computed as whole pages —
     /// <c>ceil(rows / rowsPerPage) × pageSize</c> — rather than as a flat bytes-per-row rate,
     /// because that is the term that dominates once rows approach half a page: rows too wide to
     /// pack two-per-page waste the remainder, which a linear model underestimates by ~40 %.
@@ -730,7 +734,7 @@ public class EventLogPurgeService : BackgroundService
                                     + octet_length(severity) + COALESCE(octet_length(instance_id), 0)
                                     + octet_length(source) + octet_length(message)
                                     + COALESCE(octet_length(details), 0) AS payload_bytes,
-                                octet_length(timestamp) + octet_length(event_type)
+                                (3 * octet_length(timestamp)) + octet_length(event_type)
                                     + octet_length(severity) + COALESCE(octet_length(instance_id), 0)
                                     AS indexed_bytes
                             FROM site_events LIMIT {SizeSampleRows}
@@ -742,7 +746,7 @@ public class EventLogPurgeService : BackgroundService
                                     + octet_length(severity) + COALESCE(octet_length(instance_id), 0)
                                     + octet_length(source) + octet_length(message)
                                     + COALESCE(octet_length(details), 0) AS payload_bytes,
-                                octet_length(timestamp) + octet_length(event_type)
+                                (3 * octet_length(timestamp)) + octet_length(event_type)
                                     + octet_length(severity) + COALESCE(octet_length(instance_id), 0)
                                     AS indexed_bytes
                             FROM site_events ORDER BY rowid DESC LIMIT {SizeSampleRows}

src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogSchema.cs

@@ -67,10 +67,24 @@ public static class SiteEventLogSchema
                 details TEXT
             );
             CREATE INDEX IF NOT EXISTS idx_events_timestamp ON site_events(timestamp);
-            CREATE INDEX IF NOT EXISTS idx_events_type ON site_events(event_type);
             CREATE INDEX IF NOT EXISTS idx_events_instance ON site_events(instance_id);
-            CREATE INDEX IF NOT EXISTS idx_events_severity ON site_events(severity);
+            DROP INDEX IF EXISTS idx_events_type;
+            DROP INDEX IF EXISTS idx_events_severity;
+            CREATE INDEX IF NOT EXISTS idx_events_type_timestamp ON site_events(event_type, timestamp);
+            CREATE INDEX IF NOT EXISTS idx_events_severity_timestamp ON site_events(severity, timestamp);
             """;
+        // The type and severity indexes are composites with timestamp (#123). The query
+        // service always orders by (timestamp, id) and returns one page, so a
+        // single-column index made an equality filter read EVERY matching row and sort
+        // it before returning 500 (severity=Error over a full 1 GB log: 2.3 s cold,
+        // 115 ms warm). The composite returns the rows already in timestamp order and
+        // stops at the page limit (~1 ms). The cost is ~69 bytes per row more index,
+        // which the storage-cap size model charges (EventLogPurgeService sampling SQL).
+        // The DROPs migrate a pre-#123 file in place on its first boot; dropping and
+        // building an index on a 1 GB table takes seconds and runs once. Indexes are not
+        // part of LocalDb's replicated-table digest, so a pair may upgrade one node at a
+        // time as far as this change is concerned.
+        //
         // The query service also supports keyword search via leading-wildcard
         // LIKE on message/source. A leading-wildcard LIKE cannot use a B-tree
         // index, so that path intentionally full-scans; severity/event_type/