fix(alarms): never drop a comment-trail entry on pair divergence (E5 review r1)

I-1: a node whose in-memory trail is one short of the stored trail (it missed
a command the peer applied, whose entry then replicated in) silently dropped
its own next entry — in failover, the survivor's newest ack — because the
positional "past MAX(seq)" rule treated it as already stored.

SaveAsync now reconciles the last 16 in-memory entries against the stored
rows from that position on, by content (Kind/User/Text, timestamps within a
5-minute skew window; matched rows are consumed so repeats count). The peer's
copy of the same command is recognised and not re-appended; an unmatched
entry is inserted at its own ordinal when free, otherwise appended at
MAX(seq)+1 with a Warning — kept, never silent. Bounded read, constant write.

m1: MigrateInlineComments copies only — it no longer UPDATEs the condition
row (a fresh-HLC re-stamp could beat a newer peer ack under LWW); the next
save clears comments_json. m2: it returns unparseable alarm ids and OnReady
logs a Warning per id (+ an Information tally). m3/m4: documented in the
store remarks (Remove vs lagging peer; in-memory index == durable ordinal).

Tests: store-level failover sequence (missed command -> survivor's ack and
next entry persisted once, re-save writes 0 rows, one Warning) with the
dual-write positive control (peer copy present -> 0 rows, no Warning) and a
repeated-identical-comment case; a loopback pair failover test (link down,
survivor acks, rejoin -> both entries on both peers, held); migration
no-row-rewrite, partial fill and unparseable reporting; OnReady Warning.
Both failover tests fail against the round-0 store.

Refs #748
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 16:58:46 -04:00
parent 13f1357ad2
commit dad28d37a2
6 changed files with 490 additions and 82 deletions
@@ -119,12 +119,33 @@ public static class LocalDbSetup
RemoveStaleCapture(db, logger ?? Serilog.Log.Logger);
// LAST, and only here. These are the only calls in OnReady that write rows.
// Moves any pre-split inline comment trail (alarm_condition_state.comments_json) into the
// append-only alarm_condition_comment table (review R05:4). Idempotent: a boot after the
// first finds no inline trail and writes nothing.
// Copies any pre-split inline comment trail (alarm_condition_state.comments_json) into the
// append-only alarm_condition_comment table (review R05:4). Idempotent: an ordinal already
// held is never rewritten, and the condition row itself is never touched (the alarm's next
// save clears the column). An unreadable trail is reported, not skipped silently.
AlarmConditionStateSchema.InlineCommentMigrationResult commentMigration;
using (var connection = db.CreateConnection())
{
AlarmConditionStateSchema.MigrateInlineComments(connection);
commentMigration = AlarmConditionStateSchema.MigrateInlineComments(connection);
}
var migrationLog = logger ?? Serilog.Log.Logger;
if (commentMigration.RowsMoved > 0)
{
migrationLog.Information(
"LocalDb: copied the inline comment trail of {AlarmCount} scripted alarm(s) into {Table}.",
commentMigration.RowsMoved,
AlarmConditionStateSchema.CommentsTable);
}
foreach (var alarmId in commentMigration.UnparseableAlarmIds)
{
migrationLog.Warning(
"LocalDb: scripted alarm {AlarmId} has an unreadable inline comment trail "
+ "(alarm_condition_state.comments_json is not valid JSON); its history was NOT moved to {Table} "
+ "and the alarm's state will not load until the row is repaired.",
alarmId,
AlarmConditionStateSchema.CommentsTable);
}
AlarmSfLegacyMigrator.Migrate(db, configuration);