fix(management): CAS-guard secured-write reject (WHERE Status='Pending') so reject can never falsify an executed two-person write (arch-review 2026-08-19 P6-1 High)

Reject read the row as Pending, then landed a blind full-row UpdateAsync. A
concurrent approve could win its own Pending->Approved CAS in that gap and enter
the multi-second WriteTagAsync relay; the reject's update then stamped
Status/ExecutedAtUtc/ExecutionError to Rejected mid-relay, so the two-person-control
record said Rejected for a write that physically executed at the plant. Works
cross-node (both centrals share one MS SQL).

- ISecuredWriteRepository/SecuredWriteRepository: new TryMarkRejectedAsync,
  mirroring TryMarkApprovedAsync exactly (conditional UPDATE ... WHERE
  Status='Pending', writes only the columns the reject transition owns). No schema
  change, no migration, no rowversion column.
- HandleRejectSecuredWrite: uses the CAS; on loss re-reads the row and returns the
  status that actually won (same UX shape as approve's lost-race error), and emits
  the SecuredWriteReject audit row ONLY on a won CAS.
- Audited the other lifecycle writes: approve/reject/expiry (inline + the
  opportunistic list sweep) now ALL transition only out of Pending, so winning the
  approve CAS gives that leg exclusive ownership of the row and its post-relay
  Executed/Failed update stays safe as an unconditional write. Documented in place.
- Tests: repository-level approve-vs-reject races in both orders (exactly one wins,
  loser false, row reflects the winner only) plus reject-vs-expiry; handler-level
  lost-race reject (conflict error, no write, no audit row) and approve on an
  already-rejected row (no CAS, no relay, no audit).
This commit is contained in:
Joseph Doherty
2026-08-19 13:46:59 -04:00
parent 59468399db
commit 61a14bccab
6 changed files with 322 additions and 20 deletions
@@ -86,6 +86,35 @@ public interface ISecuredWriteRepository
DateTime decidedAtUtc,
CancellationToken ct = default);
/// <summary>
/// Atomically flips a row from <c>Pending</c> to <c>Rejected</c>, stamping the
/// verifier identity, comment, and decision time, but ONLY if the row is still
/// <c>Pending</c>. The reject counterpart of <see cref="TryMarkApprovedAsync"/>, and
/// the compare-and-swap guard for the approve-vs-reject race: a stale-read reject
/// used to land a blind full-row update while a concurrent approve's multi-second
/// site relay was still in flight, marking the row <c>Rejected</c> while the plant
/// write physically executed (arch-review 2026-08-19 P6-1). The conditional
/// <c>WHERE Status='Pending'</c> makes exactly one of approve/reject/expiry win.
/// The loser observes <c>false</c> and must record nothing — no status change and no
/// reject audit row.
/// </summary>
/// <param name="id">Identity of the pending secured write.</param>
/// <param name="verifierUser">The rejecting verifier's username.</param>
/// <param name="verifierComment">Optional free-text comment from the verifier.</param>
/// <param name="decidedAtUtc">UTC instant the rejection decision was made.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>
/// A task resolving to <c>true</c> if this caller won the rejection (exactly one
/// row transitioned), or <c>false</c> if the row was no longer <c>Pending</c>
/// (already approved, rejected, or expired).
/// </returns>
Task<bool> TryMarkRejectedAsync(
long id,
string verifierUser,
string? verifierComment,
DateTime decidedAtUtc,
CancellationToken ct = default);
/// <summary>
/// Atomically flips a row from <c>Pending</c> to <c>Expired</c>, stamping the
/// expiry as the decision time, but ONLY if the row is still <c>Pending</c>.
@@ -113,6 +113,35 @@ WHERE Id = {id}
return rowsAffected == 1;
}
/// <inheritdoc />
public async Task<bool> TryMarkRejectedAsync(
long id,
string verifierUser,
string? verifierComment,
DateTime decidedAtUtc,
CancellationToken ct = default)
{
// Single-statement compare-and-swap, mirroring TryMarkApprovedAsync exactly.
// A blind full-row UpdateAsync here used to be able to land AFTER a concurrent
// approve had won its own CAS and while its multi-second WriteTagAsync relay was
// still in flight — stamping the row Rejected even though the plant write was
// physically executing, i.e. falsifying the two-person-control record
// (arch-review 2026-08-19 P6-1). The conditional WHERE Status='Pending' makes
// reject win only from Pending, so approve and reject can never both transition
// one row. Only the columns the reject transition owns are written.
var rowsAffected = await _context.Database.ExecuteSqlInterpolatedAsync(
$@"UPDATE dbo.PendingSecuredWrites
SET Status = 'Rejected',
VerifierUser = {verifierUser},
VerifierComment = {verifierComment},
DecidedAtUtc = {decidedAtUtc}
WHERE Id = {id}
AND Status = 'Pending';",
ct);
return rowsAffected == 1;
}
/// <inheritdoc />
public async Task<bool> TryMarkExpiredAsync(long id, DateTime expiredAtUtc, CancellationToken ct = default)
{
@@ -1604,6 +1604,14 @@ public class ManagementActor : ReceiveActor
row.ExecutionError = error;
// UpdateAsync overwrites all columns -> pass the fully-populated entity.
// Safe as a blind write (arch-review 2026-08-19 P6-1): winning the
// Pending->Approved CAS above gives this leg EXCLUSIVE ownership of the row from
// that point on. Every other writer transitions only out of Pending —
// TryMarkApprovedAsync, TryMarkRejectedAsync and TryMarkExpiredAsync (both the
// approve/reject inline expiry and the opportunistic list sweep) are all
// conditional on Status='Pending' — so nothing can overwrite the row while the
// multi-second relay above is in flight. Do not reintroduce an unconditional
// transition on this table.
await repo.UpdateAsync(row);
// Terminal execute outcome: Delivered (relay succeeded) maps to canonical
@@ -1646,15 +1654,37 @@ public class ManagementActor : ReceiveActor
throw new ManagementCommandException(
"A secured write cannot be verified by its submitter.");
// Compare-and-swap: guards the approve-vs-reject race exactly as approve's own
// CAS guards the two-verifier race (arch-review 2026-08-19 P6-1). The status
// check above is a stale read — commands run concurrently, each on its own DI
// scope and across both central nodes on one shared MS SQL — so a concurrent
// approve can win its CAS and enter its multi-second WriteTagAsync relay in the
// gap. The blind full-row UpdateAsync that used to sit here would then land
// mid-relay and stamp the row Rejected while the plant write physically
// executed, falsifying the two-person-control record. Reject now transitions
// ONLY from Pending; the loser records nothing.
var decidedAtUtc = DateTime.UtcNow;
if (!await repo.TryMarkRejectedAsync(cmd.Id, user.Username, cmd.Comment, decidedAtUtc))
{
// Re-read so the conflict names the status that actually won (approve,
// another reject, or the TTL expiry) — same UX shape as the stale-read
// guard above and approve's own lost-race error.
var current = await repo.GetAsync(cmd.Id);
throw new ManagementCommandException(current is null
? $"Secured write {cmd.Id} is no longer pending — already decided."
: $"Secured write {cmd.Id} is '{current.Status}', not Pending; it cannot be rejected.");
}
// We won the race. Stamp the decision locally so the returned DTO carries the
// same values the CAS committed.
entity.Status = "Rejected";
entity.VerifierUser = user.Username;
entity.VerifierComment = cmd.Comment;
entity.DecidedAtUtc = DateTime.UtcNow;
entity.DecidedAtUtc = decidedAtUtc;
// UpdateAsync overwrites all columns -> pass the fully-populated entity.
await repo.UpdateAsync(entity);
// Reject is a terminal lifecycle event (canonical Discarded outcome).
// Reject is a terminal lifecycle event (canonical Discarded outcome). Emitted
// ONLY on a won CAS, so a lost race never leaves a reject row in the audit log
// for a write that was actually approved and executed.
// Actor = the verifier; the operator is carried in the row's extra detail.
await EmitSecuredWriteAuditAsync(
sp, AuditKind.SecuredWriteReject, AuditStatus.Discarded, entity, actor: user.Username);