feat(audit): off-box chain-head export, verify-heads, maintenance login (ScadaBridge#107)

Item 1 (owner decision: no signing). AuditLog:HashChain:HeadExport (default
off): both centrals periodically read every month's chain head (LastSeq,
LastHash, purge anchor, chained rows, tombstones; one snapshot under the
chain lock SHARED) and append the changed heads to a sink the DB login cannot
write - File (append-only JSONL per node per day, local or UNC) or Syslog
(RFC 5424 over UDP / TCP octet counting). `audit verify-heads --path` reads
those records back (file or syslog-collector lines) and the new
VerifyAuditChainHeads command checks each against the database: Matches /
Extended / Purged, or Mismatch / Regressed / MissingHead / MissingLink /
CountMismatch (exit 3). AuditChainHeadStoreTests forges a chain with the
production hasher: verify-chain says Intact, verify-heads says Mismatch.

Item 3. AuditLog:MaintenanceConnectionString (${secret:}-capable, unset =
today's single login) runs the chain job, chain verify / head export, the
partition switch-out + archive and the per-channel purge on
AuditMaintenanceContext (keyed audit-maintenance repository/archive, purge
actor prefers them). AuditMaintenanceLoginTests runs the hardened setup on
SQL Server: writer (datareader/datawriter/audit_writer) ingests and is
refused chaining and deleting; the chainer+purger login does the full cycle.
Two-login grants documented in Component-AuditLog.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditCommands.cs

@@ -10,8 +10,10 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
 /// (<c>GET /api/audit/query</c>, <c>GET /api/audit/export</c>,
 /// <c>GET /api/audit/tree</c>), plus <c>verify-chain</c>, which sends the
 /// <c>VerifyAuditChain</c> management command (hash-chain tamper evidence, ScadaBridge#41),
-/// and <c>archive list</c> / <c>archive verify</c>, which send <c>ListAuditArchives</c> /
-/// <c>VerifyAuditArchive</c> (Parquet cold archive, ScadaBridge#42).
+/// <c>verify-heads</c>, which checks off-box exported chain heads with
+/// <c>VerifyAuditChainHeads</c> (ScadaBridge#107), and <c>archive list</c> / <c>archive verify</c>,
+/// which send <c>ListAuditArchives</c> / <c>VerifyAuditArchive</c> (Parquet cold archive,
+/// ScadaBridge#42).
 /// </summary>
 public static class AuditCommands
 {
@@ -32,6 +34,7 @@ public static class AuditCommands
         command.Add(BuildExport(urlOption, formatOption, usernameOption, passwordOption));
         command.Add(BuildTree(urlOption, formatOption, usernameOption, passwordOption));
         command.Add(BuildVerifyChain(urlOption, formatOption, usernameOption, passwordOption));
+        command.Add(BuildVerifyHeads(urlOption, formatOption, usernameOption, passwordOption));
         command.Add(BuildBackfillSourceNode(urlOption, formatOption, usernameOption, passwordOption));
         command.Add(BuildArchive(urlOption, formatOption, usernameOption, passwordOption));
 
@@ -323,6 +326,54 @@ public static class AuditCommands
     /// Builds the <c>audit archive</c> group (ScadaBridge#42): <c>list</c> and <c>verify</c>
     /// over the AuditLog Parquet cold archives on the central archive directory.
     /// </summary>
+    private static Command BuildVerifyHeads(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
+    {
+        var pathOption = new Option<string>("--path")
+        {
+            Description = "Export file, or directory of *.jsonl/*.log/*.txt export files (file sink or syslog collector output)",
+            Required = true,
+        };
+        var cmd = new Command("verify-heads")
+        {
+            Description = "Check AuditLog chain heads exported off-box against the database chain; exit 3 if any no longer matches",
+        };
+        cmd.Add(pathOption);
+        cmd.SetAction(async (ParseResult result) =>
+        {
+            AuditVerifyHeadsHelpers.ReadResult read;
+            try
+            {
+                read = AuditVerifyHeadsHelpers.ReadRecords(result.GetValue(pathOption)!);
+            }
+            catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
+            {
+                OutputFormatter.WriteError(ex.Message, "INVALID_ARGUMENT");
+                return 1;
+            }
+
+            if (read.Records.Count == 0)
+            {
+                OutputFormatter.WriteError(
+                    $"No chain-head records found in {read.Files} file(s) ({read.LinesSkipped} unparsable line(s)).",
+                    "INVALID_ARGUMENT");
+                return 1;
+            }
+
+            if (read.LinesSkipped > 0)
+            {
+                Console.Error.WriteLine($"warning: skipped {read.LinesSkipped} line(s) without a chain-head record.");
+            }
+
+            var format = AuditCommandHelpers.ResolveFormat(result, formatOption);
+            return await CommandHelpers.ExecuteCommandAsync(
+                result, urlOption, formatOption, usernameOption, passwordOption,
+                new VerifyAuditChainHeadsCommand(read.Records),
+                timeout: AuditVerifyHeadsHelpers.RequestTimeout,
+                onSuccess: json => AuditVerifyHeadsHelpers.Render(json, format, Console.Out));
+        });
+        return cmd;
+    }
+
     private static Command BuildArchive(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
     {
         var group = new Command("archive") { Description = "List and verify the AuditLog Parquet cold archives" };

src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditVerifyHeadsHelpers.cs

@@ -0,0 +1,160 @@
+using System.Globalization;
+using System.Text.Json;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
+
+namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
+
+/// <summary>
+/// Helpers for <c>audit verify-heads</c> (ScadaBridge#107 item 1): read AuditLog chain heads
+/// exported off-box (the <c>AuditLog:HashChain:HeadExport</c> JSON lines — straight from the
+/// file sink, or from a syslog collector's log), send them in a
+/// <c>VerifyAuditChainHeads</c> management command, and render the verdicts.
+/// </summary>
+/// <remarks>
+/// Exit codes: <c>0</c> every exported head is consistent with the database,
+/// <see cref="FailedExitCode"/> at least one is not (tamper evidence), <c>1</c> on a request
+/// or input error, <c>2</c> on an authorization failure.
+/// </remarks>
+public static class AuditVerifyHeadsHelpers
+{
+    /// <summary>Process exit code when a head check failed.</summary>
+    public const int FailedExitCode = 3;
+
+    /// <summary>Client deadline (the server bounds the call with the long-running Ask).</summary>
+    public static readonly TimeSpan RequestTimeout = TimeSpan.FromMinutes(6);
+
+    private static readonly JsonSerializerOptions ReadOptions = new(JsonSerializerDefaults.Web);
+
+    /// <summary>The result of reading export files.</summary>
+    /// <param name="Records">Distinct records (by month, sequence, hash), latest export kept.</param>
+    /// <param name="LinesRead">Lines that held a head record.</param>
+    /// <param name="LinesSkipped">Non-empty lines without a parsable record.</param>
+    /// <param name="Files">Files read.</param>
+    public sealed record ReadResult(IReadOnlyList<AuditChainHeadRecord> Records, int LinesRead, int LinesSkipped, int Files);
+
+    /// <summary>
+    /// Reads every record from <paramref name="path"/> — one file, or every <c>*.jsonl</c> /
+    /// <c>*.log</c> / <c>*.txt</c> file below a directory. Each line's JSON object is taken from
+    /// its first <c>{</c> to its last <c>}</c>, so syslog-prefixed lines parse too.
+    /// </summary>
+    /// <param name="path">File or directory.</param>
+    /// <returns>The records read.</returns>
+    /// <exception cref="FileNotFoundException">The path does not exist.</exception>
+    public static ReadResult ReadRecords(string path)
+    {
+        string[] files;
+        if (File.Exists(path))
+        {
+            files = [path];
+        }
+        else if (Directory.Exists(path))
+        {
+            files = Directory.EnumerateFiles(path, "*", SearchOption.AllDirectories)
+                .Where(f => f.EndsWith(".jsonl", StringComparison.OrdinalIgnoreCase)
+                    || f.EndsWith(".log", StringComparison.OrdinalIgnoreCase)
+                    || f.EndsWith(".txt", StringComparison.OrdinalIgnoreCase))
+                .Order(StringComparer.Ordinal)
+                .ToArray();
+        }
+        else
+        {
+            throw new FileNotFoundException($"'{path}' is neither a file nor a directory.", path);
+        }
+
+        var records = new List<AuditChainHeadRecord>();
+        int read = 0, skipped = 0;
+        foreach (var file in files)
+        {
+            foreach (var line in File.ReadLines(file))
+            {
+                if (string.IsNullOrWhiteSpace(line))
+                {
+                    continue;
+                }
+
+                var record = TryParseLine(line);
+                if (record is null)
+                {
+                    skipped++;
+                    continue;
+                }
+
+                read++;
+                records.Add(record);
+            }
+        }
+
+        var distinct = records
+            .GroupBy(r => (r.Month, r.LastSeq, Hash: r.LastHash.ToLowerInvariant()))
+            .Select(g => g.OrderByDescending(r => r.ExportedAtUtc).First())
+            .OrderBy(r => r.Month, StringComparer.Ordinal)
+            .ThenBy(r => r.LastSeq)
+            .ToList();
+        return new ReadResult(distinct, read, skipped, files.Length);
+    }
+
+    /// <summary>Parses one export line; null when it holds no valid record.</summary>
+    /// <param name="line">The line.</param>
+    /// <returns>The record, or null.</returns>
+    public static AuditChainHeadRecord? TryParseLine(string line)
+    {
+        var start = line.IndexOf('{');
+        var end = line.LastIndexOf('}');
+        if (start < 0 || end <= start)
+        {
+            return null;
+        }
+
+        try
+        {
+            var record = JsonSerializer.Deserialize<AuditChainHeadRecord>(line.AsSpan(start, end - start + 1), ReadOptions);
+            return record is { V: >= 1, Month.Length: 7, LastHash.Length: > 0 } && record.LastSeq > 0 ? record : null;
+        }
+        catch (JsonException)
+        {
+            return null;
+        }
+    }
+
+    /// <summary>Renders a <c>VerifyAuditChainHeads</c> result and maps it to the exit code.</summary>
+    /// <param name="json">Success body (an <c>AuditChainHeadVerificationResult</c>, camelCase).</param>
+    /// <param name="format">Resolved output format.</param>
+    /// <param name="output">Writer for the report.</param>
+    /// <returns><c>0</c> when verified, <see cref="FailedExitCode"/> otherwise.</returns>
+    public static int Render(string json, string format, TextWriter output)
+    {
+        using var doc = JsonDocument.Parse(json);
+        var root = doc.RootElement;
+        var verified = root.TryGetProperty("verified", out var v) && v.ValueKind == JsonValueKind.True;
+
+        if (!string.Equals(format, "table", StringComparison.OrdinalIgnoreCase))
+        {
+            output.WriteLine(json);
+            output.Flush();
+            return verified ? 0 : FailedExitCode;
+        }
+
+        output.WriteLine($"Verified        : {(verified ? "yes" : "NO")}");
+        output.WriteLine($"Heads checked   : {Num(root, "recordsChecked")}");
+        if (root.TryGetProperty("checks", out var checks) && checks.ValueKind == JsonValueKind.Array)
+        {
+            output.WriteLine($"{"MONTH",-8} {"EXPORTED SEQ",12} {"DB SEQ",8} {"STATUS",-13} DETAIL");
+            foreach (var c in checks.EnumerateArray())
+            {
+                output.WriteLine(string.Create(CultureInfo.InvariantCulture,
+                    $"{Str(c, "month"),-8} {Num(c, "exportedSeq"),12} {Num(c, "databaseSeq", "-"),8} {Str(c, "status"),-13} {Str(c, "detail")}"));
+            }
+        }
+
+        output.Flush();
+        return verified ? 0 : FailedExitCode;
+    }
+
+    private static string Str(JsonElement e, string name)
+        => e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String ? v.GetString() ?? "" : "";
+
+    private static string Num(JsonElement e, string name, string missing = "0")
+        => e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.Number
+            ? v.GetInt64().ToString(CultureInfo.InvariantCulture)
+            : missing;
+}

src/ZB.MOM.WW.ScadaBridge.CLI/README.md

@@ -2071,6 +2071,41 @@ end, `DuplicateSequence`, `IncompleteLink`). `--format table` prints a summary;
 The chain only exists once `AuditLog:HashChain:Enabled` is on for the central nodes
 (default off); see `Component-AuditLog.md`, Security & Tamper-Evidence.
 
+#### `audit verify-heads`
+
+Check AuditLog chain heads that were exported off-box (ScadaBridge#107) against the
+database chain.
+
+```sh
+scadabridge --url <url> audit verify-heads --path <file-or-directory>
+```
+
+| Option | Required | Default | Description |
+|--------|----------|---------|-------------|
+| `--path` | yes | — | An export file, or a directory read recursively for `*.jsonl` / `*.log` / `*.txt` |
+
+The central nodes write one JSON record per month head (`month`, `lastSeq`, `lastHash`,
+purge anchor, chained-row and tombstone counts, `exportedAtUtc`, `node`) when
+`AuditLog:HashChain:HeadExport:Enabled` is on — to append-only files
+(`{Directory}/{yyyy}/audit-chain-heads-{yyyy-MM-dd}-{node}.jsonl`) or to syslog (RFC 5424, the
+JSON record as the message). The CLI reads both forms (the JSON object is taken from each
+line's first `{` to its last `}`, so a syslog collector's prefixes do not matter), keeps one
+record per distinct (month, sequence, hash), and sends `VerifyAuditChainHeads`. Per record the
+server reports `Matches`, `Extended`, `Purged` (the link was removed by a partition purge —
+check the month's archive), or a failure: `Mismatch` (the chain was rewritten and re-hashed),
+`Regressed` (head rolled back), `MissingHead`, `MissingLink` (a chained row deleted outside the
+purge paths), `CountMismatch`.
+
+| Exit code | Meaning |
+|-----------|---------|
+| `0` | Every exported head is consistent with the database |
+| `3` | At least one failure — tamper evidence |
+| `1` | No records found, bad input, `CHAIN_BUSY`, server error |
+| `2` | Authorization failure (needs `Administrator` or `Viewer`, not site-scoped) |
+
+Run it with `audit verify-chain`: `verify-chain` recomputes each month's links;
+`verify-heads` proves the database still holds the heads that left the box.
+
 #### `audit archive list` / `audit archive verify`
 
 List and verify the AuditLog Parquet cold archives (ScadaBridge#42) that the retention