scadaproj/docs/plans/2026-06-02-scadabridge-audit-rearch.md

ScadaBridge audit re-architecture (Task 2.5, DEEP full 9-col) — decomposition

Companion to 2026-06-02-auth-audit-normalization-phase2-deep.md. User chose Full re-arch (pure 9-col storage) for ScadaBridge audit. Architect design pass (read-only, verified on feat/adopt-zb-audit) produced this. The full audit record becomes the library 9-field ZB.MOM.WW.Audit.AuditEvent; ~15 domain fields relocate into DetailsJson; ScadaBridge consumes the library IAuditWriter/IAuditRedactor/AuditOutcome. This is the program's largest task.

Key resolutions (from the design)

• Forwarding state machine (the crux) → resolved cleanly. It lives only in site SQLite; the central MS SQL AuditLog table is append-only (DENY UPDATE/DELETE; central rows leave ForwardState null; reconciliation is pure idempotent-insert with in-memory cursors), and the gRPC AuditEventDtoMapper already drops ForwardState/IngestedAtUtc on the wire. So central needs NO forwarding columns (pure 9-col). On the site, add a sidecar audit_forward_state table keyed by EventId (ForwardState, OccurredAtUtc, precomputed IsCachedKind, optional AttemptCount/LastAttemptUtc) — MarkForwarded/MarkReconciled UPDATE the sidecar; ReadPending* JOIN it; the canonical audit_event table is write-once. Precomputing IsCachedKind keeps the drain hot path off JSON parsing (strictly faster than today's Kind NOT IN(...)).
• Central storage migration → new table + copy (in-place collapse infeasible: partition-aligned indexes + SwitchOutPartitionAsync hard-codes a byte-identical staging column list). New 10-col table on the SAME ps_AuditLog_Month(OccurredAtUtc) scheme; per-partition data copy projecting old typed columns into DetailsJson (FOR JSON PATH); rename + role re-grant (append-only preserved). Partitioning preserved (OccurredAtUtc stays).
• Reporting queryability → persisted computed columns for hot filters. Category(=Channel) + canonical Outcome/Target/Actor/SourceNode/CorrelationId cover most filters directly. Add PERSISTED computed columns Kind/Status/SourceSiteId/ExecutionId/ParentExecutionId (JSON_VALUE(DetailsJson,'$.x')) + partition-aligned indexes so the existing index semantics + the GetExecutionTreeAsync recursive CTE survive without a JSON perf cliff.
• Redactor → ScadaBridgeAuditRedactor : IAuditRedactor on the canonical record: parse DetailsJson once, redact + byte-safe-truncate requestSummary/responseSummary/errorDetail/extra in the JSON tree, cap on canonical Category/Outcome (replacing the typed Channel/Status reads), set payloadTruncated, re-serialize. Add a fast-path that skips JSON parse when nothing to redact. SafeDefault → SafeDefaultAuditRedactor. Re-baseline the perf hot-path budgets (JSON parse/rewrite is ~2–4× the typed-field path).
• Canonical field mapping: Action = "{Channel}.{Kind}"; Category = Channel; Target/SourceNode/CorrelationId/ Actor/OccurredAtUtc direct (DateTime→DateTimeOffset UTC). Outcome: Kind==InboundAuthFailure→Denied (checked first); Status==Delivered→Success; Status∈{Failed,Parked,Discarded}→Failure; in-flight/Skipped→Success.
• DetailsJson schema (camelCase, stable): channel, kind, status, executionId, parentExecutionId, sourceSiteId, sourceInstanceId, sourceScript, httpStatus, durationMs, errorMessage, errorDetail, requestSummary, responseSummary, payloadTruncated, extra, ingestedAtUtc. One shared AuditDetailsCodec (Commons) with deterministic options is MANDATORY — the canonical record uses value-equality + consumers dedup on it, so key-order/whitespace drift would break dedup. (forwardState is NOT in DetailsJson — it's site-sidecar only.)
• Commons takes the ZB.MOM.WW.Audit package ref (the record lives in Commons; the package is a leaf canonical-types pkg, only dep Microsoft.Extensions.DependencyInjection.Abstractions). Acceptable.
• gRPC proto kept UNCHANGED — the wire AuditEventDto stays 24-field internally; AuditEventDtoMapper projects to/from DetailsJson. Avoids a proto/codegen rev + a site/central version-skew handshake. (A proto collapse is a separate later task.)

Staged decomposition (C1–C7)

Stage	Scope	Green?	Class	Risk
C1	Commons: add ZB.MOM.WW.Audit ref; new pure types AuditDetails record + AuditDetailsCodec (deterministic) + Status/Kind→AuditOutcome projection + Action/Category builders. No existing type changes.	yes	small	trivial
C2	ScadaBridgeAuditRedactor/SafeDefaultAuditRedactor : IAuditRedactor (canonical record, parse/rewrite DetailsJson, fast-path) — additive, old IAuditPayloadFilter still wired; unit-tested in isolation.	yes	standard	low
C3	ATOMIC CUT — swap the record everywhere. Commons.Entities.Audit.AuditEvent → ZB.MOM.WW.Audit.AuditEvent across ~40 src files + tests: emitters build canonical (domain→DetailsJson via codec); seams (IAuditWriter/ICentralAuditWriter/ISiteAuditQueue/IAuditLogRepository/AuditLogQueryFilter) re-type; AuditEventDtoMapper DTO↔canonical (proto unchanged); switch redactor wiring IAuditPayloadFilter→IAuditRedactor.	boundaries only	high-risk	HIGHEST
C4	Site SQLite two-table forwarding: SqliteAuditWriter → audit_event + audit_forward_state; retarget MarkForwarded/MarkReconciled/ReadPending*/GetBacklogStats/MapRow to JOIN+sidecar; precompute IsCachedKind. Telemetry/Reconciliation actors unchanged (seam stable). Site SQLite is ephemeral (7-day) → in-place schema reset, no data migration.	yes	high-risk	HIGH
C5	ATOMIC CUT — central migration. EF CollapseAuditLogToCanonical: new 10-col table on the partition scheme + per-partition data copy (old cols→DetailsJson) + persisted computed cols/indexes + rename + role re-grant; update AuditLogRepository.InsertIfNotExistsAsync + SwitchOutPartitionAsync staging list; regen ModelSnapshot. Maintenance-window; verify row-count + JSON spot-check.	boundaries only	high-risk	HIGHEST
C6	Reporting/UI/export retarget: QueryAsync/GetKpiSnapshotAsync/GetExecutionTreeAsync predicates→canonical/computed cols; AuditLogExportService+AuditEndpoints CSV + CentralUI Audit components + CLI parse DetailsJson for display.	yes	standard	med
C7	Tests + perf re-baseline + cleanup: rewrite PayloadFilterContractTests/redaction/HotPathLatencyTests to canonical+JSON + new budget; delete dead Commons.Entities.Audit.AuditEvent, 4 audit enums (or relocate behind codec), IAuditPayloadFilter/Default/SafeDefault, obsolete AddColumnIfMissing.	yes	standard	low

Atomic cuts: only C3 (shared record type changes for all callers at once) and C5's data-copy half cannot stay green continuously. All other stages are green at completion.

Top risks (carry into execution)

1. C5 partition + SwitchOutPartitionAsync + persisted computed columns — staging table must carry identical computed defs for SWITCH; add a SWITCH round-trip integration test before C5 ships. Documented fallback: if too brittle, keep Kind/Status as 2 real non-canonical columns on the central table (pragmatic, not pure-9-col) — decide at C5 implementation if blocked.
2. DetailsJson determinism — single AuditDetailsCodec (C1) is load-bearing for value-equality/dedup, not cosmetic.
3. Redactor perf — budgets move; add the no-op fast-path + empirically re-baseline in C7.
4. gRPC — keep the proto unchanged (mapper-internal projection); do NOT couple a wire change to this storage cut.
5. Action=Channel.Kind lossiness — mitigated by Category(=channel) + persisted computed Kind; ScadaBridge-internal filtering uses those, not Action parsing.

Delivery: feat/adopt-zb-audit (stacked on auth), local-only. Each stage = one implementer + classification review chain; full ScadaBridge suite at C3/C4/C5/C7.

Stage status (live)

• ✅ C1 DONE 3d77dc0 (code ✅) — AuditDetails + deterministic AuditDetailsCodec (pinned byte-exact) + AuditOutcomeProjector + AuditFieldBuilders + Commons→ZB.MOM.WW.Audit ref; 56 tests.
• ✅ C2 DONE adfb4d3 + fix 5aaf9e2 (spec ✅, code ✅ after fix) — ScadaBridgeAuditRedactor/SafeDefaultAuditRedactor : IAuditRedactor on the canonical record; redaction primitives extracted into shared AuditRedactionPrimitives/AuditRegexCache (old filter delegates, behaviour-preserved); cap-selection reads d.Status (faithful to legacy IsErrorStatus); fast-path + never-throws; review-fix hardened OverRedact to scrub ALL free-text fields + marker alignment + outer-catch never-leak test. 61 redaction + 44 payload + 88 commons-audit green.
• ✅ C3 DONE db707bb + fix c27b2c3 (spec ✅, code ✅; independently re-verified build 0/0 + AuditLog 241/Communication 201). Atomic record swap across all seams/emitters/gRPC DTO/redactor-wiring (127 files); ScadaBridgeAuditEventFactory single emit point; AuditRowProjection Decompose/Recompose transitional 24-col shim (lossless round-trip verified); proto unchanged; old IAuditPayloadFilter classes deleted (C7 pulled forward). Fix: safe enum-parse fallback in MapRow+FromDto.
• ✅ C4 DONE 946d3e2 + fix 1737d15 (spec ✅, code ✅; independently re-verified diff scope = writer+tests only, build 0/0, AuditLog 249/1-preexisting). Site SQLite → audit_event (canonical) + audit_forward_state sidecar; forwarding marks/reads on the sidecar via JOIN; IsCachedKind={CachedSubmit,ApiCallCached,DbWriteCached,CachedResolve} precomputed drain split; old AuditLog table dropped (ephemeral reset). Fix: PRAGMA foreign_keys=ON + MarkForwarded no-demote guard.
• ✅ C5 DONE 68a6bd1 (spec ✅, code ✅; a LIVE SQL Server was available so the migration + SWITCH were fully exercised — independently re-verified build 0/0 + ConfigurationDatabase 248/248). Central dbo.AuditLog collapsed to 10 canonical cols + 6 computed cols (5 PERSISTED + IngestedAtUtc non-persisted) on the preserved ps_AuditLog_Month scheme; CollapseAuditLogToCanonical new-table-and-copy migration (FOR JSON PATH projection, byte-verified round-trip; Down = documented one-way); repo writes/reads canonical directly; SwitchOutPartition staging matches the computed-col defs; append-only roles re-granted. C3 central shim retired. Forced deviations (all sound): IngestedAtUtc non-persisted, execution-id indexes unfiltered, provider-aware OnModelCreating strips JSON_VALUE for SQLite. Deferred to C7: a dedicated migration-projection test + the stale CreatesFiveNamedIndexes test name.
• ✅ C6 SUBSUMED (no commit) — reporting/UI/export/CLI retarget was already completed by the C3 record-swap (AuditEventView/AuditExportRow shims decode every domain field from DetailsJson) + the C5 repo-query retarget. Read-only explorer verdict: all consumer surfaces canonical-complete; the only flagged items (ExecutionId/ParentExecutionId not in CSV; SourceNodes not parsed in export ParseFilter) are PRE-rearch omissions, not regressions. CentralUI 595/595, ManagementService 125/125 confirm.
• ✅ C7 DONE 635461c + doc-fix bc0e5bf (review ✅; independently re-verified build 0/0, PerformanceTests 10/10, ConfigurationDatabase 251/251 incl. the 3 new migration-projection tests PASSING on live MSSQL, zero dead crefs). Perf hot-path re-baselined (canonical JSON redactor measured ~14µs/2µs — faster than the old typed walk; budgets 200/30/5µs + fast-path Assert.Same); CollapseAuditLogToCanonicalMigrationTests (seed→migrate→assert Action/Category/Outcome/Actor-null/DetailsJson-round-trip + 5 persisted computed cols); index test → CreatesNineNamedIndexes; 26 dead-<see cref> across 13 files cleaned; doc-fix corrected the "six persisted" wording (5 persisted + IngestedAtUtc non-persisted).

✅ TASK 2.5 COMPLETE — ScadaBridge audit FULL re-architecture to pure 9-col canonical (2026-06-02)

All of C1–C7 done, each spec+code reviewed, on feat/adopt-zb-audit (local-only, never pushed). ScadaBridge's audit subsystem now: the canonical ZB.MOM.WW.Audit.AuditEvent record everywhere (domain fields in DetailsJson via the deterministic AuditDetailsCodec); the library IAuditRedactor/AuditOutcome consumed; site SQLite = audit_event (canonical) + audit_forward_state sidecar (forwarding decoupled, IsCachedKind drain split); central dbo.AuditLog collapsed to 10 canonical cols + persisted computed cols on the preserved partition scheme (CollapseAuditLogToCanonical migration, MSSQL-verified); UI/export/CLI canonical-complete via AuditEventView/AuditExportRow. The gRPC proto was intentionally left unchanged (mapper-internal projection). This was the program's single largest task.