# Audit Log #23 — M3 Cached Operations + Dual-Write Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development (bundled cadence per `feedback_subagent_cadence`).

**Goal:** Cached external calls (`ExternalSystem.CachedCall`) and cached DB writes (`Database.CachedWrite`) each produce 4+ audit rows per operation (`CachedSubmit` → `ApiCallCached`/`DbWriteCached` × N attempts with statuses `Forwarded` then `Attempted` then `Delivered`/`Failed` → `CachedResolve` terminal) AND a `SiteCalls` row at central. Combined telemetry: site emits one packet per lifecycle event carrying both the AuditEvent and the SiteCalls upsert; central writes both in one MS SQL transaction. Audit-write failure never aborts the script.

**Recommended-defaults applied:**
- Telemetry proto: **new top-level RPC `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)`** (sitestream.proto), separate from the M2 `IngestAuditEvents` to keep payload shapes distinct.
- Forwarder: **separate `CachedCallTelemetryForwarder`** actor (or static dispatcher hooking into the existing `SiteAuditTelemetryActor`'s SQLite queue) — write the audit row + tracking row in one SQLite transaction, then let the existing telemetry actor drain both via the new RPC. Reuse the M2 Channel/SQLite hot-path infrastructure; do NOT introduce a parallel writer.
- Provenance: mirror M2's `ScriptRuntimeContext` wrapper pattern — ScriptRuntimeContext's cached-call helpers capture instance/script/site and feed the combined packet.
- IntegrationTests E2E: same component-level pattern as M2 Bundle H (`DirectActorSiteStreamAuditClient`), but extracted into `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/Infrastructure/` for reuse.

**M2 realities baked in (from roadmap line 446-459):**
- Use M1 vocabulary: `AuditKind.CachedSubmit` (enqueue), `AuditKind.ApiCallCached` / `AuditKind.DbWriteCached` (each attempt + post-forward), `AuditKind.CachedResolve` (terminal). `AuditStatus.Submitted` → `Forwarded` → `Attempted` × N → `Delivered`/`Failed`/`Parked`/`Discarded`. NO `CachedEnqueued`/`CachedAttempt`/`CachedTerminal` strings appear in code (those are pre-M1 spec wording the roadmap text still mentions; honor the enum vocabulary).
- NoOpSiteStreamAuditClient still in production until M6; E2E tests use the M2 Bundle H pattern.
- AuditEventMapper duplication note from M2: M3 should move the mapper into Commons (or document the gRPC inline duplication) since M3 adds a SECOND gRPC handler with the same DTO→entity translation work.
- CachedCallTelemetry message creates from scratch (additive per Commons REQ-COM-5a) — NOT renamed to CachedOperationTelemetry.

---

## Bundles

- **Bundle A — Commons types + tracking store** (T1, T2, T3, T4): TrackedOperationId, OperationTrackingStore, Tracking.Status API, CachedCallTelemetry message.
- **Bundle B — SiteCalls table EF + migration + repo** (T5, T6, T7).
- **Bundle C — SiteCallAudit project + actor** (T8).
- **Bundle D — Proto + central dual-write transaction** (T9, T10).
- **Bundle E — ESG / DB-gateway / S&F emissions** (T11, T12, T13, T14).
- **Bundle F — Host registration** (T15).
- **Bundle G — Integration tests** (T16, T17, T18).

Final cross-bundle reviewer + merge to main.

---

## Bundle A — Commons types + tracking store

### Task A1: TrackedOperationId strong-typed ID
File: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/TrackedOperationId.cs` — `public readonly record struct TrackedOperationId(Guid Value)`. Static `New()`, `Parse(string)`, `ToString()` returns Value.ToString("D"). Implicit conversion from Guid via `From(Guid)` (no operator implicit because record struct doesn't allow). Tests in `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/TrackedOperationIdTests.cs`. Commit: `feat(commons): TrackedOperationId strong type (#23 M3)`.

### Task A2: OperationTrackingStore (site-local SQLite)
File: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/IOperationTrackingStore.cs` — `RecordEnqueueAsync`, `RecordAttemptAsync`, `RecordTerminalAsync`, `GetStatusAsync(TrackedOperationId)`, `PurgeTerminalAsync(olderThanUtc)`.
File: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs` — SQLite-backed, mirror SqliteAuditWriter pattern: Channel<T> + background writer Task + write-lock. Schema:
```sql
CREATE TABLE IF NOT EXISTS OperationTracking (
  TrackedOperationId TEXT NOT NULL PRIMARY KEY,
  Kind TEXT NOT NULL,
  TargetSummary TEXT NULL,
  Status TEXT NOT NULL,
  RetryCount INTEGER NOT NULL DEFAULT 0,
  LastError TEXT NULL,
  HttpStatus INTEGER NULL,
  CreatedAtUtc TEXT NOT NULL,
  UpdatedAtUtc TEXT NOT NULL,
  TerminalAtUtc TEXT NULL,
  SourceInstanceId TEXT NULL,
  SourceScript TEXT NULL);
CREATE INDEX IF NOT EXISTS IX_OperationTracking_Status_Updated ON OperationTracking(Status, UpdatedAtUtc);
```
Tests: schema, insert+update sequence, terminal purge (only terminal rows older than threshold). Commit: `feat(siteruntime): OperationTrackingStore site-local SQLite (#23 M3)`.

### Task A3: Tracking.Status script API
File: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptRuntimeContext.cs` — add a `Tracking` accessor exposing `Status(TrackedOperationId)` reading via `IOperationTrackingStore.GetStatusAsync`. Returns a `TrackingStatusSnapshot` record (Commons/Types) with `Status`, `RetryCount`, `LastError`, `CreatedAtUtc`, `UpdatedAtUtc`, `TerminalAtUtc`. Returns null for unknown IDs.
Tests: known, unknown, terminal IDs. Commit: `feat(siteruntime): Tracking.Status script API (#23 M3)`.

### Task A4: CachedCallTelemetry Commons message
File: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Integration/CachedCallTelemetry.cs` — `public sealed record CachedCallTelemetry(TrackedOperationId TrackedOperationId, AuditEvent Audit, SiteCallOperational Operational)` plus `SiteCallOperational` record (TrackedOperationId, Channel, Target, SourceSite, Status, RetryCount, LastError, HttpStatus, CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc?).
Tests: round-trip; lifecycle-specific construction (Submit/Attempted/Resolve). Commit: `feat(commons): CachedCallTelemetry combined operational+audit packet (#23 M3)`.

---

## Bundle B — SiteCalls EF + migration + repo

### Task B1: SiteCall entity + EF mapping
File: `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Audit/SiteCall.cs` — `public sealed record SiteCall` with fields per `SiteCallOperational` plus `IngestedAtUtc`.
File: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs` — table `SiteCalls`, PK on `TrackedOperationId`, indexes `IX_SiteCalls_Source_Created` on (SourceSite, CreatedAtUtc), `IX_SiteCalls_Status_Updated` on (Status, UpdatedAtUtc).
Modify: `ScadaBridgeDbContext.cs` — `public DbSet<SiteCall> SiteCalls => Set<SiteCall>();`.
Tests as M1 pattern. Commit: `feat(configdb): map SiteCall to SiteCalls table (#23 M3)`.

### Task B2: AddSiteCallsTable migration
Generate via `dotnet ef migrations add AddSiteCallsTable --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host`. No partitioning (operational state, not audit). Use MsSqlMigrationFixture for integration test. Commit: `feat(configdb): add SiteCalls migration (#23 M3)`.

### Task B3: ISiteCallAuditRepository + EF impl
File: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ISiteCallAuditRepository.cs` — `UpsertAsync(SiteCall)` with **monotonic status progression** (later status wins; earlier status is no-op), `GetAsync(TrackedOperationId)`, `QueryAsync(filter, paging)`, `PurgeTerminalAsync(olderThanUtc)`.
File: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` — implement via `MERGE` or `INSERT ... WHERE NOT EXISTS` + `UPDATE WHERE TerminalAtUtc IS NULL AND <monotonic order check>`. Tests use MsSqlMigrationFixture. Commit: `feat(configdb): ISiteCallAuditRepository + EF impl (#23 M3)`.

---

## Bundle C — SiteCallAudit project + actor

### Task C1: ZB.MOM.WW.ScadaBridge.SiteCallAudit project + actor
Create: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ZB.MOM.WW.ScadaBridge.SiteCallAudit.csproj` (mirrors ZB.MOM.WW.ScadaBridge.AuditLog csproj style — net10.0, references Commons + ConfigurationDatabase). 
Create: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` — central singleton actor handling `UpsertSiteCallCommand(SiteCall siteCall)` by calling `ISiteCallAuditRepository.UpsertAsync` (scope-per-message via IServiceProvider, mirror AuditLogIngestActor). Idempotent via repo's monotonic upsert.
Create: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ServiceCollectionExtensions.cs` — `AddSiteCallAudit()` registering actor props factory.
Create: `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/` project.
Modify: `ZB.MOM.WW.ScadaBridge.slnx` — add src + tests entries.
Commit: `feat(scaudit): SiteCallAuditActor minimum surface (#22, #23 M3)`.

---

## Bundle D — Proto + central dual-write transaction

### Task D1: Extend sitestream.proto with IngestCachedTelemetry RPC
Follow the documented protobuf regen procedure from M2 Bundle C (temporarily uncomment ItemGroup, build, copy back, recomment). Add:
```proto
message SiteCallOperationalDto {
  string tracked_operation_id = 1;
  string channel = 2;
  string target = 3;
  string source_site = 4;
  string status = 5;
  int32 retry_count = 6;
  string last_error = 7;
  google.protobuf.Int32Value http_status = 8;
  google.protobuf.Timestamp created_at_utc = 9;
  google.protobuf.Timestamp updated_at_utc = 10;
  google.protobuf.Timestamp terminal_at_utc = 11;  // null when active
}
message CachedTelemetryPacket {
  AuditEventDto audit_event = 1;
  SiteCallOperationalDto operational = 2;
}
message CachedTelemetryBatch { repeated CachedTelemetryPacket packets = 1; }

service SiteStreamService {
  rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck);
}
```
Test round-trips. Commit: `feat(comms): IngestCachedTelemetry RPC + CachedTelemetryPacket proto (#23 M3)`.

### Task D2: Dual-write transaction in AuditLogIngestActor
File: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs` (extend) — add `IngestCachedTelemetryCommand` handler. Inside one `DbContext.Database.BeginTransactionAsync()`:
1. Call `IAuditLogRepository.InsertIfNotExistsAsync(auditEvent)` (idempotent already from M2 Bundle A).
2. Call `ISiteCallAuditRepository.UpsertAsync(siteCallOperational)` (monotonic).
3. Commit on both-success; rollback on either-throw (the central singleton SUPERVISES — actor doesn't crash).
4. Reply `IngestAck(acceptedIds)`.

Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/SiteStreamGrpcServer.cs` — implement `IngestCachedTelemetry` gRPC handler routing to actor. Same inline FromDto pattern as M2 (move to mapper if time permits per M2 reviewer recommendation).

Add: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Audit/IngestCachedTelemetryCommand.cs` and `IngestCachedTelemetryReply.cs` (Akka messages).

Tests: 
- Single packet → 1 AuditLog + 1 SiteCalls row.
- Duplicate `EventId` + same status → AuditLog no-op, SiteCalls no-op (monotonic), no error.
- Duplicate `EventId` + ADVANCED status → AuditLog no-op, SiteCalls updates.
- SiteCalls upsert throws → AuditLog rolled back (no orphan).
- AuditLog throws (non-duplicate) → SiteCalls rolled back.

Commit: `feat(auditlog): combined telemetry dual-write transaction (#23 M3)`.

---

## Bundle E — ESG / DB / S&F lifecycle emissions

### Task E1: ScriptRuntimeContext.ExternalSystem.CachedCall wrapper
Mirror M2 Bundle F's `Call` wrapper. Differences:
- Emit on enqueue: AuditEvent(Kind=CachedSubmit, Status=Submitted) + SiteCallOperational(Status=Submitted, RetryCount=0).
- Calls `_externalSystemClient.CachedCallAsync` (resolves what S&F existing API surface looks like — discover by reading ExternalSystemClient).
- Returns a `TrackedOperationId` immediately (a TrackedOperationId tracking handle).
- Hands the operation to the existing StoreAndForward retry loop.

For the per-attempt + terminal emissions, hook into the S&F dispatch loop (Bundle E2/E3).

### Task E2: S&F retry-loop emission
Find the S&F retry-attempt callback site in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`. On each attempt (success/transient/permanent):
- Build AuditEvent(Kind=ApiCallCached or DbWriteCached, Status=Attempted).
- Build SiteCallOperational(Status=Attempted, RetryCount=N, LastError, HttpStatus).
- Hand to `CachedCallTelemetryForwarder` which writes both to SQLite (AuditLog + OperationTracking tables, in one SQLite transaction) and lets SiteAuditTelemetryActor's drain loop push them.

### Task E3: S&F terminal-state emission
On final state transition (Delivered / Failed / Parked / Discarded):
- Build AuditEvent(Kind=CachedResolve, Status={final state}).
- Build SiteCallOperational(Status={final state}, TerminalAtUtc=DateTime.UtcNow).
- Forward.

### Task E4: Database.CachedWrite mirror
Same three-event pattern but Channel=DbOutbound, Kind=DbWriteCached for attempts, Kind=CachedSubmit for enqueue, Kind=CachedResolve for terminal.

Tests in ExternalSystemGateway.Tests + StoreAndForward.Tests.

Commit (bundle-level): one commit per task, descriptive messages following M2 style.

---

## Bundle F — Host registration

### Task F1: Register SiteCallAuditActor central singleton
File: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` — register `SiteCallAuditActor` central singleton + proxy alongside `AuditLogIngestActor`. Hand the proxy to `SiteStreamGrpcServer.SetSiteCallAuditActor(proxy)` (mirroring `SetAuditIngestActor`).
File: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` — call `.AddSiteCallAudit()` on the central role's services.
Tests in `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs` (extend).
Commit: `feat(host): register SiteCallAuditActor central singleton (#22, #23 M3)`.

---

## Bundle G — Integration tests

### Task G1: Extract DirectActorSiteStreamAuditClient to shared infrastructure
Move from `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/SyncCallEmissionEndToEndTests.cs` private inner class into `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/Infrastructure/DirectActorSiteStreamAuditClient.cs`. Extend to also implement the new `IngestCachedTelemetryAsync` method (mirror pattern).

### Task G2: Cached call E2E test
File: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/CachedCallCombinedTelemetryTests.cs` (use AuditLog.Tests, not IntegrationTests, because the existing IntegrationTests harness disables Akka per M2 reality). Test: cached call that fails twice then succeeds produces 5 AuditLog rows (1 Submit + 1 Forwarded + 2 Attempted + 1 Resolve) + 1 SiteCalls row (Status=Delivered) + Tracking.Status reports Delivered.

### Task G3: Cached DB write E2E test
File: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/CachedWriteCombinedTelemetryTests.cs`. Mirror G2 for DB.

### Task G4: Idempotency test
File: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/CombinedTelemetryIdempotencyTests.cs`. Send the same packet twice; assert exactly 1 AuditLog row + 1 SiteCalls row.

---

## Final cross-bundle review + merge

Same template as M1/M2.
