CBDDC/twonode.md

# Plan: Peer-Confirmed Oplog Retention and Peer Sync Health

## Objective
Move from time-only oplog pruning to peer-confirmed pruning so entries are not removed until peers have confirmed them, while also adding peer lifecycle management and health visibility.

## Requested Outcomes
- Oplogs are not cleared until confirmed by peers.
- Each node tracks latest oplog confirmation per peer.
- New peers are automatically registered when they join.
- Deprecated peers can be explicitly removed from tracking.
- Hosting health checks report peer sync status, not only store availability.

## Current Baseline (Codebase)
- Pruning is retention-time based in `SyncOrchestrator` + `IOplogStore.PruneOplogAsync(...)`.
- Push ACK is binary success/fail (`AckResponse`) and does not persist peer confirmation state.
- Peer discovery exists (`IDiscoveryService`, `UdpDiscoveryService`, `CompositeDiscoveryService`).
- Persistent remote peer config exists (`IPeerConfigurationStore`, `PeerManagementService`).
- Hosting health check only validates oplog access (`CBDDCHealthCheck` in Hosting project).

## Design Decisions
- Track confirmation as a persisted watermark per `(peerNodeId, sourceNodeId)` using HLC and hash.
- Use existing vector-clock exchange and successful push results to advance confirmations (no mandatory wire protocol break required).
- Treat tracked peers as pruning blockers until explicitly removed.
- Keep peer registration idempotent and safe for repeated discovery events.

## Data Model and Persistence Plan
### 1. Add peer confirmation tracking model
Create a new persisted model (example name: `PeerOplogConfirmation`) with fields:
- `PeerNodeId`
- `SourceNodeId`
- `ConfirmedWall`
- `ConfirmedLogic`
- `ConfirmedHash`
- `LastConfirmedUtc`
- `IsActive`

### 2. Add store abstraction
Add `IPeerOplogConfirmationStore` with operations:
- `EnsurePeerRegisteredAsync(peerNodeId, address, type)`
- `UpdateConfirmationAsync(peerNodeId, sourceNodeId, timestamp, hash)`
- `GetConfirmationsAsync()` and `GetConfirmationsForPeerAsync(peerNodeId)`
- `RemovePeerTrackingAsync(peerNodeId)`
- `GetActiveTrackedPeersAsync()`

### 3. BLite implementation
- Add entity, mapper, and indexed collection to `CBDDCDocumentDbContext`.
- Index strategy:
- unique `(PeerNodeId, SourceNodeId)`
- index `IsActive`
- index `(SourceNodeId, ConfirmedWall, ConfirmedLogic)` for cutoff scans

### 4. Snapshot support
Include peer-confirmation state in snapshot export/import/merge so pruning safety state survives backup/restore.

## Sync and Pruning Behavior Plan
### 5. Auto-register peers when discovered
On each orchestrator loop, before sync attempts:
- collect merged peer list (discovered + known peers)
- call `EnsurePeerRegisteredAsync(...)` for each peer
- skip local node

### 6. Advance confirmation watermarks
During sync session with a peer:
- after vector-clock exchange, advance watermark for nodes where remote is already ahead/equal
- after successful push batch, advance watermark to max pushed timestamp/hash per source node
- persist updates atomically per peer when possible

### 7. Gate oplog pruning by peer confirmation
Replace retention-only prune trigger with a safe cutoff computation:
- compute retention cutoff from `OplogRetentionHours` (existing behavior)
- compute confirmation cutoff as the minimum confirmed point across active tracked peers
- effective cutoff = minimum(retention cutoff, confirmation cutoff)
- prune only to effective cutoff

If any active tracked peer has no confirmation for relevant source nodes, do not prune those ranges.

### 8. Deprecated peer removal path
Provide explicit management operation to unblock pruning for decommissioned peers:
- add method in management service (example: `RemovePeerTrackingAsync(nodeId, removeRemoteConfig = true)`)
- remove from confirmation tracking store
- optionally remove static peer configuration
- document operator workflow for node deprecation

## Hosting Health Check Plan
### 9. Extend hosting health check payload
Update Hosting `CBDDCHealthCheck` to include peer sync status data:
- tracked peer count
- peers with no confirmation
- max lag (ms) between local head and peer confirmation
- lagging peer list (node IDs)
- last successful confirmation update per peer

### 10. Health status policy
- `Healthy`: persistence OK and all active tracked peers within lag threshold
- `Degraded`: persistence OK but one or more peers lagging/unconfirmed
- `Unhealthy`: persistence unavailable, or critical lag breach (configurable)

Add configurable thresholds in hosting options/cluster options.

## Implementation Phases
### Phase 1: Persistence and contracts
- Add model + store interface + BLite implementation + DI wiring.
- Add tests for CRUD, idempotent register, and explicit remove.

### Phase 2: Sync integration
- Register peers from discovery.
- Update confirmations from vector clock + push success.
- Add sync tests validating watermark advancement.

### Phase 3: Safe pruning
- Implement cutoff calculator service.
- Integrate with orchestrator maintenance path.
- Add two-node tests proving no prune before peer confirmation.

### Phase 4: Management and health
- Expose remove-tracking operation in peer management API.
- Extend hosting healthcheck output and status policy.
- Add hosting healthcheck tests for Healthy/Degraded/Unhealthy.

### Phase 5: Docs and rollout
- Update docs for peer lifecycle and pruning semantics.
- Add upgrade notes and operational runbook for peer deprecation.

## Safe Subagent Usage
- Use subagents only for isolated, low-coupling tasks with clear file ownership boundaries.
- Assign each subagent a narrow scope (one component or one test suite at a time).
- Require explicit task contracts for each subagent including input files/components, expected output, and forbidden operations.
- Prohibit destructive repository actions by subagents (`reset --hard`, force-push, history rewrite, broad file deletion).
- Require subagents to report what changed, why, and which tests were run.
- Do not accept subagent-authored changes directly into final output without primary-agent review.

## Mandatory Verification After Subagent Work
- Enforce a verification gate after every subagent-delivered change before integration.
- Verification gate checklist: independently review the produced diff; run targeted unit/integration tests for touched behavior; validate impacted acceptance criteria; confirm no regressions in pruning safety, peer lifecycle handling, or healthcheck output.
- Reject or rework any subagent output that fails verification.
- Only merge/integrate subagent output after verification evidence is documented.

## Test Plan (Minimum)
- Unit:
- confirmation store upsert/get/remove behavior
- auto-register is idempotent
- safe cutoff computation with mixed peer states
- removing a peer from tracking immediately changes cutoff eligibility
- healthcheck classification rules

- Integration (two-node focus):
- Node B offline: Node A does not prune confirmed-required range
- Node B catches up: Node A prunes once confirmed
- New node join auto-registers without manual call
- Deprecated node removal unblocks pruning

## Risks and Mitigations
- Risk: indefinite growth if a peer never confirms.
- Mitigation: explicit removal workflow and degraded health visibility.

- Risk: confirmation drift after restore/restart.
- Mitigation: snapshot persistence of confirmation records.

- Risk: mixed-version cluster behavior.
- Mitigation: rely on existing vector clock exchange first; keep protocol additions backward compatible if introduced later.

## Acceptance Criteria
- Oplog entries are not pruned while any active tracked peer has not confirmed required ranges.
- Newly discovered peers are automatically present in tracking without operator action.
- Operators can explicitly remove a deprecated peer from tracking and pruning resumes accordingly.
- Hosting health endpoint exposes peer sync lag/confirmation status and returns degraded/unhealthy when appropriate.