Files

T

Joseph Doherty d1ade5653b docs(plans): bookkeeping sync — reconcile stale trackers with merged code

A verified audit of all ~90 plan documents (2026-08-01) found ~30 .tasks.json
trackers and several plan headers still reporting 'pending'/'draft' for work
fully merged to main. Sync them so future audits don't re-litigate closed work:

- Flip ~380 stale task statuses to completed across March/May/June/July
  trackers (audit-log series, milestones M5-M10, playwright waves, stillpending,
  LocalDb, ClusterClient->gRPC DoD rows, and more), each verified against
  code/git evidence before flipping.
- Annotate obsolete-not-done rows: ClusterClient CLI transport (never built,
  HTTP shipped), TreeView Areas/Instances pages (replaced by Topology),
  template-tree drag-drop (dropped for M9 menu reorder), otopcua item C
  (premise superseded by #17).
- Flip stale headers: aggregated-live-alarm + kpi-rollups 'Draft not executed'
  -> Delivered 2026-07-10; otopcua cutover-scope SCOPING -> DECIDED;
  scadabridge-rename -> Implemented; LocalDb phase1/2 status strings ->
  merged 28ca04d7.
- Fix doc drift: T9/T10 'deferred' -> shipped as SMS (Teams dropped); waitfor
  sandbox follow-up shipped; followups #52/#53/#54/#162/#207 resolved; purge
  TODO closed by PendingDeploymentPurgeActor; live-gate pre-existing failures
  #28/#29/#31 fixed; auto-down boot-alone residual superseded by self-first
  seeds; supersession banners on keep-oldest SBR + ClusterClient-era designs;
  requirements-traceability 'Pending' clarified as frozen plan-generation
  status.

Deliberately left pending (genuinely open, tracked in the pending-work list):
opcua-tag-browser task 19 (live smoke), ipsen tasks 7-8 (vd03 verification),
selfform task 7 (vd03 overlay, user-held), live-gate observation 1
(external-system delete orphan bug), otopcua item A + maxDepth calibration.

2026-08-01 08:53:56 -04:00

2.8 KiB

Raw Blame History

Cluster Infrastructure Refinement — Design

Date: 2026-03-16 Component: Cluster Infrastructure (docs/requirements/Component-ClusterInfrastructure.md) Status: Approved — superseded in part: the keep-oldest SBR decision was replaced by the auto-down decision 2026-07-21 (docs/plans/2026-07-21-auto-down-availability-decision.md).

Problem

The Cluster Infrastructure doc covered topology and failover behavior but lacked specification for the split-brain resolver strategy, seed node configuration, failure detection timing, and dual-node failure recovery.

Decisions

Split-Brain Resolver

Keep-oldest strategy. The longest-running node stays active on partition; the younger node downs itself.
Stable-after duration: 15 seconds — prevents premature downing during startup or transient instability.
Quorum-based strategies rejected because they cause total cluster shutdown on any partition in a two-node cluster.

Seed Node Configuration

Both nodes are seed nodes. No startup ordering dependency. Whichever node starts first forms the cluster.

Failure Detection Timing

Heartbeat interval: 2 seconds.
Failure threshold: 10 seconds (5 missed heartbeats).
Total failover time: ~25 seconds (10s detection + 15s stable-after + singleton restart).
All values configurable. Defaults balance failover speed with stability.

Dual-Node Recovery

Automatic recovery, no manual intervention. First node up forms a new cluster from seed configuration.
Site clusters rebuild from SQLite (deployed configs, S&F buffer). Alarm states re-evaluate from live data.
Central cluster rebuilds from MS SQL. No message buffer state to recover.

Affected Documents

Document	Change
`docs/requirements/Component-ClusterInfrastructure.md`	Added 3 new sections: Split-Brain Resolution, Failure Detection Timing, Dual-Node Recovery. Updated Node Configuration to clarify both-as-seed.

Alternatives Considered

Static-quorum / keep-majority: Rejected — both cause total cluster shutdown on partition in a two-node cluster. Unacceptable for SCADA availability.
Single designated seed node: Rejected — creates startup ordering dependency for no benefit in a two-node cluster.
Manual recovery on dual failure: Rejected — system already persists all state needed for automatic recovery.
Fast detection (1s/5s): Rejected — too sensitive; brief network hiccups would trigger unnecessary failovers and full actor hierarchy rebuilds.
Conservative detection (5s/30s): Rejected — 30 seconds of data collection downtime is too long for SCADA.
Shorter stable-after (10s): Rejected — matching the failure threshold risks downing nodes that are slow to respond (GC pause, heavy load).

2.8 KiB Raw Blame History