dohertj2/lmxopcua
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FOCAS Tier-C PR D — supervisor + backoff + crash-loop breaker #172

Merged
dohertj2 merged 1 commits from focas-tier-c-pr-d-supervisor into v2 2026-04-20 14:19:34 -04:00
Conversation 0 Commits 1 Files Changed 6 +568
dohertj2 commented 2026-04-20 14:19:23 -04:00
Owner
Copy Link

Fourth of 5 PRs for #220. Ships the resilience harness between IFocasClient requests and the Tier-C Host process. Same shape as Galaxy Tier-C so Admin /hosts has one mental model.

New (Proxy, .NET 10)

  • Supervisor/Backoff — 5s→15s→60s ladder + 2-min StableRunThreshold reset
  • Supervisor/CircuitBreaker — 3-crashes-in-5-min → 1h→4h→manual-reset with sticky alert
  • Supervisor/HeartbeatMonitor — 3 consecutive misses = dead
  • Supervisor/IHostProcessLauncher — abstraction over spawn-process-and-connect (keeps supervisor I/O-free + testable)
  • Supervisor/FocasHostSupervisor — orchestrator with GetOrLaunchAsync + NotifyHostDeadAsync + AcknowledgeAndReset; exposes OnUnavailable, ObservedCrashes, StickyAlertActive for /hosts wiring

Tests (14 new)

  • Backoff: sequence, clamping, stable-run reset
  • CircuitBreaker: below-threshold allowed, opens at threshold, escalates cooldown, manual reset
  • HeartbeatMonitor: 3 misses dead, ack resets
  • FocasHostSupervisor: first-launch success, retry-with-backoff, breaker-open surfaces InvalidOperationException, NotifyHostDead fan-out, AcknowledgeAndReset, Dispose terminates

Totals

186/186 FOCAS driver tests (172 prior + 14 new). No IFocasClient contract changes — existing FakeFocasClient-based tests unaffected.

Next — PR E

Real Process-based IHostProcessLauncher + NSSM install scripts + MMF post-mortem + docs.

Fourth of 5 PRs for #220. Ships the resilience harness between IFocasClient requests and the Tier-C Host process. Same shape as Galaxy Tier-C so Admin /hosts has one mental model. ## New (Proxy, .NET 10) - `Supervisor/Backoff` — 5s→15s→60s ladder + 2-min StableRunThreshold reset - `Supervisor/CircuitBreaker` — 3-crashes-in-5-min → 1h→4h→manual-reset with sticky alert - `Supervisor/HeartbeatMonitor` — 3 consecutive misses = dead - `Supervisor/IHostProcessLauncher` — abstraction over spawn-process-and-connect (keeps supervisor I/O-free + testable) - `Supervisor/FocasHostSupervisor` — orchestrator with `GetOrLaunchAsync` + `NotifyHostDeadAsync` + `AcknowledgeAndReset`; exposes `OnUnavailable`, `ObservedCrashes`, `StickyAlertActive` for /hosts wiring ## Tests (14 new) - Backoff: sequence, clamping, stable-run reset - CircuitBreaker: below-threshold allowed, opens at threshold, escalates cooldown, manual reset - HeartbeatMonitor: 3 misses dead, ack resets - FocasHostSupervisor: first-launch success, retry-with-backoff, breaker-open surfaces InvalidOperationException, NotifyHostDead fan-out, AcknowledgeAndReset, Dispose terminates ## Totals **186/186 FOCAS driver tests** (172 prior + 14 new). No IFocasClient contract changes — existing FakeFocasClient-based tests unaffected. ## Next — PR E Real Process-based IHostProcessLauncher + NSSM install scripts + MMF post-mortem + docs.
dohertj2 added 1 commit 2026-04-20 14:19:25 -04:00
FOCAS Tier-C PR D — supervisor + backoff + crash-loop breaker + heartbeat monitor. Fourth of 5 PRs for #220. Ships the resilience harness that sits between the driver's IFocasClient requests and the Tier-C Host process, so a crashing Fwlib32.dll takes down only the Host (not the main server), gets respawned on a backoff ladder, and opens a circuit with a sticky operator alert when the crash rate is pathological. Same shape as Galaxy Tier-C so the Admin /hosts surface has a single mental model. New Supervisor/ namespace in Driver.FOCAS (.NET 10, Proxy-side): Backoff with the 5s→15s→60s default ladder + StableRunThreshold that resets the index after a 2-min clean run (so a one-off crash after hours of steady-state doesn't restart from the top); CircuitBreaker with 3-crashes-in-5-min threshold + escalating 1h→4h→manual-reset cooldown ladder + StickyAlertActive flag that persists across cooldowns until AcknowledgeAndReset is called; HeartbeatMonitor tracking ConsecutiveMisses against the 3-misses-kill threshold + LastAckUtc for telemetry; IHostProcessLauncher abstraction over "spawn Host process + produce an IFocasClient connected to it" so the supervisor stays I/O-free and fully testable with a fake launcher that can be told to throw on specific attempts (production wiring over Process.Start + FocasIpcClient.ConnectAsync is the PR E ops-glue concern); FocasHostSupervisor orchestrating them — GetOrLaunchAsync cycles through backoff until either a client is returned or the breaker opens (surfaced as InvalidOperationException so the driver maps to BadDeviceFailure), NotifyHostDeadAsync fans out the unavailable event + terminates the current launcher + records the crash without blocking (so heartbeat-loss detection can short-circuit subscriber fan-out and let the next GetOrLaunchAsync handle the respawn), AcknowledgeAndReset is the operator-clear path, OnUnavailable event for Admin /hosts wiring + ObservedCrashes + BackoffAttempt + StickyAlertActive for telemetry. 14 new unit tests across SupervisorTests.cs: Backoff (default sequence, clamping, RecordStableRun resets), CircuitBreaker (below threshold allowed, opens at threshold, escalates cooldown after second open, ManualReset clears state), HeartbeatMonitor (3 consecutive misses declares dead, ack resets counter), FocasHostSupervisor (first-launch success, retry-with-backoff after transient failure, repeated failures open breaker + surface InvalidOperationException, NotifyHostDeadAsync terminates + fan-outs + increments crash count, AcknowledgeAndReset clears sticky, Dispose terminates). Full FOCAS driver tests now 186/186 green (172 + 14 new). No changes to IFocasClient DI contract; existing FakeFocasClient-based tests unaffected. PR E wires the real Process-based IHostProcessLauncher + NSSM install scripts + MMF post-mortem + docs. 5033609944 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dohertj2 merged commit 446a5c022c into v2 2026-04-20 14:19:34 -04:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#172
Delete Branch "focas-tier-c-pr-d-supervisor"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?