Commit Graph

8 Commits

Author SHA1 Message Date
Joseph Doherty 55ae230fa4 ci(#606): Gitea-compatible trx upload + deterministic fixes for the CI-load test flakes
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and
  v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3
  on the test step.
- Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR
  (TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget.
- Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests
  RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process
  (abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s.
- Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start
  state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes
  host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while
  Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick).
- Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps
  (wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log.
- Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on
  Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped.
- infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 19:44:31 -04:00
Joseph Doherty 90d7f6dfca fix(deploy): hold the Applied ack until the address-space apply outcome is known (#601)
DriverHostActor no longer acks Applied the moment the revision advances. It
asks OpcUaPublishActor for the outcome (RebuildAddressSpace.ReportOutcome) and
acks Applied+Degraded(reason) when the apply was degraded; a missing outcome
times out (60 s, inside the coordinator's 2 min deadline) to a degraded ack,
never a clean one. Re-dispatching a revision whose apply was degraded or is
still unconfirmed re-applies instead of taking the clean idempotent
short-circuit - that re-apply is also what repairs the degraded nodes.
Tests with a bare publish probe use a clean-outcome auto-pilot probe.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 17:29:45 -04:00
Joseph Doherty 4a10ea1e1c fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.

- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
  Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
  survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
  re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
  fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
  real invoker's bracket across the removal and signals when its late writes are done (positive
  control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
  (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
  host test.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 09:07:11 -04:00
Joseph Doherty 53247a056a fix(runtime): malformed artifact bytes on boot/restore are no answer (#601); evict telemetry snapshots on driver removal (#609)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:53:45 -04:00
Joseph Doherty 909d75357b test(mesh-phase5): cover the ScriptedAlarmHostActor + VirtualTagActor telemetry taps
Closes two test-completeness gaps flagged in review of the seam-tap commit — the
ScriptedAlarmHostActor and VirtualTagActor hub emits had no test proving the hub
actually receives them.

- ScriptedAlarm_activation_emits_one_Alarm_item_with_matching_payload: spawns the
  host with a capturing fake hub, drives a real engine Inactive->Active transition
  through the existing ScriptedAlarms harness, and asserts exactly one
  TelemetryItem.Alarm carrying the Activated transition.
- VirtualTag_script_log_emits_one_Script_item_with_same_payload: passes a fake hub
  via Props + a publisherFactory, drives an evaluator failure, and asserts a
  TelemetryItem.Script carrying the SAME ScriptLogEntry instance handed to the
  publisher.

To make the VirtualTagActor tap observable, its hub emit moved to the TOP of
PublishLog, before the test-only publisherFactory early-return branch, so the emit
fires on BOTH the production DPS path and the factory path. Production behavior is
unchanged: production passes publisherFactory: null, so it still reaches the DPS
Tell; the emit just moved a couple lines earlier (both are fire-and-forget).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 15:38:12 -04:00
Joseph Doherty 7bdb6d00ec fix(mesh-phase5): telemetry hub — plain Dictionary under gate, concurrency stress test, priming/eviction notes
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 15:30:24 -04:00
Joseph Doherty 8e5090edf3 feat(mesh-phase5): tap the 4 telemetry publish seams into the local hub (DPS intact)
Each of the four telemetry publish seams now also emits its DPS payload into the
node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub
publish untouched:

- driver-health   AkkaDriverHealthPublisher            -> TelemetryItem.Health
- resilience      DriverResilienceStatusPublisherService -> TelemetryItem.Resilience
- alerts          ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm
- script-logs     VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script

DI services take the hub as a required ctor param; actors take an optional
nullable hub threaded through their Props and the real spawn sites
(DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor
<- DriverHostActor), so existing test Props constructions keep compiling and simply
do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe
on every node.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 15:27:29 -04:00
Joseph Doherty 9882b1d250 feat(mesh-phase5): node-local telemetry hub (snapshot-replay + drop-oldest fan-out)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 15:08:56 -04:00