Galaxy driver initialize often exceeds the 5 s initialize timeout against a real gateway (25% of restarts on VD03) #875

New Issue

2026-11-11T18:41:47-05:00

dohertj2 commented

2026-11-11 18:41:47 -05:00

Found by #761 gate F (2026-11-11, docker-dev on master 5ac55205 against the VD03 DEV mxaccessgw 2b13564e).

GalaxyDriver initialize against VD03 takes 4–5 s (session open plus the per-template platform seed) against the 5 s default driver-initialize timeout. 10 of 40 restarts failed with "initialize failed … 00:00:05". Each failure cost a 10 s retry and showed on the gateway as a killed worker plus an OpenSessionFailed/WorkerFaulted fault. The gateway cleaned up every one (no session leak, gate F PASS), so the cost is restart latency and gateway fault noise.

Fix: a per-driver initialize budget for Galaxy (default around 30 s), or have the timeout cover only the session open and finish seeding in the background. Find where the 5 s comes from (driver host initialize timeout) and make it overridable per driver type. Check it against the comms-loss watchdog and the deploy apply deadline.

Found by #761 gate F (2026-11-11, docker-dev on master `5ac55205` against the VD03 DEV mxaccessgw `2b13564e`). `GalaxyDriver` initialize against VD03 takes 4–5 s (session open plus the per-template platform seed) against the 5 s default driver-initialize timeout. 10 of 40 restarts failed with "initialize failed … 00:00:05". Each failure cost a 10 s retry and showed on the gateway as a killed worker plus an `OpenSessionFailed`/`WorkerFaulted` fault. The gateway cleaned up every one (no session leak, gate F PASS), so the cost is restart latency and gateway fault noise. **Fix:** a per-driver initialize budget for Galaxy (default around 30 s), or have the timeout cover only the session open and finish seeding in the background. Find where the 5 s comes from (driver host initialize timeout) and make it overridable per driver type. Check it against the comms-loss watchdog and the deploy apply deadline.

dohertj2 added the type:bug priority:p3 labels 2026-11-11 18:41:47 -05:00

dohertj2 commented

2026-11-11 19:01:26 -05:00

Fixed, pushed to master (95bde828, merged as $(git rev-parse --short HEAD)).

The 5 s came from the Tier A Initialize policy in DriverResilienceOptions, not from the actor; the actor's own pessimistic bound is 30 s. A driver type can now register capability defaults with its factory: DriverFactoryRegistry.Register(..., capabilityDefaults:), read back by GetCapabilityDefaults. They layer over the tier table and under the per-instance ResilienceConfig, and the Host wires them into DriverCapabilityInvokerFactory.

Galaxy registers Initialize = 20 s, with no retry and breaker threshold 2, which stays under the actor's 30 s bound. A per-instance ResilienceConfig still overrides it.

Tests: DriverTypeCapabilityDefaultsTests (layering order, registry and invoker), plus a Galaxy registration test. The full local suite is green (21 projects). Documented in docs/DriverLifecycle.md.

Fixed, pushed to master (`95bde828`, merged as `$(git rev-parse --short HEAD)`). The 5 s came from the Tier A `Initialize` policy in `DriverResilienceOptions`, not from the actor; the actor's own pessimistic bound is 30 s. A driver type can now register **capability defaults** with its factory: `DriverFactoryRegistry.Register(..., capabilityDefaults:)`, read back by `GetCapabilityDefaults`. They layer over the tier table and under the per-instance `ResilienceConfig`, and the Host wires them into `DriverCapabilityInvokerFactory`. Galaxy registers `Initialize` = 20 s, with no retry and breaker threshold 2, which stays under the actor's 30 s bound. A per-instance ResilienceConfig still overrides it. Tests: `DriverTypeCapabilityDefaultsTests` (layering order, registry and invoker), plus a Galaxy registration test. The full local suite is green (21 projects). Documented in docs/DriverLifecycle.md.

dohertj2 closed this issue

2026-11-11 19:01:26 -05:00

dohertj2 referenced this issue from a commit

2026-11-11 19:01:27 -05:00

fix(galaxy): #875 Galaxy initialize gets a 20 s budget via per-driver-type capability defaults

dohertj2 referenced this issue from a commit

2026-11-11 19:01:27 -05:00

Merge #875 Galaxy initialize 20 s budget (per-driver-type capability defaults)

dohertj2 commented

2026-11-11 19:01:35 -05:00