7 Commits

2 changed files with 34 additions and 26 deletions
@@ -306,20 +306,20 @@ Critical path ≈ 1B: **~46 weeks total**, matching the design estimate.
- [x] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate) - [x] T0.2 File dead-`IntegrationCallRequest` issue; record exclusion (28 of 29 migrate)
- [x] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients - [x] T0.3 `ControlPlaneAuthInterceptor` + `CommunicationOptions.GrpcPsk` + `SitePskProvider`; gate SiteStream; PSK attached on central's streaming + pull clients
- [x] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests - [x] T0.4 Rig dev keys (all 3 sites + central store seeds) + interceptor/provider/wiring tests
- [ ] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged - [x] Phase 0 DoD: suite green; rig unauthenticated ⇒ `PermissionDenied`, authenticated paths work; PR merged (#25, ff to `main` @ `3fa95555`; gate PASS in `2026-07-22-clusterclient-to-grpc-live-gate.md`)
**Phase 1A — central control plane** (worktree, `feat/grpc-central-control`) **Phase 1A — central control plane** (worktree, `feat/grpc-central-control`)
- [ ] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests - [x] T1A.1 `central_control.proto` (7 RPCs; checked-in codegen) + `CentralControlDtoMapper` + round-trip golden tests
- [ ] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate - [x] T1A.2 Central hosting: `AddGrpc` + per-site-PSK interceptor (`x-scadabridge-site`), `CentralGrpcPort` h2c listener (8083), `CentralControlGrpcService` (Ask existing handlers), readiness gate
- [ ] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator - [x] T1A.3 `ICentralTransport` (Akka extract + Grpc impl), `CentralChannelProvider` (sticky failover/failback, backoff, deadlines, PSK), `CentralTransport` flag default `Akka`, `CentralGrpcEndpoints` option + validator
- [ ] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified - [x] T1A.4 Tests: actor-with-fake-transport ×7, TestServer transport tests, S&F/audit/health suites pass unmodified
- [ ] 1A DoD: rig site-a on `Grpc` proves all 5 site→central paths while site-b/c stay Akka; PR merged (before 1B) - [ ] 1A DoD: rig site-a on `Grpc` proves all 5 site→central paths while site-b/c stay Akka; PR merged (before 1B)
**Phase 1B — site command plane** (worktree, `feat/grpc-sitecommand`) **Phase 1B — site command plane** (worktree, `feat/grpc-sitecommand`)
- [ ] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 + replies) - [x] T1B.1 `site_command.proto` (6 oneof RPCs / 28 commands) + `SiteCommandDtoMapper` + round-trip golden tests (all 28 + replies)
- [ ] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local) - [x] T1B.2 `SiteCommandDispatcher` refactor (actor + new `SiteCommandGrpcService` share it; parked stays node-local)
- [ ] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka` - [x] T1B.3 `ISiteCommandTransport` in `CentralCommunicationActor` (Akka extract + Grpc impl), `SitePairChannelProvider` (Site entity Grpc columns + DB refresh loop), `SiteTransport` flag default `Akka`
- [ ] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green - [x] T1B.4 Tests: dispatcher routing ×28, actor envelope/reply plumbing, TestServer service tests, existing Communication suites green
- [ ] 1B DoD: rig central on `Grpc` for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged - [ ] 1B DoD: rig central on `Grpc` for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged
**Phase 2 ∥ 3 — cutover + soak** **Phase 2 ∥ 3 — cutover + soak**
@@ -73,7 +73,7 @@
"id": "P0.DoD", "id": "P0.DoD",
"phase": "0", "phase": "0",
"subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged", "subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged",
"status": "in_progress", "status": "completed",
"activeForm": "Verifying the Phase 0 DoD", "activeForm": "Verifying the Phase 0 DoD",
"blockedBy": [ "blockedBy": [
"T0.1", "T0.1",
@@ -87,42 +87,46 @@
"id": "T1A.1", "id": "T1A.1",
"phase": "1A", "phase": "1A",
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests", "subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
"status": "pending", "status": "completed",
"activeForm": "Authoring central_control.proto and its mappers", "activeForm": "Authoring central_control.proto and its mappers",
"blockedBy": [ "blockedBy": [
"P0.DoD" "P0.DoD"
] ],
"notes": "d7455577 on feat/grpc-central-control. 7 RPCs; ingest RPCs reuse sitestream AuditEventBatch/CachedTelemetryBatch/IngestAck by import. 32 goldens, verified to have teeth by mutation. PLAN CORRECTIONS FOUND: (1) actor sends IngestAuditEventsCommand/-Reply (IReadOnlyList<Guid>), NOT the batch/IngestAck types the plan's table claims - mapper bridges; (2) CachedTelemetryEntry carries SiteCall not SiteCallOperational, needed a new SiteCallDtoMapper.ToDto(SiteCall); (3) SiteHealthReport is ~33 members and 5 are INIT-ONLY props not ctor params - FromDto needs an object initializer or they silently drop; (4) 3 collections are nullable with load-bearing null, proto3 cannot express presence on repeated/map => wrapper messages; (5) ConnectionHealth has no Unspecified member, so naive mapping puts Connected on proto3 zero - reserved 0 and unknown decodes to Error, never Connected."
}, },
{ {
"id": "T1A.2", "id": "T1A.2",
"phase": "1A", "phase": "1A",
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate", "subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
"status": "pending", "status": "completed",
"activeForm": "Hosting CentralControlService on central", "activeForm": "Hosting CentralControlService on central",
"blockedBy": [ "blockedBy": [
"T1A.1" "T1A.1"
] ],
"notes": "780bb9c3 on feat/grpc-central-control. NEW class CentralControlAuthInterceptor (one public ctor, 3-arg internal, pinned by reflection test) - central verifies per-site PSK via ISitePskProvider keyed by required x-scadabridge-site header, fail-closed on missing/blank/unknown/mismatch. CentralControlGrpcService Asks existing CentralCommunicationActor (0 handler changes), readiness via SetReady mirror, heartbeat Tell/never-gated, ingest reuses AuditIngestAskTimeout. Central branch had NO AddGrpc/Kestrel before - added h2c listener on CentralGrpcPort default 8083, :5000 untouched. Rig ports 9013/9014:8083 published. PLAN GAPS: (1) mappers throw on unset WKT fields - test DTOs must carry a timestamp (no prod impact); (2) plan gave no deadline for Submit/QueryNotification - used NotificationForwardTimeout, check T1A.3 client sets same. 17 CentralControl tests, Host.Tests 384, Communication.Tests 356."
}, },
{ {
"id": "T1A.3", "id": "T1A.3",
"phase": "1A", "phase": "1A",
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option", "subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
"status": "pending", "status": "completed",
"activeForm": "Building the site->central transport seam", "activeForm": "Building the site->central transport seam",
"blockedBy": [ "blockedBy": [
"T1A.1" "T1A.1"
] ],
"notes": "33b15f10 on feat/grpc-central-control. ICentralTransport: 7 methods, actor delegates all 7; optional ctor param (null->actor self-builds AkkaCentralTransport in PreStart, byte-identical Akka path). CentralChannelProvider = sticky failover/failback, 1s-double-cap-60s backoff, per-site PSK not needed (site's own GrpcPsk + site header). Flag CentralTransport default Akka; on Grpc the ClusterClient is not created at all. Deadlines matched per-RPC to today's Ask timeouts (table in report). PLAN CORRECTION (important): connect-refused surfaces as StatusCode.Internal + 'Error starting gRPC call'/HttpRequestException, NOT Unavailable, in this Grpc.Net version - IsConnectFailure covers Unavailable OR that Internal class, still excludes DeadlineExceeded (verified in code + DeadlineExceeded_IsNotRetriedOnThePeer test). Above-seam suites (NotificationForwarder/HealthReportSender/SiteAuditTelemetry) pass UNMODIFIED. Communication.Tests 371, Host.Tests 391."
}, },
{ {
"id": "T1A.4", "id": "T1A.4",
"phase": "1A", "phase": "1A",
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified", "subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
"status": "pending", "status": "completed",
"activeForm": "Testing the central control plane", "activeForm": "Testing the central control plane",
"blockedBy": [ "blockedBy": [
"T1A.2", "T1A.2",
"T1A.3" "T1A.3"
] ],
"notes": "Tests shipped inside T1A.1/1A.2/1A.3 per the plan folding T1A.4 into each task. Actor-with-fake-transport x7, TestServer transport tests, above-seam suites pass unmodified."
}, },
{ {
"id": "P1A.DoD", "id": "P1A.DoD",
@@ -138,42 +142,46 @@
"id": "T1B.1", "id": "T1B.1",
"phase": "1B", "phase": "1B",
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests", "subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
"status": "pending", "status": "completed",
"activeForm": "Authoring site_command.proto and its mappers", "activeForm": "Authoring site_command.proto and its mappers",
"blockedBy": [ "blockedBy": [
"P0.DoD" "P0.DoD"
] ],
"notes": "a7481174 on feat/grpc-sitecommand. 6 oneof RPCs / 28 commands / 22 replies / 18 nested, verified by reflection guards not a hand list. PLAN CORRECTIONS: (1) generic Guid?->empty-string-means-null is UNSAFE for RouteToWaitForAttributeRequest.TargetValueEncoded, a nullable STRING where '' is a real wait target distinct from null - gave it a StringValue wrapper; (2) RouteToGetAttributesResponse.Values is NON-nullable dict but RouteToCallRequest.Parameters is nullable - two decode paths (absent->empty vs absent->null); (3) DeployArtifactsCommand has 6 NULLABLE artifact collections, proto3 repeated collapses null<->empty => wrapper messages (PLAN-05 T8 class); (4) AlarmStateChanged.Condition is derived-on-read over a nullable backing field - encoding unconditionally breaks record equality, mapper omits condition==computed default. Added seam types for T1B.2/3: SiteCommandGroup, GroupOf/GroupOfReply, UnsubscribeDebugViewAck marker (unsubscribe is Tell today, unary RPC must answer). REBASE NOTE: union-conflict expected in Communication.csproj commented Protobuf ItemGroup + docker/regen-proto.sh vs 1A."
}, },
{ {
"id": "T1B.2", "id": "T1B.2",
"phase": "1B", "phase": "1B",
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)", "subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
"status": "pending", "status": "completed",
"activeForm": "Extracting the site command dispatcher", "activeForm": "Extracting the site command dispatcher",
"blockedBy": [ "blockedBy": [
"T1B.1" "T1B.1"
] ],
"notes": "cd6c20e1 on feat/grpc-sitecommand. SiteCommandDispatcher = pure routing (ResolveRoute -> Route{Disposition,Target,Reply}); actor delegates all 27 non-failover Receives to it, gRPC SiteCommandGrpcService calls the SAME dispatcher; shared via SetReady(dispatcher) hand-off like SiteStreamGrpcServer. Parked stays node-local (proven NotSame(proxy)). Interceptor: SiteCommandService added to DefaultGatedPrefixes via descriptor - NO new ctor, one-public-ctor test green. PLAN CORRECTION (important): existing actor code issues cluster Leave BEFORE the ack (fine over ClusterClient Tell, WRONG for gRPC ack-before-Leave). Solved via the pre-existing dryRun param on ClusterFailoverCoordinator.FailOverOldest: seam widened to Func<role,dryRun,addr?>, actor commits immediately (dryRun:false, byte-identical today), gRPC resolves dry-run then defers CommitLeave until after ack. Two ordering tests pin resolve-then-leave. Actor got an OPTIONAL dispatcher param so existing SiteCommunicationActorTests pass with ZERO edits. Server derives local-Ask timeout from ServerCallContext.Deadline (remaining), 2min fallback. Communication.Tests 574, Host.Tests 377."
}, },
{ {
"id": "T1B.3", "id": "T1B.3",
"phase": "1B", "phase": "1B",
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag", "subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
"status": "pending", "status": "completed",
"activeForm": "Building the central->site transport seam", "activeForm": "Building the central->site transport seam",
"blockedBy": [ "blockedBy": [
"T1B.1" "T1B.1"
] ],
"notes": "3be85f19 on feat/grpc-sitecommand. ISiteCommandTransport injected into CentralCommunicationActor; HandleSiteEnvelope->_transport.Send(env,Sender), HandleSiteAddressCacheLoaded->_transport.ReconcileSites. AkkaSiteTransport extracted verbatim (incl no-client->drop path). GrpcSiteTransport + SitePairChannelProvider (A/B from GrpcNode*Address, per-site PSK via ISitePskProvider, Invalidate on removal). Rode the EXISTING LoadSiteAddressesFromDb loop (extended SiteAddressCacheLoaded to carry gRPC cols too) - ONE poll loop. Flag SiteTransport default Akka. PLAN CORRECTIONS: (1) LoadSiteAddressesFromDb DOES exist (known); (2) plan deadline table wrong TWICE - DeploymentStateQuery uses QueryTimeout not LifecycleTimeout, TriggerSiteFailover uses QueryTimeout not LifecycleTimeout - resolver is per-command-type not per-group, verified in code; (3) SiteCommandGroup XML doc's 'shared deadline class' claim is false; (4) THIRD SiteEnvelope producer DebugStreamBridgeActor (not just CommunicationService+SiteCallAudit) - reply plumbing routes to a real actor sender, not only Ask temp actors. RetryParkedOperation/DiscardParkedOperation->QueryTimeout(30s) keeps SiteCallAudit RelayTimeout(10s)<30s. Communication.Tests 607, gRPC Host.Tests 34. Existing suites: 1 trivial helper edit (SiteAddressCacheLoaded internal->public + new dict arg)."
}, },
{ {
"id": "T1B.4", "id": "T1B.4",
"phase": "1B", "phase": "1B",
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green", "subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
"status": "pending", "status": "completed",
"activeForm": "Testing the site command plane", "activeForm": "Testing the site command plane",
"blockedBy": [ "blockedBy": [
"T1B.2", "T1B.2",
"T1B.3" "T1B.3"
] ],
"notes": "Tests shipped inside T1B.1/1B.2/1B.3 per the plan folding T1B.4 into each task. Dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, deadline theory, failover/failback, existing Communication suites green with Akka default."
}, },
{ {
"id": "P1B.DoD", "id": "P1B.DoD",