{ "plan": "docs/plans/2026-07-22-clusterclient-to-grpc-plan.md", "design": "~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md", "worktrees": { "feat/grpc-phase0-psk": "/Users/dohertj2/Desktop/ScadaBridge-phase0" }, "tasks": [ { "id": "T0.1", "phase": "0", "subject": "Delete the vestigial /user/management receptionist registration", "status": "completed", "activeForm": "Deleting the /user/management receptionist registration", "files": [ "src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs", "docs/requirements/Component-Host.md", "docs/requirements/Component-ManagementService.md", "docs/requirements/Component-Communication.md", "docs/components/ManagementService.md", "docs/components/Communication.md", "src/ZB.MOM.WW.ScadaBridge.CLI/README.md" ], "notes": "Actor stays; only the ClusterClientReceptionist.RegisterService call goes. Verified: CLI is Akka-free, so 6 docs claiming the CLI reaches ManagementActor over ClusterClient are stale and must be corrected in the same change." }, { "id": "T0.2", "phase": "0", "subject": "Record the dead IntegrationCallRequest exclusion (28 of 29 commands migrate)", "status": "completed", "activeForm": "Recording the dead IntegrationCallRequest exclusion", "files": [ "docs/known-issues/" ], "notes": "Plan says 'file a Gitea issue' \u2014 outward-facing, handed to the user. In-repo half is a known-issues note + the exclusion record." }, { "id": "T0.3", "phase": "0", "subject": "ControlPlaneAuthInterceptor + CommunicationOptions.GrpcPsk + SitePskProvider; gate SiteStream; attach PSK on central's clients", "status": "completed", "activeForm": "Building the control-plane PSK auth", "files": [ "src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs", "src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs", "src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs", "src/ZB.MOM.WW.ScadaBridge.Host/SitePskProvider.cs", "src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs", "src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs", "src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs", "src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullSiteCallsClient.cs", "src/ZB.MOM.WW.ScadaBridge.Host/Program.cs" ], "notes": "Service prefix verified from sitestream.proto: /sitestream.SiteStreamService/. Grpc 2.76 -> CallCredentials.FromInterceptor + UnsafeUseInsecureChannelCallCredentials is the async-safe attach path on h2c." }, { "id": "T0.4", "phase": "0", "subject": "Rig dev keys (3 sites + central secret seeds) + interceptor/provider/wiring tests", "status": "completed", "activeForm": "Seeding rig dev keys and writing the auth tests", "files": [ "docker/site-a-node-a/appsettings.Site.json", "docker/site-a-node-b/appsettings.Site.json", "docker/site-b-node-a/appsettings.Site.json", "docker/site-b-node-b/appsettings.Site.json", "docker/site-c-node-a/appsettings.Site.json", "docker/site-c-node-b/appsettings.Site.json", "tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ControlPlaneAuthInterceptorTests.cs" ], "notes": "Fail-closed from day one \u2014 every environment (incl. docker-env2 and the gitignored deploy/wonder-app-vd03 overlay) needs its key before upgrade. Ops item for the user." }, { "id": "P0.DoD", "phase": "0", "subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged", "status": "completed", "activeForm": "Verifying the Phase 0 DoD", "blockedBy": [ "T0.1", "T0.2", "T0.3", "T0.4" ], "notes": "Live gate PASS 2026-07-22 (all 7 checks, docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md). Suite: 29 non-Playwright suites / 6872 tests / 0 failures. Playwright 170 pass / 2 fail / 1 skip of 173 - BOTH failures root-caused and PRE-EXISTING on main, unrelated to Phase 0 (branch touches no EF/CentralUI/Transport/ManagementService file): (1) TransportImportTests - REAL production bug, BundleImporter.cs:1298 user-initiated transaction + EnableRetryOnFailure => import broken on real MS SQL, hidden by the in-memory EF provider; (2) SmsNotificationE2ETests - stale fixture SID 'ACtest123' vs the ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21); failing since then, which also silences its secret-non-leak assertion. The earlier 44-failure run is VOID (concurrent rig rebuild). REMAINING: PR + merge, HELD for the user per the plan's stop-at-DoD rule." }, { "id": "T1A.1", "phase": "1A", "subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests", "status": "completed", "activeForm": "Authoring central_control.proto and its mappers", "blockedBy": [ "P0.DoD" ], "notes": "d7455577 on feat/grpc-central-control. 7 RPCs; ingest RPCs reuse sitestream AuditEventBatch/CachedTelemetryBatch/IngestAck by import. 32 goldens, verified to have teeth by mutation. PLAN CORRECTIONS FOUND: (1) actor sends IngestAuditEventsCommand/-Reply (IReadOnlyList), NOT the batch/IngestAck types the plan's table claims - mapper bridges; (2) CachedTelemetryEntry carries SiteCall not SiteCallOperational, needed a new SiteCallDtoMapper.ToDto(SiteCall); (3) SiteHealthReport is ~33 members and 5 are INIT-ONLY props not ctor params - FromDto needs an object initializer or they silently drop; (4) 3 collections are nullable with load-bearing null, proto3 cannot express presence on repeated/map => wrapper messages; (5) ConnectionHealth has no Unspecified member, so naive mapping puts Connected on proto3 zero - reserved 0 and unknown decodes to Error, never Connected." }, { "id": "T1A.2", "phase": "1A", "subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate", "status": "completed", "activeForm": "Hosting CentralControlService on central", "blockedBy": [ "T1A.1" ], "notes": "780bb9c3 on feat/grpc-central-control. NEW class CentralControlAuthInterceptor (one public ctor, 3-arg internal, pinned by reflection test) - central verifies per-site PSK via ISitePskProvider keyed by required x-scadabridge-site header, fail-closed on missing/blank/unknown/mismatch. CentralControlGrpcService Asks existing CentralCommunicationActor (0 handler changes), readiness via SetReady mirror, heartbeat Tell/never-gated, ingest reuses AuditIngestAskTimeout. Central branch had NO AddGrpc/Kestrel before - added h2c listener on CentralGrpcPort default 8083, :5000 untouched. Rig ports 9013/9014:8083 published. PLAN GAPS: (1) mappers throw on unset WKT fields - test DTOs must carry a timestamp (no prod impact); (2) plan gave no deadline for Submit/QueryNotification - used NotificationForwardTimeout, check T1A.3 client sets same. 17 CentralControl tests, Host.Tests 384, Communication.Tests 356." }, { "id": "T1A.3", "phase": "1A", "subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option", "status": "completed", "activeForm": "Building the site->central transport seam", "blockedBy": [ "T1A.1" ], "notes": "33b15f10 on feat/grpc-central-control. ICentralTransport: 7 methods, actor delegates all 7; optional ctor param (null->actor self-builds AkkaCentralTransport in PreStart, byte-identical Akka path). CentralChannelProvider = sticky failover/failback, 1s-double-cap-60s backoff, per-site PSK not needed (site's own GrpcPsk + site header). Flag CentralTransport default Akka; on Grpc the ClusterClient is not created at all. Deadlines matched per-RPC to today's Ask timeouts (table in report). PLAN CORRECTION (important): connect-refused surfaces as StatusCode.Internal + 'Error starting gRPC call'/HttpRequestException, NOT Unavailable, in this Grpc.Net version - IsConnectFailure covers Unavailable OR that Internal class, still excludes DeadlineExceeded (verified in code + DeadlineExceeded_IsNotRetriedOnThePeer test). Above-seam suites (NotificationForwarder/HealthReportSender/SiteAuditTelemetry) pass UNMODIFIED. Communication.Tests 371, Host.Tests 391." }, { "id": "T1A.4", "phase": "1A", "subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified", "status": "completed", "activeForm": "Testing the central control plane", "blockedBy": [ "T1A.2", "T1A.3" ], "notes": "Tests shipped inside T1A.1/1A.2/1A.3 per the plan folding T1A.4 into each task. Actor-with-fake-transport x7, TestServer transport tests, above-seam suites pass unmodified." }, { "id": "P1A.DoD", "phase": "1A", "subject": "1A DoD: rig site-a on Grpc proves all 5 site->central paths while site-b/c stay Akka; PR merged before 1B", "status": "pending", "activeForm": "Verifying the 1A DoD", "blockedBy": [ "T1A.4" ] }, { "id": "T1B.1", "phase": "1B", "subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests", "status": "completed", "activeForm": "Authoring site_command.proto and its mappers", "blockedBy": [ "P0.DoD" ], "notes": "a7481174 on feat/grpc-sitecommand. 6 oneof RPCs / 28 commands / 22 replies / 18 nested, verified by reflection guards not a hand list. PLAN CORRECTIONS: (1) generic Guid?->empty-string-means-null is UNSAFE for RouteToWaitForAttributeRequest.TargetValueEncoded, a nullable STRING where '' is a real wait target distinct from null - gave it a StringValue wrapper; (2) RouteToGetAttributesResponse.Values is NON-nullable dict but RouteToCallRequest.Parameters is nullable - two decode paths (absent->empty vs absent->null); (3) DeployArtifactsCommand has 6 NULLABLE artifact collections, proto3 repeated collapses null<->empty => wrapper messages (PLAN-05 T8 class); (4) AlarmStateChanged.Condition is derived-on-read over a nullable backing field - encoding unconditionally breaks record equality, mapper omits condition==computed default. Added seam types for T1B.2/3: SiteCommandGroup, GroupOf/GroupOfReply, UnsubscribeDebugViewAck marker (unsubscribe is Tell today, unary RPC must answer). REBASE NOTE: union-conflict expected in Communication.csproj commented Protobuf ItemGroup + docker/regen-proto.sh vs 1A." }, { "id": "T1B.2", "phase": "1B", "subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)", "status": "completed", "activeForm": "Extracting the site command dispatcher", "blockedBy": [ "T1B.1" ], "notes": "cd6c20e1 on feat/grpc-sitecommand. SiteCommandDispatcher = pure routing (ResolveRoute -> Route{Disposition,Target,Reply}); actor delegates all 27 non-failover Receives to it, gRPC SiteCommandGrpcService calls the SAME dispatcher; shared via SetReady(dispatcher) hand-off like SiteStreamGrpcServer. Parked stays node-local (proven NotSame(proxy)). Interceptor: SiteCommandService added to DefaultGatedPrefixes via descriptor - NO new ctor, one-public-ctor test green. PLAN CORRECTION (important): existing actor code issues cluster Leave BEFORE the ack (fine over ClusterClient Tell, WRONG for gRPC ack-before-Leave). Solved via the pre-existing dryRun param on ClusterFailoverCoordinator.FailOverOldest: seam widened to Func, actor commits immediately (dryRun:false, byte-identical today), gRPC resolves dry-run then defers CommitLeave until after ack. Two ordering tests pin resolve-then-leave. Actor got an OPTIONAL dispatcher param so existing SiteCommunicationActorTests pass with ZERO edits. Server derives local-Ask timeout from ServerCallContext.Deadline (remaining), 2min fallback. Communication.Tests 574, Host.Tests 377." }, { "id": "T1B.3", "phase": "1B", "subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag", "status": "completed", "activeForm": "Building the central->site transport seam", "blockedBy": [ "T1B.1" ], "notes": "3be85f19 on feat/grpc-sitecommand. ISiteCommandTransport injected into CentralCommunicationActor; HandleSiteEnvelope->_transport.Send(env,Sender), HandleSiteAddressCacheLoaded->_transport.ReconcileSites. AkkaSiteTransport extracted verbatim (incl no-client->drop path). GrpcSiteTransport + SitePairChannelProvider (A/B from GrpcNode*Address, per-site PSK via ISitePskProvider, Invalidate on removal). Rode the EXISTING LoadSiteAddressesFromDb loop (extended SiteAddressCacheLoaded to carry gRPC cols too) - ONE poll loop. Flag SiteTransport default Akka. PLAN CORRECTIONS: (1) LoadSiteAddressesFromDb DOES exist (known); (2) plan deadline table wrong TWICE - DeploymentStateQuery uses QueryTimeout not LifecycleTimeout, TriggerSiteFailover uses QueryTimeout not LifecycleTimeout - resolver is per-command-type not per-group, verified in code; (3) SiteCommandGroup XML doc's 'shared deadline class' claim is false; (4) THIRD SiteEnvelope producer DebugStreamBridgeActor (not just CommunicationService+SiteCallAudit) - reply plumbing routes to a real actor sender, not only Ask temp actors. RetryParkedOperation/DiscardParkedOperation->QueryTimeout(30s) keeps SiteCallAudit RelayTimeout(10s)<30s. Communication.Tests 607, gRPC Host.Tests 34. Existing suites: 1 trivial helper edit (SiteAddressCacheLoaded internal->public + new dict arg)." }, { "id": "T1B.4", "phase": "1B", "subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green", "status": "completed", "activeForm": "Testing the site command plane", "blockedBy": [ "T1B.2", "T1B.3" ], "notes": "Tests shipped inside T1B.1/1B.2/1B.3 per the plan folding T1B.4 into each task. Dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, deadline theory, failover/failback, existing Communication suites green with Akka default." }, { "id": "P1B.DoD", "phase": "1B", "subject": "1B DoD: rig central on Grpc for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged", "status": "pending", "activeForm": "Verifying the 1B DoD", "blockedBy": [ "T1B.4", "P1A.DoD" ] }, { "id": "P2", "phase": "2", "subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean", "status": "pending", "activeForm": "Running the site->central cutover soak", "blockedBy": [ "P1A.DoD" ] }, { "id": "P3", "phase": "3", "subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity", "status": "pending", "activeForm": "Running the central->site cutover soak", "blockedBy": [ "P1B.DoD" ] }, { "id": "P4.1", "phase": "4", "subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags", "status": "pending", "activeForm": "Deleting the ClusterClient transport", "blockedBy": [ "P2", "P3" ] }, { "id": "P4.2", "phase": "4", "subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)", "status": "pending", "activeForm": "Running the deletion grep-gates and doc sweep", "blockedBy": [ "P4.1" ] }, { "id": "P5", "phase": "5", "subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md", "status": "pending", "activeForm": "Running the live gate", "blockedBy": [ "P4.2" ] } ] }