Joseph Doherty
a54602c14a
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane (heartbeat/health/notification S&F/audit) rides gRPC CentralControlService, 196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions. Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s): - single-node active-kill (central-a): sticky failover central-a:8083→b:8083 logged instantly, central-b active in 29s, buffer drained 72→101, every 5s bucket = exactly 3 through the gap, 101==101 distinct - failback: central-a rejoined ready ~5s as standby, central-b kept active (oldest-Up, no flap), traffic uninterrupted - full outage (both central down ~59s): count frozen, cold re-form central-b active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the whole dead window, 216==216 distinct — zero loss, zero dupes Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill). Rig config reverted to Akka default (defaults stay Akka until Phase 4).
249 lines
16 KiB
JSON
249 lines
16 KiB
JSON
{
|
|
"plan": "docs/plans/2026-07-22-clusterclient-to-grpc-plan.md",
|
|
"design": "~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md",
|
|
"worktrees": {
|
|
"feat/grpc-phase0-psk": "/Users/dohertj2/Desktop/ScadaBridge-phase0"
|
|
},
|
|
"tasks": [
|
|
{
|
|
"id": "T0.1",
|
|
"phase": "0",
|
|
"subject": "Delete the vestigial /user/management receptionist registration",
|
|
"status": "completed",
|
|
"activeForm": "Deleting the /user/management receptionist registration",
|
|
"files": [
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs",
|
|
"docs/requirements/Component-Host.md",
|
|
"docs/requirements/Component-ManagementService.md",
|
|
"docs/requirements/Component-Communication.md",
|
|
"docs/components/ManagementService.md",
|
|
"docs/components/Communication.md",
|
|
"src/ZB.MOM.WW.ScadaBridge.CLI/README.md"
|
|
],
|
|
"notes": "Actor stays; only the ClusterClientReceptionist.RegisterService call goes. Verified: CLI is Akka-free, so 6 docs claiming the CLI reaches ManagementActor over ClusterClient are stale and must be corrected in the same change."
|
|
},
|
|
{
|
|
"id": "T0.2",
|
|
"phase": "0",
|
|
"subject": "Record the dead IntegrationCallRequest exclusion (28 of 29 commands migrate)",
|
|
"status": "completed",
|
|
"activeForm": "Recording the dead IntegrationCallRequest exclusion",
|
|
"files": [
|
|
"docs/known-issues/"
|
|
],
|
|
"notes": "Plan says 'file a Gitea issue' \u2014 outward-facing, handed to the user. In-repo half is a known-issues note + the exclusion record."
|
|
},
|
|
{
|
|
"id": "T0.3",
|
|
"phase": "0",
|
|
"subject": "ControlPlaneAuthInterceptor + CommunicationOptions.GrpcPsk + SitePskProvider; gate SiteStream; attach PSK on central's clients",
|
|
"status": "completed",
|
|
"activeForm": "Building the control-plane PSK auth",
|
|
"files": [
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/SitePskProvider.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullSiteCallsClient.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/Program.cs"
|
|
],
|
|
"notes": "Service prefix verified from sitestream.proto: /sitestream.SiteStreamService/. Grpc 2.76 -> CallCredentials.FromInterceptor + UnsafeUseInsecureChannelCallCredentials is the async-safe attach path on h2c."
|
|
},
|
|
{
|
|
"id": "T0.4",
|
|
"phase": "0",
|
|
"subject": "Rig dev keys (3 sites + central secret seeds) + interceptor/provider/wiring tests",
|
|
"status": "completed",
|
|
"activeForm": "Seeding rig dev keys and writing the auth tests",
|
|
"files": [
|
|
"docker/site-a-node-a/appsettings.Site.json",
|
|
"docker/site-a-node-b/appsettings.Site.json",
|
|
"docker/site-b-node-a/appsettings.Site.json",
|
|
"docker/site-b-node-b/appsettings.Site.json",
|
|
"docker/site-c-node-a/appsettings.Site.json",
|
|
"docker/site-c-node-b/appsettings.Site.json",
|
|
"tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ControlPlaneAuthInterceptorTests.cs"
|
|
],
|
|
"notes": "Fail-closed from day one \u2014 every environment (incl. docker-env2 and the gitignored deploy/wonder-app-vd03 overlay) needs its key before upgrade. Ops item for the user."
|
|
},
|
|
{
|
|
"id": "P0.DoD",
|
|
"phase": "0",
|
|
"subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged",
|
|
"status": "completed",
|
|
"activeForm": "Verifying the Phase 0 DoD",
|
|
"blockedBy": [
|
|
"T0.1",
|
|
"T0.2",
|
|
"T0.3",
|
|
"T0.4"
|
|
],
|
|
"notes": "Live gate PASS 2026-07-22 (all 7 checks, docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md). Suite: 29 non-Playwright suites / 6872 tests / 0 failures. Playwright 170 pass / 2 fail / 1 skip of 173 - BOTH failures root-caused and PRE-EXISTING on main, unrelated to Phase 0 (branch touches no EF/CentralUI/Transport/ManagementService file): (1) TransportImportTests - REAL production bug, BundleImporter.cs:1298 user-initiated transaction + EnableRetryOnFailure => import broken on real MS SQL, hidden by the in-memory EF provider; (2) SmsNotificationE2ETests - stale fixture SID 'ACtest123' vs the ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21); failing since then, which also silences its secret-non-leak assertion. The earlier 44-failure run is VOID (concurrent rig rebuild). REMAINING: PR + merge, HELD for the user per the plan's stop-at-DoD rule."
|
|
},
|
|
{
|
|
"id": "T1A.1",
|
|
"phase": "1A",
|
|
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
|
|
"status": "completed",
|
|
"activeForm": "Authoring central_control.proto and its mappers",
|
|
"blockedBy": [
|
|
"P0.DoD"
|
|
],
|
|
"notes": "d7455577 on feat/grpc-central-control. 7 RPCs; ingest RPCs reuse sitestream AuditEventBatch/CachedTelemetryBatch/IngestAck by import. 32 goldens, verified to have teeth by mutation. PLAN CORRECTIONS FOUND: (1) actor sends IngestAuditEventsCommand/-Reply (IReadOnlyList<Guid>), NOT the batch/IngestAck types the plan's table claims - mapper bridges; (2) CachedTelemetryEntry carries SiteCall not SiteCallOperational, needed a new SiteCallDtoMapper.ToDto(SiteCall); (3) SiteHealthReport is ~33 members and 5 are INIT-ONLY props not ctor params - FromDto needs an object initializer or they silently drop; (4) 3 collections are nullable with load-bearing null, proto3 cannot express presence on repeated/map => wrapper messages; (5) ConnectionHealth has no Unspecified member, so naive mapping puts Connected on proto3 zero - reserved 0 and unknown decodes to Error, never Connected."
|
|
},
|
|
{
|
|
"id": "T1A.2",
|
|
"phase": "1A",
|
|
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
|
|
"status": "completed",
|
|
"activeForm": "Hosting CentralControlService on central",
|
|
"blockedBy": [
|
|
"T1A.1"
|
|
],
|
|
"notes": "780bb9c3 on feat/grpc-central-control. NEW class CentralControlAuthInterceptor (one public ctor, 3-arg internal, pinned by reflection test) - central verifies per-site PSK via ISitePskProvider keyed by required x-scadabridge-site header, fail-closed on missing/blank/unknown/mismatch. CentralControlGrpcService Asks existing CentralCommunicationActor (0 handler changes), readiness via SetReady mirror, heartbeat Tell/never-gated, ingest reuses AuditIngestAskTimeout. Central branch had NO AddGrpc/Kestrel before - added h2c listener on CentralGrpcPort default 8083, :5000 untouched. Rig ports 9013/9014:8083 published. PLAN GAPS: (1) mappers throw on unset WKT fields - test DTOs must carry a timestamp (no prod impact); (2) plan gave no deadline for Submit/QueryNotification - used NotificationForwardTimeout, check T1A.3 client sets same. 17 CentralControl tests, Host.Tests 384, Communication.Tests 356."
|
|
},
|
|
{
|
|
"id": "T1A.3",
|
|
"phase": "1A",
|
|
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
|
|
"status": "completed",
|
|
"activeForm": "Building the site->central transport seam",
|
|
"blockedBy": [
|
|
"T1A.1"
|
|
],
|
|
"notes": "33b15f10 on feat/grpc-central-control. ICentralTransport: 7 methods, actor delegates all 7; optional ctor param (null->actor self-builds AkkaCentralTransport in PreStart, byte-identical Akka path). CentralChannelProvider = sticky failover/failback, 1s-double-cap-60s backoff, per-site PSK not needed (site's own GrpcPsk + site header). Flag CentralTransport default Akka; on Grpc the ClusterClient is not created at all. Deadlines matched per-RPC to today's Ask timeouts (table in report). PLAN CORRECTION (important): connect-refused surfaces as StatusCode.Internal + 'Error starting gRPC call'/HttpRequestException, NOT Unavailable, in this Grpc.Net version - IsConnectFailure covers Unavailable OR that Internal class, still excludes DeadlineExceeded (verified in code + DeadlineExceeded_IsNotRetriedOnThePeer test). Above-seam suites (NotificationForwarder/HealthReportSender/SiteAuditTelemetry) pass UNMODIFIED. Communication.Tests 371, Host.Tests 391."
|
|
},
|
|
{
|
|
"id": "T1A.4",
|
|
"phase": "1A",
|
|
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
|
|
"status": "completed",
|
|
"activeForm": "Testing the central control plane",
|
|
"blockedBy": [
|
|
"T1A.2",
|
|
"T1A.3"
|
|
],
|
|
"notes": "Tests shipped inside T1A.1/1A.2/1A.3 per the plan folding T1A.4 into each task. Actor-with-fake-transport x7, TestServer transport tests, above-seam suites pass unmodified."
|
|
},
|
|
{
|
|
"id": "P1A.DoD",
|
|
"phase": "1A",
|
|
"subject": "1A DoD: rig site-a on Grpc proves all 5 site->central paths while site-b/c stay Akka; PR merged before 1B",
|
|
"status": "pending",
|
|
"activeForm": "Verifying the 1A DoD",
|
|
"blockedBy": [
|
|
"T1A.4"
|
|
]
|
|
},
|
|
{
|
|
"id": "T1B.1",
|
|
"phase": "1B",
|
|
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
|
|
"status": "completed",
|
|
"activeForm": "Authoring site_command.proto and its mappers",
|
|
"blockedBy": [
|
|
"P0.DoD"
|
|
],
|
|
"notes": "a7481174 on feat/grpc-sitecommand. 6 oneof RPCs / 28 commands / 22 replies / 18 nested, verified by reflection guards not a hand list. PLAN CORRECTIONS: (1) generic Guid?->empty-string-means-null is UNSAFE for RouteToWaitForAttributeRequest.TargetValueEncoded, a nullable STRING where '' is a real wait target distinct from null - gave it a StringValue wrapper; (2) RouteToGetAttributesResponse.Values is NON-nullable dict but RouteToCallRequest.Parameters is nullable - two decode paths (absent->empty vs absent->null); (3) DeployArtifactsCommand has 6 NULLABLE artifact collections, proto3 repeated collapses null<->empty => wrapper messages (PLAN-05 T8 class); (4) AlarmStateChanged.Condition is derived-on-read over a nullable backing field - encoding unconditionally breaks record equality, mapper omits condition==computed default. Added seam types for T1B.2/3: SiteCommandGroup, GroupOf/GroupOfReply, UnsubscribeDebugViewAck marker (unsubscribe is Tell today, unary RPC must answer). REBASE NOTE: union-conflict expected in Communication.csproj commented Protobuf ItemGroup + docker/regen-proto.sh vs 1A."
|
|
},
|
|
{
|
|
"id": "T1B.2",
|
|
"phase": "1B",
|
|
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
|
|
"status": "completed",
|
|
"activeForm": "Extracting the site command dispatcher",
|
|
"blockedBy": [
|
|
"T1B.1"
|
|
],
|
|
"notes": "cd6c20e1 on feat/grpc-sitecommand. SiteCommandDispatcher = pure routing (ResolveRoute -> Route{Disposition,Target,Reply}); actor delegates all 27 non-failover Receives to it, gRPC SiteCommandGrpcService calls the SAME dispatcher; shared via SetReady(dispatcher) hand-off like SiteStreamGrpcServer. Parked stays node-local (proven NotSame(proxy)). Interceptor: SiteCommandService added to DefaultGatedPrefixes via descriptor - NO new ctor, one-public-ctor test green. PLAN CORRECTION (important): existing actor code issues cluster Leave BEFORE the ack (fine over ClusterClient Tell, WRONG for gRPC ack-before-Leave). Solved via the pre-existing dryRun param on ClusterFailoverCoordinator.FailOverOldest: seam widened to Func<role,dryRun,addr?>, actor commits immediately (dryRun:false, byte-identical today), gRPC resolves dry-run then defers CommitLeave until after ack. Two ordering tests pin resolve-then-leave. Actor got an OPTIONAL dispatcher param so existing SiteCommunicationActorTests pass with ZERO edits. Server derives local-Ask timeout from ServerCallContext.Deadline (remaining), 2min fallback. Communication.Tests 574, Host.Tests 377."
|
|
},
|
|
{
|
|
"id": "T1B.3",
|
|
"phase": "1B",
|
|
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
|
|
"status": "completed",
|
|
"activeForm": "Building the central->site transport seam",
|
|
"blockedBy": [
|
|
"T1B.1"
|
|
],
|
|
"notes": "3be85f19 on feat/grpc-sitecommand. ISiteCommandTransport injected into CentralCommunicationActor; HandleSiteEnvelope->_transport.Send(env,Sender), HandleSiteAddressCacheLoaded->_transport.ReconcileSites. AkkaSiteTransport extracted verbatim (incl no-client->drop path). GrpcSiteTransport + SitePairChannelProvider (A/B from GrpcNode*Address, per-site PSK via ISitePskProvider, Invalidate on removal). Rode the EXISTING LoadSiteAddressesFromDb loop (extended SiteAddressCacheLoaded to carry gRPC cols too) - ONE poll loop. Flag SiteTransport default Akka. PLAN CORRECTIONS: (1) LoadSiteAddressesFromDb DOES exist (known); (2) plan deadline table wrong TWICE - DeploymentStateQuery uses QueryTimeout not LifecycleTimeout, TriggerSiteFailover uses QueryTimeout not LifecycleTimeout - resolver is per-command-type not per-group, verified in code; (3) SiteCommandGroup XML doc's 'shared deadline class' claim is false; (4) THIRD SiteEnvelope producer DebugStreamBridgeActor (not just CommunicationService+SiteCallAudit) - reply plumbing routes to a real actor sender, not only Ask temp actors. RetryParkedOperation/DiscardParkedOperation->QueryTimeout(30s) keeps SiteCallAudit RelayTimeout(10s)<30s. Communication.Tests 607, gRPC Host.Tests 34. Existing suites: 1 trivial helper edit (SiteAddressCacheLoaded internal->public + new dict arg)."
|
|
},
|
|
{
|
|
"id": "T1B.4",
|
|
"phase": "1B",
|
|
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
|
|
"status": "completed",
|
|
"activeForm": "Testing the site command plane",
|
|
"blockedBy": [
|
|
"T1B.2",
|
|
"T1B.3"
|
|
],
|
|
"notes": "Tests shipped inside T1B.1/1B.2/1B.3 per the plan folding T1B.4 into each task. Dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, deadline theory, failover/failback, existing Communication suites green with Akka default."
|
|
},
|
|
{
|
|
"id": "P1B.DoD",
|
|
"phase": "1B",
|
|
"subject": "1B DoD: rig central on Grpc for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged",
|
|
"status": "pending",
|
|
"activeForm": "Verifying the 1B DoD",
|
|
"blockedBy": [
|
|
"T1B.4",
|
|
"P1A.DoD"
|
|
]
|
|
},
|
|
{
|
|
"id": "P2",
|
|
"phase": "2",
|
|
"subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean",
|
|
"status": "completed",
|
|
"activeForm": "Running the site->central cutover soak",
|
|
"blockedBy": [
|
|
"P1A.DoD"
|
|
]
|
|
},
|
|
{
|
|
"id": "P3",
|
|
"phase": "3",
|
|
"subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
|
|
"status": "pending",
|
|
"activeForm": "Running the central->site cutover soak",
|
|
"blockedBy": [
|
|
"P1B.DoD"
|
|
]
|
|
},
|
|
{
|
|
"id": "P4.1",
|
|
"phase": "4",
|
|
"subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags",
|
|
"status": "pending",
|
|
"activeForm": "Deleting the ClusterClient transport",
|
|
"blockedBy": [
|
|
"P2",
|
|
"P3"
|
|
]
|
|
},
|
|
{
|
|
"id": "P4.2",
|
|
"phase": "4",
|
|
"subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)",
|
|
"status": "pending",
|
|
"activeForm": "Running the deletion grep-gates and doc sweep",
|
|
"blockedBy": [
|
|
"P4.1"
|
|
]
|
|
},
|
|
{
|
|
"id": "P5",
|
|
"phase": "5",
|
|
"subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md",
|
|
"status": "pending",
|
|
"activeForm": "Running the live gate",
|
|
"blockedBy": [
|
|
"P4.2"
|
|
]
|
|
}
|
|
]
|
|
} |